Dave Glencross

Best Darknet Diaries episodes

2022-04-13T00:00:00+00:00

Having gone through almost the entire backlog of Darknet Diaries, these were my favourite episodes and the ones I recommend to people.

I particularly enjoy the darknet marketplace episodes, and people hacking computer games or websites. I generally don’t enjoy the episodes with penetration testers quite as much as a rule, but they are still well worth listening to. The worst episodes of Darknet Diaries are still really good. It is possibly my favourite podcast (either that or Hardcore History).

I’m up to episode 111 at time of writing!

These are my absolute favourites:

EP 7: Manfred (Part 1)

EP 8: Manfred (Part 2)

EP 9: The Rise and Fall of Mt. Gox

EP 24: Operation Bayonet

EP 29: Stuxnet

EP 99: The Spy and EP 100: NSO

EP 101: Lotería

EP 102: Money Maker

EP 109: TeaMp0isoN

And these ones are also great:

EP 6: The Beirut Bank Job

EP 36: Jeremy from Marketing

EP 45: XBox Underground (Part 1)

EP 46: XBox Underground (Part 2)

EP 53: Shadow Brokers

EP 85: Cam the Carder

EP 92: The Pirate Bay

EP 104: Arya

Ad-free podcasts

2022-04-10T00:00:00+00:00

Ad-free podcasts

I listen to podcasts while dog-walking (generally 3 times a day), working out, doing chores, on the loo, falling asleep, generally moving around my house.

I also find adverts really annoying. Despite the ability to skip forward, I just much prefer listening to ad-free podcasts. I have given up on a lot of podcasts that would otherwise be great, but the adverts are just to frequent or too annoying.

Here is a selection of quality podcasts that are free of advertising. This is not an exhaustive list but they are all personal recommendations.

Ad-free does not necessarily mean free though! I am happy to pay for quality, ad-free podcasts so a number of these are paid.

I’ve also given a few honourary mentions for podcasts with advertising which isn’t too obstrusive.

These are loosely in order of best (within their categories of paid/free).

Ad-free and zero cost

Some of these might ask for you to support them, but I don’t consider that to be nearly as annoying as third-party adverts.

EconTalk

Nominally an economics podcast, but actually interviews with a wide range of interesting people on any topic.

80,000 Hours

“In-depth conversations about the world’s most pressing problems and what you can do to solve them.”

80k is the approximate number of hours the average person will work in their lives, and the idea behind this podcast is that you should spend them trying to make the biggest impact possible.

These are really in-depth (and usually really long) episodes on topics generally to do with improving the present or future.

Also worth listening to is their 80k After Hours podcast, which is on similar themes but a bit looser on structure.

Fall of Civilizations

“A podcast that explores the collapse of different societies through history.”

Similar to Hardcore History, but on a theme of failed civilisations. Episodes are pretty rare but very long - worth going through the backlog.

The History of the Twentieth Century

Unlike Fall of Civs, this podcast is comprised of many, many shorter episodes, progressing largely-chronologically through the 20th century.

I can’t actually swear this podcast never has advertising as at time of writing there are about 300 episodes, but I am on episode 61 and so far there is none. The host asks for donations via Patreon so hopefully there are none.

The Wright Show

Robert Wright interviews a load of different, interesting people. I particularly like his podcast with Mickey Kaus. They also do The Parrot Room together, which is listed under the paid podcasts.

Conversations With Tyler

Quite similar to EconTalk but with Tyler Cowen, who has a very distinct interview style which keeps the conversation moving quickly. Tyler is an economist, but like EconTalk the topics are quite broad.

The Entrepreneur First Podcast

EF is an organisation that brings aspiring company founders together and invests in them. Their podcast is various interviews with people who have gone through the process.

One of my friends has gone through EF, which is how I discovered the podcast.

If you like it, you can also try The Founder’s Mindset podcast which is on the same link.

The Bugle

I am generally not that in to comedy podcasts, but The Bugle has long been a favourite even after John Oliver left.

The Cryptid Factor

Despite just saying I don’t like comedy podcasts that much, this is a great one. Rhys Darby et al talk about Cryptozoology. Rarely released but a good backlog to work through.

Pushback with Aaron Maté

Giving an alternative view to a lot of coverage of American and worldwide issues. For example, interviews with Russian representatives before and during the war with Ukraine. Interesting to hear viewpoints that I don’t hear elsewhere, though possibly controversial. For example, Aaron had an interview with a Russian representative to the UN just before the Ukrainian invasion, who said a lot of things which turned out to be completely false once the war started. Still, entertaining to hear different viewpoints.

The Naked Pravda

Counterpoint to the one above - a Russian-made and Russia-focused podcast, but banned from Russia. Definitely anti-Putin.

Naturally Speaking

Possibly the most niche one here, scientific podcasts from the Institute of Biodiversity, Animal Health and Comparative Medicine at the University of Glasgow.

More or Less: Behind the Statistics

A BBC show which investigates any interesting numbers in the news, to see where they come from and how accurate they are. Ad-free if you are based in the UK or VPN in to it.

In Our Time

A podcast on ideas, people and events. It is an excellent podcast to fall asleep to. The BBC used to produce a lot of good ad-free podcasts (if you are based in the UK or VPN in to it) but unfortunately they’ve moved a bunch to their BBC Sounds app. I am not interested in that.

Hear This Idea

Interviews about philosophy, the social sciences and effective altruism.

Astral Codex Ten Podcast

A guy literally just reads Astral Codex Ten articles aloud. This can be very weird when there are a lot of diagrams that he dictates - if you’re into ACT though it’s a different way to consume the articles.

He has a patreon but I’m not into it nearly enough to donate.

Ad-free and paid

These podcasts are generally a few pounds/dollars a month. Some of these have free versions (but ad-supported).

Darknet Diaries

A podcast about hacking and computer security. This is a great, great podcast - the best episodes are fantastic, and even the worst episodes are better than most other podcasts. I intend to do a post just on the best episodes of Darknet Diaries.

My absolute favourite podcast, and sadly I am very close to getting through the backlog.

There is an ad-supported version so you can try it out for free first.

The Parrot Room

Every week Robert Wright and Mickey Kaus do an episode of The Wright Show for free, then do a longer subsequent episode for The Parrot Room. It’s essentially a continuation of the same conversation. If you like their Wright Show episode then it’s good value for money to keep listening.

This Week In Tech

TWiT is a podcast network of varying tech-related shows. You can get most of them ad-supported, or you can join the ‘club’ and get ad-free versions of them all, plus some bonus shows. I particularly like the following shows:

This Week in Tech

This Week in Google

Also worth a listen:

Security Now

Tech News Weekly

Oh God, What Now?

This is UK-focused, political podcast previously known as Remainiacs. Looking at the political news from the perspective of anti-Brexit, pro-European people.

The Bunker

Another UK-focused podcast, from the same people as Oh God, What Now?, less to do with Britain’s relationship to the world and more on general politics and culture.

Making Sense

Sam Harris’ podcast, often controversial but generally entertaining podcasts about current events. He is also very into meditation and the science of the brain but I don’t really get on with those episodes.

Honourable Mentions - ads not too intrusive

There are also some excellent podcasts which don’t have much advertising, or it is very unobtrusive.

Hardcore History

Dan Carlin’s Hardcore History is an amazing podcast. They consist of Dan Carlin telling a story for hours about a particular historical topic, and topics frequently run to multiple episodes.

Episodes are very rarely released but usually many hours long, and if you haven’t listened then there’s a huge backlog to get through (though most of the old ones are paid).

This almost made it to the ad-free list but he does have the occasional advert at the end of an episode. About as unobtrusive as can be. Some episodes have no adverts.

There’s also the Hardcore History Addendum for extra bits and interviews.

Common Sense with Dan Carlin

Another podcast from Dan Carlin, and another one with very rare episodes. USA-focused opinions from a libertarian perspective.

Same situation as Hardcore History with adverts, mostly at the end and not at all annoying.

Clearer Thinking

One of my favourite podcasts despite the adverts, Spencer Greenberg has really thoughtful ‘rationalist’ conversations about practical concepts and frameworks that can be applied to your own life. “Ideas that truly matter” according to the blurb.

The adverts are pretty inoffensive and often just advertise things on the Clearer Thinking website. Third-party adverts are at least for relevant products, rather than just having random auto-inserted ads.

Rationally Speaking

Quite similar to Clearer Thinking, Julia Galef interviews people about “the borderlands between reason and nonsense”. Good podcast, adverts aren’t too annoying, though there haven’t been any episodes for a while.

Lex Fridman

Lex interviews people on a wide variety of topics. Extremely in-depth conversations. I only listen to the ones that sound interesting from the titles, they are very long and I find Lex to be a little irritating - but the great guests often make up for it, and he does ask good questions.

In terms of ads, they take up about 8-10 minutes at the beginning of each episode, but Lex puts in bookmark timestamps so you can easily click to jump to the start of the actual conversation. After that, there are no more adverts.

Conclusion

That’s a pretty long list! Most of my podcast listening involves zero advertising, with some exceptions for particularly good podcasts.

I’m always looking for more to listen to so I’ll update this list when I do.

Theodore Roosevelt vs Speed Reading

2020-01-04T00:00:00+00:00

Theodore Roosevelt vs Speed Reading

The President manages to get through at least one book a day even when he is busy. Owen Wister has lent him a book shortly before a full evening’s entertainment at the White House, and been astonished to hear a complete review of it over breakfast. “Somewhere between six one evening and eight-thirty next morning, beside his dressing and his dinner and his guests and his sleep, he had read a volume of three-hundred-and-odd pages, and missed nothing of significance it contained.”

– from The Rise of Theodore Roosevelt by Edmund Morris.

This biography of Teddy Roosevelt was fascinating, especially considering I had no real interest in the topic before deciding to read it.

This may beg the question of why I read a huge book about him in the first place, for which the answer is - I played a lot of Red Dead Redemption 2, which caused me to get interested in the topic of cowboys, which led me to read “Cattle Kingdom - The Hidden History of the Cowboy West” by Christopher Knowlton (another excellent book, if that period appeals to you anyway), which features Roosevelt as a major ranch owner, which inspired me to find a biography of the guy.

He’s a fascinating character and inspiring in the sheer amount of work he gets done. It is just incomprehensible to me, a man who didn’t write a blog post for ten months due to laziness.

One thing that really stood out to me was his apparent ability to read at a prodigious rate. I read a lot of non-fiction (and some fiction) and would really like to increase the rate at which I read books. There are a lot of highly-rated books and not enough time to get through them.

I wouldn’t say I am a particularly fast or slow reader but I had heard of speed reading and decided to look into it.

However, as this blog from Scott Young describes, it seems like the whole idea of it is just impossible. Supposedly just going above 500 words a minute is improbable because of how the eye works, and getting up to that speed would sacrifice comprehension.

It seems like speed reading is just another fad idea which is not worth trying (file that along with barefoot running).

But the question I come back to is - how did Roosevelt manage to get through books so quickly and retain information? Maybe it’s an apocryphal tale, or maybe he was a genius - which does seem apparent from his biography.

On evenings like this, when he has no official entertaining to do, Roosevelt will read two or three books entire.

How??

N.B. A couple of great quotes:

Wine makes me awfully fighty - from Roosevelt’s diary

While its first edict, promising to “hang, burn or drown any man that will ask for public improvements at the expense of the County” could have been worded more diplomatically, it at least voiced sound Republican sentiments - about Wild West politics

Thames Trot Ultra Marathon Race Review

2019-11-30T00:00:00+00:00

48.3 miles || 8h32 || 37th out of 145 finishers (195 starters)

October 26th was the date of my first ultra-marathon, the Thames Trot. Officially it is called the ‘Thames Trot 50’ but in reality, it was 48 miles. It might seem like a marginal difference… but it does mean I can’t claim a 50-mile PB.

The event starts in Oxford and almost entirely follows the Thames path towards the finish in Henley-On-Thames.

My TLDR of my experience would be ‘the worst day of my life’. It was much harder than I expected and it rained all day long.

The race starts in a hotel car park but quickly gets on the Thames path, which it turns out is largely just mud. It had been raining in the run-up to the event and the path had taken the hit.

The race itself was well-organised. Although there was no on-course signage, everyone navigated by GPS watch and the aid stations were well-supplied. Plus all the marshalls were friendly and encouraging!

My longest training run was 24 miles, and although pretty tiring I felt OK afterwards. 24 miles is pretty short for the longest training run when attempting a 50ish mile race, but my main target was (and still is) the Country to Capital race in January 2020.

However, 24 miles on pavement turned out to be very different compared to the same distance on mud.

Going into the race, I was suffering a very minor calf strain and trying to rest it. I was concerned it would flair up and cause me to drop out. I have a terrible track record for pulling out of races injured.

mile 1 - feeling good and starting much too fast.
first 10 miles - I could feel my calf but was ticking off the miles pretty quickly.
20 miles - I was already in way more pain than I was expecting.
24 miles - suffering. It was around the half-way point that I started to take walk breaks. On the bright side, my whole lower half hurt so much that my calf was no longer a distinct pain.
miles 24-40 - this is where I fell to pieces. I was walking a huge amount, being overtaken by people I had passed earlier. - miles 40-47 - I regained some composure and started a pattern of running/jogging quarter of a mile, then walking for a short distance, and repeat.
mile 48 - I was so close to the end that I managed to run it in without any more walking breaks.

I was in pain so early that the second half became a real struggle, both mentally and physically. The mud just saps the leg strength so much more than running on pavement. Constantly recalculating how long you’re going to be out running in the rain was demoralising. I’m doing 11 minutes per mile - it’s going to be 4 hours. Now I’m going at 12 minutes per mile - 4 and a half!

My big mistake was running the first 20 miles too fast. In ultras, it is a very common tactic to walk the uphills as it takes too much out of your legs to run them. The Thames Trot is completely flat for the first 30 miles, so I ran without walking breaks until I could no longer keep it up, and by that time it was too late - I was wrecked.

With about 6 miles to go, I caught up to a guy who had overtaken me earlier in the day. He had just fallen over and was walking. I told him about my quarter-mile run strategy and he joined me with it. It turned out he was having an identical experience to me - first ultra, came in confident after a good long run, suffering much earlier than he expected. It didn’t stop the pain but it helped to pass the time.

The heaviest rain of the day fell during the last 6 miles. Crossing the finish line brought no real sense of satisfaction, but I was glad it was over. This was the worst experience of my life - the most difficult thing I have ever done.

Although it was a horrendous experience, I’ll be having another go in January 2020 - the Country to Capital 45 (which seems to actually be 43 miles). I’m hoping to start slower and be able to enjoy the experience a bit more!

Machine learning in Python with Scikit-learn - a crash course

2019-02-09T00:00:00+00:00

Or: how to fake your way through machine learning

At my place of work, we recently had a hackathon (during work hours!) in which we could spend 2 days trying to create something that would benefit the company. My idea was to use machine learning for this:

Predicting loan defaults

I would use existing customer data, with knowledge of who did and did not default (i.e. not pay us back), to then predict whether a currently-applying customer would default or not.

My knowledge of machine learning was almost entirely from doing Andrew Ng’s machine learning course on Coursera, but it had been about a year since I’d done it and had never implemented anything afterwards.

So my 2-day hackathon was a bit of a crash course in machine learning. Here’s how I did it:

The basics

This is labelled training. I had a large amount of data (which I obviously cannot share!) with each row containing information that we knew about a customer at their point of application, and whether or not they had defaulted.

This was a classification exercise. That means I was just trying to classify each new customer as predicted to default or not default.

The plan

I used Scikit-Learn, which seems to be a widely-popular library for doing machine learning in Python. It implements a wide range of machine learning algorithms so I would not have to do that work myself. This would be an exercise in plugging things together (I hoped!).

First steps - manipulating data

Scikit-Learn requires all inputs to be numerical. No text, no dates, not even any null/empty fields.

My data was made of mostly publicly available information (e.g. Companies House data) plus information the customer themselves provide. This included plenty of blank fields - for example, newer companies would have less information on Companies House.

Step one then - load in the data.

import pandas as pd

training_path = r'application_info.csv'

# Read in data
df = pd.read_csv(training_path)

Text/string fields

An example text field: industry category

Scikit-Learn won’t take string inputs. But you can’t just convert each string to a number - Scikit-Learn would treat that column as linearly related, but that makes no sense - if you label ‘Transport’ as 1, ‘Television’ as 2, ‘Catering’ as 3 and so on, you’re saying Television is halfway between Transport and Catering - which is meaningless.

Instead, you need to have each industry category be its own column, and set the values to be 1 (for yes, this row is in this category) or 0 (for not).

Scikit-Learn can do this for you, using a LabelBinarizer:

from sklearn.preprocessing import LabelBinarizer

lb = LabelBinarizer()
    
X = lb.fit_transform(df[label].values)
    
# add new generated columns to data
newColumn = pd.DataFrame(X, columns = [label+str(int(i)) for i in range(X.shape[1])])
    df = pd.concat([df, newColumn], axis=1)

# Delete old label, add new ones
features.remove(label)
for i in range(X.shape[1]):
   features.append(label+str(int(i)))

Getting rid of nulls

There are choices to be made here. These are your potential strategies:

delete rows with nulls. This was impractical for me - I would lose too much data
Replace nulls with something:
- 0
- The mean of the rest of the values in that column
- The median

Which makes the most sense? It depends on the data. For example, a customer’s total other loans - what does null mean in this instance? Should we assume they don’t have other loans, or just that we’re lacking that information? It’s a choice you have to make.

Whatever you do, you will affect your overall result. It’s best to play around with these (which I’ll discuss later).

Dates

You can do whatever you want here - it might just be easiest to delete dates if they are irrelevant.

I just ignored dates. But if you want to use them, you could extract numerical data from them, e.g. day of the week 1-7, time of day, year etc.

Starting to process data

What algorithm are you going to use? What parameters? How to start?

Scikit-Learn has a pattern which allows you to try a load of things at once, a method called GridSearchCV.

There is a great example of how to use this on my friend George’s Github page

from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import GridSearchCV

pipeline = Pipeline([
    ('scale', None), 
    ('predict', None)
])

# Set up some parameters
params = [
    {
    'scale':[None, StandardScaler()],
    'predict': [LogisticRegression()],
    'predict__C': [0.1, 1, 10, 100, 1000]
    },
    {
    'scale':[None, StandardScaler()],
    'predict': [SVC()],
    'predict__C': [0.1, 1, 10, 100, 1000]
}
]

gs = GridSearchCV(pipeline, params, scoring='accuracy', cv=StratifiedKFold(), return_train_score=True)
gs.fit(X, y)

print(gs.best_estimator_)

We define some parameters, define the algorithms we want to use, and search every combination of the parameters.

We have also introduced scalers, to scale the data.

This code will score each one based on accuracy (how many predictions were correct) and print the most successful combination of parameters.

Scaling data

Many machine learning algorithms require/prefer data to be scaled - i.e. all data in the same range. Scikit-Learn includes a bunch of scalers, and you can try out what works. I ended up using a MinMaxScaler, scaling all my data to between 0 and 1. This is something to play around with (and your experimentation can be automated, as above).

Scoring

Scoring by accuracy can be a bad idea - and in my case, it was. The % of people who default on loans is pretty small. Imagine it was 5% (not the real value in my data). Then any algorithm which predicts no one to default will score 95% accuracy.

This is exactly what happened to me - very high accuracy, absolutely useless result.

Alternative scoring methods are available.

This one is more useful in the instance where you have very few positive results:

Precision and recall

Precision and recall are alternative ways of evaluating models:

precision = true positives / (true positives + false positives) Conceptually, this means - of the customers I said would default, how many actually did?

recall = true positives / (true positives + false negatives) Conceptually, this means - of the customers who defaulted, how many did I correctly identify?

Example: 35 people default. You predict 50 will default, of which 25 are correctly identified and 25 are falsely accused. You also miss 10 defaulters, who you mark as good customers. Precision = 25 / (25 + 25) = 50% Recall = 25 / (25 + 10) = 71%

For further reading: Toward Science blog

F1 score

The F1 score tries to balance precision and recall, so is very useful in this case - and indeed in any case where your distribution of results is not evenly weighted.

Confusion matrix

A confusion matrix can be a really great way of visualising how your model is doing.

When using a confusion matrix, your aim is to maximise the numbers in the top left and bottom right - these are your true negatives and true positives. The bottom left and top right represent mis-categorised customers, so should be minimised.

In our case, we start by assuming no one defaults (before starting this project). Therefore all our customers are on the left - the right side of the matrix contains no one.

Our aim is to move as many people as possible from the bottom left (people who you say are good customers but actually default) to the bottom right (number of people you correctly identify as being defaulters) without increasing the number in the top right (number of people you accuse of being defaulters who are not).

You can get a text confusion matrix by comparing your test results with your test predictions:

from sklearn.metrics import confusion_matrix

matrix = confusion_matrix(y_test, y_pred)
print (matrix)

And draw an image as I have done like this:

import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
sns.set()

df_cm = pd.DataFrame(matrix, index = [i for i in "01"],
                  columns = [i for i in "01"])
plt.figure(figsize = (10,7))
sns.heatmap(df_cm, annot=True, fmt='g', cmap="PiYG", center=1)

train_test_split

To prove what you’re doing is right, you need to split your data into a training section and a test section. You ‘fit’ the algorithm using the training data, and test how well it works using the test data.

Anything else is cheating - you can get very good results by testing against the whole data set, but you cannot learn anything from it - it will not be a good predictor of future unseen data.

Scikit-Learn has a ‘train_test_split’ method which will divide up your data for you (see below).

Putting it all together

Once I’d decided on my algorithm with the most success - a RandomForestClassifier - I put it all together:

from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

# Now do the rest
y = df.Defaulted.values

X = df[features]

columns = X.columns
index = X.index

classifier = RandomForestClassifier(n_estimators = 10)

pipeline = Pipeline(
[
    ('impute', SimpleImputer(missing_values=np.nan, strategy='constant', fill_value=0)),
    ('scale', MinMaxScaler(feature_range=(0, 1))),
    ('predict', classifier)
])

x_train, x_test, y_train, y_test = train_test_split(X,y, test_size= 0.2, random_state=27)

pipeline.fit(x_train, y_train)

# make predictions against test data
y_pred = pipeline.predict(x_test)

So I’ve told Scikit-Learn to impute (fill) missing values with 0, scale all my data to between 0 and 1, and run a RandomForestClassifier with 80% of the data as my training set.

You can then plot a confusion matrix and see how well you’ve done.

Prediction confidence levels

Alternative prediction method:

# make predictions against test data
y_pred = pipeline.predict_proba(x_test)

This will give you the results along with confidence levels of the algorithm. Each ‘prediction’ has an estimate for its confidence of the negative and positive result, e.g. [0.2, 0.8]. These will always add up to 1.

If it’s important to be very confident, you could filter out everything under (for example) 80% confidence.

Conclusion

So that’s it - in 2 days I put together a fairly good estimator for whether someone would default on their loan or not, using the above code.

It doesn’t really require a huge understanding of machine learning, just the basics.

In summary, your key steps:

Massage data, remove nulls etc
Scale your data
Separate out training and test data
Fit model to training data
Test model against test data
Evaluate your results

And that’s it - from a basic understanding of machine learning, but with no experience of using it, you should be able to hack your way through a simple project like this.

Book Review: “Architects of Intelligence: The truth about AI from the people building it” by Martin Ford

2019-02-05T00:00:00+00:00

TLDR of book

Interviews with big names in the world of artificial intelligence
Most of them are very sceptical about ‘artificial general intelligence’ emerging any time soon
No one’s too bothered about Nick Bostrom’s worries written about in ‘Superintelligence’
Random change

Mildly interesting, very long

Sometimes I read a really highly-rated book, do not get on with it at all, and wonder if I completely missed the point. A recent example is “The Master and Margarita” by Mikhail Bulgakov, supposedly one of the best Russian novels of the twentieth century. I struggled. Afterwards, I looked up summaries of it and discovered that yes, I had completely missed the point. It’s full of symbols from Freemasonry. It is apparently a ‘response to aggressive atheistic propaganda’. I missed all this.

‘Architects of Intelligence’ is very highly rated on Goodreads, about the same as ‘The Master and Margarita’, and much like that, I struggled.

As a software developer, and someone with an interest in machine learning (not a huge interest, but an interest), I thought this would be really good. I enjoyed Nick Bostrom’s ‘Superintelligence’, although bits of it were a bit heavy going. He is, in fact, one of those interviewed in this book.

My key issues with this book are basically that the author asks the same questions to all the interviewees. Alright, that is what he set out to do - he explicitly says that’s what he’s doing in the intro. And I do get it - it means you hear what all these different experts think on the same topics, and you can see the differences in their opinions.

I am not so interested in AI that I actually had any idea who most of these people were. A friend who works in this area was quite impressed by the list of interviewees - he said the book started strong (in terms of the interviewees) and got weaker as it went on when I showed him the table of contents. For me though, I had only heard of a very small number of these - Nick Bostrom, Ray Kurzweil. That’s about it.

So for me, it was a lot of people I’d never heard of, giving largely similar answers to the same questions. Ford emphasises how different the opinions are on the question of when Artificial General Intelligence will emerge. I got the sense though, that there are such unsolved problems that need to be figured out before AGI can emerge, that no one really had any idea. Plus, most of them didn’t even want to make a prediction.

Good bits

Despite my being very negative, there were some interesting ideas:

every interviewer asked about self-driving cars was much more pessimistic about them than I would have thought. My knowledge is solely based on news/tech articles about self-driving cars, and from that I got the impression that they were a few years away. The interviewees as a group were much more likely to predict at least 10 years. That doesn’t bode well for Uber.
no one was impressed by the Turing test, which in itself is not a surprise. But one interviewee spoke about the ‘coffee test’. A robot is placed in front of a normal house, one that it has never seen before, and has to go in and make a cup of coffee. The robot would have to find the kitchen, find all the coffee-making apparatus etc.
Nick Bostrom spoke about the risk of amateurs flying drone across airports - something we’ve seen recently

Conclusion

I feel a bit guilty saying I didn’t enjoy this book, and I wonder whether it’s just not aimed at me. But I think I could have learned the same information from survey results. Most interviews were too similar to be interesting and the whole book dragged for me. Though based on the Goodreads reviews, this is not a popular opinion. Unless you are particularly fascinated by AI and have heard of some of these people - I could not recommend this book.

Running SQL Files from other SQL Files with MSSQL

2019-01-17T00:00:00+00:00

During my work with RoundhousE, I wanted to ensure the SQL scripts ran in the order I wanted. Foreign keys referenced things which had to exist first (for example an account has to belong to a customer, so the customer has to be created first).

RoundhousE, within a given folder, will just run scripts in alphabetical order. So you could name them “0001….sql”, “0002….sql” but what happens if you have 100 scripts and you want to insert one in the middle? Then renaming becomes a pain. Instead, we write one ‘master’ SQL file which will call the others (note - RoundhousE will only load SQL scripts, which is why I had to do it like this).

The data scripts I had generated were just lots of “insert …” statements, so nothing complicated. But this was a slightly more difficult problem to solve than I first expected.

Solution

Essentially we want to read in the file as a string, and then execute it as dynamic SQL.

Loading the file

DECLARE @FileContents  NVARCHAR(MAX)


SELECT @FileContents=BulkColumn
FROM   OPENROWSET(BULK'C:\temp\myfile.sql',SINGLE_NCLOB) x;

A CLOB is a Character Large Object in SQL, basically a huge list of characters. NCLOB is the same but in nvarchar format.

You might now see this message:

Error Number: 4809, Message: SINGLE_NCLOB requires a UNICODE (widechar) input file. The file specified is not Unicode.

OPENROWSET wants the files you read to be in Unicode, specifically UTF-16. I use Notepad++, so I clicked Encoding -> ‘Encode in UCS-2 BE BOM’ (which is essentially equivalent to UTF-16).

With that done, the file loads successfully.

Now we run it:

exec sp_executesql @FileContents

And your SQL in the other file (myfile.sql) should be run.

You can add multiple commands like this to run any SQL file you want (provided the encoding is correct!).

I thought this problem would be way easier than it turned out to be, so writing it up in case anyone else comes across this.

Integration testing with C# and RoundhousE

2019-01-16T00:00:00+00:00

Recently at work, I was tasked with creating some integration tests from scratch for a project which only had unit tests. The current way we run integration tests on other projects was too slow (adding data via Entity Framework) so they wanted a new approach. A colleague suggested RoundhousE.

RoundhousE

https://github.com/chucknorris/roundhouse

RoundhousE (that E is really annoying me) is, in their own words “a database migrations engine that uses plain old SQL Scripts to transition a database from one version to another”. It has versioning so you can easily know whether a database needs upgrading, or rollback to a previous version if necessary (and you’ve set it up properly).

However, all I want it to do is set up local databases for integration testing, then delete them afterwards.

Creating a database

Firstly, wherever you’re running this should have MS SQL Server installed and running.

RoundhousE packages rh.exe, which when called will do the following:

create a database if it doesn’t already exist (wherever you tell it to)
in a pre-determined order, run through folders looking for SQL scripts and run any it finds (check out https://github.com/chucknorris/roundhouse/wiki/RoundhousE-Script-Order for the ordering)

Easy right? I needed 2 databases for my integration tests, so I ran through the process twice.

This is the bat script for creating a database:

    SET DIR=%~d0%~p0%

    SET database.name=%1
    SET sql.files.directory=%DIR%db\%1
    SET server.database="(local)"

    :: roundhouse call
    "%DIR%console\rh.exe" /d=%database.name% /f=%sql.files.directory% /s=%server.database% /silent /drop
    "%DIR%console\rh.exe" /d=%database.name% /f=%sql.files.directory% /s=%server.database% /silent /simple

Note that I am dropping the table first, just in case it is still hanging around (for example, if the last run failed before tear down).

Generating data scripts

I’m using Microsoft SQL Server, so I used its functionality to generate scripts for creating the tables. One of my databases was very small, so I dumped the entirety of the data into scripts for RoundhousE to use.

The other database - our main database at work - was much too large to dump the data. Technically it would work, but the tests would be too slow to set up. I dumped individual tables of static data and created some scripts of specific user data. This was quite slow - I need a better tool to do this. After my proof of concept is finished, I will try to find something better.

Ignoring versioning

Part of the appeal of RoundhousE is its versioning tools. If you try to deploy the same version number to a database that you already deployed there, and any of the files have changed, RoundhousE will complain. Equally, if it detects the files haven’t changed, it will assume it doesn’t need to update.

In my use case, I just want it to deploy afresh each time. RoundhousE stores this versioning info in tables in the database, so before each test, I drop the databases (if they don’t exist then it just carries on). This way it redeploys every time and I pay no attention to RoundhousE’s versioning tools.

Piecing it all together

I used XUnit to run my tests. For this, you define a ‘database fixture’ like so:

    [CollectionDefinition("Database collection")]
    public class IntegrationTestBase : ICollectionFixture<DatabaseFixture>
    {
        // deliberately left empty
    }

We define our DatabaseFixture class which is responsible for the creation and dropping of databases. In the constructor, I call the bat file for creating the database (one call for each database).

DatabaseFixture will implement IDisposable, so in your Dispose() method you can add calls to a bat file to drop these databases (this is assuming you want them dropped after each run).

For any test class, we annotate it with:

    [Collection("Database collection")]

This way XUnit treats them as part of the same collection and will run the creation/deletion once.

End result - all working.

Using RoundhousE to run tests in parallel

If you want to run tests in parallel, just decouple the database setup and tear down from the tests running. Say you have 4 processes all running different tests, you would create the database, run each process, and once they’ve all finished, drop the database again. I did play around with creating different databases for each test run and got it working - but it was a bit messy and ultimately pointless.

Book Review: “Developer Hegemony: The Future of Labor” by Eric Dietrich

2019-01-08T00:00:00+00:00

TLDR of book

There are more software development jobs than there are developers - demand outweighs supply
Being a programmer in a corporation is a terrible financial deal, and the path to climbing the ladder through hard work is too slow to achieve within your lifetime. Corporations are not designed properly for software developers
As a developer, the only way to get ahead in a corporation is abandoning programming and becoming an opportunist - but this means sacrificing your ethics
Developers should quit corporations and create their own “efficiencer” firms, in which companies come to them with problems and these small firms design the solutions and create the software

Highlights

Opening chapters - fictional description of how the work of the future might look
Talking about ridiculous (but widespread) hiring/performance review/compensation practices
How to get ahead as an opportunist/sociopath in the corporate world (though why you shouldn’t)
A vision for the future of work, and advice about how you can get there

“Programming is not a calling, and it’s not a craft. It’s just automation that increases top line revenue through product or reduces bottom line costs through efficiency.” from chapter 32, titled “The Programmer’s Escape Plan”.

This blunt statement is one of a number of examples in the book about the future of development work that made me reflect about my own career in software. I’ve never really been one to be obsessed with ‘the craft’ of development, but I am sure that sentence would annoy a number of people I have met before.

There was plenty of accurate skewering of my own attitudes though. The pragmatist viewpoint is summarised by someone I work with - his internal profile on an app we use says something along the lines of “I have a real passion for staying alive and earning money allows me to accommodate that.” Pragmatists, according to Dietrich, accept the status quo and justify their lack of ambition by saying they don’t care about the job anyway:

“Pragmatists have their real thing that they care about, and it isn’t the job they’re doing. This is an entirely rational means of ego salve, similar to a teenager making a big to-do over how he doesn’t “believe in” the prom because of some philosophy or another that he’s adopted. Getting dates isn’t easy and the attempt may mean embarrassment, so it’s a lot safer to create a choice narrative around not trying in the first place.”

I.e. Pretend not to care, because it’s easier than accepting that you aren’t going to make a difference in your company. Not that Dietrich accuses ‘pragmatists’ of being lazy, but the descriptions were fairly spot on in places and made me laugh in self-recognition.

Getting ahead in the corporate world

Although Dietrich is at pains to state that he does not endorse any of the methods he gives for corporate advancement, the section where he talks about how to become an ‘opportunist’ within a corporation is fascinating in a morbid kind of way.

A memorable example he gives is of this: say you’ve taken over management of a team, and you realise they aren’t going to deliver a project on time. You have to spin the narrative that makes yourself look good, so you decide that the narrative will be that this team is overcommitting and underdelivering, but you will sort the problem out afterwards. To really make the point, you identify a team member who vocally commits to getting work done, talk to them privately, and encourage them to keep up the behaviour. Now this team member is even more vocal, and you can make the point to superiors that this problem already existed in the team, you didn’t identify it quickly enough, but you will sort it out - rather than the alternative narrative that you just couldn’t organise a team. There’s no evidence of your conversation with the team member - but plenty of evidence of him/her committing to too much work in meetings.

This example (definitely not advice!) is fairly immoral but I found this whole section to be really interesting, and in this case - pretty funny.

In fact, some of the most entertaining sections are Dietrich describing what is so wrong with the current state of work for software developers. I won’t go into much detail but you can tell just from some of the chapter names:

“Interviews, Induction, and Nonsense” “Performance Reviews and Advancement Theater” “Your Company Doesn’t Care About You” “The Madness Of It All”

In the chapter about interviewing:

“the job interview is a process that was dreamed up on a whim about a century ago, never worked in the first place, and hasn’t been altered since. Unsurprisingly, they haven’t magically started working.”

Although Dietrich acknowledges that he still interviews people because there isn’t (yet) an ideal alternative (although one is given in the ideal future work scenario that he envisages).

Any developer will have experienced the kinds of nonsense he describes in these chapters, and I took some definite pleasure in him being so scathing about it.

Sidenote : interviews

Personally, I think the current best way of doing this is to do real work with an interviewee - get them to do a project for 2/3 hours and then come in and talk it through. Possibly with a bit of pair programming between an employee and interviewee. However, this is really time-consuming for a candidate who has to do this for even a moderate amount of companies. Dietrich says he does a similar process.

A Corporation of One

One of the most interesting bits of advice is to stop thinking of yourself as an employee of a particular company, and that opportunists think of themselves as “single-person corporate entities unto themselves”:

“This may seem like a silly mental exercise at first blush, but its impact is profound. If you’re a software engineer, your boss asking you to put in a few fifty-hour weeks ahead of the upcoming release is perfectly reasonable, if something of a bummer. But if you’re a free agent, your client asking you to work ten hours a week for free is perfectly preposterous. The difference in thinking is that it’s now reasonable is for you to ask, “Why would I do that? What’s in it for me?””

He is keen to emphasise that his methods for getting ahead in the corporate world are not advice, simply fact. His real advice is for how to move towards his vision of working in very small firms which are essentially contractors.

He gives practical advice about how and why you should promote yourself and how to do it cough blog cough, and other more dramatic advice such as leaving your current job if you are a programmer because you are already pigeon-holed and the only way to escape that is to escape the company.

Conclusion

This book was a great mix of scepticism about the current environment and positivity about the future. Dietrich’s descriptions of future app dev firms (and what they should morph into) really grabbed me and I will be trying to do some of the things he advises to move in that direction.

I would highly recommend reading this book, especially if you are a software developer.