Yudkowsky is a strange character; he’s a key figure in my own book, The AI Does Not Hate You, about a group of people who worry about (among other things) an AI apocalypse. But he and they are also interested in why we are so bad at fixing the world’s problems.

Dominic Cummings, the senior adviser to the Prime Minister, is also interested in these ideas. I know this because we mix in similar internet circles. His latest post is a job ad, asking for maths nerds, data scientists, project managers and general “weirdos” to come and work with him in No 10. His first line is a quote from a 2017 Yudkowsky tweet, about “the bad Nash equilibrium in which the US and many other governments are stuck”. 

A few lines later, he mentions “low-hanging fruit — trillion dollar bills lying on the street”, an apparent reference to an ongoing theme in Inadequate Equilibria, based on an old joke about economics. The joke goes like this. There are two economists walking down the street; one of them says “Hey look! Someone dropped a £20 note.” And the other one says: “Well, it can’t be a real £20 note, or someone would have picked it up already.”

Suggested reading

The 10 brains behind Dominic Cummings

By Peter Franklin

The joke refers to the efficient markets hypothesis. If you think you can get rich quick by buying stocks, you’re probably wrong, because millions of smart people are also trying to do the same thing. And they’ll all buy stocks in the companies that they think will do well, so the stocks of those companies will go up. If a stock is even a tiny bit underpriced, hedge funds can make billions by buying it, and they’re watching every second, so it’s very unlikely that you — Random Citizen X — will have better information than they do. 

The efficient markets hypothesis obviously isn’t completely true. It is possible for people to make money in stocks; in reality, you do sometimes find notes on the floor. But it is mainly true: most stocks are appropriately priced, and if you were to see a £20 on the floor of King’s Cross Station in London with thousands of people walking past every hour, and it had been there all week, then you could confidently predict that something weird was going on.

But, says Yudkowsky, if you take this idea on a bit, you can end up accidentally proving that nothing can be wrong with the world. If you think that, say, your city’s public transport is overpriced and inefficient, then obviously someone could open a cheaper, more efficient bus service, undercutting the existing ones. It would make lots of money and put the other ones out of business. So saying “the public transport system is bad” is a bit like saying “there is a £20 note on the floor of King’s Cross and no one has picked it up”. Medicine can’t be improved because, if it could be, someone would already have done it. Science can’t be broken because if it was, someone would have made an un-broken version and won all the Nobel prizes. House prices can’t be too high. If you’re not careful, you end up as Leibniz or Dr Pangloss, saying that all is for the best in the best of all possible worlds. 

And yet, Yudkowsky points out among several other examples, 33% of babies with short bowel syndrome die in the US because the standard FDA-approved intravenous nutrient fluid has the wrong mix of lipids. The correct mix has been known for decades, is just as cheap, and prevents all the deaths. No one wants babies to die. But somehow it doesn’t get fixed. This is very hard to square with the Leibnizian position.

By the same author

Dominic Cummings is no chicken

By Tom Chivers

I think this is what Cummings is driving at. It’s hard to tell, because it’s just a job ad, and even his longer pieces seem to me, usually, like a grab-bag of loosely connected tech/management/AI buzzwords, rather than a coherent argument. But I think he’s interested in why things go wrong even when no one wants them to. It’s easy to blame bad people — the Tories, or billionaires, say — and maybe that’s sometimes true, but I suspect it’s usually a comforting self-deception; it’s much gloomier to think it’s a systemic failing of human thought.

Yudkowsky suggests that there are three main ways in which societies can get stuck in these “inadequate equilibria”, bad situations that no one wants but which we can’t get out of. 

The first is bad incentives. The efficient markets hypothesis, and the free market in general, relies on people acting self-interestedly. If everyone knew that the £20 lying on the floor of King’s Cross would have to be donated to HMRC if they picked it up, then people would be less likely to do so. 

An example of this that Yudkowsky uses is the Japanese central bank, which for years kept a tight monetary policy. Many economic bloggers, including Scott Sumner, suggested that printing more money would be better. But because the Japanese central bank’s monetary policy was set by a committee of a few bankers, none of whom would make any extra money if Japan’s economy improved, the market had no way of influencing it.

Suggested reading

Will maths succumb to the woke wave?

By Douglas Murray

The second cause is bad information. If I am buying a used car from you, I might ask if the car is any good. You will say “yes”. But you would probably say that even if it weren’t. So I can’t work out what the right price to pay for it is. So the used-car market becomes a lottery: good cars which should sell for more money don’t, and bad cars get sold for more than they’re worth. You can’t tell, among all the people saying “this used car is great”, which ones are telling the truth. 

Similarly, in cases like the short bowel syndrome babies, there probably are hundreds or thousands of parents and doctors screaming: “Look, babies are dying because of this simple mistake.” But loads more people are also screaming about babies dying because of vaccines or austerity or any number of more or less spurious reasons. And it’s hard to tell which ones are real and which ones aren’t. So lots of real ones get ignored and lots of fake ones get picked up. You can’t tell which ones are telling the truth. 

The third and final cause is the “bad Nash equilibria” mentioned at the beginning. A Nash equilibrium, in game theory, is a stable state in which the players cannot improve the situation by changing strategies. For instance, in the prisoner’s dilemma (if played only once), the two players will always do best if they defect, even though that’s worse than if both of them cooperate, because if one cooperates while the other defects it’s worse for the sucker. You need to coordinate both players to cooperate at the same time, and that’s hard.

An analogous situation (lifted from Scott Alexander) is that no one wants to be on Facebook — it’s terrible and incomprehensible and it steals your data. But no one can leave, because going off and forming your own social media system isn’t going to work. You’d just be there on your own. To leave, you need to coordinate millions of people to leave at the same time, and that’s hard. Google tried it, and failed utterly; if Google can’t do it, you can’t.

suggested reading

What the AI doomsters get wrong

By Graeme archer

I think (I hope) that Cummings’s job ad is about trying to get people who think in these terms into places of government. I don’t know whether it will make a serious difference. But there are problems in the governance of the UK which analogise quite well to these situations. 

For instance, incentives. No one I’ve ever spoken to who’s involved in drug policy — not academics, not clinicians, not police — thinks that it’s a good idea to imprison people for possession offences. And after they leave power, every politician seems to say how important it is that we stop doing it: even former home secretaries who could once have done something about it. But still, we do it, because the people in power are not incentivised to save the exchequer money or to save lives – they’re incentivised not to look soft on crime.

Or, bad information. I notice that Cummings mentions Philip Tetlock in his blog post. Tetlock, whom I’ve written about before, is the thinker behind the Good Judgement Project and “superforecasting”. The groups he works with get the best results — better than CIA analysts — at predicting future world events, and there are fairly simple methods that they use to improve those skills. A better skill at forecasting the future would be immensely useful for the UK government. But there are thousands of cranks pushing “one weird trick”-type solutions, and it’s very hard for government advisers or ministers to tease out which are good and which are bad.

And finally, Nash equilibria. I think most people who care about the subject nowadays agree that our First Past The Post (FPTP) voting system is no longer fit for purpose. But to change the system, you first have to win under FPTP, and once you’ve won under FPTP, your best strategy is to keep playing under it. To get out of it, you have to coordinate all the players to change at the same time, and that’s hard.

By the same author

How politics makes us irrational

By Tom Chivers

Bringing in maths nerds and weirdos won’t fix any of this on its own. The incentives and bad information and coordination problems will still exist. But it’s pleasing that someone close to government is thinking about it. Anecdotally, lots of nerdy oddballs in DM groups and internet forums I hang around in, people who would never consider themselves for government jobs normally, are thinking about applying; I saw a post on an effective altruism site calling for people to apply. Maybe it’s just that these are my sort of people, but I’d be happy to see more nerdy, techie types in government.

It could go wrong. Lots of very sensible clever people I admire have written things pointing out that a naive belief in the power of maths to fix problems won’t get us anywhere, and that the Civil Service is actually much less lacking in maths-ability than the ministers who make the decisions. And that’s true — and I also think that there’s probably a lot of hard-to-understand local knowledge in the Civil Service, and if you sweep all that aside and replace it with nerdy 23-year-olds it’ll go badly wrong (Chesterton’s fence and all that). It’s a bit unfair to say Cummings is only interested in maths and science nerds — only one of the seven job categories he mentions is explicitly STEM, the “data scientist” – but that’s clearly his focus.

But I do think there are some relatively simply fixed problems in the governance of this country which could, in theory, be fixed quite easily, by a government with a big majority which doesn’t need to care about short-term popularity and which is willing to do some fairly unusual things.

What about, say, some proper A-B testing of policies? It can, as I’ve mentioned before, do real good but which is weirdly unpopular in political situations. Or how about taking some hot-button things out of direct political control (I’ve seen people suggest the NHS!), in the same way the Bank of England was by Blair? That would avoid some of the bad Nash equilibria, and the bad incentives – especially if combined with reforms that made those directly in charge of it responsible for whether it works or not.

But don’t ask me. I’m not weird enough to apply. Maybe we should just pick up a few of those £20 notes lying on the floor of King’s Cross Station instead.