It may sound obvious that you should take both: you get the extra thousand in either case. But the twist is that, before Omega filled the boxes, it predicted what you would do. And if it predicted you would take just the opaque box, it put the million pounds in there. If it predicted you’d take both, it left the opaque box empty. It knows you know this. It’s done this to 100 other people already, and it’s successfully predicted their choice 99 times. What is the rational, optimal thing to do? Do you two-box or one-box? 

This thought experiment is called Newcomb’s paradox, and it has been a source of heated philosophical debate for more than 50 years. The philosopher Robert Nozick, in a long and agonised discussion of the problem, wrote in 1969:

“I have put this problem to a large number of people, both friends and students in class. To almost everyone it is perfectly clear and obvious what should be done. The difficulty is that these people seem to divide almost evenly on the problem, with large numbers thinking that the opposing half is just being silly.”

More from this author

The dangers of festive over-indulgence

By Tom Chivers

The book Rationality: From AI to Zombies mentions this problem. Its author, Eliezer Yudkowsky, acknowledges the philosophical difficulties. But he has a robust and sensible approach. The rational course of action, he says, is the course of action that gives you a million pounds, rather than the one that doesn’t.

In the early 2000s, the young Yudkowsky had something of an epiphany. He was only 20 or so, but he was already quite a well-known figure in certain subcultures on the early modern internet: the transhumanist, technophile set who talked about cryonics and uploading our brains into the cloud.

This was before social media, so these subcultures existed as mailing lists. Great long nerdy conversations went on in people’s emails. Yudkowsky set up his own, SL4; it was about how to bring about the technological Singularity, where advances in computing and AI will render human life unrecognisable. He also founded what was then known as the Singularity Institute (now the Machine Intelligence Research Institute), with the same goal. 

The idea was to bring about superintelligent AI as quickly as possible, because, he thought, being superintelligent, it would be clever enough to fix all the world’s problems: “I have had it,” he wrote, as a 17-year-old in 1996.

“I have had it with crack houses, dictatorships, torture chambers, disease, old age, spinal paralysis, and world hunger. I have had it with a planetary death rate of 150,000 sentient beings per day. I have had it with this planet. I have had it with mortality.

“None of this is necessary. The time has come to stop turning away from the mugging on the corner, the beggar on the street. It is no longer necessary to look nervously away, repeating the mantra: ‘I can’t solve all the problems of the world.’ We can. We can end this.”*

The reasoning went, very roughly: human intelligence has made human life much better, over the last few centuries; if we make something much more intelligent, it will fix the problems that remain.

Suggested reading

Are humans heading for the knacker’s yard?

By Peter Franklin

But some time between 2000 and 2002, Yudkowsky had the aforementioned epiphany: just because an AI is intelligent, doesn’t mean it will do what we want it to do. Instead, it will do what we tell it to do. And the difference between those two statements is huge, and potentially disastrous. AI could, he said, ultimately destroy humanity, if we are not immensely careful about how we build it: not because it “goes rogue”, or “becomes sentient”, but simply because it follows its instructions to the letter, rather than in spirit.

Yudkowsky had, though, a devil of a time convincing people of this. If it’s intelligent, it would know what we wanted it to do, his critics argued. It would know the right thing to do.

So, in 2006, he started to write a blog, to explain why they were all wrong. He had to explain why artificial intelligence wouldn’t be like human intelligence. But, he realised, to explain that, he had to explain why human intelligence wasn’t exactly like rational thought — it is full of biases and shortcuts and systematic errors.

But to explain why human thought wasn’t like rational thought, he found, he had to explain what rational thought actually was — in the decision-theory sense, of making optimal decisions with the available information. And then, he found, he had to explain all the ways in which human thought differed from rational thought: he had to list all those biases and shortcuts and systematic errors.

And to explain those, he had to explain — almost everything. At one point, he wanted to write a relatively unimportant post using an analogy of AI utility functions. But in order to do that, he felt he ought to first explain a few other concepts. To explain those, he realised, he needed to explain a few more. But to explain those…

Anyway, long story short, over the following month he ended up writing about two dozen blog posts explaining evolution by natural selection. And then he could write the not-especially-central-to-his-thesis post about the utility functions.

Suggested reading

The horror of deepfake nudes

By Nina Schick

The whole series of blog posts, which became known as the Sequences, sprawled out to a million words. It was edited down into an ebook version, Rationality, which is a more manageable 600,000 or so — though still rather longer than War and Peace, or than all three volumes of The Lord of the Rings put together.

The Sequences are the founding text for what would, later, become known as the “rationality community”, or the “rationalists”. I’ve written about them, and their AI concerns, a few times in the past; my own first book, The Rationalist’s Guide to the Galaxy, is about them. (It’s also just 80,000 words and gets most of the key ideas across, if you are pressed for time. All good bookshops!) They are a strange bunch: clever, thoughtful, sometimes paranoid; they get accused of being a cult, with the Sequences as their holy scripture and Yudkowsky as a sort of messianic figure (although I think that is not true.)

But the community is that rare thing, on the internet: a place where people can disagree and argue calmly and in good faith. And it sprang up around Yudkowsky’s great sprawling series of blog posts, because he explicitly encouraged that norm.

As well as — or as part of — trying to lay the foundations for AI safety, it tries to establish what it means to be rational, and to make us more so. In other words, the aim is to help us be “less wrong”, as the blog the Sequences are published on is called. Following decision theorists like Judea Pearl, he defined rationality in two forms.

First is epistemic rationality, establishing true beliefs about the world — maximising the extent to which my mental model of the universe, the “map” in my head, corresponds to the universe itself, the “territory” of reality. I believe that there is a door behind me, and that black holes emit Hawking radiation: those are facts about my brain. Whether there actually is a door behind me, or whether black holes really do emit Hawking radiation, are facts about reality.

Epistemic rationality is about trying to make the beliefs line up with the facts. “This correspondence between belief and reality is commonly called ‘truth,” says Yudkowsky, “and I’m happy to call it that.”

More from this author

Dominic Cummings is no chicken

By Tom Chivers

The second is instrumental rationality, choosing the course of action that is most likely, given what you know now, to bring about the things you want to achieve. If you want to achieve Goal X — wealth, world domination, world peace, preventing climate change, Oxford United winning the Champions League — what steps are most likely to achieve that? Rationality, he says, means winning. Winning at whatever you want to win at, whether it’s saving the world or making money. But winning.

I can’t begin to go into the whorls and sprawls of its reasoning: in its efforts to ground morality and rationality on Bayes’ theorem and utilitarian calculus, Rationality reminds me of one of those huge Enlightenment-era works of philosophy, by Spinoza or Leibniz or someone, that start out with some guy sitting in an armchair and end up trying to establish a Theory of Everything. But I do want to come back to Newcomb’s problem, because I think it is key.

In 1969, Nozick concluded, after much agonising, that you should take both boxes. Imagine the far side of the opaque box is transparent, he says, and that a friend of yours can see into both. Whatever Omega has done, whether the £1 million is there or not, she would be hoping that you take both: if it’s not there, you at least gain £1,000; if it is, you get a bonus £1,000 on top of your million. It’s already happened. The money is in the box, or it isn’t.

Yudkowsky, though, disagrees. The rational thing to do, he says, assuming that you would like a million pounds, is the thing that is most likely to make you a million pounds. Almost everyone who chose to one-box got a million pounds. Almost everyone who chose to two-box didn’t. Only a very clever person could convince themselves that the sensible thing to do is the option that will almost certainly cost you £1 million.

Suggested reading

Facts vs feelings in the BLM debate

By David Goodhart

That’s why I like the book. It may address very serious, high-concept ideas — AI, transhumanism, morality, alien gods visiting the planet and playing strange parlour games with boxes — but it gets to them all in a robust, commonsense way. The moral thing to do is, usually, the thing that kills the fewest people, or makes the largest number happy. As he writes, in a wonderful section on how it seems obvious that consciousness has an effect on the world, because philosophers write books about consciousness, “You can argue clever reasons why this is not so, but you have to be clever.” 

But following these robust, commonsense views to their logical conclusion often ends up in strange places. If you accept some fairly uncontroversial premises, you can end up with AIs destroying the world, brains uploaded to the cloud, galaxy-spanning civilisations of quasi-immortal demigods.

It’s been especially relevant this year. The rationalist community was well ahead of the public, and of many scientists, in spotting the dangers of Covid-19: they are used to data, to exponential curves, to reasoning under uncertainty. They, and the wider tech community, were using masks and stocking up on essential goods, even as others were saying to worry about the flu instead, or mocking tech workers for avoiding handshakes. And now, when we should be working out the risks to ourselves and our families of going home for Christmas, a bit of extra familiarity with Bayes’ theorem and probability theory would probably not be a bad idea.

In the 12 years or so since the Sequences ended, Yudkowsky himself has withdrawn somewhat, working at MIRI or on other projects, occasionally cropping up on Facebook or Twitter. But they remain as a fascinating piece of writing, pulling together a thousand strands of thought into one place. And, whether or not you agree with their conclusions on AI or rationality, they have created a small, safe harbour for peaceable disagreement on the internet. That alone is a legacy worth having.

*Precocious 17-year-olds can be quite annoying.

Virgil, Aeneid.

Magic is a belief in short cuts. Whether it is turning lead into gold, making beautiful women fall in love with you, or transforming lepers into Olympic athletes, magic promises an instant route to where we want to go – quick as a flash, abracadabra. This fantasy of the philosopher’s stone appeals just as much today as it ever has, perhaps even more so.

That magical thinking persists in an age of science is not so much that people continue to be inexplicably stupid and superstitious, refusing to understand and accept the success of the scientific revolution; but rather that they continue to be, as humans always have been, wishful – a word that describes a point on the spectrum that slides dangerously between hopeful and deluded. At its core, a belief in magic is a belief in quick power. This is why magic is a ‘dark technology’. That is why magic is widely seen as dangerous.

In Marlow’s play, the ageing academic Dr Faustus – frustrated with conventional wisdom – summons a dark power in order to extend his life. He renounces his baptism and makes a deal with the devil. He has found a short cut. But this short cut is ultimately disastrous.

If Marlow were writing Faustus today he might make the hubristic academic someone like Ray Kurzweil, the director of engineering at Google. Kurzweil promises that the accelerating power of technology will one day conquer death itself. The day is coming, this modern-day necromancer insists, when we will be able to upload our core being into some cloud of software and live forever. We don’t think of this as magic because we have been persuaded that science is the very opposite of magic.

Further reading

Who wants to live forever?

By Nigel Cameron

This assumption seems patently obvious to people. As the sociologist Max Weber claimed, the modern, scientific world-view is one in which:

“…no mysterious forces are in play, eluding calculation, and that our calculations can (in theory) master everything… We need no longer to rely on magic as a device for mastering spirits or pleading with them – unlike the savages for whom such forces used to exist. Calculation and technical equipment do the job.”

It is a telling passage, for while it imagines magic as being overthrown by “calculation and technology” it also accepts that they “do the [same] job”. That is why the futurist and science fiction writer Arthur C Clarke was correct when he claimed: “Any sufficiently advanced technology is indistinguishable from magic.”

To be fair, the persistence of magic in an age of science has nothing to do with proper science or proper technology. Rather, it has to do with our over-inflated expectations of their power. Most of us non-scientists now rely on things for our everyday existence that we do not begin to understand. When we text our friends, surf the web, take a flight, or even turn on a light switch.

Further reading

Why flawed science should concern us all

By Tom Chivers

Some of us have a sketchy idea how these things work. But for most of us muggles, we rely on that which is beyond our understanding. And so we get used to the idea that there are forces beyond our ken that have this massive power to fix things. We end up trusting in the (to us) mysterious to solve our problems. We don’t know how it works but we believe it anyway. We end up believing in magic.

Perhaps the most dangerous form of magical thinking today comes when it is claimed that technology will be able to sort out the existential crisis of climate change. As scientists agree, the world has warmed up by 1.5 degrees since the advent of the Industrial Revolution, that coalition of technological ‘advance’ and international capitalism. The magical thinking of technology is that we can continue to dig for coal, fly on planes, eat meat; that we can continue to go on living beyond our environmental means, and that it doesn’t really matter because, one day, someone will come up with some (magical) technological fix. This is wishful thinking.

Our culture has surrendered unquestioningly to the power and promise of technology. It has become the omnipresent grid by which we chart our way in the world – chips with everything – offering us salvation from the threat of the unknown. Through technology we imagine ourselves to be masters of the universe. And that we are more advanced than those Weber called the “savages” that went before us. Technology flatters us with the fantasy of our specialness. We are now a super species, approaching the power of the gods themselves.

Further reading

Have our scientists run out of steam?

By Peter Franklin

Forget Harry Potter. The place where that magical thinking is flourishing today is within the world of technology. The lead magicians of our times are fêted as public figures. They work in NASA and Apple and Tesla. Elon Musk imagines we can evade human extinction by founding a colony on Mars. Post-humanists imagine Homo sapiens evolving into some entirely new species; stronger, faster, longer-lasting.

Algorithms curate our reality, determining our online experience, knowing us better than we know ourselves. Technology in agriculture will provide more food for the needy. Artificial intelligence surpasses human intelligence, promising potentially limitless solutions to the human condition. DNA wizardry imagines it can create life itself in a petri dish. “Thy kingdom come,” we all intone. As with magic, we are mastering the unknown. There is no need for us to be afraid of the dark any more. Technology is the new sacred, the new salvation. All hail!

Technology, once a means to an end, has now become an end in itself, and an object of worship. And woe betide those who question this new realm of the sacred, for they will be met with the same angry reaction as those, throughout history, who have dared to question the gods: “All criticism of it brings down impassioned, outraged, and excessive reactions,” as the great Jacques Ellul wrote in The Technological Society.

Further reading

All hail the AI dictatorship

By James Bloodworth

Why is magic so dangerous? Because it refuses to accept the existence of limits. Faustus is a morality tale about a man who transgressed the limits of what it is to be human. Likewise, climate change is turning into a morality tale about a planet that has transgressed the limits of sustainability. Both are tales of human hubris: and with climate change we have become like Icarus, flying too close to the sun.

Real science rightly pushes against the boundaries of human ignorance. But there are certain boundaries that it cannot wipe away because they are the constituent conditions of what it is to be a human being. The boundaries Faustus perceived to be a limit were, in fact, the defining features of what it is to be human. Think of it a bit like this: Faustus pushed over the walls of his existence, thinking they imprisoned him, not appreciating they sustained the roof over his head. This is how magic destroys us. And why, for centuries, people have been afraid of it. Remember: some things are just too good to be true.

From the same author

Why we need more superstition

By Giles Fraser