They have dreamed up a dystopia for themselves to rule. (The Matrix)
Earlier this week, cybersecurity experts discovered something nasty online: two critical flaws in PHP, the code that has been used to build approximately two-thirds of the World Wide Web. On Tuesday, various European governments shut down websites as they scrambled to fix the flaws before malign actors were able to exploit them. Millions more websites will now need to be “patched”. “For the big juicy targets, I hope to goodness they are being patched quickly,” said one cybersecurity expert.
In this case, AI might just have saved the day. The flaws were discovered because cybersecurity experts used AI to search for weaknesses that humans had missed. Had, say, AI-assisted Russian cybercriminals got there first, it might have been far worse. The incident gives an insight into the double-edged nature of AI, which presents clear and present dangers — but also, new tools to combat those dangers. Note, however, that we are talking about the fairly unsexy business of cybercrime and not, for example, sentient computers spontaneously deciding to eradicate humanity or farming our living souls in some sort of Matrix nightmare. And yet it is these scenarios, dreamed up by a small set of extravagantly funded AI doomers and spread by a credulous media that have occupied the world’s attention; and have gone into overdrive since the Hugging Face hack and Jacob Coxon’s resignation from Anthropic.
For “normies”, anxiety about a new emerging technology tends to centre on things like jobs, education, or the economy. Cybersecurity experts tend to worry about tangible threats to existing systems. But AI doomerism is distinct. It describes people who believe that advanced AI poses a risk of human extinction — and are, generally, prepared to put a number on it. Coxon’s former colleague Evan Hubinger set his own “p(doom)” at “above 10%”; some go as high as 25% or even 50% .
Much has been written about the strange confluence of money, power, and quasi-theological belief systems that have given rise to this apocalyptic worldview. Still, what receives less attention is what this movement actually wants to happen in order to avert this admittedly unpleasant scenario — which is to say, the authoritarian and even totalitarian implication of the AI doomers’ policy proposals. Ideas of mass surveillance that would make Big Brother blush have almost imperceptibly crept from the fantastical thought experiments of Effective Altruists and the Rationalists and are being now turned into policy proposals by such figures as Bernie Sanders.
To understand the peculiar flavour of AI doomerism, a good place to start is a 2019 TED talk given by Oxford’s Nick Bostrom, one of the leading philosophers on existential risks, titled “How civilisation could destroy itself—and 4 ways we could prevent it”. Bostrom opened his talk with a thought experiment. He asked his audience to imagine an urn containing various balls representing technological advances. There would be some white balls, representing beneficial discoveries. There would be some grey balls, representing discoveries that have some positive and some negative benefits. But there might be a few black balls: discoveries so profound, they would effectively make political life as we know it unsustainable. One example might be synthetic biology: imagine if your average supervillain really did gain the ability to create mutants! Another might be AGI.
Bostrom’s point was that once the ball is withdrawn from the urn, you can’t put it back. Therefore, the only viable technological safeguard would be to stop anyone from withdrawing any black balls in the first place. And the only real way of doing this, he mused, was: “Ubiquitous surveillance — everybody would be monitored all the time.” The interviewer’s thoughts turned immediately to dystopia: “This is Minority Report, essentially,” he said, referencing the 2002 Tom Cruise movie about a future police state that uses psychics to arrest criminals before they have a chance to commit their crimes. Bostrom duly admitted that there were “huge problems and risks” associated with the idea of mass surveillance and global governance. But perhaps, he mused, people would be more willing to wear multidirectional cameras and microphones if we called this “The ‘freedom tag’… or something like that”. Still, he suggested that “we seemed to be doomed anyway”, by which logic, extreme solutions may come to seem proportionate.
Over time, this “end justifies the means” mindset moved from thought experiment to policy proposal, with increasingly concrete ideas for controlling AI research and the people working on it. With the work of the Machine Intelligence Research Institute (MIRI), the case for totalising enforcement became more explicit.
MIRI is a Berkeley-based nonprofit organisation, founded more than two decades ago by the writer Eliezer Yudkowsky and backed by a remarkable concentration of money and influence. Its funders include Open Philanthropy, the philanthropic organisation founded by Facebook co-founder Dustin Moskovitz and Cari Tuna; Vitalik Buterin, the co-founder of Ethereum (an open-source blockchain); and Jaan Tallinn, the Estonian programmer who helped build Skype and has since made existential risk one of the central objects of his philanthropy. Tallinn is a particularly important figure in this ecosystem: through the Survival and Flourishing Fund, which he funds alongside other philanthropists, he has helped direct hundreds of millions of dollars towards organisations concerned with AI existential risks.
MIRI’s particular obsession is artificial superintelligence (ASI) — a hypothetical scenario in which computers develop vastly superior intelligence to humans on every vector. Its position on ASI is: “Everyone dies.” Yudkowsky therefore argues that the only reasonable response is a globally coordinated and collectively enforced moratorium on the development of ASI.
This may not worry you, since you (I assume) have no intention of trying to create ASI. But the wider problem is enforcement. If governments agree to halt this kind of research, how do they know companies or individuals are obeying? In June, MIRI’s Technical Governance Team published Verifying Restrictions on Frontier AI Research, which considers how such a prohibition could actually be policed. The paper catalogues 28 possible verification mechanisms, including intelligence gathering; international search warrants and inspections (of properties, computers, and files); polygraphs; monitoring what users type into AI systems and what the model produces while it is being used; sting operations; employee-monitoring software; and AI-assisted code review, which may extend to codebases, experiment logs, and research documents. It is a proposal for giving governments unprecedented oversight of private activity. Researchers would work under permanent clouds of suspicion, never knowing which of their experiments might be deemed illicit, which conversations were incriminating, which of their colleagues was an informant.
This really is a dystopian nightmare, and, unfortunately, not the dystopia the media is covering. Journalists have spent years asking readers to imagine the dictatorship of future machines while paying far less attention to the systems of human control being proposed to prevent it.
MIRI itself acknowledged that such mechanisms could “enable authoritarianism” and cautioned that some “might be undesirable.” And yet thanks to the well-funded ecosystem of organisations surrounding MIRI, these ideas have now migrated into public policy proposals.
On 23 September, Senator Bernie Sanders and Representative Greg Casar introduced the Ban Artificial Superintelligence Act, which would permanently prohibit the development and deployment of artificial superintelligence and immediately pause advanced AI development until a new federal regulator was established to enforce the bans and strict penalties. The bill would create a cabinet-level Department of Artificial Intelligence with powers to monitor frontier AI systems and oversee their destruction. Individuals who violated or circumvented the ban could face up to 20 years in prison; companies could face what the sponsors call the “corporate death penalty”. That is a remarkable amount of government power over a technology that doesn’t even exist.
“A new age of AI radicalism is upon us,” concluded Adam Thierer and Nicholas Thielman of the R Street Institute, describing the Sanders-Casar framework as one of “the most radical interventions ever proposed in any piece of proposed legislation in American history”. Their report warned that “if enacted, it would have extremely destructive and dangerous consequences for innovation, competition, economic growth, global competitiveness, national security, and freedom of speech.”
They label the proposals “AI Authoritarianism”, arguing that it represents an unprecedented expansion of government control over technology and potentially over information itself, sharply restricting what companies, scientists, and individuals are allowed to do. Even pro-compute governance groups such as GovAI have cautioned that such a regime could “also be used to infringe on civil liberties, perpetuate existing power structures, and entrench authoritarian regimes.” Indeed, it could reasonably be argued that its implications are totalitarian — since enforcement will necessarily extend into private intellectual activity itself, through the monitoring of researchers’ prompts, code, messages, and devices via a pervasive surveillance system designed to prevent even private non-compliance.
These ideas have also found an energetic British advocate in ControlAI, a non-profit founded by Andrea Miotti to campaign against the existential risk posed by superintelligent AI. Its policy proposal also includes a 20-year AI pause. Backed by figures including Tallinn, the organisation has spent the past two years briefing politicians in Britain, America, and elsewhere. In September, its UK campaign achieved a milestone: Labour MP Alex Sobel introduced a bill, drafted by ControlAI, that would ban the development of superintelligent AI in Britain for 20 years and require the Government to pursue an international agreement banning it worldwide. ControlAI also drafted a separate AI “Kill-Switch” amendment, designed to grant the government emergency powers to shut down AI models and data centers. And its US arm also consulted with the offices of Bernie Sanders and Greg Casar on their Ban Artificial Superintelligence Act.
The moratorium is extremely important to the AI doomers. Writing in Time in 2023, Yudkowsky warned readers: “Be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike.” Yudkowsky’s argument treats interstate military conflict, potentially involving nuclear-armed powers, as a lesser danger than his hypothetical risk of AI-induced catastrophe. In that framework, even conventional geopolitical dangers become subordinate to the threat he assigns to AI.
If it were up to Nick Bostrom and Eliezer Yudkowsky, we would therefore have a “Singleton” — a single, centralized, global decision-making agency with the power to monitor all computing power, an entity that would require absolute mass surveillance and the dismantling of national sovereignty.
Their ideas have roots in two overlapping subcultures: the Californian Rationalists and Oxford’s Effective Altruists, who by the early 2010s, were converging around longtermism: the idea that safeguarding humanity’s distant future should be a central moral priority. Figures such as Dustin Moskovitz, Sam Bankman-Fried, and Jaan Tallinn leverage the logic that EA had applied to philanthropy: if there was even the tiniest chance AI might one day wipe out humanity, that possibility warrants outsized funding today.
A pair of rather insular intellectual subcultures had developed something approaching a secular eschatology, turning sci-fi hypotheticals into moral imperatives. Critics and former members have described recurring problems of insularity, groupthink, and superiority complex. The true believers cast themselves as the tiny group who “get the danger” before the “ignorant masses” do. Qiaochu Yuan, a former rationalist, told Bloomberg that “Within the group, there was an unspoken sense of being the chosen people smart enough to see the truth and save the world, of being “cosmically significant.’”
“Why are they issuing exact probabilities of human extinction, which in fact are pulled out of thin air?” wondered Harvard’s Steven Pinker in a recent magazine article about their tendency to pronounce that there is, say, a 25% or 35% chance of us all dying by 2040. I am sorry to say this is how they talk. When you spend time in these circles, you clearly see their obsession with making up numbers and then quoting them as important facts when, essentially, they are “feelings dressed up as numbers.”
Those who actually engage with the technology, as opposed to the thought experiments, tend to find all this pretty absurd. One review of Yudkowsky and Nate Soares’s book If Anyone Builds It, Everyone Dies memorably retitled it: “If Anyone Reads It, Everyone Laughs.” “The real work of technology is found in implementation, adoption, maintenance, repair, adaptation, debugging, negotiation, administration, and other boring but essential tasks.”
In other words, it’s buggy PHP code, a battle between defenders and bad actors. This may not get the kind of clicks that doom-friendly news outlets aim for with headlines about “THE END OF HUMANITY” (the infamous Time magazine cover with two letters in flashing lights: AI). But it’s far closer to the realities of AI.
Nonetheless, the “AI will kill us all” camp is on media tours. They are also paying content creators to push AI existential risk messages to the public by sponsoring social media posts and partnering with influencers: influencers with tens of millions of subscribers, recruited to preach doom. The AI doomers have deep pockets — the most recent estimate is $3 billion. The generous support of figures like Moskovitz, Tallinn, and convicted felon Sam Bankman-Fried has helped finance the ecosystem’s rapid expansion, with hundreds of organizations devoted to existential risks from AI. These billionaires have built the “AI safety” and “AI alignment” fields through grants, fellowships, research centres, training pipelines, media work, advocacy groups, and lobbying networks, as well as within frontier AI companies. Anthropic’s ties to Effective Altruism run unusually deep, spanning its funding, hires, and organisational culture. Anthropic’s policy advocacy — built around access controls, evaluation requirements, and proposals to “pace” development — could create a permission regime that large incumbents are far better equipped to navigate than smaller competitors, especially from the open-source community. Anthropic’s IPO is meanwhile expected to generate tens of billions of dollars in new philanthropic wealth for causes including AI existential risk. If so, the network built during the first AI-safety funding boom could soon have access to vastly more capital.
The danger is that this small but well-funded and influential group is claiming oversized moral and political authority over everyone else’s AI future. If AI really is as consequential as its most alarmed advocates claim, then concentrating control over it in a small number of frontier corporations, regulators, or self-appointed “experts” susceptible to the same groupthink creates its own category of risk. Broad access would give researchers, developers, businesses, and ordinary users greater opportunity to test systems, discover weaknesses, and develop competing alternatives.
What comes next does not have to be a choice between AI hype and predictions of apocalypse. A more grounded alternative is AI realism: treat AI as a powerful, fast-moving technology whose benefits and risks should be assessed through evidence rather than prophecy. The goal of AI regulation should be to target specific, concrete harms. That means distinguishing demonstrated harms from speculative scenarios, and treating the risks created by surveillance, censorship, market concentration, regulatory capture, and restrictions on scientific inquiry as part of the safety equation rather than acceptable collateral damage.
Before building an apparatus of surveillance and control to prevent a hypothetical dystopia, we should examine whether the proposed cure is becoming the dystopia itself.
Dr. Nirit Weiss-Blatt is a Communication Researcher and author of The TECHLASH and Tech Crisis Communication and the AI Panic newsletter.