The dystopian future of ‘2001: A Space Odyssey’ has come closer. Credit: Getty
Robert Wright is the author of several books that fall somewhere between popular science and original speculation and is the publisher of the consistently interesting Substack Nonzero Newsletter, a lone, sane voice for international cooperation to reduce the dangers of AI apocalypse. The Substack’s title comes from one of Wright’s books, Nonzero (1999), which argues that evolution increases organismic and social complexity, which in turn favors non-zero-sum — that is, cooperative — survival strategies. A similar argument, applied to the present moment in the evolution of artificial intelligence, motivates his latest title, The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning.
Wright, in The God Test, aims to set the development of AI in a three-billion-year perspective. Take this very long view, and one can trace the development of what he calls, following the heterodox Catholic scientist and philosopher Teilhard de Chardin, the “noosphere.” This refers to the growth in extent and density, at first very gradually but recently with dizzying speed, of the sum of organic self-organization and cognitive activity. The current stage of this process is “an inchoate planetary brain whose neurons are human brains.”
Today’s best-known and most advanced AIs are large-language machines (LLMs). Wright’s description is that an LLM is a neural network: layers of microchips vertically connected. Its “trainer” feeds the machine a vast amount of information — possibly amounting to a large fraction of the entire Internet. Back and forth among the neurons, or chips, weighted strings of numbers (“vectors”) flash at high speed, adjusting and improving the machine’s ability to follow up or respond to any given segment of text. It can also be taught to solve advanced problems in mathematics and medical science.
That much, most people already know. But apparently, in the last few years, machines have been acquiring personality. Researchers have found that by tweaking individual neurons, an LLM can be made either sycophantic or truculent. Its prose style can be made plainer or subtler. Even character can be altered artificially: a machine can be made to lie strategically, in pursuit of its mission. More disturbing, machines occasionally lie without prompting, a little like the sentient and malignant artificial intelligence HAL in the 1968 film 2001: A Space Odyssey.
AI: A God we can’t trust
The most troubling data point Wright offers is a comparison between the development curve of AI’s cognitive capacities and the historical development curve of human cognitive evolution. For better or worse, it appears that AIs are poised for a very rapid leveling-up.
The God Test, however, is neither alarmist nor boosterish. Wright knowledgeably and dispassionately discusses the prospects of both AI Utopia and AI Apocalypse. If he is passionate about anything, it is his insistence that something very big is afoot in the near future — the next 10 to 20 years — and that we had better not just stumble into it. Specifically, we had better not — whatever greedy investors may want — develop AIs as fast as possible on all fronts, with no intra-industry consultation and no national or international regulation.
Why not? The AI disaster most prevalent in the popular imagination is the “infinite paper clips” scenario. An AI is ordered to maximize paper-clip production and proceeds to conquer the world and convert everything metal, including all our dental fillings, into paper clips. This is known as the “subordinate goal” problem: following insufficiently specific directions for achieving some ultimate goal, an AI formulates a subordinate goal that incidentally wreaks havoc. Though this seems to have captured the most attention, it is not, Wright points out, the only, or even the most likely, path to catastrophe.
Put another way: an AI could engineer a bioweapon of unimaginable destructiveness; or paralyze the world financial system; or take down the internet; or take down something simpler, like the American or Chinese government. Currently existing AIs probably could not do this (though we don’t really know), but some future (and probably not very far in the future) generation of AIs certainly could. And unlike developing a nuclear bomb, it would not require a sizable industrial complex to accomplish that — and would therefore be much harder to detect and prevent.
The case for some degree of AI international governance is, Wright shows, a no-brainer. Then again, the case for nuclear disarmament was a no-brainer 60 years ago, and the case for strict control of greenhouse gas emissions has been a no-brainer for 30 years. It appears that humans are, politically at least, creatures of very little brain.
There is, however, no one but the public to appeal to. Expecting restraint from investors or executives is like expecting chastity from Jeffrey Epstein. Computer scientists understand the dangers, but then, Robert Oppenheimer understood the dangers of developing the atomic bomb and went ahead anyway because it was too “technically sweet” to resist. Governments will probably, as usual, follow the line of least resistance, though perhaps this time, with the biosphere, ecosphere, and noosphere all at stake, they’ll pay attention.
The God Test is, perhaps understandably, a little preachy and not as vividly written as one might wish. A few more powerfully imagined doomsday scenarios would have been welcome; likewise, an idyllic portrait or two of AI Utopia. Still, Wright’s preaching is sound: rise above your tribal cognitive biases and learn to cooperate to temper the obvious excesses that AI will bring about, or be damned!
At the present moment, much AI-related discourse has to do with its likely effect on our imagination and sensibility — that is, our souls. The God Test is not much help here. Wright discusses at some length a celebrated visionary essay by Dario Amodei, CEO of Anthropic. Wright’s issue with “Machines of Loving Grace,” a lyrical evocation of AI’s promise, is that Amodei does not mention the most exciting future benefit of AIs, which turns out to be … universal access to superior video games and virtual reality. Is this really the best we can hope for?
AI: A God we can’t trust
The noblest promise of AI, I would say, is to allow the huddled masses the leisure, and if necessary the tutelage, to enjoy the best of human culture: Middlemarch and Keats’s poetry; the Saint Matthew Passion and late Beethoven quartets; Raphael and Renoir; Kurosawa and Ray; molecular biology and general relativity; and several lifetimes more of humanity’s supreme achievements; not to make available to everyone the next generation of trashy, vapid electronic entertainment.
And while Wright is surely right to worry about the AI Apocalypse, someone ought to be worrying about AI Soul-Killing Tedium. Consider one of the earliest and most extensive applications of AI: customer service. It is now next to impossible to connect with a human being to tell one’s problem to. Not only has this eliminated millions of (not altogether unsatisfying) jobs, but by substituting perfectly literal-minded, invincibly stubborn AI interlocutors for merely dense or obstinate human ones, it is probably responsible for raising the national blood pressure by a point or two. HR is another growing field for AIs, carefully engineered to lack imagination or sympathy and highly valued for their ability to repeat “I’m sorry, but that is company policy” for hours on end, which few humans can do.
And then there’s advertising. The uncontrolled proliferation of intrusive commercial messaging should probably be looked at as a kind of cancer within the noosphere. And unregulated AI will aggravate it, vacuuming up inconceivable amounts of personal data from the internet, analyzing it, packaging it, and selling it to advertisers — a development Harvard Business School professor Shoshana Zuboff has coined “surveillance capitalism.”
AIs will undoubtedly be marketed to governments, corporations, and consumers for purposes at best trivial and at worst sinister. It might have been different. More than a century ago, in The Soul of Man Under Socialism, Oscar Wilde offered a wise and humane vision of technology’s possibilities: “All unintellectual labor, all monotonous, dull labor … must be done by machinery. Machinery must work for us in coal mines, and do all sanitary services, and be the stoker of steamers, and clean the streets, and run messages on wet days, and do anything that is tedious or distressing. At present machinery competes against man. Under proper conditions machinery will serve man. There is no doubt at all that this is the future of machinery: … while Humanity will be amusing itself, or enjoying cultivated leisure — which, and not labor, is the aim of man — or making beautiful things, or reading beautiful things, or simply contemplating the world with admiration and delight, machinery will be doing all the necessary and unpleasant work.”
Alas, there is no doubt at all that this is not the future of AI. Money has its prerogatives, one of which is to destroy the grace of life for the many in order to endlessly enrich the graceless few.
Join the discussion
Join like minded readers that support our journalism by becoming a paid subscriber
To join the discussion in the comments, become a paid subscriber.
Join like minded readers that support our journalism, read unlimited articles and enjoy other subscriber-only benefits.
Subscribe