Wright, in The God Test, aims to set the development of AI in a three-billion-year perspective. Take this very long view, and one can trace the development of what he calls, following the heterodox Catholic scientist and philosopher Teilhard de Chardin, the “noosphere.” This refers to the growth in extent and density, at first very gradually but recently with dizzying speed, of the sum of organic self-organization and cognitive activity. The current stage of this process is “an inchoate planetary brain whose neurons are human brains.” 

Today’s best-known and most advanced AIs are large-language machines (LLMs). Wright’s description is that an LLM is a neural network: layers of microchips vertically connected. Its “trainer” feeds the machine a vast amount of information —  possibly amounting to a large fraction of the entire Internet. Back and forth among the neurons, or chips, weighted strings of numbers (“vectors”) flash at high speed, adjusting and improving the machine’s ability to follow up or respond to any given segment of text. It can also be taught to solve advanced problems in mathematics and medical science. 

That much, most people already know. But apparently, in the last few years, machines have been acquiring personality. Researchers have found that by tweaking individual neurons, an LLM can be made either sycophantic or truculent. Its prose style can be made plainer or subtler. Even character can be altered artificially: a machine can be made to lie strategically, in pursuit of its mission. More disturbing, machines occasionally lie without prompting, a little like the sentient and malignant artificial intelligence HAL in the 1968 film 2001: A Space Odyssey. 

Suggested reading

Invasion of the literary bots

By Vincenzo Barney

The most troubling data point Wright offers is a comparison between the development curve of AI’s cognitive capacities and the historical development curve of human cognitive evolution. For better or worse, it appears that AIs are poised for a very rapid leveling-up.

The God Test, however, is neither alarmist nor boosterish. Wright knowledgeably and dispassionately discusses the prospects of both AI Utopia and AI Apocalypse. If he is passionate about anything, it is his insistence that something very big is afoot in the near future — the next 10 to 20 years — and that we had better not just stumble into it. Specifically, we had better not — whatever greedy investors may want — develop AIs as fast as possible on all fronts, with no intra-industry consultation and no national or international regulation.  

Why not? The AI disaster most prevalent in the popular imagination is the “infinite paper clips” scenario. An AI is ordered to maximize paper-clip production and proceeds to conquer the world and convert everything metal, including all our dental fillings, into paper clips. This is known as the “subordinate goal” problem: following insufficiently specific directions for achieving some ultimate goal, an AI formulates a subordinate goal that incidentally wreaks havoc. Though this seems to have captured the most attention, it is not, Wright points out, the only, or even the most likely, path to catastrophe. 

Put another way: an AI could engineer a bioweapon of unimaginable destructiveness; or paralyze the world financial system; or take down the internet; or take down something simpler, like the American or Chinese government. Currently existing AIs probably could not do this (though we don’t really know), but some future (and probably not very far in the future) generation of AIs certainly could. And unlike developing a nuclear bomb, it would not require a sizable industrial complex to accomplish that — and would therefore be much harder to detect and prevent. 

The case for some degree of AI international governance is, Wright shows, a no-brainer. Then again, the case for nuclear disarmament was a no-brainer 60 years ago, and the case for strict control of greenhouse gas emissions has been a no-brainer for 30 years. It appears that humans are, politically at least, creatures of very little brain. 

There is, however, no one but the public to appeal to. Expecting restraint from investors or executives is like expecting chastity from Jeffrey Epstein. Computer scientists understand the dangers, but then, Robert Oppenheimer understood the dangers of developing the atomic bomb and went ahead anyway because it was too “technically sweet” to resist. Governments will probably, as usual, follow the line of least resistance, though perhaps this time, with the biosphere, ecosphere, and noosphere all at stake, they’ll pay attention.

The God Test is, perhaps understandably, a little preachy and not as vividly written as one might wish. A few more powerfully imagined doomsday scenarios would have been welcome; likewise, an idyllic portrait or two of AI Utopia. Still, Wright’s preaching is sound: rise above your tribal cognitive biases and learn to cooperate to temper the obvious excesses that AI will bring about, or be damned! 

At the present moment, much AI-related discourse has to do with its likely effect on our imagination and sensibility — that is, our souls. The God Test is not much help here. Wright discusses at some length a celebrated visionary essay by Dario Amodei, CEO of Anthropic. Wright’s issue with “Machines of Loving Grace,” a lyrical evocation of AI’s promise, is that Amodei does not mention the most exciting future benefit of AIs, which turns out to be … universal access to superior video games and virtual reality. Is this really the best we can hope for? 

Suggested reading

Was AI called by Cthulhu?

By Michael Cuenco

The noblest promise of AI, I would say, is to allow the huddled masses the leisure, and if necessary the tutelage, to enjoy the best of human culture: Middlemarch and Keats’s poetry; the Saint Matthew Passion and late Beethoven quartets; Raphael and Renoir; Kurosawa and Ray; molecular biology and general relativity; and several lifetimes more of humanity’s supreme achievements; not to make available to everyone the next generation of trashy, vapid electronic entertainment. 

And while Wright is surely right to worry about the AI Apocalypse, someone ought to be worrying about AI Soul-Killing Tedium. Consider one of the earliest and most extensive applications of AI: customer service. It is now next to impossible to connect with a human being to tell one’s problem to. Not only has this eliminated millions of (not altogether unsatisfying) jobs, but by substituting perfectly literal-minded, invincibly stubborn AI interlocutors for merely dense or obstinate human ones, it is probably responsible for raising the national blood pressure by a point or two. HR is another growing field for AIs, carefully engineered to lack imagination or sympathy and highly valued for their ability to repeat “I’m sorry, but that is company policy” for hours on end, which few humans can do.

And then there’s advertising. The uncontrolled proliferation of intrusive commercial messaging should probably be looked at as a kind of cancer within the noosphere. And unregulated AI will aggravate it, vacuuming up inconceivable amounts of personal data from the internet, analyzing it, packaging it, and selling it to advertisers — a development Harvard Business School professor Shoshana Zuboff has coined “surveillance capitalism.” 

AIs will undoubtedly be marketed to governments, corporations, and consumers for purposes at best trivial and at worst sinister. It might have been different. More than a century ago, in The Soul of Man Under Socialism, Oscar Wilde offered a wise and humane vision of technology’s possibilities: “All unintellectual labor, all monotonous, dull labor … must be done by machinery. Machinery must work for us in coal mines, and do all sanitary services, and be the stoker of steamers, and clean the streets, and run messages on wet days, and do anything that is tedious or distressing. At present machinery competes against man. Under proper conditions machinery will serve man. There is no doubt at all that this is the future of machinery: … while Humanity will be amusing itself, or enjoying cultivated leisure — which, and not labor, is the aim of man — or making beautiful things, or reading beautiful things, or simply contemplating the world with admiration and delight, machinery will be doing all the necessary and unpleasant work.” 

Alas, there is no doubt at all that this is not the future of AI. Money has its prerogatives, one of which is to destroy the grace of life for the many in order to endlessly enrich the graceless few.