The existential risks of AI

But the fundamental threat posed by AI is “existential” – it’s a threat to human existence as we know it. Because we simply have no idea what will happen if machines really do become explosively smart and “take over.” Not just filch our jobs, not just give a handful of super-wealthy individuals the whip hand, but of they become the masters of the human species. Famed inventor and Google executive Ray Kurzweil, whose door-step volume The Singularity is Near has defined the debate, is as optimistic about the pace of change as he is about its benevolent outcomes.

Mathematics professor and sci-fi writer Vernor Vinge (pronounced Vingey), who is credited with coining the term “singularity” in this sense, agrees that “it will probably occur faster than any technical revolution seen so far”, but is less optimistic:

And what happens a month or two (or a day or two) after that? I have only analogies to point to: The rise of humankind. We will be in the Posthuman era. And for all my technological optimism, I think I’d be more comfortable if I were regarding these transcendental events from one thousand years’ remove . . . instead of twenty.²

Founding editor of Wired, Kevin Kelly, is more skeptical. A long-time critic of Kurzweil and those who think like him, Kelly punches back against the core idea of the singularity – that machines will take charge and then be able to solve all our problems, including mortality itself.  He even suggests that Kurzweil’s obsession with the singularity is at root the result of his fear of death (the computer scientists uses such stratagems as caloric restriction and scoffing scores of vitamin tablets to try to slow the ageing process). And while Kelly agrees with the singularity crowd and the Open Letter crowd that we need to take the possibility of AI risk seriously, he’s a lot less worried about a singularity apocalypse.

This critique is especially notable, since Kelly is not simply a tech enthusiast, he’s something of a fatalist about the future. His latest book, a roundup of the twelve tech trends shaping what comes next, is aptly titled The Inevitable. But in singularity he sees a “cargo cult“, based on false assumptions. He accuses singularity’s proponents of what he calls “thinkism” – the idea that human thinking is the be-all and end-all, and so if machines can do it better than we can they will solve all our problems in a burst of super-human intelligence. He calls it “the fallacy that future levels of progress are only hindered by a lack of thinking power.”

For one thing, he points out, the world does not lack for extraordinarily smart people. But curing cancer cannot come about through smart thoughts; however smart they are, it will need years of trials of treatments and medications. Moreover, there are many kinds of intelligence; it’s not just one thing.

So the question is, where is the limit of intelligence? We tend to believe that the limit is way beyond us, way “above” us, as we are “above” an ant. Setting aside the recurring problem of a single dimension, what evidence do we have that the limit is not us? Why can’t we be at the maximum? Or maybe the limits are only a short distance away from us? Why do we believe that intelligence is something that can continue to expand forever?³

In a thousand years’ time, he reckons, we’ll still be expecting the singularity to occur soon.

Concerns about AI risk have spawned a series of efforts to keep this smartest of human inventions under some kind of human control. Kelly himself calls for us to engineer our values into AIs. U.S. congressman Brad Sherman once asked the Science and Technology Committee to fund “non-ambitious AI,” and then convened a hearing on the need for a non-proliferation approach to such technologies. ⁴

The Hawking-Musk-Gates letter has helped boost other initiatives aimed at ensuring the safe and ethical development of AI, such as Cambridge University’s Centre for the Study of Existential Risk, the Leverhulme Centre for the Future of Intelligence, and the Berkeley Center for Human-Compatible AI.

“Welcome, singularitarians!” was how the conference chair opened proceedings at a cult-like singularity event I attended a decade back in New York City. Yet even among those most committed to the “singularity” analysis, there are clear differences of opinion, even if they are united in believing that this event will occur relatively soon (Kurzweil has suggested various dates; his latest prediction is for 2043). But they don’t all see things turning out quite as clearly as he does. Peter Thiel, long-time Kurzweil financial backer (and Facebook board member), made it plain when he addressed one recent singularity conference that he is more of a gradualist in his vision (in other words, like Kelly, he doesn’t really believe in the sudden, discrete, singularity idea at all – as some in the audience were quick to point out.) And they are by no means all as optimistic that its arrival will be good news.

So let’s give the last word to Vinge, who’s a much more interesting thinker than Kurzweil: “The longer we humans have our hand on the tiller, the better.”

---

Nigel Cameron writes about technology, society, and the future. In 2007 he founded the Washington think tank The Center for Policy on Emerging Technologies. His most recent book is Will Robots Take Your Job?

nigelcameron