<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	xmlns:unherd="https://www.unherd.com"
	>

	<channel>
		<title>Superintelligent AI - UnHerd</title>
		<atom:link href="https://unherd.com/tag/superintelligent-ai/feed/" rel="self" type="application/rss+xml" />
		<link></link>
		<description>think again</description>
		<lastBuildDate>Thu, 18 Sep 2025 22:34:12 +0000</lastBuildDate>
		<language>en-US</language>
		<sy:updatePeriod>hourly</sy:updatePeriod>
		<sy:updateFrequency>12</sy:updateFrequency>
		<generator>https://wordpress.org/?v=7.1.5</generator>

<image>
	<url>https://unherd.com/wp-content/uploads/2019/09/cropped-favicon.jpg?w=32</url>
	<title>Superintelligent AI - UnHerd</title>
	<link></link>
	<width>32</width>
	<height>32</height>
</image> 
<site xmlns="com-wordpress:feed-additions:1">241381703</site>				<item>
					<title>What the tech giants aren’t telling us</title>
					<link>https://unherd.com/2025/09/what-the-tech-giants-arent-telling-us/?edition=us</link>
											<comments>https://unherd.com/2025/09/what-the-tech-giants-arent-telling-us/?edition=us#comments</comments>
										<unherd:featured_image>https://unherd.com/wp-content/uploads/2025/09/atlmanov.jpg?w=517</unherd:featured_image>
										<unherd:main_theme>Review</unherd:main_theme>
					<dc:creator>
						<![CDATA[Tom Chivers]]>
					</dc:creator>
					<unherd:creator_img>https://unherd.com/wp-content/uploads/2019/09/chivers.jpg?w=96</unherd:creator_img>
																<unherd:article_ranking>2</unherd:article_ranking>
																<unherd:image_caption>
							<![CDATA[Are we underreacting to the risks of AI?Photo: Kevin Dietsch/Getty.]]>
						</unherd:image_caption>
						<unherd:twitter_bio>
							<![CDATA[<p><strong>Tom Chivers</strong> is a science writer. His second book, <em>How to Read Numbers</em>, is out now.</p>
]]>
						</unherd:twitter_bio>
						<unherd:twitter_handle>TomChivers</unherd:twitter_handle>
															<unherd:author_bio>
						<![CDATA[ is a science writer. His second book, <em>How to Read Numbers</em>, is out now.]]>
					</unherd:author_bio>
										<pubDate>Tue, 16 Sep 2025 23:02:51 +0000</pubDate>
					<updated>Thu, 18 Sep 2025 22:34:12 +0000</updated>
							<category><![CDATA[Uncategorized]]></category>
		<category><![CDATA[ai]]></category>
		<category><![CDATA[Extinction]]></category>
		<category><![CDATA[Science]]></category>
		<category><![CDATA[Superintelligent AI]]></category>
		<category><![CDATA[Tech bros]]></category>
		<category><![CDATA[UK]]></category>
					<guid isPermaLink="false">https://unherd.com/?p=768708</guid>
																												<description>
								<![CDATA[Superintelligent AI could still kill us]]>
							</description>
												
																									<content:encoded>
								<![CDATA[<p>Outside the AI company Anthropic’s offices in San Francisco, a guy is on a hunger strike. He’s been there two weeks. Another was doing something similar outside Google DeepMind in London, but never quite hit his straps: he <a href="https://x.com/MichaelTrazzi/status/1964078661188886746" target="_blank" rel="noopener">started tweeting about it on the first day</a>, which seems a grandiose way of saying “I skipped breakfast”, and then <a href="https://x.com/michaeltrazzi/status/1966285365758628240?s=46" target="_blank" rel="noopener">gave up after doctors told him it was dangerous</a> — a surprise to those of us who rather thought that was the point.</p>
<p>Jokes aside, the intention is to stop the tech companies from building superintelligent AI, which they say “<a href="https://forum.effectivealtruism.org/posts/zwYjE5ZFwn3KaFq6v/hunger-strike-in-front-of-anthropic-by-one-guy-concerned-1" target="_blank" rel="noopener">threatens to destroy life on Earth</a>”. But here’s the thing: if these protesters believe that superintelligent AI has a high chance of killing everyone, then they’re not overreacting. If anything, they’re underreacting. Since I, too, think that <a href="https://unherd.com/2025/06/ais-1930s-moment/">there is a non-trivial chance of AI killing everyone</a>, I should really be applauding them, not sneering.</p>
<p>A couple of other people who aren’t sneering are Eliezer Yudkowsky and Nate Soares, who have just published <em><a href="https://www.penguin.co.uk/books/474267/if-anyone-builds-it-everyone-dies-by-soares-eliezer-yudkowsky-and-nate/9781847928924" target="_blank" rel="noopener">If Anyone Builds It, Everyone Dies</a></em>. They are, respectively, the founder and president of the Machine Intelligence Research Institute, which, long ago — way back in about 2002, when OpenAI’s ChatGPT was not even a twinkle in Sam Altman’s eye, and in fact Sam Altman himself was just starting high school — became what was probably the world’s first organized group dedicated to preventing AI from destroying the world.</p>
<p>Yudkowsky and Soares are so worried about the deadly potential of AI that they’re after something much more dramatic than a hunger strike. They want a global agreement, backed up with the sort of teeth that global anti-nuclear-proliferation treaties have, banning all frontier AI research. And yes, that would mean that if some country was found to be breaking that treaty, its data centers should be bombed.</p>
<div class="ref-ar"><a href="https://unherd.com/2025/06/ais-1930s-moment/?edition=us&#038;ref=refinnar" class="text-decoration-none border-0"><a href="https://unherd.com/2025/06/ais-1930s-moment/?edition=us&#038;ref=refinnar" class="text-decoration-none border-0"><img decoding="async" src="https://unherd.com/wp-content/uploads/2025/06/GettyImages-517350684-e1750172245534.jpg?w=300" /></a><div class="content"><a href="https://unherd.com/2025/06/ais-1930s-moment/?edition=us&#038;ref=refinnar" class="text-decoration-none border-0"><span class="meta">More from this author</span></a><a href="https://unherd.com/2025/06/ais-1930s-moment/?edition=us&#038;ref=refinnar" class="text-decoration-none border-0"><h5>AI’s 1930s moment</h5></a><p class="author"><a href="https://unherd.com/2025/06/ais-1930s-moment/?edition=us&#038;ref=refinnar" class="text-decoration-none border-0"><span>By</span> Tom Chivers</a></p></div></a></div>
<p>And if you accept the premise, that’s not an overreaction either. Elon Musk — who is, don’t forget, the head of a major AI lab — has said that AI is just as dangerous as nukes, and as recently as 2023 that it could lead to “<a href="https://news.sky.com/story/elon-musk-tells-sky-news-ai-is-a-risk-to-humanity-12998024" target="_blank" rel="noopener">civilization destruction</a>”. Musk’s former friend Donald Trump is very much still surrounded by AI leaders, though: he is accompanied on his ongoing UK state visit by several Silicon Valley CEOs, including Altman. Trump ordered the bombing of Iranian nuclear facilities earlier this year, because they might be close to making a bomb; should he consider blowing up his friends’ data centers at some point?</p>
<p>Yudkowsky and Soares lay the risks out pretty clearly. First, if AI companies succeed in making a superintelligent AI — which, to reiterate, they are actively trying to do — then that AI will be more powerful than us, in the way that we are more powerful than chimpanzees. Yes, an individual chimp could pummel me to death, but humans as a species have nearly driven chimps extinct without even meaning to. If a superintelligent AI wants to eradicate us, or to achieve some goal which would lead to our eradication as a side effect, then we would not be able to do a great deal about it.</p>
<p>In any case, the pair argues, we don’t actually know what’s going on inside AIs — much as we don’t truly know how the human brain works. There are trillions of simulated connections between billions of simulated neurons in each AI, far too many for any human to check. “The relationship that biologists have with DNA,” they say, “is pretty much the relationship that AI engineers have with the numbers inside an AI.” That is, we can see that some genes correlate somewhat with some behavior or disease, but we can’t just read out an embryo’s DNA and say “this will be a 6’2” guy who’s interested in maths and can play the guitar”. Similarly, AI engineers cannot look into an AI’s nodes and say “this AI is truthful and human-aligned”, although there are efforts to become better at doing that.</p>
<p>Another problem they lay out is that when we try to give them goals, we don’t really know what we’re doing. We train AI to want to be helpful and honest and so on, but only in a very high-level way. It’s comparable, Y&amp;S argue, to how evolution trained us to “want” to reproduce, but accidentally created humans who “want” to have orgasms, and often actively take steps to stop ourselves reproducing with contraception. An AI’s goals could just as easily be disconnected from the goals we gave it. And even though AIs talk like humans, they are nothing like humans — their minds are alien. The things they want will be weird, and the very basic things that humans want — like staying alive — might not be important to them, despite our best efforts.</p>
<p>Finally, they argue, we’ll only have one chance to get it right. A superintelligent AI could, they say, kill us if it tried. So it must never try, not even on the first time we boot it up. But we can’t test the superintelligent AI — because if you get it wrong the first time it will kill everyone — so you have to test it on stupider, weaker versions, and hope those tests apply. It is, they say, like a space rocket: you can do all the lab testing you like, but it will never be quite like the stress of launching into space at seven miles a second. About 30% of new rocket types explode on their first flight.</p>
<div class="su-pullquote su-pullquote-align-left">&#8220;We train AI to want to be helpful and honest and so on, but only in a very high-level way.&#8221;</div>
<p>One objection sometimes made at this point is that AI does not want things; it just does what we ask of it. But that is a pointless semantic dispute, as far as Y&amp;S (and I) are concerned: a chess-playing AI acts as though it wants to win at chess. You can tell, by how it protects its king and tries to checkmate yours. If you don’t want to call it “wanting”, fine. But there’s an old AI proverb: &#8220;The question of whether a computer can think is no more interesting than the question of whether <a href="https://swimmingsubmarine.blogspot.com/2007/10/question-of-whether-computer-can-think.html" target="_blank" rel="noopener">a submarine can swim</a>.&#8221; That’s true of “wanting”, too. As AIs get cleverer and more powerful, they’ll be more able to pursue the goals we give them, and will be driven to do so. It will look exactly like “wanting”. Call it what you like.</p>
<p>Once, you might have said that this doesn’t matter: AIs are just some programming in a box, and don’t have power in the world. But we’ve long since connected AIs to the internet, used them to run factories and weapons; one AI got connected to Twitter, was given some money to put into crypto, and <a href="https://www.lesswrong.com/posts/buiTYy75KJDhckDgq/truth-terminal-a-reconstruction-of-events" target="_blank" rel="noopener">is now worth $50,000,000</a>. Money can be exchanged for goods and services. The idea that we could just switch them off, or that they are powerless to affect the real world, was hard enough to sustain 15 years ago; now it is silly.</p>
<p>So taking the points together, Yudkowsky and Soares argue compellingly that if things continue apace, unchecked, we may have, in the coming years, a hyperintelligent machine capable of manipulating the real world, which has weird, alien goals which are unlikely to be compatible with humans being alive; a planet with an oxygen-rich atmosphere and a surface temperature below the boiling point of water is great for us, but not particularly important to an AI. &#8220;The AI does not hate you, nor does it love you, <a href="https://everythingstudies.com/2020/03/20/review-the-ai-does-not-hate-you/" target="_blank" rel="noopener">but you are made out of atoms which it can use for something else</a>.&#8221;</p>
<p>Yes, Y&amp;S are, it should be said, outliers on the “AI doom” scale. They think — as their book title explicitly says — that if anyone builds it, everyone dies. But there are others, such as Scott Alexander, who <a href="https://www.astralcodexten.com/p/book-review-if-anyone-builds-it-everyone" target="_blank" rel="noopener">have much lower estimates of the risk</a>: he describes himself as a “boring moderate” because he thinks there’s less than a 25% chance (!) that AI will kill everyone. Part of the disagreement here seems to be that Y&amp;S think that there could be a sudden, staggeringly rapid takeoff, in which the AI goes from not-dangerous to dangerous within a few hours.</p>
<p>The AI researcher Quintin Pope, for one, thinks <a href="https://www.lesswrong.com/posts/FyChg3kYG54tEN3u6/evolution-is-a-bad-analogy-for-agi-inner-alignment" target="_blank" rel="noopener">a key premise of Yudkowsky’s argument is false</a>: he believes training AI is nothing like evolution, and that it’s more like human learning, so accidentally instilling weird, dangerous values into the AI is unlikely. Pope and his colleague Nora Belrose reassuringly argue that AI “<a href="https://optimists.ai/2023/11/28/ai-is-easy-to-control/" target="_blank" rel="noopener">is easy to control</a>” — although even they, self-described AI optimists, think “a catastrophic AI takeover is roughly 1% likely”.</p>
<div class="ref-ar"><a href="https://unherd.com/2025/03/why-im-boycotting-ai/?edition=us&#038;ref=refinnar" class="text-decoration-none border-0"><a href="https://unherd.com/2025/03/why-im-boycotting-ai/?edition=us&#038;ref=refinnar" class="text-decoration-none border-0"><img decoding="async" src="https://unherd.com/wp-content/uploads/2025/03/Kahn.jpg?w=300" /></a><div class="content"><a href="https://unherd.com/2025/03/why-im-boycotting-ai/?edition=us&#038;ref=refinnar" class="text-decoration-none border-0"><span class="meta">Suggested reading</span></a><a href="https://unherd.com/2025/03/why-im-boycotting-ai/?edition=us&#038;ref=refinnar" class="text-decoration-none border-0"><h5>Why I’m boycotting AI</h5></a><p class="author"><a href="https://unherd.com/2025/03/why-im-boycotting-ai/?edition=us&#038;ref=refinnar" class="text-decoration-none border-0"><span>By</span> Sam Kahn </a></p></div></a></div>
<p>So while it might be fair to think that Yudkowsky and Soares are overconfident in their predictions of doom, we are nonetheless in a situation where the “boring moderates” believe we have a one-in-four chance of dying and the &#8220;optimists” think it’s a mere one-in-100. Would you play Russian roulette using a revolver with 100 chambers?</p>
<p>As I was reading <em>If Anyone Builds It, Everyone Dies</em>, I realized that it was not the book I wanted to be reading. The book I wanted to be reading was by someone like Dario Amodei, the CEO of Anthropic, or Demis Hassabis, the CEO of Google DeepMind, or maybe Altman himself. The book would be called something like <em>I’m Going to Build It, And I Don’t Think We’re Going to Die</em>. It would be a chapter-by-chapter rebuttal of this book.</p>
<p>I know that all those men understand the arguments Yudkowsky and Soares are making. Hassabis met his DeepMind cofounder Shane Legg at a conference organized by MIRI; Altman is on record as saying Yudkowsky was critical in his decision to start OpenAI. Anthropic was founded explicitly with AI safety goals which stemmed from Yudkowsky’s work. These guys know this stuff, most of which long predates the publication of this book, and which has been echoed by many other thinkers. They have spoken about the risks. Hassabis has said that AI could do “incredible things for humanity” but also have existential risks if mismanaged; Amodei likewise has written long essays on AI’s enormous potential upsides such as <a href="https://www.darioamodei.com/essay/machines-of-loving-grace" target="_blank" rel="noopener">curing disease, ending climate change</a> and wrote <a href="https://arxiv.org/abs/1606.06565" target="_blank" rel="noopener">a seminal 2016 paper on the potential risks</a>, but has not, to my knowledge, ever explained why he is confident now that the risks are worth taking.</p>
<p>I suspect that Yudkowsky and Soares are too confident that AI means doom. But even the “moderate” position that there’s a 25% chance is at least three orders of magnitude too high for me. I think the guy doing the hunger strike outside DeepMind would say the same, if he hadn’t got peckish and gone home for a snack. The only people who might reasonably explain why we shouldn’t worry are the ones building the superintelligent AIs. But for some reason they’re not telling us.</p>
]]>
							</content:encoded>
																											<wfw:commentRss>https://unherd.com/2025/09/what-the-tech-giants-arent-telling-us/?edition=us/feed/</wfw:commentRss>
															<post-id xmlns="com-wordpress:feed-additions:1">768708</post-id>				</item>
									<item>
					<title>Coronavirus is the wake-up call we need</title>
					<link>https://unherd.com/2020/03/will-we-be-ready-for-the-next-pandemic/?edition=us</link>
											<comments>https://unherd.com/2020/03/will-we-be-ready-for-the-next-pandemic/?edition=us#respond</comments>
										<unherd:featured_image>https://unherd.com/wp-content/uploads/2020/03/GettyImages-93247292.jpg?w=507</unherd:featured_image>
										<unherd:main_theme></unherd:main_theme>
					<dc:creator>
						<![CDATA[Tom Chivers]]>
					</dc:creator>
					<unherd:creator_img>https://unherd.com/wp-content/uploads/2019/09/chivers.jpg?w=96</unherd:creator_img>
																<unherd:article_ranking></unherd:article_ranking>
																<unherd:image_caption>
							<![CDATA[Innoculations during the A(H1N1) pandemic in 2009. Credit: AFP/AFP/Getty ]]>
						</unherd:image_caption>
						<unherd:twitter_bio>
							<![CDATA[<p><strong>Tom Chivers</strong> is a science writer. His second book, <em>How to Read Numbers</em>, is out now.</p>
]]>
						</unherd:twitter_bio>
						<unherd:twitter_handle>TomChivers</unherd:twitter_handle>
															<unherd:author_bio>
						<![CDATA[ is a science writer. His second book, <em>How to Read Numbers</em>, is out now.]]>
					</unherd:author_bio>
										<pubDate>Tue, 10 Mar 2020 00:10:00 +0000</pubDate>
					<updated>Tue, 10 Mar 2020 11:43:25 +0000</updated>
							<category><![CDATA[Uncategorized]]></category>
		<category><![CDATA[ai]]></category>
		<category><![CDATA[Climate change]]></category>
		<category><![CDATA[Coronavirus]]></category>
		<category><![CDATA[Foot and Mouth]]></category>
		<category><![CDATA[Nick Bostrom]]></category>
		<category><![CDATA[Superintelligent AI]]></category>
					<guid isPermaLink="false">https://old.unherd.com/?p=31191</guid>
																												<description>
								<![CDATA[The outbreak could help us avoid the complete annihilation of humanity ]]>
							</description>
												
																									<content:encoded>
								<![CDATA[<p><span style="font-weight: 400;">Here are some numbers that are worth knowing. There are about 7.5 billion humans who are currently alive. That’s quite a lot: certainly more than have ever been alive at any point in the past. But it’s probably </span><a href="https://www.prb.org/howmanypeoplehaveeverlivedonearth/" target="_blank" rel="noopener noreferrer"><span style="font-weight: 400;">only about 7%</span></a><span style="font-weight: 400;"> of all the modern humans, </span><i><span style="font-weight: 400;">Homo sapiens,</span></i><span style="font-weight: 400;"> who have ever lived; there have been about 108 billion of those in the last 50,000 years.</span></p>
<p><span style="font-weight: 400;">(Depending where you draw the arbitrary line around “modern human”, of course.)</span></p>
<p><span style="font-weight: 400;">That’s just the beginning, though. What we really want to know is: how many will come after us?</span></p>
<p><span style="font-weight: 400;">The philosopher Nick Bostrom, building on work by the great Derek Parfit, </span><a href="https://www.existential-risk.org/concept.pdf" target="_blank" rel="noopener noreferrer"><span style="font-weight: 400;">had a go at answering this question</span></a><span style="font-weight: 400;">; you can also read about it </span><a href="https://smarturl.it/AIDoesNotHateYouHB" target="_blank" rel="noopener noreferrer"><span style="font-weight: 400;">in my own book</span></a><span style="font-weight: 400;">. The planet Earth will be able to support complex life for about a billion years. Say that the human population stabilises at a roomy one billion in the next thousand years or so; Bostrom works out that there will be about 10,000,000,000,000,000 people who follow us. That’s 10 quadrillion; about 1.3 million times as many people are currently alive.</span></p>
<p><span style="font-weight: 400;">Of course, that’s assuming we stay on Earth. If we leave the planet – if, at some point in the next million years or even more, we develop genuine space capability – it gets much more dramatic. If we can build craft that travel at 50% of lightspeed, we could reach about six quintillion stellar systems before the expansion of the universe puts the rest out of our reach; if we can reach 99% of lightspeed, that figure is more like 100 quintillion. </span></p>
<p><span style="font-weight: 400;">If, says Bostrom, 10% of those stars have habitable planets, each capable of supporting one billion humans for one billion years, then the number of human lives that could be supported is 1 followed by 35 zeroes. I could tell you how many times the current human population that is, but it would involve writing out some very long strings of noughts.</span></p>
<p><span style="font-weight: 400;">(Where it gets really weird, of course, is if we eventually learn how to build our own habitats from space rocks, so we’re not limited to existing planets; or upload human brains into computers, so we each take up a few picometres of circuitry rather than some number of thousands of square metres of planetary surface. Then you get some seriously huge numbers. But I don’t want to stretch your credulity here.)</span></p>
<div class="ref-ar"><a href="https://unherd.com/2020/02/why-we-should-overreact-to-coronavirus/?edition=us&#038;ref=refinnar" class="text-decoration-none border-0"><a href="https://unherd.com/2020/02/why-we-should-overreact-to-coronavirus/?edition=us&#038;ref=refinnar" class="text-decoration-none border-0"><img decoding="async" src="https://unherd.com/wp-content/uploads/2020/02/GettyImages-1198465048.jpg?w=300" /></a><div class="content"><a href="https://unherd.com/2020/02/why-we-should-overreact-to-coronavirus/?edition=us&#038;ref=refinnar" class="text-decoration-none border-0"><span class="meta">Suggested reading</span></a><a href="https://unherd.com/2020/02/why-we-should-overreact-to-coronavirus/?edition=us&#038;ref=refinnar" class="text-decoration-none border-0"><h5>Why we should be worried about coronavirus</h5></a><p class="author"><a href="https://unherd.com/2020/02/why-we-should-overreact-to-coronavirus/?edition=us&#038;ref=refinnar" class="text-decoration-none border-0"><span>By</span> Peter Franklin</a></p></div></a></div>
<p><span style="font-weight: 400;">So, assuming we avoid killing ourselves or otherwise getting wiped out, there could be a lot of humans, or human-like beings, that follow us. If we think that those humans have any sort of moral value; if we think it is a good thing that they exist; that their joy or suffering matters, even a fraction as much as the joy or suffering of people alive now; then the lives of people who follow us matter an awful lot. </span></p>
<p><span style="font-weight: 400;">And what </span><i><span style="font-weight: 400;">that </span></i><span style="font-weight: 400;">means is that it really, really matters that we don’t let humanity go extinct. Some disaster – a nuclear war or pandemic, say – that killed 100% of humanity is not 1.0101…% worse than a disaster that killed 99%: it’s incomparably worse; millions of times worse.</span></p>
<p><span style="font-weight: 400;">The coronavirus outbreak is a disaster in its own right; the death toll could very easily reach the millions. The forecasting site Metaculus, which crowdsources predictions for wisdom-of-the-crowds purposes, has </span><a href="https://www.metaculus.com/questions/3530/how-many-people-will-die-as-a-result-of-the-2019-novel-coronavirus-covid-19-before-2021/" target="_blank" rel="noopener noreferrer"><span style="font-weight: 400;">a median estimate</span></a><span style="font-weight: 400;"> of 2.1 million deaths by 2021; its 95% certainty intervals are 210,000 and 27 million. This really could be the most dangerous pandemic in at least 100 years. </span></p>
<p><span style="font-weight: 400;">But it’s also a reminder that we could one day face something much worse. </span></p>
<p><span style="font-weight: 400;">Researchers at Oxford’s </span><a href="https://www.fhi.ox.ac.uk/" target="_blank" rel="noopener noreferrer"><span style="font-weight: 400;">Future of Humanity Institute</span></a> (FHI)<span style="font-weight: 400;">, and their Cambridge counterparts </span><a href="https://www.cser.ac.uk/research/" target="_blank" rel="noopener noreferrer"><span style="font-weight: 400;">the Centre for the Study of Existential Risk</span></a> (CSER)<span style="font-weight: 400;">, both look at ways that humanity could be destroyed. Climate change and, should it happen, nuclear war are highly likely to cause enormous damage, but</span> <a href="https://unherd.com/2020/01/its-time-for-the-doom-mongers-to-clock-off/" target="_blank" rel="noopener noreferrer"><span style="font-weight: 400;">best estimates</span></a><span style="font-weight: 400;"> seem to suggest that they are unlikely to kill us all. We know that civilisation-destroying asteroids can’t happen all that often, because if there was even a 1% risk of one per century, it would be very unlikely that our species would have survived the 500 centuries it has so far.</span></p>
<div class="ref-ar"><a href="https://unherd.com/2020/02/how-coronavirus-will-change-our-culture/?edition=us&#038;ref=refinnar" class="text-decoration-none border-0"><a href="https://unherd.com/2020/02/how-coronavirus-will-change-our-culture/?edition=us&#038;ref=refinnar" class="text-decoration-none border-0"><img decoding="async" src="https://unherd.com/wp-content/uploads/2020/02/GettyImages-1208765870.jpg?w=300" /></a><div class="content"><a href="https://unherd.com/2020/02/how-coronavirus-will-change-our-culture/?edition=us&#038;ref=refinnar" class="text-decoration-none border-0"><span class="meta">Suggested reading</span></a><a href="https://unherd.com/2020/02/how-coronavirus-will-change-our-culture/?edition=us&#038;ref=refinnar" class="text-decoration-none border-0"><h5>Will coronavirus change the world?</h5></a><p class="author"><a href="https://unherd.com/2020/02/how-coronavirus-will-change-our-culture/?edition=us&#038;ref=refinnar" class="text-decoration-none border-0"><span>By</span> Tom Chivers</a></p></div></a></div>
<p><span style="font-weight: 400;">Both CSER and FHI consider pandemics — specifically, genetically engineered pandemics, released either deliberately or accidentally — to be one of the two most likely ways for humanity to annihilate itself. The other is </span><a href="https://unherd.com/2018/07/disney-shows-ai-apocalypse-possible/" target="_blank" rel="noopener noreferrer"><span style="font-weight: 400;">superintelligent AI</span></a><span style="font-weight: 400;">. </span><span style="font-weight: 400;">COVID-19 is not a genetically engineered pandemic. But it, among other things, can tell us a bit about how prepared the world is for one. The world has a much stronger response now than it did </span><a href="https://unherd.com/thepost/chinas-coronavirus-will-not-be-the-next-black-death/"><span style="font-weight: 400;">even five years ago</span></a><span style="font-weight: 400;">; it is more coordinated, it shares data better. But there are still various things that are profoundly lacking. </span></p>
<p><span style="font-weight: 400;">The effective altruism movement focuses on the most effective ways to do good in the world, whether through your career or through charitable donations. The charity evaluator </span><a href="https://www.givewell.org/"><span style="font-weight: 400;">Givewell</span></a><span style="font-weight: 400;">, for instance, suggests that we ought to donate to charities that provide bed nets in countries at risk of malaria, because conservative estimates suggest that you can save a life for a couple of thousand pounds by doing so. But the “</span><a href="https://www.effectivealtruism.org/articles/cause-profile-long-run-future/"><span style="font-weight: 400;">long-termist</span></a><span style="font-weight: 400;">” parts of the movement focus on what we can do to reduce the risk of catastrophic disasters, because even a small reduction in the risk has a huge potential outcome when you’re dealing with numbers as big as those mentioned above. </span></p>
<p><span style="font-weight: 400;">Talking on the 80,000 Hours podcast, run by the Centre for Effective Altruism, FHI’s Cassidy Nelson, who studies catastrophic biological risks, lists</span><a href="https://80000hours.org/podcast/episodes/cassidy-nelson-12-ways-to-stop-pandemics/"><span style="font-weight: 400;"> 12 things</span></a><span style="font-weight: 400;"> that would help stop the next big outbreak. </span></p>
<p><span style="font-weight: 400;">At the moment, for example, it takes at least a year to get from new disease to patient-ready vaccine; that is a huge improvement over even a few years ago, but still, the timetable needs to come down enormously, probably via research into generic vaccines that can have the gene sequences of new diseases “plugged in”. Similarly, there are very few broad-spectrum antivirals in the same way that there are antibiotics. Screening for pathogens is very specific at the moment; improving ways to screen people and areas for any unknown new disease is vital. </span></p>
<p><span style="font-weight: 400;">Biosecurity, too, on even the most “secure” labs, is far short of what we’d want – the second foot-and-mouth outbreak in Britain was caused when </span><a href="https://en.wikipedia.org/wiki/Merial#Investigations"><span style="font-weight: 400;">a leaky pipe</span></a><span style="font-weight: 400;"> at a lab dripped active virus into the water supply of a farm. That lab was supposedly biosafety level 4 – the highest level. If it had been doing work on genetically engineered super-smallpox, it would not have been rated higher. And bafflingly, labs are not required to disclose accidents – such as that one – and are incentivised to keep them quiet so they don’t lose their BSL4 ratings. </span><span style="font-weight: 400;">There’s a huge lack of regulation over “dual-use” research – research into things like pathogens that could be used to help people but could also be used to cause harm.</span></p>
<p><span style="font-weight: 400;">There’s also the problem that synthetic biology – engineering DNA in labs – lacks oversight. In 2006 a <em>Guardian</em> reporter </span><a href="https://www.theguardian.com/science/2006/jun/23/weaponstechnology.guardianweekly"><span style="font-weight: 400;">ordered small sections</span></a><span style="font-weight: 400;"> of smallpox DNA from a private lab. It wasn’t enough to create the virus but it should have raised alarm bells. DNA sequencing is much more powerful now; regulation is even more important.</span></p>
<div class="ref-ar"><a href="https://unherd.com/2020/03/how-coronavirus-plays-into-trumps-hands/?edition=us&#038;ref=refinnar" class="text-decoration-none border-0"><a href="https://unherd.com/2020/03/how-coronavirus-plays-into-trumps-hands/?edition=us&#038;ref=refinnar" class="text-decoration-none border-0"><img decoding="async" src="https://unherd.com/wp-content/uploads/2020/03/GettyImages-1148074927.jpg?w=300" /></a><div class="content"><a href="https://unherd.com/2020/03/how-coronavirus-plays-into-trumps-hands/?edition=us&#038;ref=refinnar" class="text-decoration-none border-0"><span class="meta">Suggested reading</span></a><a href="https://unherd.com/2020/03/how-coronavirus-plays-into-trumps-hands/?edition=us&#038;ref=refinnar" class="text-decoration-none border-0"><h5>How coronavirus plays into Trump&#039;s hands</h5></a><p class="author"><a href="https://unherd.com/2020/03/how-coronavirus-plays-into-trumps-hands/?edition=us&#038;ref=refinnar" class="text-decoration-none border-0"><span>By</span> Giles Fraser</a></p></div></a></div>
<p><span style="font-weight: 400;">None of these changes will be easy to make. Vaccines and antiviral drugs are not good money-spinners for pharmaceutical companies; encouraging their research will involve government or World Bank support. Perhaps incentivising them by creating research prizes would be a way to do it. Stronger world standards on biosecurity will need global coordination and leadership: labs and countries are often loath to reveal their own mistakes. But every small change we make will slightly increase the chance that humanity gets to enjoy the far future.</span></p>
<p><span style="font-weight: 400;">It’s worth talking about the huge potential future of humanity, as I did above, because we can get lost, sometimes, thinking about what are, in the end, short-term, local problems – Brexit, Trump, free-speech freakouts at university, trans rights. These things are important but they occupy an enormous amount of our attention, probably a disproportionate amount. But things that are much less fun to argue about get much less attention. Those things are sometimes prosaic, like antimalarial bed nets, and we don’t concentrate on them because </span><a href="https://slatestarcodex.com/2014/12/17/the-toxoplasma-of-rage/"><span style="font-weight: 400;">it doesn’t involve a big, satisfying fight</span></a><span style="font-weight: 400;">. But sometimes they’re big and weird, like existential risk; we get nervous of taking a position on them because we don’t want to look mad.</span></p>
<p><span style="font-weight: 400;">I don’t want to suggest that</span><span style="font-weight: 400;"> governments and research funding bodies</span><span style="font-weight: 400;"> drop everything and start focusing solely on preparing for the next pandemic; that would be as stupid as not doing anything at all. But I think it’s fair to say that civilisation is not as well prepared for big outbreaks as it could be, and that now is a good time – while everyone’s minds are focused – to improve things.</span><span style="font-weight: 400;"> Humans could be around for a billion years, or more, if we don’t screw it up. Coronavirus won’t be the thing that kills us all, but it’s a bloody good illustration of how something could.</span></p>
]]>
							</content:encoded>
																											<wfw:commentRss>https://unherd.com/2020/03/will-we-be-ready-for-the-next-pandemic/?edition=us/feed/</wfw:commentRss>
															<post-id xmlns="com-wordpress:feed-additions:1">31191</post-id>				</item>
						</channel>
</rss>
