How vulnerable is the world?

How vulnerable is the world?

 

Sooner or later a technology capable of wiping out human civilisation might be invented. How far would we go to stop it?

 

 

Nick Bostrom & Matthew van der Merwe

AEON

 

 

Nick Bostrom

is professor in the Faculty of Philosophy at the University of Oxford where he is director of the Future of Humanity Institute. His books include Anthropic Bias (2002) and Superintelligence: Paths, Dangers, Strategies (2014).

 

Matthew van der Merwe

is a research assistant at the Future of Humanity Institute at the University of Oxford. He writes about AI policy for the weekly newsletter Import AI.

 

 

 

 

One way of looking at human creativity is as a process of pulling balls out of a giant urn. The balls represent ideas, discoveries and inventions. Over the course of history, we have extracted many balls. Most have been beneficial to humanity. The rest have been various shades of grey: a mix of good and bad, whose net effect is difficult to estimate.

 

What we haven’t pulled out yet is a black ball: a technology that invariably destroys the civilisation that invents it. That’s not because we’ve been particularly careful or wise when it comes to innovation. We’ve just been lucky. But what if there’s a black ball somewhere in the urn? If scientific and technological research continues, we’ll eventually pull it out, and we won’t be able to put it back in. We can invent but we can’t un-invent. Our strategy seems to be to hope that there is no black ball.

 

Thankfully for us, humans’ most destructive technology to date – nuclear weapons – is exceedingly difficult to master. But one way to think about the possible effects of a black ball is to consider what would happen if nuclear reactions were easier. In 1933, the physicist Leo Szilard got the idea of a nuclear chain reaction. Later investigations showed that making an atomic weapon would require several kilos of plutonium or highly enriched uranium, both of which are very difficult and expensive to produce. However, imagine a counterfactual history in which Szilard realised that a nuclear bomb could be made in some easy way – over the kitchen sink, say, using a piece of glass, a metal object and a battery.

 

 

Szilard would have faced a dilemma. If he didn’t tell anyone about his discovery, he would be unable to stop other scientists from stumbling upon it. But if he did reveal his discovery, he would guarantee the further spread of dangerous knowledge. Imagine that Szilard confided in his friend Albert Einstein, and they decided to write a letter to the president of the United States, Franklin D Roosevelt, whose administration then banned all research into nuclear physics outside of high-security government facilities. Speculation would swirl around the reason for the heavy-handed measures. Groups of scientists would wonder about the secret danger; some of them would figure it out. Careless or disgruntled employees at government labs would let slip information, and spies would carry the secret to foreign capitals. Even if by some miracle the secret never leaked, scientists in other countries would discover it on their own.

 

Or perhaps the US government would move to eliminate all glass, metal and sources of electrical current outside of a few highly guarded military depots? Such extreme measures would meet with stiff opposition. However, after mushroom clouds had risen over a few cities, public opinion would shift. Glass, batteries and magnets could be seized, and their production banned; yet pieces would remain scattered across the landscape, and eventually they would find their way into the hands of nihilists, extortionists or people who just want ‘to see what would happen’ if they set off a nuclear device. In the end, many places would be destroyed or abandoned. Possession of the proscribed materials would have to be harshly punished. Communities would be subject to strict surveillance: informant networks, security raids, indefinite detentions. We would be left to try to somehow reconstitute civilisation without electricity and other essentials that are deemed too risky.

 

That’s the optimistic scenario. In a more pessimistic scenario, law and order would break down entirely, and societies would split into factions waging nuclear wars. The disintegration would end only when the world had been ruined to the point where it was impossible to make any more bombs. Even then, the dangerous insight would be remembered and passed down. If civilisation arose from the ashes, the knowledge would lie in wait, ready to pounce once people started again to produce glass, electrical currents and metal. And, even if the knowledge were forgotten, it would be rediscovered when nuclear physics research resumed.

 

In short: we’re lucky that making nuclear weapons turned out to be hard. We pulled out a grey ball that time. Yet with each act of invention, humanity reaches anew into the urn.