Abstract
A classification of the global catastrophic risks of AI is presented, along with a comprehensive list of previously identified risks. This classification allows the identification of several new risks. We show that at each level of AI’s intelligence power, separate types of possible catastrophes dominate. Our classification demonstrates that the field of AI risks is diverse, and includes many scenarios beyond the commonly discussed cases of a paperclip maximizer or robot-caused unemployment. Global catastrophic failure could happen at various levels of AI development, namely, (1) before it starts self-improvement, (2) during its takeoff, when it uses various instruments to escape its initial confinement, or (3) after it successfully takes over the world and starts to implement its goal system, which could be plainly unaligned, or feature-flawed friendliness. AI could also halt at later stages of its development either due to technical glitches or ontological problems. Overall, we identified around several dozen scenarios of AI-driven global catastrophe. The extent of this list illustrates that there is no one simple solution to the problem of AI safety, and that AI safety theory is complex and must be customized for each AI development level.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Alexander S (2016) Ascended economy? Star Slate Codex. http://slatestarcodex.com/2016/05/30/ascended-economy/. Accessed 27 Apr 2018
Anderson M (2017) RethinkX: self-driving electric cars will dominate roads by 2030. In: IEEE Spectrum: technology, engineering and science news. http://spectrum.ieee.org/cars-that-think/transportation/self-driving/rethinkx-selfdriving-electric-cars-will-dominate-roads-by-2030. Accessed 17 Jul 2017
Angel.co (2017) Artificial intelligence startups. https://angel.co/artificial-intelligence. Accessed 27 Apr 2018
Armstrong S (2017) Good and safe uses of AI Oracles. ArXiv171105541 Cs
Auerbach D (2014) The Most Terrifying Thought Experiment of All Time. In: Slate. http://www.slate.com/articles/technology/bitwise/2014/07/roko_s_basilisk_the_most_terrifying_thought_experiment_of_all_time.html. Accessed 27 Apr 2018
Baker BH (2000) The gray matter: the forgotten story of the telephone. Telepress, Kent, WA
Bardi U (2008) The Universal Mining Machine. http://europe.theoildrum.com/node/3451. Accessed 27 Apr 2018
Barrett AM, Baum SD (2017) A model of pathways to artificial superintelligence catastrophe for risk and decision analysis. J Exp Theor Artif Intell 29:397–414
BBC (2017) Cyber-attack: europol says it was unprecedented in scale—BBC News. http://www.bbc.com/news/world-europe-39907965. Accessed 17 Jul 2017
Bender J (2014) Russia may still have an automated nuclear launch system aimed across the northern hemisphere. In: Bus. Insid. https://www.businessinsider.com.au/russias-dead-hand-system-may-still-be-active-2014-9. Accessed 17 Jul 2017
Blair BG (2011) The logic of accidental nuclear war. Brookings Institution Press, Washington, DC
Boles KS, Kannan K, Gill J et al (2017) Digital-to-biological converter for on-demand production of biologics. Nat Biotechnol 35:672–675 2017
Bostrom N (2002) Existential risks: analyzing human extinction scenarios and related hazards. J Evol Technol 9(1):1–30
Bostrom N (2003a) Astronomical waste: The opportunity cost of delayed technological development. Utilitas 15:308–314
Bostrom N (2003b) Are you living in a computer simulation? Publ Philos Q 53(211):243–255
Bostrom N (2006) What is a singleton. Linguist Philos Investig 5:48–54
Bostrom N (2009) Pascal’s mugging. Analysis 69(3):443–445
Bostrom N (2011) Infinite ethics. Anal Metaphys 9–59
Bostrom N (2014) Superintelligence. Oxford University Press, Oxford
Bradbury RJ (2001) Matrioshka brains. preprint. http://www.aeiveos.com/bradbury/MatrioshkaBrains/MatrioshkaBrains.html
Carrigan RA Jr (2006) Do potential SETI signals need to be decontaminated? Acta Astronaut 58:112–117
Chalmers DJ (2002) Does conceivability entail possibility? In: Gendler T, Hawthorne J (eds) Conceivability possibility. Oxford University Press, New York pp 145–200
Chiew KL, Yong KSC, Tan CL (2018) A survey of phishing attacks: their types, vectors and technical approaches. Expert Syst Appl 106:1–20
Christiano P (2016) Prosaic AI alignment. https://ai-alignment.com/prosaic-ai-control-b959644d79c2. Accessed 27 Apr 2018
Clavero M, García-Berthou E (2005) Invasive species are a leading cause of animal extinctions. Trends Ecol Evol 20:110
Cole DD, Denkenberger D, Griswold M et al (2016) Feeding everyone if industry is disabled. In: Proceedings of the 6th international disaster and risk conference. Davos, Switzerland
Critch A (2017) Toward negotiable reinforcement learning: shifting priorities in Pareto optimal sequential decision-making (arXiv:1701.01302)
Daniel M (2017) S-risks: why they are the worst existential risks, and how to prevent them (EAG Boston 2017). https://foundational-research.org/s-risks-talk-eag-boston-2017/. Accessed 27 Apr 2018
Dennett DC (1978) Why you can’t make a computer that feels pain. Synthese 38:415–456
Ellison H (1967) I have no mouth, and i must scream. Galaxy Publishing Corp, New York
Enserink M (2011) Scientists brace for media storm around controversial flu studies. In: Sciencemag. http://www.sciencemag.org/news/2011/11/scientists-brace-media-storm-around-controversial-flu-studies. Accessed 27 Apr 2018
Freitas R (2000) Some limits to global ecophagy by biovorous nanoreplicators, with public policy recommendations. Foresight Institute Technical Report
Future of Life Institute (2016) Accidental nuclear war: a timeline of close calls. https://futureoflife.org/background/nuclear-close-calls-a-timeline/. Accessed 4 Nov 2017
Futureworld (2013) Airplane “crashes” as hacker gets control. In: Futureworld. http://www.futureworld.org/PublicZone/MindBullets/MindBulletsDetails.aspx?MindBulletID=498. Accessed 27 Apr 2018
Gildert S (2011) Why “computronium” is really “unobtanium” IO9. http://io9.gizmodo.com/5758349/why-computronium-is-really-unobtanium. Accessed 27 Apr 2018
Goertzel B (2012) Should humanity build a global ai nanny to delay the singularity until it’s better understood? J Conscious Stud 19(1–2):96–111
Grace K, Salvatier J, Dafoe A et al (2017) When will AI exceed human performance? evidence from AI experts. (arXiv:1705.08807 [cs.AI])
Granoff J (2016) Donald trump is an existential threat to America and the world. Time
Gwern (2016) Why tool AIs want to be agent AIs. https://www.gwern.net/Tool-AI
Hanson R (2008) Catastrophe, social collapse, and human extinction. In: Bostrom N, Cirkovic MM (eds) Global catastrophic risks. Oxford University Press, Oxford, p 554
Hanson R (2016) The age of Em: work, love, and life when robots rule the earth. Oxford University Press, Oxford
Hines N (2016) Neural implants could let hackers hijack your brain. In: Inverse. https://www.inverse.com/article/19148-neural-implants-could-let-hackers-hijack-your-brain. Accessed 17 Jul 2017
Hume D (1739) A treatise of human nature. Oxford: Clarendon Press, London, UK
Hutter M (2000) A theory of universal artificial intelligence based on algorithmic complexity. ArXiv Prepr Cs0004001
Jenkins A (2018) Uber may not be to blame for self-driving car death in Arizona. Fortune, New York
Joy B (2000) Why the future doesn’t need us. Wired, San Francisco, CA
Kahn H (1959) On thermonuclear war. Princeton University Press, Princeton
Kardashev NS (1985) On the inevitability and the possible structures of supercivilizations. Reidel Publishing Co., Dordrecht, pp 497–504
Karpathy A (2015) The unreasonable effectiveness of recurrent neural networks. Andrej Karpathy Blog. http://karpathy.github.io/2015/05/21/rnn-effectiveness/
Kushner D (2013) The real story of stuxnet. IEEE Spectr 50:48–53
LaVictoire P, Fallenstein B, Yudkowsky E et al (2014) Program equilibrium in the prisoner’s dilemma via Löb’s theorem. MIRI
LaVictorie P (2015) An Introduction to Löb’s Theorem in MIRI Research. MIRI, San Francisco CA. http://intelligence.org/files/lob-notes-IAFF.pdf
Lem S (1961) Return from the stars. Houghton Mifflin Harcourt, Boston, US
Lem S (1963) Summa technologiae. Suhrkamp, Berlin, Germany
Lem S (1973) The Invincible: science fiction. Sidgwick & Jackson, London, UK
Lenat DB, Brown JS (1984) Why AM and EURISKO appear to work. Artif Intell 23:269–294
LoPucki LM (2017) Algorithmic ENTITIES. Social Science Research Network, Rochester
Lubin G (2016) Data reveals the 20 most popular TV shows of 2016. Business Insider
Mennen A (2017) Existential risk from AI without an intelligence explosion. http://lesswrong.com/lw/p28/existential_risk_from_ai_without_an_intelligence/
Menzel C (2017) Actualism. In: Zalta EN (ed) The stanford encyclopedia of philosophy, 2014th edn. Metaphysics Research Lab, Stanford University, Stanford
Meuhlhauser L (2014) How big is the field of artificial intelligence? (initial findings). https://intelligence.org/2014/01/28/how-big-is-ai/. Accessed 27 Apr 2018
Muehlhauser L (2011) Intelligence explosion FAQ. https://intelligence.org/ie-faq/. Accessed 27 Apr 2018
Mullin G (2017) What is the Blue Whale suicide challenge, how many deaths has the game been linked to so far and is it in the UK? TheSun
Oberhaus D (2017) Watch ‘Slaughterbots’, a warning about the future of killer bots. In: Motherboard. https://motherboard.vice.com/en_us/article/9kqmy5/slaughterbots-autonomous-weapons-future-of-life. Accessed 17 Dec 2017
Omohundro S (2008) The basic AI drives. In: Wang P, Goertzel B, Franklin S (eds) Proceedings of the 2008 conference on Artificial General Intelligence 2008: proceedings of the First AGI Conference. IOS Press Amsterdam, The Netherlands
Orwell G (1948) 1984. Houghton Mifflin Harcourt, Boston, US
Pinker S (2011) The better angels of our nature: The decline of violence in history and its causes. Penguin, London
Reason J (2000) Human error: models and management. BMJ 320:768–770
Russell S (2017) 3 principles for creating safer AI. https://www.youtube.com/watch?v=EBK-a94IFHY. Accessed 27 Apr 2018
Saito T, Angles J (2013) Hikikomori: adolescence without end. Univesity Of Minnesota Press, Minnesota
Sarma GP, Hay NJ (2016) Mammalian value systems. (arXiv:1607.08289 [cs.AI])
Schneier B (2017) Perspective | The next ransomware attack will be worse than WannaCry. Wash, Post
Shakirov V (2016) Review of state-of-the-arts in artificial intelligence with application to AI safety problem. (ArXiv Prepr ArXiv160504232)
Shulman C (2010) Omohundro’s “basic AI drives” and catastrophic risks. http://intelligence.org/files/BasicAIDrives.pdf. Accessed 27 Apr 2018
Shulman C (2011) Arms races and intelligence explosions. Singularity Hypotheses. Springer, New York
Sotala K (2016) Decisive strategic advantage without a hard takeoff. http://kajsotala.fi/2016/04/decisive-strategic-advantage-without-a-hard-takeoff/#comments. Accessed 27 Apr 2018
Sotala K (2017) Disjunctive AI scenarios: Individual or collective takeoff? http://kajsotala.fi/2017/01/disjunctive-ai-scenarios-individual-or-collective-takeoff/. Accessed 27 Apr 2018
Sotala K, Yampolskiy R (2014) Responses to catastrophic AGI risk: a survey. Phys Scr 90:018001
Srugatsky N, Strugatsky B (1985) The time wanderers. Richardson & Steirman, New York, US
Strugatsky A, Strugatsky B (1976) The final circle of paradise, Translated by Leonid Renen. DAW, New York
Taylor A (2017) Flying around the world in a solar powered plane—the Atlantic
The Telegraph (2009) Russian spacecraft landed on moon hours before Americans. The telegraph. http://www.telegraph.co.uk:80/science/space/5737854/Russian-spacecraft-landed-on-moon-hours-before-Americans.html. Accessed 27 Apr 2018
Torres P (2014) Why running simulations may mean the end is near. https://ieet.org/index.php/IEET2/more/torres20141103. Accessed 27 Apr 2018
Torres P (2016) Problems with defining an existential risk. IEET. https://ieet.org/index.php/IEET2/more/torres20150121. Accessed 27 Apr 2018
Turchin A (2018) The risks connected with possibility of finding alien AI code during SETI. Rev J Br Interplanet Soc. Manuscript, https://philpapers.org/rec/TURCSW
Turchin A, Denkenberger D (2017) Levels of self-improvement. Manuscript, University of Louisville, TN
Turchin A, Denkenberger D (2018a) Military AI as convergent goal of the self-improving AI. In: Yampolskiy R (ed) Artificial intelligence safety and security. CRC Press, Baca Raton
Turchin A, Denkenberger D (2018b) Could slaughterbots wipe out humanity? Assessment of the global catastrophic risk posed by autonomous weapons. Manuscript
Turchin A, Green B, Denkenberger D (2017) multiple simultaneous pandemics as most dangerous global catastrophic risk connected with bioweapons and synthetic biology. Rev Health Secur
Turing AM (1937) On computable numbers, with an application to the Entscheidungsproblem. Proc Lond Math Soc 2:230–265
Velicovich B (2017) I could kill you with a consumer drone. Defense one, Washington, DC
Watkins J (2016) “Shut up and dance”—“Black mirror” series
Wei D (2013) Outside view(s) and MIRI’s FAI endgame. http://lesswrong.com/lw/ig9/outside_views_and_miris_fai_endgame/. Accessed 27 Apr 2018
Wootson J (2017) Elon Musk doesn’t think we’re prepared to face humanity’s biggest threat: artificial intelligence. Wash, Post
Yampolskiy R (2014) Utility function security in artificially intelligent agents. J Exp Theor Artif Intell JETAI 373–389. https://doi.org/10.1080/0952813X.2014.895114
Yampolskiy R (2015a) Artificial superintelligence: a futuristic approach. CRC Press, Boca Raton
Yampolskiy R (2015b) Taxonomy of pathways to dangerous AI. (ArXiv Prepr ArXiv151103246)
Yampolskiy R, Spellchecker M (2016) artificial intelligence safety and cybersecurity: a timeline of AI failures. (ArXiv Prepr ArXiv161007997)
Yudkowsky E (2001) Creating friendly AI 1.0: the analysis and design of benevolent goal architectures. MIRI, San Francisco, CA, pp 1–282
Yudkowsky E (2002) The AI-Box Experiment. http://yudkowsky.net/singularity/aibox. Accessed 27 Apr 2018
Yudkowsky E (2003) HUMOR: friendly AI critical failure table. http://www.sl4.org/archive/0310/7163.html. Accessed 27 Apr 2018
Yudkowsky E (2004) Coherent extrapolated volition. http://intelligence.org/files/CEV.pdf. Accessed 27 Apr 2018
Yudkowsky E (2008) Artificial intelligence as a positive and negative factor in global risk, in global catastrophic risks. Oxford University Press, Oxford
Yudkowsky E (2015) From AI to zombies. MIRI, San Francisco, CA
Yudkowsky E (2017) Comment on paper clip maximiser scenario. http://www.jefftk.com/p/examples-of-superintelligence-risk#fb-886930452142_886983450932. Accessed 27 Apr 2018
Yudkowsky E, Hanson R (2008) The Hanson-Yudkowsky AI-foom debate. In: MIRI Technical report
Yudkowsky E, Herreshoff M (2013) Tiling agents for self-modifying AI, and the Löbian obstacle. Early Draft MIRI
Acknowledgements
We would like to thank Roman Yampolskiy and Seth Baum for their interesting ideas in this article. This article represents views of the authors and does not necessarily represent the views of the Global Catastrophic Risk Institute or the Alliance to Feed the Earth in Disasters. No external sources of funding were used for this work.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Turchin, A., Denkenberger, D. Classification of global catastrophic risks connected with artificial intelligence. AI & Soc 35, 147–163 (2020). https://doi.org/10.1007/s00146-018-0845-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00146-018-0845-5