IEET > Rights > HealthLongevity > GlobalDemocracySecurity > Vision > Contributors > Phil Torres > Philosophy > Futurism > SciTech
Problems with Defining an Existential Risk
Phil Torres   Jan 21, 2015   Ethical Technology  

What is an existential risk? The general concept has been around for decades, but the term was coined by Nick Bostrom in his seminal 2002 paper, here. Like so many empirical concepts – from organism to gene to law of nature, all of which are still debated by philosophically-minded scientists and scientifically-minded philosophers – the notion of an existential risk turns out to be more difficult to define than one might at first think.

In his 2002 paper, Bostrom defines the term a couple of ways. The most prominent definition specifies an existential risk as “one where an adverse outcome would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential” (my italics). Bostrom provides a matrix to graphically represent this definition, Figure A, which comes from his 2008 co-edited book Global Catastrophic Risks. As you can see, an existential risk is defined as being transgenerational in scope and terminal in intensity.

By “transgenerational,” Bostrom means that it affects “not only the current world population but all generations that could come to exist in the future.” We’ll examine some problems with this term below. By “terminal,” Bostrom doesn’t mean “an end to our lineage,” as one might assume given the standard dictionary definition of the word. If this were the case, the conception of existential risks as transgenerational-terminal events would directly contradict the second part of the definition above, which counts certain survivable events as existential risks as long as they involve the permanent and drastic compromising of our potential. The idiosyncratic definition adopted by Bostrom characterizes “terminal” in terms of either “causing death or permanently and drastically reducing quality of life,” as he puts it. Essentially, “terminal” can also mean “endurable,” with some extra conditions. So the prima facie contradiction between these two definitions disappears by stipulation: a “terminal” event is one resulting in either death or a lasting reduction of life-quality. Let’s call this Definition #1.

In my reading of the literature, the term “existential risks” is more often used in a different way, as referring to scenarios in which our species kicks the bucket. For example, the Centre for the Study of Existential Risks states on its website that “an existential risk is one that threatens the existence of our entire species.” Bostrom himself gestures at this definition when he writes, in his 2002 paper, that “an existential risk is one where humankind as a whole is imperiled” (my italics).i Thus, this definition focuses on Homo sapiens in particular, rather than “Earth-originating intelligent life” more generally, which could include a multiplicity of beings in addition to humanity. Let’s call this Definition #2.

Notice that Definition #2 has objective criteria, whereas Definition #1 is thoroughly normative. This isn’t necessarily a bad thing, but it does introduce some complications. For example, what exactly does “intelligent life” mean? If humanityii dies out in a decades-long volcanic winter but chimpanzees survive, would this count as an existential risk? Chimps don’t perform as well as humans on IQ tests, but surely it would be wrong to claim they aren’t “intelligent.”

One might respond that the potential of earth-originating life would be severely compromised if chimps were to take over the planet. But as before, what exactly does “potential” mean? One answer comes from transhumanism: reaching our potential has something to do with becoming posthuman, with continuing a trend of ever-increasing productivity, efficiency, information processing, economic and population growth, and so on.iii But not everyone agrees with these measures of “progress.” The anarcho-primitivist, for instance, might assert that life will reach its full potential once humanity returns to nature, adopting the old hunter-gatherer mode of subsistence we abandoned 12,000 years ago. Others might argue, as Naomi Oreskes and Erik Conway (implicitly) do in their book The Collapse of Western Civilization, that the relevant measure of progress shouldn’t be gross domestic product, but “the Bhutanian concept of gross domestic happiness.”

While one can put forth more or less cogent arguments for these positions, the point is that there exist no “facts of the matter” at which one can point to settle disputes about what “potential” means. The same goes for “intelligent life”: where do we draw the line? Is counting human beings but not chimps as intelligent life too anthropocentric? These are non-trivial issues, as they call into question the core meaning of Definition #1.

(One might also object that the word “permanent” is too strong. Why? Because it disqualifies scenarios in which human civilization is severely crippled for millions, even billions of years, but then recovers, from counting as existential risks. As long as the damage isn’t forever, the risk ain’t existential, even if it’s nearly forever. The further out one goes in time, the less plausible the word “permanent” appears.)


There are more pressing issues, though: both Definition #1 and #2 are susceptible to Gettier-like counterexamples, or outcomes that quack like a duck but aren’t, and ones that don’t quack like a duck but are. Beginning with Definition #2, consider the fact that in evolutionary biology every state is an in-between state. There are no final forms, only transitional ones. This means that (even bracketing the phenomenon of cyborgization) Homo sapiens is not a permanent fixture of the biosphere: we are a mere link between the not-yet-extant and the already extinct.

Imagine for a moment that millions of years pass and natural selection drastically modifies our phenotypes. We become a new species, call it Gedanken experimentus. Imagine further that this species is more intelligent, wise, physically fit, and altruistic than we have ever been. For example, it figures out how to create dual use technologies that could destroy the universe, while also figuring out how to prevent them from ever being used this way. Gedanken experimentus is, in a word, better than us, relative to some non-controversial criteria of evaluation.

The problem is that if one equates an existential risk with human extinction, as Definition #2 does, the above situation counts as an existential risk, since the appearance of Gedanken experimentus millions of years from now through anagenesis (let’s say) would mean the extinction of Homo sapiens. Yet this scenario looks positively desirable. In fact, transhumanism explicitly promotes the extinction of our species; this is what the creation of posthumanity is all about. We “transcend” our current nature, which is “a work-in-progress, a half-baked beginning that we can learn to remold in desirable ways” (from Bostrom’s “Transhumanist Values”). Let’s call this the Problem of Good Extinction.iv

An exactly opposite issue arises for Definition #1. Consider this possible future: let’s say we successfully upload a conscious mind to a supercomputer. Because this mind supervenes upon a silicon (or carbon nanotube) substrate, it’s able to think orders of magnitude faster than we can. In addition, given the multiple realizability of cognitive systems, let’s say this mind clones itself to produce a large population of artificial intelligences (identical pasts, but unique futures). This new society of simulated brains, operating on a faster timescale than humanity, quickly takes over the task of creating a superintelligence, and through the positive feedback process of recursive self-improvement they rapidly become far more clever than us.

Suddenly, then, humanity is confronted by superbeings that tower over us to the extent that we tower over the guinea pig, cognitively speaking. Here’s the catch, though: the first emulated brain – the one from which all the others were spawned – came from a religious extremist. Consequently, the entire population of superbeings inherited its radical beliefs about the way the world is and ought to be. This may seem implausible, but it’s perfectly consistent with Bostrom’s “orthogonality thesis,” according to which end goals and intelligence are not tightly coupled. In other words, a superintelligence could be combined with just about any set of aims and motivations, however strange, irrational, and dangerous they may appear to us.

Because of their religious extremism, then, these superbeings – which qualify as posthumans on Bostrom’s definitionv – quickly destroy humanity. Furthermore, because of their cleverness, they go on to realize all the marvels of technology, foment unprecedented economic growth, colonize the cosmos, and generally live happily ever after. (This gestures at some scenarios that I discuss in a recent IEET article, in which human annihilation by future AIs may actually be morally recommendable.)

The point is that we have a situation here in which Earth-originating intelligent life is neither annihilated, nor is its potential curtailed in a permanent and drastic It follows that no existential risk has occurred, even though our species – indeed, our entire lineage – has been violently destroyed. Bostrom himself counts this as an existential risk, and goes into great detail in his excellent book Superintelligence about how we can prevent our cognitive children from engaging in an act of parricide. Let’s call this the Problem of Bad Extinction.vii

It’s worth noting here that the appearance of “human extinction” in the top right box of Figure A is a bit misleading. While human extinction could constitute an existential catastrophe, this will only be the case if we are the only instance of “Earth-originating intelligent life.” This term is broader than us, though: it could include both Gedanken experimentus, as well as the technoprogressive (but malicious) posthumans mentioned above. Thus, subsequent depictions of this typology should make clear that the top right box includes but is not limited to human extinction scenarios.


Finally, there is a nontrivial problem with Bostrom’s typology of risks itself. Take a close look at the y-axis of Figure A. Bostrom divides the scope of an event’s consequences into personal, local, global, and transgenerational categories. These are presented as alternative possibilities, but not all of them are: the first three concern geography, so to speak, whereas the fourth concerns history. And different geographies can be mixed with different kinds of histories.

​We can put it like this: just as a risk is defined as the probability of an event times its consequences, and the consequences of an event can be analyzed into scope and intensity, so too can the scope of an event’s consequences be analyzed into spatial and temporal components. Thus, we can ask: “For a given time slice, how many people has this event affected?” and “Over a given time segment, how many people has this event affected?” These are completely distinct questions, since the first concerns the spatial scope of a consequence, whereas the second concerns its temporal scope.

For example, a germline mutation could, if survivable, be classified as a personal-endurable risk. But because it affects the germ cells, it is transgenerational in (temporal) scope: in a given time slice it might affect only one person, but peering across time it could affect many people (namely that individual’s descendants). By contrast, a somatic mutation that results in cancer, if survivable, would be a personal-endurable risk that’s generational in scope, since it wouldn’t affect “all generations that could come to exist in the future” (within the relevant spatial scope). Moving up a level, the bombing of Hiroshima could be classified as a local-endurable event, since it caused significant harm but didn’t destroy the quality of life completely (to borrow Bostrom’s definition of “endurable”viii). And its effects have clearly been transgenerational – something dealt with by multiple generations. In contrast, Superstorm Sandy was a local-endurable event in New York City that was merely generational in (temporal) scope.

Once we get to the spatial category of “global,” the difference between generational and transgenerational is the difference between individuals in the population and the population itself. (Where a species is the largest population possible of a biological kind.) This makes sense of the phenomenon of aging, which prompted Bostrom to expand his typology between his 2002 and 2008 publications. On the present account, aging is unproblematic: it constitutes a global-terminal risk whose temporal scope is generational. That is to say, every individual is affected by death, but the population itself isn’t. A runaway greenhouse effect or grey-goo disaster, in contrast, would count as a global-terminal risk whose temporal scope is transgenerational: it affects everyone everywhere forever.

An existential risk can thus be redefined more precisely as any catastrophe that’s spatially global, temporally transgenerational, and terminal in intensity. This fixes the typology, but how does it deal with the two counterexamples above? To address these problems, we need to be specific about the particular population of beings affected by the existential event, since both problems derive from how “everyone everywhere” is construed. (In Definition #1 it’s too broad; in Definition #2 it’s too narrow.)

Perhaps a solution is to specify the relevant group as either our current population or any future population that we care about.ix By defining the group affected in terms of what we value, the problems of good and bad extinction can be effectively solved: insofar as Gedanken experimentus is a desirable species to become, this scenario doesn’t count as an existential risk; and insofar as we don’t want our lineage to be destroyed by a race of superintelligent posthumans (however superior their moral status or aligned with transhumanist values of infinite progress), this scenario does constitute such a risk. Let’s call this Definition #3.

This offers a modified account both of what an existential risk’s consequences are, and of what the relevant group affected by these consequences is. I don’t put it forward only as a tentative idea – a point of departure for further discussion. I have argued that there are a number of problems with the two primary definitions currently being used in the literature – problems that, to my knowledge, no one hast yet pointed out. Perhaps Definition #3 is a small step in the right direction.

(For more on this and a number of related issues, see my forthcoming book The End: What Religion and Science Tell Us About the Apocalypse.)

i One finds this in other papers of his as well, as in the abstract of Bostrom’s “Existential Risk Prevention as Global Priority.

ii Here I’ll use the non-anthropology sense of Homo sapiens specifically, rather than our entire genus.

iii See Bostrom’s “Transhumanist Values.” Search for “technological progress.”

iv Notice that Definition #1 handles this scenario perfectly well. It doesn’t count it as an existential risk, which seems right.

v See “Why I Want to be a Posthuman When I Grow Up.”

vi Accepting, say, transhumanism’s definition of “potential.”

vii Notice that Definition #2 handles this scenario perfectly well. It counts it as an existential risk, which seems right. Also notice that if bad extinction weren’t a problem, the project of Friendly AI would be pointless.

viii We could, alternatively, simply say that the risk was endured by the locality, which is, after all, the relevant (spatial) scope.

ix Or would care about if we knew about them.

Phil Torres is an author and artist. His forthcoming book is called The End: What Science and Religion Tell Us About the Apocalypse (Pitchstone Publishing). You can contact him here:


You raise some thoughtful and, I believe, very valid objections to Nick Bostrom’s classification of existential risks. In fact, I think your Definition #3 only scratches the surface of possible exceptions and contingencies. It could turn out to be an even be a broader category than the first two. The term existential is very elastic, and bisecting it into transgenerational and terminal does little to tighten it up. Extinction, as defined by the physical death of all members of a species, is pretty easy to grasp (though still not an open-shut case). But what constitutes a “lasting reduction in life quality?” Here are a number of conceivable scenarios that don’t fit into Bostrom’s pigeonholes.

Homo Sapiens Park: What if the remnant of humanity were kept on a bio-preserve by a higher-order race like the Gedanken experimentus? Sort of like a Jurassic Park for homo sapiens where they are free to roam the grounds of their walled-off garden but their freedoms are entirely circumscribed. In this scenario, homo sapiens are physically alive and thriving, perhaps even in large numbers. Like a breed of animals that exists only in the zoo, humanity would now exist only as a captive species, even if the captivity is a highly benevolent one. Indeed, homo sapiens under the protective wing of the Gedanken may not be allowed to go extinct—but neither could it further progress and evolve. While this may seem like Matrix-style dystopia, it makes perfect logical sense. How could mere humans, with their ponderous biological brains and unstable psyches hope to contribute anything of value to a posthuman civilization with god-like minds and abilities? Humans would be like toddlers fumbling with electrical outlets, able to offer no new insight and only apt to do themselves harm. The paternal Gedankus would have every reason not to entrust them with quantum loop-time paradoxology which, wrongly applied, could fizz the universe into Hawking radiation; instead, the human toddlers would be given the posthuman equivalent of “fancy toys” to play with. Humans would be provided with everything they want or desire including health and longevity; their quality of life would be far superior to anything conceivable today. There may still be a way out of Homo Sapiens Park but that way would likely require a human to be elevated to Gedanken, at which point he would cease to be a member of the human race.

DNA Bank: What if homo sapiens becomes extinct insofar as there are no longer any living members but an extensive DNA bank is preserved along with a record of human civilization, a sort of Voyager record for the Information Age? In this state, humanity is not permanently extinct but could be “on ice” for millions or even billions of years.

Universal Mind uploads: Another spin on the scenario you introduced. What if the brain patterns of every living human could be instantiated onto a computational substrate where they occupy avatars in a simulation so rich that it is indistinguishable from reality—better even. Eventually, the physical humans die off, leaving behind a vibrant and dynamic race of human-equivalent avatars. Does this count as extinction?

Radical speciation: What if humanity were to splinter into numerous sub and super-species? Just like all biological life on Earth branched out from a single progenitor, what if homo sapiens were to become the seed of a great flowering of sentient beings vastly different both in intellect and morphology? Biological engineering could result in races of merman, centaurs, pixies, leprechauns, cyborgs, human pterodactyls, almost any form conceivable. As for intellect, there could be hive-minds, super-minds, proto-minds, mind splinters and fragments, augmented and hybrid minds, bio-minds and computer minds. The combinations are literally boundless and, just like the multitudes of extant biological species, they could co-exist and interact in complex and interdependent ways. Humanity, as we know it, would be an undistinguished member of a larger sentient community, largely irrelevant except for its role as the progenitor race. Is it a diminution of humanity to play only a bit part in the future it brought about?

The isolated offshoot: What if humanity were to colonize another star system where, over the course of several millennia, local conditions caused an accelerated mutation rate, creating a race of homo pleaides radically different from the Earth adapted homo sapiens? Then a rogue black hole plows into the Sol system like a cosmic wrecking ball, obliterating everything in its path. Homo pleaides is all that remains.

DNA assimilation: The Neanderthals went extinct about 40,000 years ago. Or did they? Recent DNA findings estimate that the DNA of modern humans have a Neanderthal component of 1-4%. Does that mean Neanderthals are only 96-99% extinct? What if the component was much higher, say 20-30%? What if the tables are turned and a future race of evolved Earthlings drives homo sapiens to extinction but, through interbreeding, carries 40% of their DNA along for the ride?

Interesting ideas, Knotanumber. Thanks for sharing them!

The harm to individuals can be justified in many ways via fancy theories and language spins. The immediate harm can be downplayed and ignored. The immediate harm of tech is to every cell of every living thing on Earth from the life-blood of popular tech, the same tech needed to power robots, and transhumanist plans. Radiofrequency/microwave radiation is not compatible with biological integrity. It is being allowed to flourish without full disclosure to individuals. Futurist visions are in process. People are already becoming victims to the RF/MW life-blood of the scheme.

Morality? It is not existential, it is actual, here and now. Illness, damage, death. Independent science tells the truth. Examples: 2014 Children’s Health and EMR by Dr. Erica Mallery-Blythe at youtube. Work of Leif Salford on blood brain barrier and cell phones at youtube. For consumers, BabySafe Project. 2012 BioInitiative Report. For weapons buffs, Barrie Trower talks on microwave weapons of the cold war. Wifi uses same frequency.

Wireless radiation, deadly life-blood of futurist wetdreams, is being spilled over all. People are not informed. Morality? none here. not if individuals count. not if the natural world counts. not if preventing harm and suffering counts. not if human rights count.

We won’t make it to deal with “super-beings:”  The deadly emissions will cut down the numbers of the living before that dawn.

YOUR COMMENT Login or Register to post a comment.

Next entry: “Surviving the 21st Century” (1hr 30min)

Previous entry: Existential Risk: Future of Humanity Institute | University of Oxford