This is a cross-post from my blog post.
Over the past decade, the effective altruism movement has become increasingly focused on “existential risk,” the idea that, this century, there’s a significant chance that humanity will go extinct, become permanently disempowered, or otherwise lose almost all of its future potential.
In this post, I’m not going to argue that, given what we know, we should believe that existential risk is high or low. Instead, what I’m going to argue is that existential risk is an extraordinary claim and that, as such, we must have extraordinary evidence to be concerned about it. Whether we have such evidence I’ll leave to the reader.
My most general argument for this is simply that existential risk represents a dramatic state change in the world and that, the more dramatic of a state change someone is arguing for, the more evidence they must supply to convince us of it. If your friend tells you that one of your windows could get broken this year because of the upcoming hail storm, you have good reason to believe them. But, if they tell you that every single house in your entire state will be completely destroyed, they need to provide extreme evidence to substantiate such a claim.
Extinction
The dominant existential risk that effective altruists worry about is human extinction.
But, notably, in order for this to happen, there has to be an extremely specific set of conditions. It can’t merely be that some event, say a nuclear war or a pandemic, causes a large number of people to die. Instead, it has to be that, for some reason, every single last human is killed. The only mechanisms I can really think of that would cause this is either that some entity actively seeks out and kills every human or that every part of the Earth becomes uninhabitable for a significant duration of time.
One way some argue this could occur is that humanity will develop a superintelligent AI that will systematically seek out and kill every last human. But, to believe that this could happen, we have to believe the claims that humanity can make such an intelligence, that humanity will make such an intelligence, and that, if we do, it will seek to destroy us.
Another way some argue that this could occur is that humanity will develop engineered viruses that infect and then kill everyone. But, to believe that this could happen, we need to have good evidence to think that such a virus (one that spreads rapidly and has 100% fatality) can be engineered, that people who are unreachable (because they’re in bunkers or outer space or Antartica) will also die, and that humanity will be unable to create a cure for such a virus in time.
The last way that some argue this could occur is that two great powers will engage in a nuclear war that causes a nuclear winter. This could cause crop failures around the world, and billions would certainly die. But, it’s important to remember that nuclear winters are not expected to last more than a decade. Given this, we need to believe not only that total war between nuclear powers will occur and that this war will result in a nuclear winter, but also that those living underground or in protected areas will also perish before the surface of the Earth becomes capable of agriculture again.
Some may take the claims of a few experts or their own “vibes” as evidence that we should be very worried about existential risk, but it’s important to remember that humans have a natural proclivity towards believing that their generation could be the last. (According to Pew, “39% of adults believe that humanity is ‘living in the end times.’”) Given this, we can’t merely rely on the beliefs of a few experts or our personal opinions. Instead, we need the concern of a significant portion of experts to really justify concern. Some may believe they have strong enough evidence to worry, but it’s important to remember the mountain of learning that has to be done to be at all confident about these things. (For instance, it seems pretty hard to say whether extremely deadly viruses could be engineered without knowing a lot about virology.)
Disempowerment
Another existential risk effective altruists worry about is disempowerment.
The basic concern is that, if superintelligence is developed, it actually may not eliminate humanity because that would be a waste of resources. Instead, it would simply disempower us by eliminating our ability to make choices about the future of our species. Whether this is more or less likely than the extinction scenario I’ll leave to the reader.
Irreversible Civilizational Collapse
The last existential risk effective altruists worry about is irreversible civilizational collapse.
Some effective altruists believe that, even if the risks in the extinction section wouldn’t actually kill everyone, they would still kill enough people that humanity would be unable to re-industrialize afterwards, possibly because it has gone through all of the easily accessible fossil fuels on Earth. (The thought is that, if you can’t get coal, you might not be able to make a steam engine and then not be able to make almost anything else.)
I think this is also an extraordinary claim because it supposes that there’s a significant risk of almost everyone dying, that, if almost everyone died, civilization would become set back to pre-industrial development, and that, if this were to occur, humans would never be able to re-industrialize despite having vast amounts of knowledge, technology, and creativity.
You say that you’re not arguing if existential risk is high or low, but that’s it’s an extraordinary claim. But your arguments to me just seem to be arguing that the risk is low. Which is fine, I’m just confused by the distinction you’re trying to draw here.
You say we need to have extraordinary evidence to be concerned about it, but I don’t really get this. I think the amount of concern we should have about x-risk should be proportional to both the risk level and the amount that we think would be lost if a catastrophe occurred. There are people who think the risk is 1% (which is pretty low) and who are still extremely concerned because there’s just so much at stake.
Basically I just see your post as arguing that the risk seems low and presumably therefore that you think there are higher priority things we should be doing. That’s fine, you’re just framing your argument in a confusing way.
Thanks for explaining how you perceived the article. There's some better way I could have written this. I argued that x-risk is an extraordinary claim not that x-risk is low because I'm much more confident in the first claim than the second.
My impetus for writing is that I think that people tend to be overly confident about x-risks because they fail to appreciate how much must be argued in order to be really justified in worrying about them.
Also, for clarity, I would consider a 1% x-risk to be extremely high. I don't think that we should lower our epistemic standards merely because the stakes are high as this could lead us to constantly invest resources in action when we know very little and we should instead be investing in further research.
In this context I don’t understand the difference between x-risk being an extraordinary claim and the risk being low.
One could distinguish this in a pop-bayesian way: an extraordinary claim is one that we should put an extremely low prior probability on being true, based on the nature claim alone. So, for example, quantum physics is extremely weird and out of line with what came before, before the quantum revolution we would place an insanely low probability of it being true.
But then you assess the evidence, and calculate the posterior probability based on that evidence. In order to be confident that an extraordinary claim is true, you need an extraordinary amount of evidence, like what we found for quantum physics. So I would say quantum physics is an extraordinary claim that is nonetheless extremely likely to be true.
This is not exactly rigorous (but none of this stuff is anyway). I do think existential risk is an extraordinary claim in this sense, and the prior probability for near-term extinction really should be extremely low. I personally don't think the evidence offered by the x-risk community comes anywhere close to "extraordinary", and so I believe that the overall risk is low as well.
The claim "Teleportation will be invented next year" is extraordinary. If 20% of physicists believe the claim to be true, we should think there's a significant chance it will occur. The claim "AI will cause human extinction" is also extraordinary, but, if a significant portion of AI researchers believe it is true, then we should probably think there's a significant chance (with no additional knowledge) that this will occur. Something can be extraordinary and true. General relativity is extraordinary and true.
Part of what I'm struggling with is that "existential risk" isn't a claim, it's a concept. So saying "existential risk is an extraordinary claim" makes no sense. "Teleportation will be invented next year" is a very specific claim I suppose you could call extraordinary. I'm not sure what specific claim you're evaluating. Maybe it's the claim that "there is a realistic chance of an existential catastrophe occurring in the next 5 years" or something like that?
I also don't know what criteria you use to call something "extraordinary" or not. What is it that actually makes general relativity extraordinary?
Are you essentially saying our prior belief in an existential catastrophe occurring should be very low because something so drastic has never happened before, and the evidence for it is not strong enough to shift us to a position where it should be a priority for us? This would be a framing that makes more sense to me.
I think that the statement "There is a one in six chance that humanity will permanently destroy its future potential value" is an extraordinary claim.
What makes something extraordinary is definitely vibes-based and meant as a framing. For me, it is essentially whether the claim involves a vast change in our typical understanding of the world or a vast change in how our behavior should be in context to our world. The claim "There is a second world where people live forever" is an extraordinary claim. The claim "Lifespans in this part of Afghanistan are decreasing" is not an extraordinary claim.
Yeah, your last paragraph is how I think I should have re-framed the article. We have very little precedence for x-risks, given that the ones people worry about all have essentially no historical precedence. Additionally, since humans are the most intelligent, resilient, and creative species, we should expect that the risks that humans face must be truly vast in scale and unable to be defended against.
I generally worry that, because our brains have a proclivity towards believing we're in the end times, we may waste resources by lowering our epistemic standards when the concept of existential risk is raised.