Toby Ord’s The Precipice is an ambitious and excellent book. Among many other things, Ord attempts to survey the entire landscape of existential risks humanity faces. As part of this, he provides, in Table 6.1, his personal estimates of the chance that various things could lead to existential catastrophe in the next 100 years. He discusses the limitations and potential downsides of this (see also), and provides a bounty of caveats, including:
Don’t take these numbers to be completely objective. [...] And don’t take the estimates to be precise. Their purpose is to show the right order of magnitude, rather than a more precise probability.
Another issue he doesn’t mention explicitly is that people could anchor too strongly on his estimates.
But on balance, I think it’s great that he provides this table, as it could help people to:
- more easily spot where they do vs don’t agree with Ord
- get at least a very approximate sense of what ballpark the odds might be in
In this post, I will:
- Present a reproduction of Table 6.1
- Discuss whether Ord may understate the uncertainty of these estimates
- Discuss an ambiguity about what he’s actually estimating when he estimates the risk from “unaligned AI”
- Discuss three estimates I found surprisingly high (at least relative to the other estimates)
- Discuss some adjustments his estimates might suggest EAs/longtermists should make to their career and donation decisions
Regarding points 2 and 4: In reality, merely knowing that these are Ord’s views about the levels of uncertainty and risk leads me to update my views quite significantly towards his, as he’s clearly very intelligent and has thought about this for much longer than I have. But I think it’s valuable for people to also share their “independent impressions” - what they’d believe without updating on other people’s views. And this may be especially valuable in relation to Ord’s risk estimates, given that, as far as I know, we have no other single source of estimates anywhere near this comprehensive.
I’ll hardly discuss any of the evidence or rationale Ord gives for his estimates; for all that and much more, definitely read the book!
The table
Here’s a reproduction of Table 6.1:
Understating uncertainty?
In the caption for the table, Ord writes:
There is significant uncertainty remaining in these estimates and they should be treated as representing the right order of magnitude - each could easily be a factor of 3 higher or lower.
Lighthearted initial reaction: Only a factor of 3?! That sounds remarkably un-uncertain to me, for this topic. Perhaps he means the estimates could easily be a factor of 3 higher or lower, but the estimates could also be ~10-50 times higher or lower if they really put their backs into it?
More seriously: This at least feels to me surprisingly “certain”/“precise”, as does his above-quoted statement that the estimates’ “purpose is to show the right order of magnitude”. On the other hand, I’m used to reasoning as someone who hasn’t been thinking about this for a decade and hasn’t written a book about it - perhaps if I had done those things, then it’d make sense for me to occasionally at least know how many 0s should be on the ends of my numbers. But my current feeling is that, when it comes to existential risk estimates, uncertainties even about orders of magnitude may remain appropriate even after all that research and thought.
Of course, the picture will differ for different risks. In particular:
-
for some risks (e.g., asteroid impacts), we have a lot of at least somewhat relevant actual evidence and fairly well-established models.
- But even there, our evidence and models are still substantially imperfect for existential risk estimates.
-
And for some risks, the estimated risk is already high enough that it’d be impossible for the “real risk” to be two orders of magnitude higher.
- But then the real risk could still be orders of magnitude lower.
I’d be interested in other people’s thoughts on whether Ord indeed seems to be implying more precision than is warranted here.
What types of catastrophe are included in the “Unaligned AI” estimate?
Ord estimates a ~1 in 10 chance that “unaligned artificial intelligence” will cause existential catastrophe in the next 100 years. But I don’t believe he explicitly states precisely what he means by “Unaligned AI” or “alignment”. And he doesn’t include any other AI-related estimates there. So I’m not sure which combination of the following issues he’s estimating the risk from:
-
AI systems that aren’t even trying to act in accordance with the instructions or values of their operator(s) (as per Christiano’s definition).
- E.g., the sorts of scenarios Bostrom’s Superintelligence focuses on, where an AI actively strategises to seize power and optimise for its own reward function.
-
AI systems which are trying to act in accordance with the instructions or values of their operator(s), but which make catastrophic mistakes in the process.
- I think that, on some definitions, this would be an “AI safety” rather than “AI alignment” problem?
-
AI systems that successfully act in accordance with the instructions or values of their operator(s), but not of all of humanity.
- E.g., the AI systems are “aligned” with a malicious or power-hungry actor, causing catastrophe. I think that on some definitions this would be a “misuse” rather than “misalignment” issue.
-
AI systems that successfully act in accordance with something like the values humanity believes we have, but not what we truly value, or would value after reflection, or should value (in some moral realist sense).
-
“Non-agentic” AI systems which create “structural risks” as a byproduct of their intended function, such as by destabilising nuclear strategies.
In Ord’s section on “Unaligned artificial intelligence”, he focuses mostly on the sort of scenario Bostrom’s Superintelligence focused on (issue #1 in the above list). However, within that discussion, he also writes that we can’t be sure the “builders of the system are striving to align it with human values”, as they may instead be trying “to achieve other goals, such as winning wars or maximising profits.” And later in the chapter, he writes:
I’ve focused on the scenario of an AI system seizing control of the future, because I find it the most plausible existential risk from AI. But there are other threats too, with disagreement among experts about which one poses the greatest existential risk. For example, there is a risk of a slow slide into an AI-controlled future, where an ever-increasing share of power is handed over to AI systems and an increasing amount of our future is optimised towards inhuman values. And there are the risks arising from deliberate misuse of extremely powerful AI systems.
I’m not sure whether those quotes suggest Ord is including some or all of issues #2-5 in his definition or estimate of risks from “unaligned AI”, or if he’s just mentioning “other threats” as also worth noting but not part of what he means by “unaligned AI”.
I’m thus not sure whether Ord thinks the existential risk:
(A) from all of the above-mentioned issues is ~1 in 10.
(B) from some subset of those issues is ~1 in 10, while the risk from the other issues is negligible.
- But how low would be negligible anyway? Recall that Table 6.1 includes a risk for which Ord gives only odds of 1 in a billion; I imagine he’d seen the other AI issues as more risky than that.
(C) from some subset of those issues is ~1 in 10, while the risk from others of those issues is non-negligible but (for some other reason) not directly estimated
Personally, and tentatively, it seems to me that at least the first four of the above issues may contribute substantially to existential risk, with no single one of the issues seeming more important than the other three combined. (I’m less sure about the importance of structural risks from AI.) Thus, if Ord meant B or especially C, I may have reason to be even more concerned than the ~1 in 10 estimate suggests.
I’d be interested to hear other people’s thoughts either on what Ord meant by that estimate, or on their own views about the relative importance of each of those issues.
“Other environmental damage”: Surprisingly risky?
Ord estimates a ~1 in 1000 chance that “other environmental damage” will cause existential catastrophe in the next 100 years. This includes things like overpopulation, running out of critical resources, or biodiversity loss, and not climate change.
I was surprised that that estimate was:
- that high
- as high as his estimate of the existential risk from each of nuclear war and climate change
- 10 times higher than his estimate for the risk of existential catastrophe from “‘naturally’ arising pandemics”
I very tentatively suspect that this estimate of the risk from other environmental damage is too high. I also suspect that, whatever the “real risk” from this source is, it’s lower than that from nuclear war or climate change. E.g., if ~1 in 1000 turns out to indeed be the “real risk” from other environmental damage, then I tentatively suspect the “real risk” from those other two sources is greater than ~1 in 1000. That said, I don’t really have an argument for those suspicions, and I’ve spent especially little time thinking about existential risk from other environmental damage.
I also intuitively feel like the risks from ‘naturally’ arising pandemics is larger than that from other environmental damage, or at least not 10 times lower. (And I don’t think that’s just due to COVID-19; I think I would’ve said the same thing months ago.)
One the other hand, Ord gives a strong argument that the per-century extinction risks from “natural” causes must be very low, based in part on our long history of surviving such risks. So I mostly dismiss my intuitive feeling here as quite unfounded; I suspect my intuitions can’t really distinguish “Natural pandemics are a big deal!” from “Natural pandemics could lead to extinction, unrecoverable collapse, or unrecoverable dystopia!”
On the third hand, Ord notes that that argument doesn’t apply neatly to ‘naturally’ arising pandemics. This is because changes in society and technology have substantially changed the ability for pandemics to arise and spread (e.g., there’s now frequent air travel, although also far better medical science). In fact, Ord doesn’t even classify ‘naturally’ arising pandemics as a natural risk, and he places them in the “Future risks” chapter. Additionally, as Ord also notes, that argument applies most neatly to risks of extinction, not to risks of “unrecoverable collapse” or “unrecoverable dystopia”.
So I do endorse some small portion of my feeling that the risk from ‘naturally’ arising pandemics is probably more than a 10th as big as the risk from other environmental damage.
“Unforeseen” and “other” anthropogenic risks: Surprisingly risky?
By “other anthropogenic risks”, Ord means risks from
- dystopian scenarios
- nanotechnology
- “back contamination” from microbes from planets we explore
- aliens
- “our most radical scientific experiments”
Ord estimates the chances that “other anthropogenic risks” or “unforeseen anthropogenic risks” will cause existential catastrophe in the next 100 years are ~1 in 50 and ~1 in 30, respectively. Thus, he views these categories of risks as, respectively, ~20 and ~33 times as existentially risky (over this period) as are each of nuclear war and climate change. He also views them as in the same ballpark as engineered pandemics. And as there are only 5 risks in the “other” category, this means he must see at least some of them (perhaps dystopian scenarios and nanotechnology?) as posing much higher existential risks than do nuclear war or climate change.
I was surprised by how high his estimates for risks from the “other” and “unforeseen” anthropogenic risks were, relative to his other estimates. But I hadn’t previously thought about these issues very much, so I wasn’t necessarily surprised by the estimates themselves, and I don’t feel myself inclined towards higher or lower estimates. I think my strongest opinion about these sources of risk is that dystopian scenarios probably deserve more attention than the longtermist community typically seems to give them, and on that point it appears Ord may agree.
Should this update our career and donation decisions?
One (obviously imperfect) metric of the current priorities of longtermists is the problems 80,000 Hours recommends people work on. Their seven recommended problems are:
- Positively shaping the development of artificial intelligence
- Reducing global catastrophic biological risks
- Nuclear security
- Climate change (extreme risks)
- Global priorities research
- Building effective altruism
- Improving institutional decision-making
The last three of those recommendations seem to me like they’d be among the best ways of addressing “other” and “unforeseen” anthropogenic risks. This is partly because those three activities seem like they’d broadly improve our ability to identify, handle, and/or “rule out” a wide range of potential risks. (Another top contender for achieving such goals would seem to be “existential risk strategy”, which overlaps substantially with global priorities research and with building EA, but is more directly focused on this particular cause area.)
But as noted above, if Ord’s estimates are in the right ballpark, then:
- “other” and “unforeseen” anthropogenic risks are each (as categories) substantially existentially riskier than each of nuclear war, climate change, or ‘naturally’ arising pandemics
- at least some individual “other” risks must also be substantially higher than those three things
- “other environmental damage” is similarly existentially risky as nuclear war and climate change, and 10 times more so than ‘naturally’ arising pandemics
So, if Ord’s estimates are in the right ballpark, then:
- Perhaps 80,000 Hours should write problem profiles on one or more of those specific “other” risks? And perhaps also about “other environmental damage”?
- Perhaps 80,000 Hours should more heavily emphasise the three “broad” approaches they recommend (global priorities research, building EA, and improving institutional decision-making), especially relative to work on nuclear security and climate change?
- Perhaps 80,000 Hours should write an additional “broad” problem profile on existential risk strategy specifically?
- Perhaps individual EAs should shift their career and donation priorities somewhat towards:
- those broad approaches?
- specific “other anthropogenic risks” (e.g., dystopian scenarios)?
- “other environmental damage”?
Of course, Ord’s estimates relate mainly to scale/impact, and not to tractability, neglectedness, or number of job or donation opportunities currently available. So even if we decided to fully believe his estimates, their implications for career and donation decisions may not be immediately obvious. But it seems like the above questions would be worth considering, at least.
From memory, I don’t think Ord explicitly addresses these sorts of questions, perhaps because he was writing partly for a broad audience who would neither know nor care about the current priorities of EAs. Somewhat relevantly, though, his “recommendations for policy and research” (Appendix F) include items specifically related to nuclear war, climate change, environmental damage, and “broad” approaches (e.g., horizon-scanning for risks), but none specifically related to any of the “other anthropogenic risks”.
As stated above, I thought this book was excellent, and I’d highly recommend it. I’d also be excited to see more people commenting on Ord’s estimates (either here or in separate posts), and/or providing their own estimates. I do see potential downsides in making or publicising such estimates. But overall, it seems to me probably not ideal how many strategic decisions longtermists have made so far without having first collected and critiqued a wide array of such estimates.
This is one of a series of posts I plan to write that summarise, comment on, or take inspiration from parts of The Precipice. You can find a list of all such posts here.
This post is related to my work with Convergence Analysis, but the views I expressed in it are my own. I’m grateful to David Kristoffersson for helpful comments on an earlier draft.
Note: I haven't read the book. Also, based on your other writing, MichaelA, I suspect much of what I write here won't be helpful to you, but it might be for other readers less familiar with Bayesian reasoning or order of magnitude calculations.
On uncertainty about Bayesian estimates of probabilities (credences), I think the following statement could be rewritten in a way that's a bit clearer about the nature of these estimates:
But these are Ord's beliefs, so when he says they could be a factor of 3 higher or lower, I think he means that he think there's a good chance that he could be convinced that they're that much higher or lower, with new information, and since he says they "should be treated as representing the right order of magnitude", he doesn't think he could be convinced that they should be more than 3x higher or lower.
I don't think it's meaningful to say that a belief "X will happen with probability p" is accurate or not. We could test a set of beliefs and probabilities for calibration, but there are too few events here (many of which are extremely unlikely according to his views and are too far in the future) to test his calibration on them. So it's basically meaningless to say whether or not he's accurate about these. We could test his calibration on a different set of events and hope his calibration generalizes to these ones. We could test on multiple sets of events and see how his calibration changes between them to get an idea of the generalization error before we try to generalize.
On the claim, it seems like for many of his estimates, he rounded to the nearest 1 in 10k, on a logarithmic scale, since the halfway point between 10k and 10k+1 on a log scale is 10k+1/2≃3.16×10k, so he can only be off by a factor of √10∼3.16∼3. If he were off by more than a factor of 3, then he would have had to round to a different power of 10. The claim that they represent the right orders of magnitude is equivalent to them being correct to within a factor of about 3. (Or that he thinks he's unlikely to change his mind about the order of magnitude with new information is equivalent to him believing that new information is unlikely to change his mind about these estimates by more than a factor of 3.)
I'd be curious to know if there are others who have worked as hard on estimating any of these probabilities and how close their estimates are to his.
I definitely share this curiosity. In a footnote, I link to this 2008 "informal survey" that's the closest thing I'm aware of (in the sense of being somewhat comprehensive). It's a little hard to compare the estimate, as that was for extinction (or sub-extinction events) rather than existential catastrophe more generally, and was for before 2100 rather than before 2120. But it seems to be overall somewhat more pessimistic than Ord, though in roughly the same ballpark for "overall/total risk", AI, and engineered pandemics at least.
I don't off the top of my head know anything comparable in terms of amount of effort, except in the case of individual AI researchers estimating the risks from AI, or specific types of AI catastrophe - nothing broader. Or maybe a couple 80k problem profiles. And I haven't seen these collected anywhere - I think it could be cool if someone did that (and made sure the collection prominently warned against anchoring etc.).
A related and interesting question would be "If we do find past or future estimates based on as much hard work, and find that they're similar to Ord's, what do we make of this observation?" It could be taken as strengthening the case for those estimates being "about right". But it could also be evidence of anchoring or information cascades. We'd want to know how independent the estimates were. (It's worth noting that the 2008 survey was from FHI, where Ord works.)
Update: I'm now creating this sort of a collection of estimates, partly inspired by this comment thread (so thanks, MichaelStJules!). I'm not yet sure if I'll publish them; I think collecting a diversity of views together will reduce rather than exacerbate information cascades and such, but I'm not sure. I'm also not sure when I'd publish, if I do publish.
But I think the answers are "probably" and "within a few weeks".
If anyone happens to know of something like this that already exists, and/or has thoughts on whether publishing something like this would be valuable or detrimental, please let me know :)
Update #2: This turned into a database of existential estimates, and a post with some broader discussion of the idea of making, using, and collecting such estimates. And it's now posted.
So thanks for (probably accidentally) prompting this!
Thanks for the comment!
Yes, it does seem worth pointing out that these are Bayesian rather than "frequency"/"physical" probabilities. (Though Ord uses them as somewhat connected to frequency probabilities, as he also discusses how long we should expect humanity to last given various probabilities of x-catastrophe per century.)
To be clear, though, that's what I had in mind when suggesting that being uncertain only within a particular order of magnitude was surprising to me. E.g., I agree with the following statement:
...but I was surprised to hear that, if Ord does mean that the way it sounds to me, he thinks he could only be convinced to raise or lower his credence by a factor of ~3.
Though it's possible he instead meant that they could definitely be off by a factor of 3, which that wouldn't surprise him at all, but it's also plausible they could be off by even more.
I think there's something to this, but I'm not sure I totally agree. Or at least it might depend on what you mean by "accurate". I'm not an expert here, but Wikipedia says:
I think a project like Ord's is probably most useful if it's at least striving for objectivist Bayesian probabilities. (I think "the epistemic interpretation" is also relevant.) And if it's doing so, I think the probabilities can be meaningfully critiqued as more or less reasonable or useful.
I agree that this is at least roughly correct, given that he's presenting each credence/probability as "1 in [some power of 10]". I didn't mean to imply that I was questioning two substantively different claims of his; more just to point out that he reiterates a similar point, weakly suggesting he really does mean that this is roughly the range of uncertainty he considers these probabilities to have.
I'm also not an expert here, but I think we'd have to agree about how to interpret knowledge and build the model, and have the same priors to guarantee this kind of agreement. See some discussion here. The link you sent about probability interpretations also links to the reference class problem.
I think we can critique probabilities based on how they were estimated, at least, and I think some probabilities we can be pretty confident in because they come from repeated random-ish trials or we otherwise have reliable precedent to base them on (e.g. good reference classes, and the estimates don't vary too much between the best reference classes). If there's only really one reasonable model, and all of the probabilities are pretty precise in it (based on precedent), then the final probability should be pretty precise, too.
Just found a quote from the book which I should've mentioned earlier (perhaps this should've also been a footnote in this post):
And I'm pretty sure there was another quote somewhere about the complexities with this.
As for your comment, I'm not sure if we're just using language slightly differently or actually have different views. But I think we do have different views on this point:
I would say that, even if one model is the most (or only) reasonable one we're aware of, if we're not certain about the model, we should account for model uncertainty (or uncertainty about the argument). So (I think) even if we don't have specific reasons for other precise probabilities, or for decreasing the precision, we should still make our probabilities less precise, because there could be "unknown unknowns", or mistakes in our reasoning process, or whatever.
If we know that our model might be wrong, and we don't account for that when thinking about how certain vs uncertain we are, then we're not using all the evidence and information we have. Thus, we wouldn't be striving for that "evidential" sense of probability as well as we could. And more importantly, it seems likely we'd predictably do worse in making plans and achieving our goals.
Interestingly, Ord is among the main people I've seen making the sort of argument I make in the prior paragraph, both in this book and in two prior papers (one of which I've only read the abstract of). This increased my degree of surprise at him appearing to suggest he was fairly confident these estimates were of the right order of magnitude.
I agree that we should consider model uncertainty, including the possibility of unknown unknowns.
I think it's rare that you can show that only one model is reasonable in practice, because the world is so complex. Mostly only really well-defined problems with known parts and finitely many known unknowns, like certain games, (biased) coin flipping, etc..