This is the second part of "Mistakes in the moral mathematics of existential risk", a series of blog posts by David Thorstad that aims to identify ways in which estimates of the value of reducing existential risk have been inflated. I've made this linkpost part of a sequence.

In the decades to come, advanced bioweapons could threaten human existence. Although the probability of human extinction from bioweapons may be low, the expected value of reducing the risk could still be large, since such risks jeopardize the existence of all future generations. We provide an overview of biotechnological extinction risk, make some rough initial estimates for how severe the risks might be, and compare the cost-effectiveness of reducing these extinction-level risks with existing biosecurity work. We find that reducing human extinction risk can be more cost-effective than reducing smaller-scale risks, even when using conservative estimates. This suggests that the risks are not low enough to ignore and that more ought to be done to prevent the worst-case scenarios.

Millett and Snyder-Beattie, “Existential risk and cost-effective biosecurity”

1. Introduction

This is Part 2 of a series based on my paper “Mistakes in the moral mathematics of existential risk”.

Part 1 introduced the series and discussed the first mistake: focusing on cumulative rather than per-unit risk. We saw how bringing the focus back to per-unit rather than cumulative risk was enough to change a claimed `small’ risk reduction of one millionth of one percent into an astronomically large reduction that would drive risk to almost one in a million per century.

Today, I want to focus on a second mistake: ignoring background risk. The importance of modeling background risk is one way to interpret the main lesson of my paper and blog series “Existential risk pessimism and the time of perils.” Indeed, in Part 7 of that series, I suggested just this interpretation.

It turns out that blogging is sometimes a good way to write a paper. Today, I want to expand my discussion in Part 7 of the existential risk pessimism series to clarify the second mistake (ignoring background risk) and to show how it interacts with the first mistake (focusing on cumulative risk) in a leading discussion of cost-effective biosecurity.

Some elements of this discussion are lifted verbatim from my earlier post. In my defense, I remind my readers that I am lazy.

2. Snyder-Beattie and Millett on cost-effective biosecurity

Andrew Snyder-Beattie holds a DPhil in Zoology from Oxford, and works as a Senior Program Officer at Open Philanthropy. Snyder-Beattie is widely considered to be among the very most influential voices on biosecurity within the effective altruist community.

Piers Millett is a Senior Research Fellow at the Future of Humanity Institute. Millett holds advanced degrees in science policy, research methodology and international security, and has extensive industry experience in biosecurity.

Millett and Snyder-Beattie’s paper, “Existential risk and cost-effective biosecurity”, is among the most-cited papers on biosecurity written by effective altruists. The paper argues that even very small reductions in existential risks in the biosecurity sector (henceforth, `biorisks’) are cost-effective by standard metrics.

Millett and Snyder-Beattie estimate the cost-effectiveness of an intervention as C/(NLR), where:

• C is the cost of the intervention.
• N is “the number of biothreats we expect to occur in 1 century”.
• L is “the number of life-years lost in such an event”.
• R is “the reduction in risk [in this century only] achieved by spending … C”.

Millett and Snyder Beattie estimate these quantities as follows:

• C is fixed at $250 billion.
• N is estimated in min/max ranges using three different approaches.
• L is calculated assuming that humanity remains earthbound, with an annual population of 10 billion people, lasting for a million years, so that L = 10¹⁶ life years.
• R is estimated at a 1% relative reduction, i.e. risk is reduced from N to .99N.

Because N is estimated using three different models, Millett and Snyder-Beattie estimate cost-effectiveness as C/(NLR) on each model, giving:

Standard government cost-effectiveness metrics in the United States value a life-year in the neighborhood of a few hundred thousand dollars, so Millett and Snyder-Beattie conclude that across models, a small intervention (such as a 1% relative risk reduction in this century) is cost-effective even at a high price (such as $250 billion).

3. A complaint to set aside

There are many complaints that could be made about this model. One such complaint was nicely formalized by Joshua Blake in a comment on my earlier presentation of the MSB estimate.

MSB want to estimate the expected cost of saving a life, E[C/NLR]. By separately estimating each of C, N, L, and R, they have in effect estimated E[C]/(E[N]E[L]E[R}).

This move is legal if, and only if, C,N,L,R are probabilistically independent. [Edit: Whoops, that’s still not enough to bail them out. See comment by Mart.] However, N and L are highly negatively correlated: the riskier the future is, the fewer humans we should expect to live in it, because humanity becomes more likely to meet an early grave.

I want to set this complaint aside for now. Perhaps I am not really setting this complaint aside, since it will be one way of explaining how the MSB estimate commits the first mistake, a point to which I return below. But for now, I mention this complaint to illustrate first that there can be other mistakes beyond those mentioned in this paper, and second that my readers are quite helpful and good at math.

4. Second mistake: Ignoring background risk

When we intervene on some given risks (in this case, existential biorisk), we leave other risks largely unchanged. Call these unchanged risks background risks.

The lesson of my paper and blog series “Existential risk pessimism and the time of perils” is that background risk matters quite a lot. However, the MSB model does not say anything about background risk. Building background risk into the MSB model will reduce the MSB cost-effectiveness estimates, particularly when background risk is large.

If we assume biological and nonbiological risks cannot occur in the same century, then we can split per-century risk r into its biological component b and non-biological component n as:

r = b + n.

MSB envision an intervention X which provides a 1% relative reduction in biorisk, shifting risk to:

r_X = 0.99b + n.

Prior to intervention, how many life-years did the future hold in expectation? On the MSB model, a century involves a stable population of 10¹⁰ lives, for a total of 10¹² life-years. We make it through this century with probability (1-r), survive the first two centuries with probability (1-r)², and so on, so that the expected number of future lives (over a million years, or ten thousand centuries) is:

$E\left[L\right]={10}^{12}{\sum }_{i=1}^{10,000}(1-r{)}^i.$

Our intervention X increases the expected number of future lives by reducing per-century risk from r to rX, giving a post-intervention expected number of lives of:

$E\left[L|X\right]={10}^{12}{\sum }_{i=1}^{10,000}(1-r_X{)}^i.$

Intervention X adds, in expectation, E[L|X] – E[L] lives, which works out to (see appendix):

$E\left[L|X\right]-E\left[L\right]={10}^{12}\times \frac{0.01b}{r(r-0.01b)}.$

In rough outline, X provides ten billion additional life-years, scaled down by the initial biorisk b, but scaled up (approximately) by 1 / the square of background risk r. If background risk is low, this may be a large boost indeed, but if background risk is high, things become more dire.

Below, I’ve revised the MSB estimates across two levels of background risk: a 20%/century risk close to that favored by many effective altruists, and a more optimistic 1%/century risk.

For comparison, GiveWell estimates that the best short-termist interventions save a life for about $5,000. Hence even if we assume that short-termist interventions do not have positive knock-on effects (an assumption we should not make), a good short-termist benchmark will be on the order of $50-200/life-year.

Already, we see that under high levels of background risk, the MSB estimate at best ties the short-termist benchmark, and at worst falls significantly below it. By contrast, on low levels of background risk, the MSB estimate may fare well.

Does this mean that we can salvage the MSB estimate by being optimistic about background risk? Not so fast.

5. First mistake: Focusing on cumulative risk

MSB, like Bostrom, are concerned with cumulative risk. They treat risk reduction as providing an increased chance that all future people throughout a million-year stretch will come to exist, not just that people in this century will come to exist.

We saw in Part 1 of this series that this is a mistake. Focusing on cumulative risk dramatically overstates the value of existential risk reduction, and also takes us away from the policy-relevant question of how we should intervene on risks in nearby centuries.

Let us replace MSB’s stylized intervention X with an intervention X’ that provides a 1% relative reduction in biorisk, not in all centuries at once, but rather in our own century. That is, X’ reduces risk in this century to:

r_X’ = 0.99b + n

but leaves risk in future centuries at r = b + n.

How many additional lives does X’ provide in expectation. It turns out (see paper for details) that:

$E\left[L|X^{\prime }\right]-E\left[L\right]={10}^{12}\times \frac{0.01b}{r}.$

Whereas before, this expression was divided through (roughly) by r², now it divides through only by r. That will tend to reduce the cost-effectiveness of X’, since r is a number between 0 and 1. Importantly, the penalty is worse for low values of r. The loss of a second r in the denominator shaves a factor of approximately five off this expression for a pessimistic r = 0.2, but a whopping two orders of magnitude off in the case that r = 0.01.

Below, I’ve revised the MSB estimates to incorporate not only background risk, but also risk reduction within a single century rather than across all time.

This is bad news. Only the most optimistic model (Model 1 with r=0.01) makes biosecurity competitive with the short-termist benchmark of $50-200/life-year. Across most other models, biosecurity turns out not only to be less cost-effective than the best short-termist interventions, but often many orders of magnitude less cost-effective.

Again, we see that mistakes in moral mathematics matter. Correcting the first and second mistakes within the MSB model took biosecurity from robustly cost-effective to robustly cost-ineffective in comparison to short-termist benchmarks.

6. Wrapping up

So far, we have met two mistakes in the moral mathematics of existential risk:

• First mistake: Focusing on cumulative risk rather than per-unit risk.
• Second mistake: Ignoring background risk.

We looked at a leading model of cost-effective biosecurity due to Piers Millett and Andrew Snyder-Beattie. We saw that the model commits both mistakes, and that correcting the mistakes leads to a near-complete reversal of Millett and Snyder-Beattie’s conclusion: it makes biosecurity look robustly less cost-effective than leading short-termist interventions, rather than robustly more cost-effective.

In the next post, I introduce a final mistake in the moral mathematics of existential risk: neglecting population dynamics.