Comments
Thresholds for funding existential risk interventions
This is some very old work I never published in the EA forum; publishing it now because I want to add it to my Estimating value series, mark the end of an era & reference it elsewhere. I don't think I could produce this work today since I've lost the youthful innocence to think it'd be worth the effort given the idosyncrasies of funding institutions. Thanks to SoGive for commissioning this piece a few years ago; you can see the original here.
The following are strategies we might consider using to determine a reasonable cost-effectiveness threshold for philanthropy to reduce existential risk.
| Strategy | Feasible? | Recommended? |
|---|---|---|
| 1. Randomized trials | Impossible | No |
| 2. Shapley values | Even more impossible | No |
| 3. Natural experiments | Unlikely | No |
| 4. Direct reduction in x-risk | With effort | Maybe |
| 5. Relative value comparison | Yes | Maybe |
| 6. Estimate intermediate outputs | Yes | Yes |
| 7. Sanity checks or thresholds | Yes | Yes |
| 8. Other funders as a benchmark | Yes | Not really |
| 9. Expert intuition | Yes | In moderation |
| 10. Impact rubric | Yes | No |
Randomized trials are the gold standard in medicine and global health and development. However, they do have the problem of double-counting or triple-counting impact. Shapley values address that limitation.
However, randomized trials are impossible1 for the case of existential risk2; we can't boot up different possible worlds and try different existential risk reduction strategies in each. Shapley values are even more impossible, because we'd have to boot up many different counterfactual worlds.
Still, randomized trials and Shapley values could represent an aspiration: they are what other methods will seek to approximate. In particular, to very roughly approximate Shapley values, we can remember to adjust estimates of impact downwards in proportion to the number of people needed, or the number of steps in the pathway to impact.
One first step away from the ideal randomized trial would be to look at proxies and intermediate outputs, rather than at whether a catastrophe was averted. For example, instead of running an impossible randomized trial on whether existential risk is averted, one could run a less impossible randomized trial looking at an intermediate outcome, like whether laws are passed that seek to reduce existential risk. There is some sense in which that is less impossible, because one could look at lobbying across different times and places. But in practice, this method doesn't work either, because the vast majority of existential risk interventions have a small sample size. Still, for some interventions, like fellowships, one could try to aggregate multiple cohorts.
What's more, not many existential risk organizations have carried out randomization to estimate their impact. Still, we could hope for natural experiments. For instance, for a given fellowship, we could look at fellows right above and right below their bar for acceptance, and see if those who were barely accepted did better or worse than those who didn't.
Directly estimating how much catastrophic risk or existential risk a given intervention avoids is probably going to be too complicated. Still, one could look at the pathway to impact of the intervention, estimate their chances of success at each step, and estimate the magnitude of success if they do succeed.
In doing so, you will end up leaving the confines of scientific practice and enter the region of subjective Bayesian estimation, where you are quantifying your subjective beliefs of impact, but you might not convince others that your chosen quantification makes sense. Push even further, and you yourself might not trust the estimates you have produced.
Still, the quantification effort could be interesting, and you could learn something along the way. And direct existential risk estimation seems enticing: even though it will be very speculative and uncertain, it would provide a straightforward point of comparison between different interventions.
We could also ask several people about how much they value different interventions relative to each other, as in here or here. From these estimates, we can construct a linear scale and then place the different interventions in it to represent how much we value them relative to each other. This will capture participants' intuitions of value. But is time-intensive, and it can be opaque without even more effort.
The academic literature has explored similar methods, under the keyword "discrete choice theory". However, they typically ask about binary choices, not about estimates of distributional relative value. Their objective is sometimes to estimate the value of features, e.g., of a renovated bathroom or bigger windows in the market for houses in a given city.
Beyond direct existential risk estimation and relative values, we could make estimates of intermediate outputs, perhaps using midway units, like quality-adjusted:
Each of these is a proxy for the ultimate impact -- and we'd hope that these proxies scale roughly linearly, or that if some organization has better proxies than another organization, they will be better overall.
But when comparing what quantity of one unit is better than what quantity of another unit, we are back to needing some sort of relative value or direct existential risk comparison.
We could remember that ultimately, we want to make some decision -- for example, whether to fund a project -- and we could directly look for sanity checks, or thresholds which would make our decision clear:
Particularly for the later two questions, it would help to have a threshold above which some existential risk intervention should definitely be funded, and a threshold below which it should definitely not be funded.
Although the above methods are sometimes used, it's more common to use lesser methods. There is a temptation to not do quantified estimates, and instead settle for impact rubrics. By this I mean aggregating some observable metrics around the world using some sort of checklist or Likert scale. Outside of longtermism, Animal Charity Evaluators and Charity Entrepreneurship have used this. The problem with this approach is that values aren't linear: an intervention that scores twice as high on the checklist isn't therefore twice as valuable. And it might even be worse, if the checklist doesn't weigh more valuable factors comparatively heavily.
More often, grant evaluators simply rely on expert intuition, which relies on an implicit bar for funding. This could work if experts have good judgment. In practice, there isn't enough estimation infrastructure that one can move away from relying on the judgment of individuals soon.
Finally, there is the temptation of using other funders as a benchmark, for instance the Long-Term Future Fund. This makes sense in that, if you can't do better than a fund, it makes sense to donate to it. But on the other hand, other funders may have idiosyncratic constraints or different beliefs, and taking other funders as a threshold could lead to game of chicken dynamics3 or recursion problems4. In the case of the LTFF in particular, they tend to spend around an hour per grant, so it's not clear that it's worth deferring to them.
There are various ways you can put a dollar value to existential risk prevention:
For deciding whether to donate to more well-trodden global health and development interventions or to more speculative existential risk interventions, the relevant question to be asking is #2.
However, when deciding to which existential risk intervention you should donate, the relevant question is #3.
I suspect that there has been substantial confusion on this topic, leading to inflated valuations of longtermist projects. As an analogy, I at some point estimated that good headphones provided me with 6K to 100K of value, in terms of better mental health and increased productivity (this is high, but they are absurdly effective for me in both cases). If I had no other option, I'm pretty sure I would pay $6k for good headphones. However, I do have other options, because you can get some excellent headphones for less than $500. Similarly, if you "value" an expected future life at $5k, the value of speculative existential risk interventions would seem to be very high. But that is not the correct calculation to be making for willingness to pay. For willingness to pay, you want to be spending your whole budget paying as low a fraction of a penny per future life as possible5.
Having settled on "willingness to pay" as the relevant question in our case, we proceed to estimate an upper and lower bound thereof from first principles. See section 3.3 for links to the models discussed here and their outputs.
3.1.1. A minimum willingness to pay
Is there some amount such that one should be willing to pay at least that amount to avert a given amount of existential risk? I say yes. We can build such a minimum willingness to pay bound by noticing that it would be a good outcome if the existential risk community spent all its money6 to bring existential risk7 down to zero. If we do so, we arrive at a threshold of ~$5.4M (1.49M to 13.2M) per basis point (0.01%) of existential risk reduced.
This threshold should be understood as an ambitious amount: if you can find an opportunity that buys 1 basis point for ~$5.4M, my guess is that that funders seeking to reduce existential risk should be eager to take it.
The two basic building blocks of the model are estimates of total funding and total risk. As the capital seeking to reduce existential risk rises or the amount of risk decreases, it can afford to relax its cost-effectiveness while still meeting its raison d'être. As one increases capital, at some point one would reach a point where one would want to solve existential risk and then have extra funding left over. Then this threshold would no longer be valid.
3.1.2. A maximum willingness to pay
Humanity's ability to coordinate to avert catastrophes is bounded. Its ability to coordinate would be highest for an unambiguous, verifiable threat, like an asteroid heading directly towards Earth, or an unmistakable alien invasion which declared war beforehand. Maybe in those scenarios, humanity would be able to mobilize the whole world, for instance under a greatly expanded United Nations for multiple decades. In that sense, we could say that humanity's willingness to pay to avoid existential could be some multiple of the world's current GDP ($~100T/year). For example, we could say that humanity's willingness to pay to avoid existential was $1,000T if humanity was able to coordinate for ten years to spend all of its manpower deflecting an asteroid.
In practice, humanity would not be able to bring that level of coordination to threats like powerful and misaligned AI, pandemics, or nuclear war. In that sense, the maximum amount that humanity could spend on those types of threats is much lower than many multiples of the world's GDP.
So an upper bound of one's willingness to pay is given by the largest check that one could conceivably cash. Consider a "mega Manhattan project" that mobilized 10 to 100 million people, paid each wages of $20k to $200k for 5 to 50 years, and was modeled to bring risk down from 100% to 0% risk. This seems like the best humanity could do in the worst scenario. This would correspond to a maximum willingness to pay for a basis point of ~$3B (400M to 23B).
However, we've arrived at that amount considering civilizational willingness to pay. But it's unclear if that can directly translate to willingness to pay for a much smaller donor. Strictly speaking, that upper bound on civilizational willingness to pay doesn't logically imply an upper bound for much smaller donors. For example, it could be that a much smaller community is able to coordinate more tightly and make different trade-offs, for instance because it has much lower discount rates and so values the future much more highly.
However, while that could be possible, in practice getting less than a basis point of reduction for 400M to 23B means that the solutions wouldn't be able to scale to tackle the existential risk being addressed even if the whole of humanity put its efforts into it. This seems like a loud warning sign of gross inefficiency, and hence the upper threshold stands. More generally, I just expect reasonable existential risk interventions to exceed this threshold, and so I think it is useful to quickly discard potential interventions.
The previous thresholds were based on abstract reasoning. We can complement that abstract reasoning with comparisons with existing projects. In particular, if we find some existing, scalable project whose returns don't diminish or diminish slowly with scale, then we can check that a proposed project meets that bar.
This might be akin to how GiveWell compares proposed top interventions against GiveDirectly. However, the situation is not analogous, since existential risk interventions are by and large much more speculative than GiveDirectly. And so we may not want to endorse potential points of comparison as much as GiveWell endorses GiveDirectly, and not imply that returns diminish analogously slowly as for GiveDirectly.
Initially, I was keen on using the LTFF as a reference point.
However, in practice I found out that this wasn't a great idea:
In addition, I hold some idiosyncratic beliefs which reinforce this decision:
Consider funding a few talented PhD students to do technical AI safety research, for instance as in MATS. This intervention seems reasonably scalable and robust8, so we might want to use it as a threshold as well. That is, if a proposed intervention isn't as good as technical AI safety research, we might as well fund technical AI safety research instead.
One way to model the value of such research is by thinking about what the positive value of the AI safety community has been, estimate the total size of that community, and then estimate the value of new entrants as a proportion. Here, a problem is that maybe the AI safety community has had negative value. However, because AI technical safety research is relatively harmless, I think it makes sense to consider the positive component.
With this in mind, if we take:
Then we arrive at a cost per basis point of between $1.8M and $1.25B.
(Editor's note: SoGive has uncertainties here. We suspect:
Now, I like this model because thinking in terms of the impact of the whole AI safety community, their total impact will be large enough that I can intuitively grasp it. In contrast, if I were directly modeling the impact of one individual researcher, I would have to multiply the chance that they are working in an agenda that has some chance of being conceptually correct or adequate, and then some chance of counterfactually avoiding existential or catastrophic risk. That later number would be pretty small and difficult to conceptualize. I prefer the top down in this case.
We can then take MATS11 in particular as proof that an intervention that fits the above desiderata exists—though there will be more of such interventions, e.g., restricted funding to the LTFF for technical AI safety, another new AI safety fund, etc. MATS in particular, due to its selection effects, technical focus, strong mentorship program, etc., is probably averting existential risk at a cost-effectiveness point equal or higher than the cost-effectiveness of the AI safety community on average. This is important because we might think that most of the impact of the AI safety community comes from highly selected outliers, and so it's important that MATS does seem to contain outlier technical talent as well.
One problem with using the AI safety community as a threshold, though, is that although I model it as so above, it's not clear that its value is positive. For instance, through its existence it could bring to the attention of amoral actors that AI is likely to be powerful, and those actors might seek to invest in AGI development, maybe because at some level they value their private benefit over the collective existential externality. Consider that the rationality community was involved in the creation of each of DeepMind, OpenAI and Anthropic, and that their successes have attracted more amoral mimics, like Mistral, Inflection AI or Stable Diffusion.
To address the possibility of that downside, one could:
A final difficulty is that this threshold changes with the amount of funding available. For instance, another billionaire could come in and fully fund all plausible AI technical safety research. If so, this threshold would have to be updated or discarded.
| Threshold | Meaning | Decision criteria |
|---|---|---|
| Minimum willingness to pay threshold - Output | Ambitious threshold, beating this is a very good sign. | If an intervention beats this, fund enthusiastically. |
| Robust AI technical safety work threshold - Output | If you don't meet this threshold, there is a reasonably scalable intervention you could donate to that will have better results. | If an intervention beats this, fund. This depends on donor preferences, and could be tweaked. |
| Maximum willingness to pay threshold - Output | Very weak threshold, not beating this is a sign you might be Pascal mugged or grossly inefficient. | If an intervention doesn't beat this, definitely don't fund. Useful to discard interventions early on in research. |
| Threshold | Mean value in M$/bp | Median value in M$/bp | Std in M$/bp | Mean value in bp/M$ | Median value in bp/M$ | Std in bp/M$ |
|---|---|---|---|---|---|---|
| Minimum willingness to pay threshold | 5.41 | 4.21 | 4.46 | 0.288 | 0.237 | 0.201 |
| Robust AI technical safety work threshold | 9.38e4 | 26.7 | 7.04e7 | 0.1358 | 0.0374 | 0.382 |
| Maximum willingness to pay threshold | 6.59e3 | 3.16e3 | 1.21e4 | 6.59e-4 | 3.17e-4 | 1.2e-3 |
(Note that in general, the mean of an inverse distribution is not the inverse of the mean).
In a recent revision, I've tried to give some more specifics about how I'm arriving at my estimates. In the interest of replicability, here are some notes of possible approaches to take next time SoGive is trying to do some of these estimates:
