Interests:

4

You can give me feedback __here__ (anonymous or not). You are welcome to answer any of the following:

- Do you have any thoughts on the value (or lack thereof) of my posts?
- Do you have any ideas for posts you think I would like to write?
- Are there any opportunities you think would be a good fit for me which are either not listed on 80,000 Hours' job board, or are listed there, but you guess I might be underrating them?

Feel free to check my posts, and see if we can collaborate to contribute to a better world. I am open to part-time volunteering and paid work. In this case, I typically ask for 20 $/h, which is roughly equal to 2 times the global real GDP per capita.

121

1415

25

Hi David,

Existential catastrophe, annual | 0.30% | 20.04% | David Denkenberger, 2018 |

Existential catastrophe, annual | 0.10% | 3.85% | Anders Sandberg, 2018 |

Based on my adjustments to CEARCH's analysis of nuclear and volcanic winter, the expected annual mortality of nuclear winter as a fraction of the global population is 7.32*10^-6. I estimated the deaths from the climatic effects would be 1.16 times as large as the ones from direct effects. In this case, the expected annual mortality of nuclear war as a fraction of the global population would be 1.86 (= 1 + 1/1.16) times the expected annual mortality of nuclear winter as a fraction of the global population, i.e. 0.00136 %(= 1.86*7.32*10^-6). So the annual losses in future potential mentioned in the table above are 221 (= 0.0030/(1.36*10^-5)) and 73.5 (= 0.0010/(1.36*10^-5)) times my expected annual death toll, whereas I would have expected the annual loss in future potential to be much lower than the expected annual death toll.

Great points, Matt.

I think essentially all (not just many) pathways from AI risk will have to flow through other more concrete pathways. AI is a general purpose technology, so I feel like directly comparing AI risk with other lower level pathways of risk, as 80 k seems to be doing somewhat when they describe the scale of their problems, is a little confusing. To be fair, 80 k tries to account for this talking about the indirect risk of specific risks, which they often set to 10 times the direct risk, but these adjustments seem very arbitrary to me.

In general, one can get higher risk estimates by describing risk at a higher level. So the existential risk from LLMs is smaller than the risk from AI, which is smaller than the risk from computers, which is smaller than the risk from e.g. subatomic particles. However, this should only update one towards e.g. prioritise "computer risk" over "LLM risk" to the extent the ratio between the cost-effectiveness of "computer risk interventions" and "LLM risk interventions" is proportional to the ratio between the scale of "computer risk" and "LLM risk", which is quite unclear given the ambiguity and vagueness of the 4 terms involved^{[1]}.

To get more clarity, I believe it is be better to prioritise at a lower level, assessing the cost-effectiveness of specific classes of interventions, as Ambitious Impact (AIM), Animal Charity Evaluators (ACE), the Centre for Exploratory Altruism Research (CEARCH), and GiveWell do.

^{^}"Computer risk", "LLM risk", "computer risk interventions" and "LLM risk interventions".

Here is an example with text in a table aligned to the left (select all text -> cell properties -> table cell text alignement).

Statistic | Annual epidemic/pandemic deaths as a fraction of the global population |

Mean | 0.236 % |

Minimum | 0 |

5th percentile | 1.19*10^-6 |

10th percentile | 3.60*10^-6 |

Median | 0.0276 % |

90th percentile | 0.414 % |

95th percentile | 0.684 % |

Maximum | 10.3 % |

Thanks for the post! I wonder whether it would also be good to have public versions of the applications (sensible information could be redacted), as Manifund does, which would be even less costly than having external reviewers.

Hi Mike.

Los Alamos find that firestorms are highly unlikely to form under nuclear detonations, even at very high fuel loads, and so lofting is negligible. They only look at fission scale weaponry.

I think this may well misrepresent Los Alamos' view, as __Reisner 2019____ does not find significantly more lofting, and they did model firestorms__. I estimated 6.21 % of emitted soot being injected into the stratosphere in the 1st 40 min from the rubble case of __Reisner 2018__, which did not produce a firestorm. __Robock 2019__ criticised this study, as you did, __for not producing a firestorm. In response__, __Reisner 2019__ run:

Two simulations at higher fuel loading that are in the firestorm regime (Glasstone & Dolan, 1977): the first simulation (4X No-Rubble) uses a fuel load around the firestorm criterion (4 g/cm2) and the second simulation (Constant Fuel) is well above the limit (72 g/cm2).

Crucially, they say (emphasis mine):

Of note is that the

Constant Fuel case is clearly in the firestorm regimewith strong inward and upward motions of nearly 180 m/s during the fine-fuel burning phase. This simulation included no rubble, and since no greenery (trees do not produce rubble) is present, the inclusion of a rubble zone would significantly reduce BC production and the overall atmospheric response within the circular ring of fire.

These simulations led to a soot injected into the stratosphere in the 1st 40 min per emitted soot of 5.45 % (= 0.461/8.454) and 6.44 % (= 1.53/23.77), which are quite similar to the 6.21 % of __Reisner 2018__ for no firestorm I mentioned above. This suggests a firestorm is not a sufficient condition for a high soot injected into the stratosphere per emitted soot under Reisner's view?

In my analysis, I multiplied the 6.21 % emitted soot that is injected into the stratosphere in the 1st 40 min from __Reisner 2018__ by 3.39 in order to account for soot injected afterwards, but this factor is based on estimates __which do not involve firestorms__. Are you implying the corrective factor should be higher for firestorms? I think __Reisner 2019____ implicitly argues against this. Otherwise, they would have been dishonest by replying to ____Robock 2019__ with an incomplete simulation whose results differ from that of the full simulation. In my analysis, I only adjusted the results from Reisner’s and Toon’s views in case there was explicit information to do so^{[1]}, i.e. I did not assume they concealed key results.

As a result, blending together the Los Alamos model with that of Rutgers doesn’t really work as a baseline, they’re based on a very different binary concerning firestorms and lofting and you exclude other relevant analysis, like that of Lawrence Livermore.

In my analysis, I also did not integrate evidence from __Wagman 2020__ (whose main author is affiliated with Lawrence Livermore National Laboratory) to estimate the soot injected into the stratosphere per countervalue yield. As far as I can tell, they do not offer independent evidence from Toon's view. Rather than estimating the emitted soot as __Reisner 2018__ and __Reisner 2019__ did, they set it to the soot injected into the stratosphere in __Toon 2007__:

Finally, we choose to release 5 Tg (5·10^12 g) BC into the climate model per 100 fires, for consistency with the studies of Mills et al. (2008, 2014), Robock et al. (2007), Stenke et al. (2013), Toon et al. (2007), and Pausata et al. (2016). Those studies use an emission of 6.25 Tg BC and assume 20% is removed by rainout during the plume rise, resulting in 5 Tg BC remaining in the atmosphere.

^{^}For example, I adjusted downwards the soot injected into the stratosphere from Reisner 2019 (based on data from Denkenberger 2018), as it says (emphasis mine):

**Table 1.**Estimated BC Using an Idealized Diagnostic Relationship (**BC Estimates Need to be Reduced by a Factor of 10–100**) and Fuel Loadings From the Simulations Shown in Reisner et al. and Two New Simulations for 100 15-kt Detonations

Great points, Stan!

Yes, I agree that the crux is

whether firestorms will form.

I am not confident this is the crux.

The main thing I would like people to take away is that we remain uncertain what would be more damaging about a nuclear conflict: the direct destruction, or its climate-cooling effects.

I arrived at the same conclusion in my analysis, where I estimated the famine deaths due to the climatic effects of a large nuclear war would be 1.16 times the direct deaths.

Thanks for the relevant points, Joshua. I strongly upvoted your comment.

Could you please expand on why you think a Pareto distribution is appropriate here?

I did not mean to suggest a Pareto distribution is appropriate, just that it is worth considering.

Tail probabilities are often quite sensitive to the assumptions here, and it can be tricky to determine if something is truly power-law distributed.

Agreed. In my analysis of conflict deaths, for the method where I used fitter:

The 5th and 95th percentile annual probability of a conflict causing human extinction are 0 and 5.02 % [depending on the distribution]

When I looked at the same dataset, albeit processing the data quite differently, I found that a truncated or cutoff power-law appeared to be a good fit. This gives a much lower value for extreme probabilities using the best-fit parameters. In particular, there were too few of the most severe pandemics in the dataset (COVID-19 and 1918 influenza) otherwise; this issue is visible in fig 1 of Marani et al. Could you please add the data to your tail distribution plot to assess how good a fit it is?

I did not get what you would like me to add to my tail distribution plot. However, I added here the coefficients of determination (R^2) of the regressions I did.

A final note, I think you're calculating the probability of extinction

in a single yearbut the worst pandemics historically have lasted multiple years. The total death toll from the pandemic is perhaps the quantity most of interest.

Focussing on the annual deaths as a fraction of the global population is useful because it being 1 is equivalent to human extinction. In contrast, total epidemic/pandemic deaths as a fraction of the global population in the year in which the epidemic/pandemic started being equal to 1 does not imply human extinction. For example, a pandemic could kill 1 % of the population each year for 100 years, but population remain constant due to births being equal to the pandemic deaths plus other deaths.

However, I agree interventions should be assessed based on standard cost-effectiveness analyses. So I believe the quantity of most interest which could be inferred from my analysis is the expected annual epidemic/pandemic deaths. These would be 2.28 M (= 2.87*10^-4*7.95*10^9) multiplying:

- My annual epidemic/pandemic deaths as a fraction of the global population based on data from 1900 to 2023. Earlier years are arguably not that informative.
- The population in 2021.

The above expected death toll would rank as 6th in 2021.

For reference, based on my analysis of conflicts, I get an expected death toll of conflicts based on historical data from 1900 to 2000 (also adjusted for underreporting), and the population in 2021 of 3.83 M (= 2.87*10^-4*7.95*10^9), which would rank above as 5th.

Here is a graph with the top 10 actual causes of death and expected conflict and epidemic/pandemic deaths:

Thanks for the update, Toby. I used to defer to you a lot. I no longer do. After investigating the risks myself in decent depth, I consistently arrived to estimates of the risk of human extinction orders of magnitude lower than your existential risk estimates. For example, I understand you assumed in The Precipice an annual existential risk for:

In addition, I think the existential risk linked to the above is lower than their extinction risk. The worst nuclear winter of Xia et. al 2022 involves an injection of soot into the stratosphere of 150 Tg, which is just 1 % of the 15 Pg of the Cretaceous–Paleogene extinction event. Moreover, I think this would only be existential with a chance of 0.0513 % (= e^(-10^9/(132*10^6))), assuming:

exponential distributionwith a mean of 132 M years (= 66*10^6*2) represents the time between i) human extinction in such catastrophe and ii) the evolution of an intelligent sentient species after such a catastrophe. I supposed this on the basis that:exponential distributionwith a mean of 66 M years describes the time between:becomeshabitable.