Bio

Participation
4

How others can help me

You can give me feedback here (anonymous or not). You are welcome to answer any of the following:

  • Do you have any thoughts on the value (or lack thereof) of my posts?
  • Do you have any ideas for posts you think I would like to write?
  • Are there any opportunities you think would be a good fit for me which are either not listed on 80,000 Hours' job board, or are listed there, but you guess I might be underrating them?

How I can help others

Feel free to check my posts, and see if we can collaborate to contribute to a better world. I am open to part-time volunteering and paid work. In this case, I typically ask for 20 $/h, which is roughly equal to 2 times the global real GDP per capita.

Comments
1415

Topic contributions
25

Thanks for the update, Toby. I used to defer to you a lot. I no longer do. After investigating the risks myself in decent depth, I consistently arrived to estimates of the risk of human extinction orders of magnitude lower than your existential risk estimates. For example, I understand you assumed in The Precipice an annual existential risk for:

  • Nuclear war of around 5*10^-6 (= 0.5*10^-3/100), which is 843 k (= 5*10^-6/(5.93*10^-12)) times mine.
  • Volcanoes of around 5*10^-7 (= 0.5*10^-4/100), which is 14.8 M (= 5*10^-7/(3.38*10^-14)) times mine.

In addition, I think the existential risk linked to the above is lower than their extinction risk. The worst nuclear winter of Xia et. al 2022 involves an injection of soot into the stratosphere of 150 Tg, which is just 1 % of the 15 Pg of the Cretaceous–Paleogene extinction event. Moreover, I think this would only be existential with a chance of 0.0513 % (= e^(-10^9/(132*10^6))), assuming:

  • An exponential distribution with a mean of 132 M years (= 66*10^6*2) represents the time between i) human extinction in such catastrophe and ii) the evolution of an intelligent sentient species after such a catastrophe. I supposed this on the basis that:
    • An exponential distribution with a mean of 66 M years describes the time between:
      • 2 consecutive such catastrophes.
      • i) and ii) if there are no such catastrophes.
    • Given the above, i) and ii) are equally likely. So the probability of an intelligent sentient species evolving after human extinction in such a catastrophe is 50 % (= 1/2).
    • Consequently, one should expect the time between i) and ii) to be 2 times (= 1/0.50) as long as that if there were no such catastrophes.
  • An intelligent sentient species has 1 billion years to evolve before the Earth becomes habitable.

Hi David,

Existential catastrophe, annual0.30%20.04%David Denkenberger, 2018
Existential catastrophe, annual0.10%3.85%Anders Sandberg, 2018

Based on my adjustments to CEARCH's analysis of nuclear and volcanic winter, the expected annual mortality of nuclear winter as a fraction of the global population is 7.32*10^-6. I estimated the deaths from the climatic effects would be 1.16 times as large as the ones from direct effects. In this case, the expected annual mortality of nuclear war as a fraction of the global population would be 1.86 (= 1 + 1/1.16) times the expected annual mortality of nuclear winter as a fraction of the global population, i.e. 0.00136 %(= 1.86*7.32*10^-6). So the annual losses in future potential mentioned in the table above are 221 (= 0.0030/(1.36*10^-5)) and 73.5 (= 0.0010/(1.36*10^-5)) times my expected annual death toll, whereas I would have expected the annual loss in future potential to be much lower than the expected annual death toll.

Great points, Matt.

I think essentially all (not just many) pathways from AI risk will have to flow through other more concrete pathways. AI is a general purpose technology, so I feel like directly comparing AI risk with other lower level pathways of risk, as 80 k seems to be doing somewhat when they describe the scale of their problems, is a little confusing. To be fair, 80 k tries to account for this talking about the indirect risk of specific risks, which they often set to 10 times the direct risk, but these adjustments seem very arbitrary to me.

In general, one can get higher risk estimates by describing risk at a higher level. So the existential risk from LLMs is smaller than the risk from AI, which is smaller than the risk from computers, which is smaller than the risk from e.g. subatomic particles. However, this should only update one towards e.g. prioritise "computer risk" over "LLM risk" to the extent the ratio between the cost-effectiveness of "computer risk interventions" and "LLM risk interventions" is proportional to the ratio between the scale of "computer risk" and "LLM risk", which is quite unclear given the ambiguity and vagueness of the 4 terms involved[1].

To get more clarity, I believe it is be better to prioritise at a lower level, assessing the cost-effectiveness of specific classes of interventions, as Ambitious Impact (AIM), Animal Charity Evaluators (ACE), the Centre for Exploratory Altruism Research (CEARCH), and GiveWell do.

  1. ^

    "Computer risk", "LLM risk", "computer risk interventions" and "LLM risk interventions".

Here is an example with text in a table aligned to the left (select all text -> cell properties -> table cell text alignement).

StatisticAnnual epidemic/pandemic deaths as a fraction of the global population
Mean0.236 %
Minimum0
5th percentile1.19*10^-6
10th percentile3.60*10^-6
Median0.0276 %
90th percentile0.414 %
95th percentile0.684 %
Maximum10.3 %

Thanks for the post! I wonder whether it would also be good to have public versions of the applications (sensible information could be redacted), as Manifund does, which would be even less costly than having external reviewers.

Thanks, Will! Relatedly, I noted the importation makes the text in tables go from aligned to the centre in docs to aligned to the left/right on the EA Forum editor.

Hi Mike.

Los Alamos find that firestorms are highly unlikely to form under nuclear detonations, even at very high fuel loads, and so lofting is negligible. They only look at fission scale weaponry.

I think this may well misrepresent Los Alamos' view, as Reisner 2019 does not find significantly more lofting, and they did model firestorms. I estimated 6.21 % of emitted soot being injected into the stratosphere in the 1st 40 min from the rubble case of Reisner 2018, which did not produce a firestorm. Robock 2019 criticised this study, as you did, for not producing a firestorm. In response, Reisner 2019 run:

Two simulations at higher fuel loading that are in the firestorm regime (Glasstone & Dolan, 1977): the first simulation (4X No-Rubble) uses a fuel load around the firestorm criterion (4 g/cm2) and the second simulation (Constant Fuel) is well above the limit (72 g/cm2).

Crucially, they say (emphasis mine):

Of note is that the Constant Fuel case is clearly in the firestorm regime with strong inward and upward motions of nearly 180 m/s during the fine-fuel burning phase. This simulation included no rubble, and since no greenery (trees do not produce rubble) is present, the inclusion of a rubble zone would significantly reduce BC production and the overall atmospheric response within the circular ring of fire.

These simulations led to a soot injected into the stratosphere in the 1st 40 min per emitted soot of 5.45 % (= 0.461/8.454) and 6.44 % (= 1.53/23.77), which are quite similar to the 6.21 % of Reisner 2018 for no firestorm I mentioned above. This suggests a firestorm is not a sufficient condition for a high soot injected into the stratosphere per emitted soot under Reisner's view?

In my analysis, I multiplied the 6.21 % emitted soot that is injected into the stratosphere in the 1st 40 min from Reisner 2018 by 3.39 in order to account for soot injected afterwards, but this factor is based on estimates which do not involve firestorms. Are you implying the corrective factor should be higher for firestorms? I think Reisner 2019 implicitly argues against this. Otherwise, they would have been dishonest by replying to Robock 2019 with an incomplete simulation whose results differ from that of the full simulation. In my analysis, I only adjusted the results from Reisner’s and Toon’s views in case there was explicit information to do so[1], i.e. I did not assume they concealed key results.

As a result, blending together the Los Alamos model with that of Rutgers doesn’t really work as a baseline, they’re based on a very different binary concerning firestorms and lofting and you exclude other relevant analysis, like that of Lawrence Livermore.

In my analysis, I also did not integrate evidence from Wagman 2020 (whose main author is affiliated with Lawrence Livermore National Laboratory) to estimate the soot injected into the stratosphere per countervalue yield. As far as I can tell, they do not offer independent evidence from Toon's view. Rather than estimating the emitted soot as Reisner 2018 and Reisner 2019 did, they set it to the soot injected into the stratosphere in Toon 2007:

Finally, we choose to release 5 Tg (5¡10^12 g) BC into the climate model per 100 fires, for consistency with the studies of Mills et al. (2008, 2014), Robock et al. (2007), Stenke et al. (2013), Toon et al. (2007), and Pausata et al. (2016). Those studies use an emission of 6.25 Tg BC and assume 20% is removed by rainout during the plume rise, resulting in 5 Tg BC remaining in the atmosphere.

  1. ^

    For example, I adjusted downwards the soot injected into the stratosphere from Reisner 2019 (based on data from Denkenberger 2018), as it says (emphasis mine):

    Table 1. Estimated BC Using an Idealized Diagnostic Relationship (BC Estimates Need to be Reduced by a Factor of 10–100) and Fuel Loadings From the Simulations Shown in Reisner et al. and Two New Simulations for 100 15-kt Detonations

Great points, Stan!

Yes, I agree that the crux is whether firestorms will form.

I am not confident this is the crux.

The main thing I would like people to take away is that we remain uncertain what would be more damaging about a nuclear conflict: the direct destruction, or its climate-cooling effects.

I arrived at the same conclusion in my analysis, where I estimated the famine deaths due to the climatic effects of a large nuclear war would be 1.16 times the direct deaths.

Thanks for making this! I think it is valuable, although I should basically just donate to whatever I think is the most cost-effective given my strong endorsement of expected total hedonistic utilitarianism and maximising expected choice-worthiness.

Thanks for the relevant points, Joshua. I strongly upvoted your comment.

Could you please expand on why you think a Pareto distribution is appropriate here?

I did not mean to suggest a Pareto distribution is appropriate, just that it is worth considering.

Tail probabilities are often quite sensitive to the assumptions here, and it can be tricky to determine if something is truly power-law distributed.

Agreed. In my analysis of conflict deaths, for the method where I used fitter:

The 5th and 95th percentile annual probability of a conflict causing human extinction are 0 and 5.02 % [depending on the distribution]


When I looked at the same dataset, albeit processing the data quite differently, I found that a truncated or cutoff power-law appeared to be a good fit. This gives a much lower value for extreme probabilities using the best-fit parameters. In particular, there were too few of the most severe pandemics in the dataset (COVID-19 and 1918 influenza) otherwise; this issue is visible in fig 1 of Marani et al. Could you please add the data to your tail distribution plot to assess how good a fit it is?

I did not get what you would like me to add to my tail distribution plot. However, I added here the coefficients of determination (R^2) of the regressions I did.

A final note, I think you're calculating the probability of extinction in a single year but the worst pandemics historically have lasted multiple years. The total death toll from the pandemic is perhaps the quantity most of interest.

Focussing on the annual deaths as a fraction of the global population is useful because it being 1 is equivalent to human extinction. In contrast, total epidemic/pandemic deaths as a fraction of the global population in the year in which the epidemic/pandemic started being equal to 1 does not imply human extinction. For example, a pandemic could kill 1 % of the population each year for 100 years, but population remain constant due to births being equal to the pandemic deaths plus other deaths.

However, I agree interventions should be assessed based on standard cost-effectiveness analyses. So I believe the quantity of most interest which could be inferred from my analysis is the expected annual epidemic/pandemic deaths. These would be 2.28 M (= 2.87*10^-4*7.95*10^9) multiplying:

  • My annual epidemic/pandemic deaths as a fraction of the global population based on data from 1900 to 2023. Earlier years are arguably not that informative.
  • The population in 2021.

The above expected death toll would rank as 6th in 2021.

For reference, based on my analysis of conflicts, I get an expected death toll of conflicts based on historical data from 1900 to 2000 (also adjusted for underreporting), and the population in 2021 of 3.83 M (= 2.87*10^-4*7.95*10^9), which would rank above as 5th.

Here is a graph with the top 10 actual causes of death and expected conflict and epidemic/pandemic deaths:

Load more