Updates on the EA catastrophic risk landscape

My guess is Ben's referring to people like Holden Karnofsky, who went from working in finance to co-founding and -running GiveWell and then Open Phil to now doing research at a think tank.

I think there is more value in separating out AI vs bio vs nuclear vs meta GCR than having posts/events marketed as GCR but be mainly on one topic. Both from the perspective of the minor causes and the main cause which would get more relevant attention. 

Also the strategy/marketing of those causes will often be different and so it doesn't make as much sense to lump them together unless it is about GCR prioritisation or cross cause support.

It seems possible that both of these are neglected for similar reasons. 

It seems surprising the funding would be the bottleneck (which means you can't just have more of both). But that has been my experience surprisingly often, i.e. core orgs are willing to devote many highly valuable staff hours to collaborating on survey projects, but balk at ~$10,000 survey costs.

I don't know if they're making a mistake - my question wasn't meant to be rhetorical.

I take your point about capacity constraints, but if no-one else is stepping up, it seems like it might be worth OP stepping up their capacity constraints.

I continue to think the EA movement systematically underestimates the x-riskiness of nonextinction events in general and nuclear risk in particular by ignoring much of the increased difficulty of becoming interstellar post-destruction/exploitation of key resources. I gave some example scenarios of this here (see also David's results) - not intended to be taken too seriously, but nonetheless incorporating what I think are significant factors that other longtermist work omits (e.g. in The Precipice, Ord defines x-risk very broadly, but when he comes to estimate the x-riskiness of 'conventional' GCRs, he discusses them almost entirely in terms of their probability of making humans immediately go extinct, which I suspect constitutes a tiny fraction of their EV loss).

It doesn't seem too conceptually murky. You could imagine a super-advanced GPT, which when you ask it any questions like 'how do I become world leader?' gives in-depth practical advice, but which never itself outputs anything other than token predictions.

I might quantify the value of the talent pool around another $10bn, so again, you only need a ~10% increase here to be worth a billion, and over centralisation seems like one of the bigger problems.

I find it plausible that a strong fix to the funder-diversity problem could increase the value of the talent pool by 10% or even more. However, having a new independent funder with $1B in assets (spending much less than that per year) feels more like an incremental improvement.

$1bn is only 5% of the capital that OP has, so you'd only need to find a 1 grant for every 20 that OP makes that they've missed with only 2x the effectiveness of marginal OP grants in order to get 2x the value.

You'd need to do that consistently (no misses, unless counteracted by >2x grants) and efficiently (as incurring similar overhead as OP with $1B of assets would consume much of the available cash flow). That seems like a tall order. 

Moreover, I'm not sure if a model in which the new major funder always gets to act "last" would track reality very well. It's likely that OP would change its decisions, at least to some extent, based on what it expected the other funder to do. In this case, the new funder would end up funding a significant amount of stuff that OP would have counterfactually funded.

Thanks for the comment, Christian!

I think the 80K profile notes (in a footnote) that their $1-10 billion guess includes many different kinds of government spending. I would guess it includes things like nonproliferation programs and fissile materials security, nuclear reactor safety, and probably the maintenance of parts of the nuclear weapons enterprise -- much of it at best tangentially related to preventing nuclear war. 

Most of these are relevant to preventing nuclear war (even if you do not think they are the best way of doing it):

• More countries having nuclear weapons makes nuclear war more likely.
• Fissile materials are an input to making nuclear weapons.
• Malfunctioning nuclear systems/weapons could result in accidents.

So I think the number is a bit misleading (not unlike adding up AI ethics spending and AI capabilities spending and concluding that AI safety is not neglected). You can look at the single biggest grant under "nuclear issues" in the Peace and Security Funding Index (admittedly an imperfect database): it's the U.S. Overseas Private Investment Corporation (a former government funder) paying for spent nuclear fuel storage in Maryland... 

According to the footnote in the 80,000 Hours' profile following what I quoted, the range is supposed to refer to the spending on preventing nuclear war (which is not to say the values are correct):

The resources dedicated to preventing the risk of a nuclear war globally, including both inside and outside all governments, is probably $10 billion per year or higher. However, we are downgrading that to $1–10 billion per year quality-adjusted, because much of this spending is not focused on lowering the risk of use of nuclear weapons in general, but rather protecting just one country, or giving one country an advantage over another. Much is also spent on anti-proliferation measures unrelated to the most harmful scenarios in which hundreds of warheads are used. It is also notable that spending by nongovernment actors represents only a tiny fraction of this, so they may have some better opportunities to act.

A way to get at a better estimate of non-philanthropic spending might be to go through relevant parts of the State International Affairs Budget, the Bureau of Arms Control, Deterrence and Stability (ADS, formerly Arms Control, Verification, and Compliance), and some DoD entities (like DTRA), and a small handful of others, add those up, and add some uncertainty around your estimates. You would get a much lower number (Arms Control, Verification, and Compliance budget was only $31.2 million in FY 2013 according to Wikipedia -- don't have time to dive into more recent numbers rn).

For reference, the range mentioned by 80,000 Hours suggests the spending on nuclear risk is 4.87 % (= 4.04/82.9) of the 82.9 billion $ spent on nuclear weapons in 2022.

All of which is to say that I think Ben's observation that "nuclear security is getting almost no funding" is true in some sense both for funders focused on extreme risks (where Founders Pledge and Longview are the only ones) and for the field in general 

I think one had better assess the cost-effectiveness of specific interventions (as GiveWell does) instead of focussing on spending. You estimated doubling the spending on nuclear security would save a life for 1.55 k$, which corresponds to a cost-effectiveness around 3.23 (= 5/1.55) times that of GiveWell's top charities. I think corporate campaigns for chicken welfare are 1.44 k times as cost-effective as GiveWell's top charities, and therefore 446 (= 1.44*10^3/3.23) times as cost-effective as what you got for doubling the spending on nuclear security.

Thanks, Abby. I knew MacArthur had left the space, but not that Carnegie Endowment had recently decreased funding. In any case, I feel like discussions about nuclear risk funding often implicitly assume that a large relative decrease in philanthropic funding means a large increase in marginal cost-effectiveness, but this is unclear to me given it is only a small fraction of total funding. According to Founders Pledge's report on nuclear risk, "total philanthropic nuclear security funding stood at about $47 million per year ["between 2014 and 2020"]". So a 100 % reduction in philantropic funding would only be a 1.16 % (= 0.047/4.04) relative reduction in total funding, assuming this is 4.04 G$, which I got from the mean of a lognormal distribution with 5th and 95th percentile equal to 1 and 10 G$, corresponding to the lower and upper bound guessed in 80,000 Hours’ profile on nuclear war.

Thanks for the comment! I commented below that:

In any case, I feel like discussions about nuclear risk funding often implicitly assume that a large relative decrease in philanthropic funding means a large increase in marginal cost-effectiveness, but this is unclear to me given it is only a small fraction of total funding. According to Founders Pledge's report on nuclear risk, "total philanthropic nuclear security funding stood at about $47 million per year ["between 2014 and 2020"]". So a 100 % reduction in philantropic funding would only be a 1.16 % (= 0.047/4.04) relative reduction in total funding, assuming this is 4.04 G$, which I got from the mean of a lognormal distribution with 5th and 95th percentile equal to 1 and 10 G$, corresponding to the lower and upper bound guessed in 80,000 Hours’ profile on nuclear war.

More importantly, I believe the global catastrophic risk community had better assess the cost-effectiveness of specific interventions (as GiveWell does) instead of focussing on spending. Christian Ruhl from Founders Pledge estimated doubling the spending on nuclear security would save a life for 1.55 k$, which corresponds to a cost-effectiveness around 3.23 (= 5/1.55) times that of GiveWell's top charities. I think corporate campaigns for chicken welfare are 1.44 k times as cost-effective as GiveWell's top charities, and therefore 446 (= 1.44*10^3/3.23) times as cost-effective as what Christian got for doubling the spending on nuclear security.

I see. I think it is better to consider spending from other sources because these also contribute towards decreasing risk. In addition, I would not weight spending by cost-effectiveness (and much less give 0 weight to non-EA spending), as this is what one is trying to figure out when using spending/neglectedness as an heuristic.