Just want to signal boost the subreddit for s-risk discussion.

The fellowship will cover what we currently consider to be the most important sources of s-risk (TAI conflict, risks from malevolent actors).

Any reason CLR believes that to be the case specifically? For instance, it's argued on this page that botched alignment attempts/partially aligned AIs (near miss) & unforeseen instrumental drives of an unaligned AI are the 2 likeliest AGI-related s-risks, with malevolent actors (deliberately suffering-aligned AI) currently a lesser concern. I guess TAI conflict could fall under the second category, as an instrumental goal derived risk.

I agree nuclear winter risk is overblown and I'm glad to see more EAs discussing that. But I think you're also overrating the survivability of SSBNs, especially non-American ones. They are not a One Weird Trick, Just Add Water for unassailable second strike capability, with upkeep/maintenance only being one aspect of that. Geography plays a huge role in how useful they are, with the US deciding to base most of its warheads on SSBNs because they have the most favourable conditions for them (unrestricted access to and naval dominance of two oceans). In contrast, Russia has much less room to play with (mainly some parts of the Arctic ocean) and suitable ports to deploy the subs from, and China's situation is even worse. The seas surrounding it have unfavourable bathymetry (very shallow) and the only paths to open ocean are chokepoints.^[1] It's not as hard to detect a submarine as one might think, otherwise noise-quieting measures like pumpjets, reactor cooling design and tiles as you mentioned wouldn't be such a huge deal. Most importantly, the US has a large fleet of advanced attack subs (SSNs) the others lack, which pose an enormous threat to SSBNs. They could pick up a tail without knowing it and be destroyed before they can launch their missiles.^[2]

OTOH American SSBNs should be fine at least for the time being as long as they don't do anything stupid like try to sneak up close to another country. But emerging technologies like Magnetic Anomaly Detectors and such will make concealing SSBNs even more difficult and increase reliance on land-based forces in the future.

Look into suffering-focused AI safety which I think is extremely important and neglected (and s-risks).

I disagree with the claim that the overall accident risk is going down. While it's probably true early warning systems are getting more reliable (though the actual degree of this is really hard to gauge due to their complexity)^[1], a third party (China) adopting launch on warning arguably raises the risk at least 50%, if not more due to initial kinks. Also, as many have pointed out, the emerging trilateral dynamic of three nuclear peers is unprecedented in history and less stable.

Also, what would count as an accidental nuclear war? I think e.g. the US launching a large salvo of low-observable cruise missiles deep into the Chinese mainland during a conventional war could easily be mistaken as an attack on the silo fields and trigger a nuclear launch.

Little known detail about the Arkhipov incident. Unsure if true, but if so it sounds like he agreed to fire the torpedo and it all came down to the coincidence of the light getting wedged in the hatch making a few second-difference. Something that may not have happened if the signals officer's motor neurons had fired just slightly differently.

I think the OP is advocating a prize for solving the whole problem, not specific subproblems, which is a novel and interesting idea. Kind of like the $1M Millennium Prize Problems (presumably we should offer far more).

If you offer a prize for the final thing instead of an intermediate one people may also take more efficient paths to the goal than the one we're looking at. I see no downside to doing it, I mean you don't lose any money unless someone actually presents a real solution.

Hey Michael, sorry I didn't get around to commenting on this before you published haha. Long thought dump below:

I'm not sure if they count as "technological developments", but 2 of the largest things I see contributing to nuclear risk are development of ballistic missile defence (BMD) and proliferation of tactical nuclear weapons (TNWs).

The dangers from BMD are manyfold. One is being the cause of a conventional conflict. E.g. As the US continues to develop its maritime ICBM intercept capability, it'll pose a major threat to foreign arsenals. If a significant number of USN ships surround China, it may feel the need to sink them preemptively to ensure its ICBMs can get through. Russia has threatened similar against the land-based counterpart.^[1] Some even want to revive Brilliant Pebbles over the New Hysterical Threat in the east. Given that it (and SDI in general) were the most destabilizing things in history (you think the USSR would've just sat there and watched as it lost its strategic deterrence?), that's not great.
Another risk is making a first strike by either party likelier. As said, a nation will behave unpredictably if it sees its nuclear capability slipping away (now or never?). And something often pointed out is even if the BMD system doesn't work, as long as leaders believe it might, they may be emboldened to strike (hoping their BMD can mop up the rest).
Lastly, it'll spur nations to develop ever more destabilizing offensive systems to maintain their deterrence.^[2] Crazy nuclear-powered cruise missiles like Skyfall are just the start, I have some (very infohazardous) ideas for better nuclear delivery others may have thought of too. Also, one way to counter BMD is simply to increase arsenal like China's doing, as it's generally cheaper to make delivery than interception systems. Risks from larger stockpiles are obvious.

On TNWs: I think the odds of nuclear war would be far lower if they didn't exist. The fact that they're always present as an option during any conventional conflict makes the odds of crossing the nuclear threshold so much higher than if only strategic weapons existed, it's hard to overstate. For instance, the US is developing (additional) undersea TNW options clearly intended to be used against China over Taiwan.^[3] Biden even called it a "bad idea" while campaigning but is now forging ahead with it. In fact one of my likeliest concrete NW scenarios is one of US first use after its in-theatre bases and fleet suffer decimation at Chinese hands in such a war. Weapons on the Chinese side exacerbate this, like the DF-26 IRBM, but those are purely in response to US TNWs.^[4] You can't just expect them to have no response: indeed, lack of a symmetrical response ability is likely more destabilizing. Under China's previous city-buster-only force composition, the US would be both likelier to employ TNWs without credible threat of in-kind retaliation, and if China did retaliate it'd escalate straight to countervalue.

Agree with the goal of reducing silos: they are highly destabilizing, as I've written before. Stationary targets are vulnerable, even if you try to mitigate this with Launch on Warning (LoW). Stealthy cruise missiles are one such threat. Another is an obscure idea called "x-ray pindown" that could suppress them from firing by continuously detonating warheads overhead; it should be possible to combine it with other counterforce weapons^[5] and destroy the silos, thus defeating LoW. There's an exceptional 11-pg analysis from SciAm in 1984 diving deep into the problems with LoW including pindown (I have the full copy for anyone interested). If you could reduce or eliminate silos and have the nuclear powers' force compositions switched entirely to mobile & sub, that would greatly improve strategic stability & lower nuclear risk, as there would be almost no targets vulnerable to a first strike^[6]. Downside is if one still broke out it may go countervalue more easily due to there being fewer rungs on the escalation ladder, from the paucity of meaningful counterforce targets.^[7] But this consideration probably isn't large due to being outweighed by the elimination of the overwhelming majority of accidental nuclear risk (no more LoW), plus they can still always target military bases, there'll never be a shortage of those. Also, you mentioned you're "unsure where Chinese mobile warheads are", well the answer is they're in PLARF bases in the middle of nowhere, which is great from the smoke-producing perspective.^[8]

On the point about reducing entanglement: Much has been made about intermingling of conventional/nuclear forces, like China's "hot-swappable" DF-26. But I think it's not as huge a problem as claimed. These are not intercontinental-range forces, they would be flying in-theatre, not headed towards the enemy homeland, so e.g. the US could afford to wait to see whether it was conventional before retaliating since it wouldn't be threatening their own nuclear forces, in contrast to an inbound ICBM where you'd have minutes to decide whether to launch on warning or risk losing your own. Similarly, even if the US targeted China's IRBMs (unlikely anyway because they're mobile), China probably wouldn't feel that its strategic deterrent were being degraded (and thus much use-it-or-lose-it pressure).

On hypersonics: I think this excellent chart is the most concise thing I've seen to dispel the hype around them. They don't really contribute to nuclear risk imo^[9], in fact they do the opposite by preserving deterrence. The associated report is a great read.

ASATs are indeed dangerous. So much so that if we saw mass satellite warfare, I'd expect a high chance of a nuclear exchange following shortly. Once your space based early warning is blinded, you'd be extremely vulnerable & face considerable pressure to use (indeed countries may interpret it as a prelude to first strike & launch). This is only compounded as HGVs continue to proliferate & make up a larger portion of deployed arsenals because of their lower flight path, enemy land-based forces will be sitting ducks without early warning satellites.

This paper is a good resource on your question of whether Poseidon is salted. Remember that the neutrons used to activate cobalt would've otherwise been hitting a fissile/fissionable uranium jacket, so they'd be incurring a massive yield handicap.

A lot of nuclear risk reduction just strikes me as highly intractable though. It just seems that if a nuclear war does happen, it'll be due to forces and military pressures much too strong and preordained for us to influence one way or another. E.g., these are similarly implausible:
-Making the big 3 give up their silos. Lots have tried and failed in the US. Even unlikelier for China: it's an attractive basing option as cheap install capacity for a country looking to expand arsenal + great for soaking up enemy warheads.
-For the US (the main developer of BMD) to give up its obsession with neutralizing the nuclear threat its enemies pose it.^[10]
-Eliminating TNWs (that have been around for 70 years): Want the US to? Good luck with that when Russia has thousands of them (and relies on them due to conventional military weakness).
-Prevent countries from developing ASAT: countries are even willing to incur costs like worldwide condemnation & endangering their allies to strengthen this key capability. No great power will stop developing in the space domain.
Even something as simple as a NFU pledge has proven impossible, even more of a political nonstarter after recent events. Overall, I think it's a great decision for EAs to focus more on AI. There's already a huge community of capable folks working on risk reduction from every angle. Some pathways to x-risk from nukes also seem quite unrealistic. E.g. Alexey Turchin voiced to me concerns about gigaton-scale salted bombs. There's no way a single salted bomb large enough to be x-risky would be deliverable, so the country would have to detonate it on their own territory ("backyard delivery"). But it's very unlikely (and unprecedented) for countries to devise strategies explicitly harming let alone sterilizing their own nations. The optimal cost-effective strategy that maximizes deterrence while minimizing self-harm the great powers have stabilized on is simply many warheads that maximize blast radius (i.e. the typical diversified MIRV'd ICBM forces you see today), which is why you haven't seen any doomsday devices created. Seeing as the main one left is nuclear winter, I guess that should be the main focus of research despite the (imo) low probability. One potential research project I'd like to see is assessing the independent components of the hypothesis, i.e. doing a complete analysis and incorporating all available evidence into each one (e.g. generating an updated probability in light of the observations of Gulf oil well plumes, wildfires etc. for the lofting assumption) for optimal probabilities of each independent piece, then multiplying them together, to get overall conjunctive likelihoods. Allfed's Denkenberger said he got 20% chance of agricultural collapse using a Monte Carlo model, I haven't gotten to read the paper yet but I'm significantly lower, maybe ~5%? Another potential thing to investigate longer-term is I guess detection of SSBNs (as that's probably the largest conceivable strategic stability killer, other than an infallible BMD system), but again massive public & private (MIC) R&D will always be going into improving & ensuring the survivability of their undersea deterrents so I don't see how we'd help.

Finally, I've been thinking about your 3rd point in the 9 mistakes doc. Maybe there's a tendency for wars between nuclear powers to draw in the rest, if after suffering enough damage, nations decide it's best to bring others down with them. E.g., perhaps China, after being devastated in an exchange with the US in 2028 where each expended 1000 warheads, decides to glass regional enemy India with its remaining 500 to ensure it doesn't emerge unscathed as a massive future threat, calculating that cost of the additional Indian retaliation isn't meaningful in light of the destruction already suffered.^[11] I can easily see the US doing this to China after a full exchange with Russia (that China wasn't involved in), to ensure that even if it's gonna lose its hegemony, the hated ChiComs won't take its place. So who knows, scenarios depicting an exchange confined to only 2 powers may not be so realistic after all.^[12][13] The only scenario this "cascading nuclear war" fear of mine doesn't seem at all plausible in is if a large nuclear power is attacked by a small one: e.g. if the US is only hit by a couple dozen NK nukes, I can't see them knowingly choose to nuke China too and forfeit the rest of their country.

Just one thought: there are so many ways for a nuclear war to start accidentally or through miscalculations (without necessarily a conventional war) that it just seems so absurd to see estimates like 0.1%. A big part of it is even just the inscrutable failure rate of complex early warning systems composed of software, ground/space based sensors and communications infrastructure. False alarms are much likelier to be acted on during times of high tension as I pointed out. E.g., during that incident Yeltsin, despite observing a weather rocket with a similar flight profile as a Trident SLBM and having his nuclear briefcase opened, decided to wait a bit until it became clear it was arcing away from Russia. But that was during a time of hugs & kisses with the West. Had it been Putin, in a paranoid state of mind, after months of hearing Western leaders call him an evil new Hitler who must be stopped, you're absolutely sure he wouldn't have decided to launch a little earlier?

Another thing is I'm uncomfortable with is the tunnel-vision focus on US-Russia (despite current events). As I also pointed out, China joining the other 2 in adopting launch on early warning raises the accident risk by at least 50% - at least since it's probably higher at first given their inexperience/initial kinks in the system that haven't been worked out, etc.^[1]

As a last note, cities are likelier to be hit than many think. For one the not explicitly targeting civilians thing isn't even true. Plus the idea doesn't pass the smell test. If some country destroys all the US's cities in a massive attack, you think the US would only hit counterforce targets in retaliation? No, at a minimum the US would hit countervalue targets in response to a countervalue attack. Even if there were no explicit "policy" planning for that (impossible), a leader would simply order that type of targeting in that eventuality anyway.

One point about the Pentagon figure, they said "700 warheads by 2027 and at least 1000 by 2030"^[1]. Meaning most likely over 1k. I actually made a bet that they will revise the estimate up again in this/next year's report, e.g. this year we may see "900-1000 by 2027 and 1200-1500 by 2030". They said the stockpile was only expected to double over the decade to the ~400s in the 2020 report to Congress, so by increasing estimates more gradually they're probably mitigating a loss of face & hard questions over the massive intelligence failure. Signs point to China simply seeking parity with the US/Rus (~1500 deployed), I don't see why they'd expand only to 1000 and stop there.

Comments by DoD people support this hypothesis^[2]. Common sense tells us this too: Even if the ~300 new DF-41 silos discovered last year are each armed with only 3 warheads (the missile can carry ~10 max), and no other silos are built/discovered, that's still 900 warheads on top of the ~400 already in service. China's nuclear force has traditionally been mobile-based, and TEL (truck)/rail^[3]-basing are probably expanding similarly; silos are just the most visible. (Plus, submarines.)

I know a lot of ways to reduce China-US nuclear risk even without non-starters to the pro-democracy crowd (e.g. giving up defence commitments to certain US allies). There seems to be some major civilizational inadequacy in this area; i.e. obvious ways to have a major reduction on the risk that just nobody's bothered to implement. I don't think economic tensions/trade wars are very relevant to nuclear risk compared to more important factors in the grand scheme of things to be frank.