Thanks for laying these out. A month later, some look different.
My take on your points:
We are not short of warning shots (1). Labs are reviewing tens of thousands of incidents, and journalists can't keep up.
A pause can cover compute and ideas too (3). Krueger et al.'s 210-page study finds a hardware-enforced pause feasible.
The AI that helps with alignment strategy is close to the AI that could take over (4). This is your strongest point, but that capability cuts both ways.
I agree (6) and (7) are strong points against a naive pause, but they argue for a verified, mutual slowdown, not against slowing down.
And the fact that Finland co-initiated the call, signed by 22 countries, to keep AI 'under human control' shows that middle powers have an important role in normalizing existential safety in the diplomatic discourse
When I first saw the statistics, I was really surprised to see that of the 340 alumni covered by the assessment, about 186 now work in AI safety, and that the best estimate is that about half of this work is counterfactual![1] Wow, especially after only one week of an in-person program, and only £9k for each counterfactual FTE, which, if I’m not mistaken, is very competitive in the AI safety field-building space.
And this doesn’t even count the communities alumni founded: at least 12 so far, including the Buenos Aires AI Safety Hub, which went from nothing in July 2024 to 350+ members and 6 full-time AI safety roles. Field-building compounds quickly, so the idea that it only pays off if timelines are long seems largely wrong to me.
We got similar statistics after the first ML4Good bootcamps in France and Switzerland, but to be honest, I was expecting those first bootcamps to be exceptional, because we would only be picking the low-hanging fruit that had accumulated for years. Seems like I was wrong!
This is mainly due to the hard work of the team, especially Nia, Jack, and Linda, over the last few years, and to all the teachers who shared their passion and motivation to help make this world a better and safer place. And 98% of participants would recommend it!
"Their next question is almost always ‘what do you propose to do about this?’ over ‘is this actually a problem?’." --> I wrote a big post about this :)
I want to push back specifically on P1 as it applies to X-risk reduction.
The longtermist case for X-risk reduction doesn't require estimating "all effects until the end of time"
Your list of "crucial factors" to consider (digital sentience, space colonization, alien civilizations, etc.) frames longtermism as requiring a comprehensive estimate of net welfare across all possible futures. But this mischaracterizes the actual epistemic claim behind X-risk reduction work.
The claim is narrower: we can identify a specific, measurable problem (probability of extinction) and implement interventions that plausibly reduce it. This is structurally similar to how we approach any tractable problem — not by modeling all downstream effects, but by identifying a variable we care about and finding levers that move it in the right direction.
You might respond: "But whether reducing extinction probability is good requires all those judgment calls about future welfare." I'd argue this conflates two questions:
Can we reduce P(extinction)? (Empirical, tractable)
Is reducing P(extinction) net positive? (The judgment call you're highlighting)
For (2), I'd note that the judgment call isn't symmetric. Extinction forecloses all option value — including the option for future agents to course-correct if we've made mistakes. Survival preserves the ability to solve new problems. This isn't a claim about net welfare across cosmic history; it's a claim about preserving agency and problem-solving capacity.
The civilizational paralysis problem
If universal adoption of cluelessness would cause civilizational collapse, this isn't just a reductio — it suggests cluelessness fails as action-guidance at the collective level. You might say individuals can act on non-longtermist grounds while remaining longtermist-clueless. But this concedes that something must break the paralysis, and I'd argue that "preserve option value / problem-solving capacity" is a principled way to do so that doesn't require the full judgment-call apparatus you describe.
I'm genuinely uncertain here and would be curious how you'd respond to the option-value framing specifically.
I strongly agree with this assessment, and this is why we created a new version of ML4Good, which is non-technical, ML4Good governance, that is no longer targeted for wanna be researcher
Thanks for laying these out. A month later, some look different.
My take on your points:
Congratulations Meeri!
And the fact that Finland co-initiated the call, signed by 22 countries, to keep AI 'under human control' shows that middle powers have an important role in normalizing existential safety in the diplomatic discourse
If the likely outcome of an AI pause is that we unpause too early and everyone dies - I'd say that the likely outcome of not pausing is dying earlier
Pausing, slowing down and gaining time is instrumentaly convergent in such a setting
When I first saw the statistics, I was really surprised to see that of the 340 alumni covered by the assessment, about 186 now work in AI safety, and that the best estimate is that about half of this work is counterfactual![1] Wow, especially after only one week of an in-person program, and only £9k for each counterfactual FTE, which, if I’m not mistaken, is very competitive in the AI safety field-building space.
And this doesn’t even count the communities alumni founded: at least 12 so far, including the Buenos Aires AI Safety Hub, which went from nothing in July 2024 to 350+ members and 6 full-time AI safety roles. Field-building compounds quickly, so the idea that it only pays off if timelines are long seems largely wrong to me.
We got similar statistics after the first ML4Good bootcamps in France and Switzerland, but to be honest, I was expecting those first bootcamps to be exceptional, because we would only be picking the low-hanging fruit that had accumulated for years. Seems like I was wrong!
This is mainly due to the hard work of the team, especially Nia, Jack, and Linda, over the last few years, and to all the teachers who shared their passion and motivation to help make this world a better and safer place. And 98% of participants would recommend it!
(Disclaimer: I co-founded ML4Good in 2022)
Meaning that, by participants’ own estimates, about half of this work wouldn’t be happening without ML4Good.
Thanks for writing this!
"Their next question is almost always ‘what do you propose to do about this?’ over ‘is this actually a problem?’." --> I wrote a big post about this :)
LessWrong: https://www.lesswrong.com/posts/EexsebbYhbe2gXkPP/the-current-bottleneck-is-political-will-not-research
EA Forum : https://forum.effectivealtruism.org/posts/nmQpZyybjHrunDxFT/the-current-bottleneck-is-political-will-not-research
This is a good comment - thanks!
I want to push back specifically on P1 as it applies to X-risk reduction.
The longtermist case for X-risk reduction doesn't require estimating "all effects until the end of time"
Your list of "crucial factors" to consider (digital sentience, space colonization, alien civilizations, etc.) frames longtermism as requiring a comprehensive estimate of net welfare across all possible futures. But this mischaracterizes the actual epistemic claim behind X-risk reduction work.
The claim is narrower: we can identify a specific, measurable problem (probability of extinction) and implement interventions that plausibly reduce it. This is structurally similar to how we approach any tractable problem — not by modeling all downstream effects, but by identifying a variable we care about and finding levers that move it in the right direction.
You might respond: "But whether reducing extinction probability is good requires all those judgment calls about future welfare." I'd argue this conflates two questions:
For (2), I'd note that the judgment call isn't symmetric. Extinction forecloses all option value — including the option for future agents to course-correct if we've made mistakes. Survival preserves the ability to solve new problems. This isn't a claim about net welfare across cosmic history; it's a claim about preserving agency and problem-solving capacity.
The civilizational paralysis problem
If universal adoption of cluelessness would cause civilizational collapse, this isn't just a reductio — it suggests cluelessness fails as action-guidance at the collective level. You might say individuals can act on non-longtermist grounds while remaining longtermist-clueless. But this concedes that something must break the paralysis, and I'd argue that "preserve option value / problem-solving capacity" is a principled way to do so that doesn't require the full judgment-call apparatus you describe.
I'm genuinely uncertain here and would be curious how you'd respond to the option-value framing specifically.
I strongly agree with this assessment, and this is why we created a new version of ML4Good, which is non-technical, ML4Good governance, that is no longer targeted for wanna be researcher
Great post