Right, so to be a bit more concrete, maybe you're imagining that after HuggingFace 3.0 the push for generalized AIs stall out and we see some impressive specialized innovations like (say) self-driving cars and specialized biology models but no more general advances? Seems plausible.

Such a world I think look a lot like some combination of my #1 and #3, except with probably less available funding for AI safety than those worlds (though likely still higher than 2025 and 2026).

I think this question is pretty hard for me to answer because it's hard for me to imagine what the future would look like if we have AIs that are actually much more capable than today but the world mostly stays normal. I think it's much more plausible to me that I'm wrong because AIs don't become much capable, than because they do become much more capable and nothing external much changes.

1. If I'm imagining that world, the most likely hypothesis that comes to mind is that we had some significant global governance to either slow down further AI development or to manage the risks imposed by the existing very powerful models (likely both).
   1. In such a world, whether I should work on AI safety or not depends a lot on the details of whether the global governance is in a good place and/or benefits from my help.
2. Another possibility (that seems less likely to me but not impossible) is that we were wrong about the difficulty of aligning AI and we have some form of "alignment by default" world where the AIs are actually just much more cognitively capable than us but they're aligned to our intentions.
   1. In that world, what I should work on is in some sense not up to me (and certainly not up to my 2026-self!). I should instead be asking the new and better AIs what I should be doing, to the extent my efforts matter at all.
3. A third possibility is that capabilities stalled at a level slightly higher than today's, but not generally dangerous (for example if AI conquered a few fields the way it conquered math and cybersecurity, but otherwise stopped). And we've seen a few years with minimal further progress. In particular, this is a world where we have convincing evidence that AI R&D is not automatable, robotics has stalled out, human persuasion and long-run autonomy seems to have stalled, etc.
   1. In such a world I think probably some people should still work on AI safety but mostly it's manageable. Maybe people should go back to theory and conceptual stuff, if it looks clear that prosaic AI won't get to true AGI/ASI. Probably I'd personally work on something else. I'm not sure.
4. A fourth possibility is that AI capabilities are at an insanely high level and still growing, and alignment and governance still are far from solved, but we aren't yet doomed. In that world (which I think is the one you're most worried about/think is most central?) I basically think it's reasonable to think AI x-risk is just around the corner and we should still be working really hard on it.
   1. If we had, say, 3-5 more years of capabilities with AIs that are superhumanly good at science, long-term planning, real-world robotics (or side channels like persuasion and nanotech), but no major catastrophes, than I'm probably wrong about this for some key reason. #1 or #2 are the most obvious that comes to mind but maybe there's a secret third thing here that I didn't account for.

At a meta-level, it seems very hard to tell if a forum is falling from in-the-moment subjective experience: I feel like every online forum I've ever interacted with for nontrivial amounts of time (EA Forum, LessWrong, Quora, Facebook, Twitter, ACX comments, substack, reddit?, even the comments sections of my own blog) always constantly feels like they're in freefall and it always feels like the discussion forum in question is dying. This is true even if in retrospect the active period in question is looked back upon as a (relative) golden age of deep thought and inquiry (eg Quora in 2013-2014).

And yet at some point most of those forums really were dying, and at some point most of these forums truly did die^[1]. This is probably true even if it's hard to see it from obvious metrics -- websites can have a zombified existence for a long time (Eg if I go to the front page of my Quora today, most of the posts aren't even Q&A posts, just vague clickbait/slop and images that is probably AI generated. A far cry from the halcyon old days). 

Regardless, being around the block a few times, I've grown somewhat cynical of my ability, and that of other users, in discussion forums to pinpoint the likely causes and solutions to dying forums until long after a forum has died.

Still worth trying, naturally.

1. ^{^}

   This has happened often enough that occasionally I wonder if I'm the common factor, though I currently lean against. 

I mostly agree with the conclusion (though have not vetted the analysis in detail). This is one of the reasons why I'm more concerned with takeover risk and extreme concentration of power, than with random wanton destruction by terrorists (human or otherwise).

Hmm. To the extent that my intro isn't very good, I think it's a skill issue on my part! Some sequence of words surely must exist that conveys the same points well. 

If you're interested, I'd encourage you to try to write the same article yourself here. :) 

Re 1a, I'm happy to advise or mentor somebody interested in writing this up, I probably thought about climate change more than the vast majority of people in EA (not as much as actual experts like Johannes Ackva or John Halstead). 

The ideal candidate is somebody with legible climate credentials (eg worked as a climate scientist or policymaker and is pivoting/have pivoted to AI safety.

I think I won't be a good fit for writing this up myself because I think this is one of the messages where the messenger matters and also there are cultural nuances I won't be as familiar with.

Earlier notes here (though tbc it's backed by much more than just the bullet points and I also thought about it some since). 

People here might enjoy my short fiction You're Absolutely Right, an epistolary dark comedy/science-fiction horror story about corporate incentives and motivated reasoning at an imagined frontier AI lab. 

I wrote more about it here: https://linch.substack.com/p/openai-huggingface-hack

My summary: In a cybersecurity evaluation, OpenAI’s models, apparently autonomously and without any direct human direction, escaped their sandbox and successfully hacked a third-party company (HuggingFace).

The process involved leveraging a zero-day exploit to escape their sandbox, moving laterally across different OpenAI servers until they found a node with internet access, searching the internet and determining that the answers they wanted might be stored at HuggingFace, then leveraging multiple novel zero-day exploits to hack HuggingFace.

HuggingFace claimed that the models took thousands of independent actions across a swarm of short-lived sandboxes, “comprised of more than 17,000 recorded events.”

While technically a security evaluation with reduced safeguards, these actions are clearly out of bounds even in that context. It’s like being told to be creative and then breaking into your professor’s house and stealing the answer key. Worse than that, it’s not even your professor in this case, more like your professor’s friend.

Any human security researcher or engineer in a similar position would be fired on the spot. There is absolutely no valid reason to steal evaluation answers from an unaffiliated third party.

Furthermore, if I'm reading between the lines correctly, OpenAI did not address the issue (and perhaps didn't even know about their models doing this) until after HuggingFace's public blog post.

This leads me to suspect that there might be other major autonomous cybersecurity incidents that we do not yet know about. 

I think a lot of ppl studying empirical persuasion results of AI tend to overestimate the ecological validity and generalizability of those results. This isn't to say that studying empirical persuasion results is useless, just a) we should move towards better study designs in the future, and b) when looking at empirical persuasion studies you need to think carefully about not just the methodology and results but also what's reasonable to generalize and not.

I think a lot of people understand in the abstract that "ecological validity is important" but don't appreciate how difficult this is for persuasion results.

• it's actually just very hard to design a study that captures plausible real-world interactions, for practical, procedurally ethical, and infohazard-y reasons.
• It's very easy for empirical results to understate the influence of existing AI persuasion (see below)
• It's also very easy for empirical results to overstate the influence of existing AI persuasion (see below)
• Our studies give us very limited ability to predict the timing of the superpersuasive regime.
  • There's no clean analytic tool that allows us to draw lines on graphs to nearly the level of the METR time horizon graph
    • despite its many problems, I think the METR graph is much better at capturing something deeply true about software automation, compared to the studies we have at capturing persuasion ability or automation.
  • tbh I'm not sure (and currently lean against) the ability of the existing study methodologies can even allow us to reliably retrodict superhuman persuasion
    • as in, I don't think the existing tools can tell us that the AIs are reliably superhuman even after it occurs.
    • This seems concerning!
  • Furthermore, the AI labs aren't optimizing as much for AI persuasion as they are for other capabilities like coding, so we should expect some latent un-optimized for capabilities. It means that we shouldn't be shocked if sudden leaps happen.
• Our studies give us very limited ability to predict the shape of future superhuman capabilities.
  • For example, in Hackenburg (2026), when AIs outpersuade expert human debaters in single-session debates, they do so by throwing a ton of facts at you. Do we expect this to be the capability profile of future true superhuman persuaders?
    • Obviously not imo. Just seems really implausible that this is one of the empirical results that'd generalize well to the superhuman regime.

The field is already aware of some of these issues (see Appendix A), however in practice (in conversations and in the study methodology) I think many empirical researchers underestimate how big a deal these problems pose. I also think other people (eg researchers in other fields, or non-researchers) citing the studies are even less careful, and are even worse at understanding the ecological validity and generalization issues.

Appendix A (notes from Yang et. al 2026)

• “Research on impacts lags behind increased adoption of increasingly powerful AI. The mechanisms of epistemic decline discussed in Section 2 hinge on AI systems’ ability to persuade and to automate (Hackenburg et al., 2025; Durmus et al., 2024). Both of these are increasing (Anthropic, April 2025; Park et al., 2024a).
• Ecosystem-level feedback dynamics are inherently difficult for individual studies to capture. The recursive coupling of human beliefs, AI outputs, and training data described in Section 2.3 operates across platforms, training pipelines, and populations simultaneously. It is difficult for single studies, which are often limited in the number of platforms/models/populations they can test, to observe dynamics that emerge from the interaction of all three.
• Measured evidence is usually a lagging indicator of complex real-world phenomena. Realworld situations are difficult to specify and translate into neat taxonomies and variables for modeling that are currently required for much of high-quality research. For example, it is difficult to assess how much personalization boosts persuasiveness. In research settings, persuasiveness can only be measured by proxy. In doing so, some studies find no effect (Hackenburg et al., 2025; Argyle et al., 2025), others find small effects (Kelley and Riedl, 2026; Matz et al., 2024a), and others find strong effects (Salvi et al., 2025a). This disagreement may itself reflect the difficulty of measuring effects that depend on sustained, naturalistic interaction rather than one-shot exposure.
• The scientific endeavor also risks overstating these effects. Laboratory settings are not always representative of real-world environments; experimental designs may act as high-fidelity amplifiers that maximize pressure to achieve statistical rigor. In the lab, moving a participant’s opinion on a policy issue by a few percentage points may be a controlled success with effect size, but in the actual chaos of a political cycle even massive advertising spending can struggle to move the needle at all (Coppock et al., 2020).”