Thank you to Katja Grace for discussion that inspired this post. The views here are my own.
In September I wrote that there are 9 reasons against a near-term AI slow-down, which left me highly uncertain about when it would be best to slow down AI. This is an update to say that I now believe the evidence has shifted strongly in favor of trying to get a coordinated slow-down or pause on AI immediately.
The short version: AI timelines now look much shorter than I expected, and a well-coordinated slow-down could take months or years to set up. If we wait for a better moment to start, we may be too late.
This comes in the wake of METR’s report on AI R&D at Anthropic. Based on internal data, METR now thinks Anthropic is likely improving its models about twice as fast as it was before October 2025, and that most of that speed-up (perhaps 80%) comes from AI accelerating research. They didn’t find any clear bottleneck that would stop this from compounding, though they’re not confident about that.
Unfortunately, Anthropic redacted a majority of the evidence METR found most informative, and METR is quite uncertain itself (they wouldn’t be surprised by 1.3x or by 3x). Furthermore, Anthropic is only one company, and this effect is only measured on well-specified tasks. Still, many leading thinkers on the topic have updated toward much, much shorter timelines. More detail here:
- Ajeya Cotra, best quick summary: “AI is dramatically accelerating AI research right now”. She calls the report the most important update on AI risk in the last couple of years, and notes that if the acceleration compounds, the naive extrapolation reaches superintelligence within a year.
- The AI Futures Project, “Q3.5 Timelines Update”. All of its authors now think a superhuman AI researcher will likely arrive in 2027. The medians they chart for AI that dominates top human experts at virtually all tasks fall between September 2027 and September 2028. In August, they ranged from late 2028 to early 2035.
- Ryan Greenblatt, in this post: “a big update towards ASI within 1-2 years.”
- Benjamin Todd, “Trend extrapolation could get us to superhuman AGI in 18 months”. His best guess, absent a pause, is “digital worker” AGI within 2-4 years, but he gives a 30%+ chance that recursive self-improvement has already started and will bring us to broadly beyond-human AGI within 6 to 18 months.
- Paul Christiano, in a November 16th Dwarkesh Podcast interview, says there’s “maybe a 1/3 chance that we have superintelligence within one year” on the current trajectory if we don’t slow down.
So I now think there’s a significant chance that AI matches or exceeds humans at all cognitive tasks in only 1-2 years if nothing changes.
If you just want to get the basics, I’d encourage you to read Ajeya’s summary, which is short and easily readable, or listen to the first ten minutes of Paul’s interview.
Revisiting my nine reasons
Here is an update on each of my nine reasons against a near-term slow-down:
- Warning shots. Getting an international AI slow-down may take months or years of planning and effort. Even if we start now, by the time an agreement is enforced, the level of “warning shot” we get may already be dangerously close to AI takeover, and a warning shot that is too big could cause panic and public responses that backfire. With timelines this short, and AI companies working hard to contain AI, we may not get a better warning shot than the Hugging Face Incident before it’s too late. And if we do, we should at least have all the planning, political, and technical machinery for a thoughtful, well-coordinated slow-down ready before that warning shot occurs.
- Boy-calls-wolf effect. With timelines this short, it seems more likely that those calling for a pause now will look prescient, and win political capital for having called it early. If these timelines are right, then even if we slow down AI a lot, progress could still seem frighteningly fast.
- Overhang. As Charbel-Raphael pointed out on my previous post, a pause can include compute. When it comes to the other types of overhang, though, this is one area where I’m less certain. Intuitively, having AGI in 1-2 years instead of 3-10 years feels a little bit safer, because it seems AI is most existentially dangerous if it has access to better robots, biological research capabilities, and autonomous weapons at the time it’s created. Cutting against this, a well-coordinated and thoughtful slow-down could give us more time to develop the defenses we need against these threats.
- Helpful AI. If AI is speeding up AI research by enough that AI will dominate top human experts across all fields within 1-2 years, then it’s probably already strong enough to help significantly with alignment. AI may be more helpful for capabilities than alignment anyway, in which case we need to slow down overall progress so that alignment can keep up, and focus on whatever seems like it will differentially accelerate alignment, perhaps AI’s conceptual reasoning. For other things like epistemics and coordination, it seems like we’re dramatically underutilizing current AI, and by slowing down, we could shift more attention to getting help with these using current AI capabilities.
- Model organisms. This is another argument I feel ambivalent about, as it does seem like the AI that’s most dangerous is going to be significantly different from current AI. But again, if it takes several months to actually get an international slow-down, then we could be much closer to dangerous AI by that time. And a slow-down doesn’t mean stopping. We could have much slower and more deliberate AI progress until we approach models that are starting to seem potentially dangerous, and then pause and take as long as we need to study these models, until we feel like we have the safety techniques needed to progress further.
- Existentially-concerned differential uplift. I’m surprised timelines are so short, and yet there actually seems to be much less general uplift from AI than I would expect, so this point seems especially weak. As AI grows dramatically more powerful, I still think it will be very important for existentially-concerned actors (AI safety people, EAs, rationalists, longtermists, etc.) to make the best use of AI. But as far as slowing down is concerned, it seems like what’s important is getting humanity overall more uplift from AI, by slowing down AI progress and lengthening the centaur period (when humans working with AI still beat AI alone), rather than trying to play galaxy brain chicken in order to give any specific group some differential “centaur advantage.”
- Safety-focused AI companies are leading. Because safety-focused AI companies are in the lead, it’s important that we get a coordinated international slow-down, or if not that, at least a coordinated US slow-down. But we are running out of time for the safety-focused AI companies’ lead to be sacrificed by a coordinated slow-down; they're likely going to have to slow down anyway within the next year or two on current timelines. Dario Amodei has made it very clear that he is in favor of building the machinery for pacing the frontier now, with Sam Altman and Elon Musk publicly agreeing, which suggests they all likely think this may be the best course of action soon.
- Saving people and near-term benefits. With powerful AI approaching this rapidly, it seems like, in expectation, we’re risking more lives by continuing AI at the current pace than we are saving by bringing forward longevity escape velocity, even from a person-affecting view. If alignment goes unexpectedly well, then we can still have relatively fast AI progress, even with an AI slow-down. I feel the heaviness of this argument, given loved ones I know who will die soon. But I don’t think it makes sense to risk everyone else’s lives and the entire future of humanity. I think we should go as fast as we safely can (even if that is very slowly) and differentially accelerate the benefits of AI over the risks as much as we can.
- Political situation. Unfortunately, it looks like AGI will quite possibly be here under the Trump administration and the CCP. But Democrats just won both chambers of Congress, and take control in January, which could at least make legislating AI somewhat easier. While it currently looks difficult to get a thoughtful slow-down, political winds may change as AI progress ramps up, and whether or not it’s easy, it looks like this may be more or less a necessity for getting safe AI.
What to do?
I think that anyone who wants to make the world a better place should seriously consider focusing on this. If the next 1-2 years were the last ever opportunity to end factory farming, solve global poverty, or prevent any existential risk other than AI, and there were a large chance that, if we didn’t solve it now, the problem would exist forever, or the world would end and prevent us from ever solving any problems again, then I think it would make sense for a very large fraction of the population to shift their focus to that.
I also think keeping good epistemics and being conscientious and socially thoughtful is really important here. It’s at least as important that we figure out how to get a thoughtful, well-coordinated slow-down as it is that we get a slow-down at all. For example, if the US unilaterally banned all AI research indefinitely, leading to China developing AGI instead, I think this would be obviously net-negative in expectation, because China is less likely to develop it safely, and even if it somehow were aligned, China would then be first to get world-dominating superhuman AI. There are probably a lot of other subtle ways for a pause or slow-down to actively do harm, so I think all aspects of work on this need to be very careful.
That is not to say that work on this should not also be extremely ambitious and urgent. I think it needs to be both. As I like to say, “move fast and fix things.”
I think section V of Scott Alexander’s “Against SURA” has a useful frame, and good concrete actions that people concerned with making AI go well should be putting massive effort into. I think that’s more useful to read than what I have to say. I’d encourage anyone to read that first if you want to make a difference on this.
I don’t know exactly what we should be doing about this, and I’m not an expert, so I’d appreciate any comments on what we should do to make it more likely we get a thoughtful, coordinated AI slow-down. Here are a few easy, and a few ambitious ideas:
- Write a post or make other content carefully laying out your thoughts on this. I think genuine emotions are great, but equally or more important is epistemic conscientiousness, and thinking about how we do this right: what agentic-and-ambitious-but-careful-and-thoughtful actions are most likely to increase the chances of getting a good coordinated AI slow-down, and minimize the chances of getting a misguided one.
- Then again, maybe the most important part is just overcoming the bystander effect, cutting through the miasma, and letting other people know you’re seriously concerned, to establish common knowledge that we need to work together to figure out what to do. So I think short "I'm concerned about this" posts could be useful as well.
- Talk to other thoughtful people. Someone suggested finding the most formidable person you remember from your undergrad and telling them about this. I think trying to spread the word in general is good, although some people will be more open to this message than others, and some will be more helpful than others, and it’s important to focus on those people.
- Create a repeating Pace AI meet-up, where people discuss and, more importantly, take action on the highest-leverage things they could be doing. Examples so far are the Pace AI Picnic series in San Francisco, and a 30-minute online call where people called their Congresspeople together and then discussed how to scale this up. Other kinds of community building or activism could be good here too.
- I think we should have two small armies: (1) a small, well-coordinated army of the most thoughtful people thinking about and getting clear on the best policies, and (2) a small army of people taking the best of those policy ideas and calling their representatives, voicing their concerns about the risks, and requesting the policies that would be most likely to be helpful. callcongress.ai makes the calling part easy by looking up your representatives and giving you a starter script.
- Try to set up a large initiative to get AI companies to sign a slow-down agreement. As Scott says: “By stop in a coordinated manner, I mean get them to sign a commitment saying that if every other major company in every country agrees to stop or slow by some definition, they’ll also stop or slow by that definition.” Senior people at most of the major AI companies have already backed coordinated pacing in spirit, but as Scott points out, nothing binding has come of it. I think what we need now is a binding version. And for the case where not all AI companies sign, Scott suggests a fallback: “We should get companies to once again consider legally binding weak merge-and-assist clauses. These could say that if a certain authority says we’re less than X months to a certain type of dangerous superintelligence, they will jointly stop training and focus on alignment, while sharing all of their alignment secrets.”
- The point at which they slow down could be some aspect of recursive self-improvement, or some predicted number of months from top-human-expert-dominating AI. I've heard numerous options thrown around and don't know how to make sense of them all. Perhaps we could have a public discussion of this. One of the EA Forum’s “debate weeks” could work, though it might make more sense to do this on LessWrong or the Alignment Forum. I want to see all the options fleshed out, and then people carefully debating all of the different pros and cons and subtle differences between agreement types, including what the slow-down or pause itself looks like. Then, once everything is completely fleshed out, people vote on each major aspect of what form of coordinated slow-down or pause seems to make the most sense. Then we should have a coordinated effort, with all of the top people banding together behind whatever the outcome of this process is, putting aside their differences and putting their collective force of credibility into asking AI companies to agree to it.
- Get more funding to slowing down AI and AI safety more broadly. According to Nan Ransohoff’s “Third Wave” post, as of May there was something like $370 billion in philanthropic assets tied to OpenAI and Anthropic, much of which may end up focused on making AI go well. In addition to slowing down AI, we also need to speed up alignment, safety, security, policy, and transparency work. Compared to timelines of 1-2 years and the amount of money coming in, our grantmaking is moving at a glacial pace. I have a lot more to say about this, which I’ll save for a separate post.
- Increase transparency in general. In particular, I think we need to better understand the evidence METR sees, and get more people looking at it. The AI Futures Project has urged Anthropic to release more of it, and Ajeya, Ryan, and Paul have all pointed out how much currently has to be taken on trust. I think work on this could include advocating for embedded evaluators at companies, advocacy for increased government capacity and regulation on this, and in general, more people trying to make sense of and model what's going on.
- Get the machinery ready now. The Pacing the Frontier statement itself says the world currently lacks the technical and governance tools to deliberately pace the frontier. Things like verification, compute monitoring, and figuring out how companies can legally coordinate without breaking antitrust law all take time. I think this could be high leverage work if you have the skills to help build any of this.
- Work on model security. If China won’t agree to a slow-down or pause, US companies may need to coordinate a US-only slow-down and burn their lead to stay ahead of China while doing safety work on advanced models. For this to work, it's essential that US companies have sufficient security to protect model weights and training secrets, a level of security AI companies currently fall far short of. Without such security, it seems like China could very likely steal the weights and secrets and race ahead with their own intelligence explosion with a much lower regard for safety. The AI Futures Project argues the US slowing down capabilities now and sprinting on security might actually leave the US with a bigger lead at higher capability levels. In “Against SURA”, Scott calls this argument the one case he’s heard for an immediate, unilateral-ish pause that makes sense. Paul Christiano makes a similar case in his Dwarkesh interview, which is one of the best recent arguments I’ve heard for this. This could mean that AI company security is another one of the highest-leverage areas for pacing AI.
- Create a “Pace AI Fellowship”. (I myself am working on a different fellowship, the Flourishing Futures Fellowship, which I think is neglected and made more urgent by short timelines. But if someone wants to launch something like this, I’d be happy to give advice.) This could be run by universities, BlueDot, or by a new organization. Fellows could do a couple weeks of readings to make sure they have a good understanding of the issue, then they could do several weeks of taking various kinds of actions from the above list, for example:
- Writing to or calling policy officials to let them know what’s happening and advocate for a slow-down or other policies
- Creating various kinds of media to inform the public and recruit more high-impact people to work on this
- Running protests and other events
- Further research on any of the above topics
- Coming up with their own ideas, using whatever their own personal superpower is to figure out how to push forward the likelihood of getting a slow-down
- Considering professional work on related areas that could be useful such as operations, technical verification, security, transparency, and grantmaking
~
I am scared. I have not publicly freaked out about AI risk until now. I want to accurately convey the emotions I’m feeling, and the fact that I feel they are fully appropriate. There’s a funny thing where waiting to freak out about AI seems to have a nice little benefit, so that when you finally do, it’s that much more credible to everyone else.
The human era may be on track to end in 1-2 years. I don’t think we can prevent the shift itself, but we can help ensure it’s navigated wisely, and I think the most critical part of that is just getting AI progress to slow down. There are lots of other important parts too, and I support work on all of those. But now is the time to pull the fire alarm, overcome the bystander effect, and let everyone know that this is real. This is not a drill. There is a very good chance that, on our current trajectory, top-expert-dominating-AI is 1 to 2 years away.
I should also say that I’m not sure about these timelines, but find them highly plausible and have a lot of respect for the people making these arguments. The risk is high enough that we should be more careful and many more of us should be trying our best to get an AI slow-down, so that if we are in a world where we need to slow down to make AI safe, we get the slow-down we need.