I agree with David Manheim's post at a high level. I especially agree that a pause on large training runs is needed, that “We absolutely cannot delay responding”, and that we should be focusing on a pause mediated by “a multilateral agreement centered on countries and international corporations”. I also agree that if we can’t respond to the fire today, we should at least be moving fast to get a “sprinkler system”.
The basic reason we need a (long) pause, from my perspective, is that we are radically unprepared on a technical level for smarter-than-human AI. We have little notion of how to make such systems reliable or safe, and we’ll predictably have very little time to figure this out once smarter-than-human AI is here, before the technology proliferates and causes human extinction.
We need far, far more time to begin building up an alignment field and to develop less opaque approaches to AI, if we’re to have a realistic chance of surviving the transition to smarter-than-human AI systems.
My take on AI risk is similar to Eliezer Yudkowsky’s, as expressed in his piece in TIME and in the policy agenda he outlined. I think we should be placing more focus on the human extinction and disempowerment risks posed by AGI, and should be putting a heavy focus on the arguments for that position and the reasonably widespread extinction fears among ML professionals.
I have disagreements with some of the specific statements in the post, though in many cases I’m unsure of exactly what Manheim’s view is, so the disagreement might turn out to be non-substantive. In the interest of checking my understanding and laying out a few more of my views for discussion, I’ll respond to these below.[1]
So the question of whether to stop and how to do so depends on the details of the proposal - but these seem absent from most of the discussion.
This is not apparent to me. I think it would take a pretty unusual proposal in order for me to prefer the status quo over it, assuming the proposal actually pauses progress toward smarter-than-human AI.
It’s important to get this right, and the details matter. But if a proposal would actually work then I’m not picky about the additional implementation details, because there’s an awful lot at stake, and “actually working” is already an extremely high bar.
An immediate, temporary pause isn’t currently possible to monitor, much less enforce, even if it were likely that some or most parties would agree.
A voluntary and temporary moratorium still seems like an obviously good idea to me; it just doesn't go far enough, on its own, to macroscopically increase our odds of surviving AGI. But small movements in the right direction are still worthwhile.
Similarly, a single company, or country announcing a unilateral halt to building advanced models is not credible without assurances,
“Not credible” sounds too strong here, though maybe I’m misunderstanding your claim. Scientists have voluntarily imposed restrictions on their own research in the past (e.g., Asilomar), and I don’t think this led to widespread deception. Countries have banned dangerous-but-profitable inventions without pursuing those inventions in secret.
I don't think it would be that hard for many companies or countries to convince me that they're not building advanced models. It might be hard for me to (for example) get to 95% confidence that DeepMind has suspended frontier AI development, merely on DeepMind's say-so; but 75% confidence seems fairly easy to me, if their say-so is concrete and detailed enough.
(Obviously some people will pursue such research in secret, somewhere in the world, given the opportunity. If we rely purely on organizations’ say-so, then eventually this will get us killed. If that’s all you mean by “not credible”, then I agree.)
and is likely both ineffective at addressing the broader race dynamics
If current industry leaders suspended operations, then this would “address the broader race dynamics” in the sense that it would be a very positive step in the right direction. This could potentially buy us years of additional time to develop and implement a global, internationally enforced pause.
It doesn't "address the broader race dynamics" in the sense of instantly saving the world, though. A few years (or even months) of delay could prove decisive, but if so its decisiveness will certainly hinge on whether the world uses that extra time to implement a pause.
and differentially advantages the least responsible actors.
To a first approximation, I don't think this matters. I don't think the future looks any brighter if the most responsible orgs develop AGI first than if the least responsible ones do.
The most responsible orgs might successfully avoid destroying the world themselves — while not being able to safely utilize AGI to address the proliferation of AGI tech.
But in that case they're not helping the world any more than they would have by just shutting down, which is a fool-proof way to not destroy the world yourself.
What Does a Moratorium Include?
There is at least widespread agreement on many things that aren’t and wouldn’t be included. Current systems aren’t going to be withdrawn - any ban would be targeted to systems more dangerous than those that exist.
“Targeted” maybe suggests more precision than may be possible. It’s very hard to predict in advance which systems will be existentially dangerous, and algorithmic progress means that a given compute threshold may be very-likely-safe today while being plausibly unsafe tomorrow.
Regarding rolling back current systems: I think some people at Conjecture have given arguments for rolling back GPT-4, on the basis that we don't yet know what scaffolding we can get out of GPT-4, nor what dangerous insights can be learned by gaining a better grasp of how GPT-4 works internally. This doesn’t seem important enough to me to make it a focus, but a rollback does seem like the kind of policy that would be adopted (or at least be under serious consideration) in a generally well-run world that was seriously grappling with the risk that a GPT-5 or a GPT-8 might get us all killed.
Regardless, if Conjecture staff’s views are relevant then it can’t be said that there’s full consensus here.
The thing I care more about is leaving open that it might be necessary to ban systems at the same scale as GPT-4 at a future date; we can expect algorithms to get more efficient in the future, and it’s hard to predict what will be technologically possible multiple years in the future, which is an argument for some conservatism (with everyone’s lives at risk).
We’re not talking about banning academic research using current models, and no ban would stop research to make future systems safer, assuming that the research itself does not involve building dangerous systems.
If we end up solving the alignment problem at all, then I expect some alignment research to eventually yield significant capabilities insights. (See Nate Soares’ If interpretability research goes well, it may get dangerous.)
On the current margin, I think it's net-positive to pursue the most promising alignment research moonshots; but in the long run we’d definitely need to be asking about how capabilities-synergistic different alignment research directions are, rather than giving a permanent free pass to all research that’s useful for alignment. And I think we should definitely be preparing for that now, rather than treating algorithmic progress as nonexistent or alignment and capabilities research as disjoint categories.
However, I don’t think there’s a concrete proposal to temporarily or permanently pause that I could support - we don’t have clear criteria, we don’t have buy-in from the actors that is needed to make this work, and we don’t have a reasonable way to monitor, much less enforce, any agreement.
As a rule, I don't think it's a good idea to withhold support for policies on the basis that they lack "buy-in" from others. The general policy "only support things once lots of others have publicly supported them" often prevents good ideas from beginning to gain traction, and locks Overton windows in place. I'd instead usually advise people to state their actual beliefs and their rough preference ordering over policy options (including unrealistic-but-great ones). Then we can talk about feasibility and compromise from a standpoint of understanding everyone's actual views.
Part of why I recommend this is that I think any policy that prevents human extinction will need to be pretty extreme and novel. If we limit ourselves to what's obviously politically feasible today, then I think we're preemptively choosing death; we need to take some risks and get more ambitious in order to have any shot at all.
(This is not to say that all small incremental progress is useless, or that everything needs to happen overnight. But a major part of how smaller marginal progress gets parlayed into sufficient progress is via individuals continuously discussing what they think is needed even though it's currently outside the Overton window, throughout the process of iterating and building on past successes.)
Yes, companies could voluntarily pause AI development for 6 months, which could be a valuable signal.
It could also slow progress toward smarter-than-human AI for some number of months, which is useful in its own right. Time is needed to implement effective policy responses, and even more time would be needed to find a solution to the alignment problem.
(Or would be so if we didn’t think it would be a smokescreen for 'keep doing everything and delay releases slightly.')"
It sounds like you're much more cynical about this than I am? I'd be very happy to hear concrete commitments from ML organizations to pause development, and I think they should be encouraged to do so, even though it's not sufficient on its own.
Lying happens, but I don't think it's universal, especially when it would require a conspiracy between large numbers of people to cover up a very clear and concrete public lie. (Obviously if the stated commitment is very vague or clearly insufficient, then that’s another story.)
And acting too soon is costly
Acting too late destroys all of the value in the future. Is there a commensurate cost to acting too quickly? (I’ll assume for now that you don't think there is one, and are just acknowledging nonzero cost.)
Just like a fire in the basement won’t yet burn people in the attic, AI that exists today does not pose immediate existential risks[2] to humanity - but it’s doing significant damage already, and if you ignore the growing risks, further damage quickly becomes unavoidable.
This seems like a weak case for acting now, since it's vulnerable to the obvious response "AI today is doing significant damage, but also producing significant benefits, which very likely outweigh the damage."
The real reason to act now is that future systems will likely disempower and kill humanity, we don't know the threshold at which that will happen (but there's a fair bit of probability on e.g. 'the next 5 years', and quite a lot on 'the next 15 years'), and it may take years of work to develop and implement an adequate policy response.
This post is part of AI Pause Debate Week. Please see this sequence for other posts in the debate.
I find myself agreeing with Nora on temporary pauses - and I don't really understand the model by which a 6-month, or a 2-year, pause helps, unless you think we're less than 6 months, or 2-years, from doom.
First, my perception is that progress in AI so far has been combining various advances into very large models. If the companies are working on other things in ML during those 6 months, this creates algorithmic overhang as soon as they put them together into larger models. This is separate from hardware overhang, which I think is even more concerning.
Second, there are lots of parts to putting these models together. If companies are confident the pause is 6 months long, they continue to build infrastructure and curate datasets. (This is in contrast to the status if we're building comprehensive regulatory oversight, where training the planned future large models may not be approved, and further capital investments might be wasted.)
After writing this, another model occurs to me that someone might think makes a short pause useful - if we are playing a PR game, and think that the sudden advances after the pause will galvanize the public into worrying more. (But I don't think that you're proposing playing clever-sounding but fragile strategic moves, and think this type of pause would obviously be worse than trying for a useful governance regime.)
Edit: If the "(long) pause" you're suggesting is actually an indefinite moratorium on larger models, I think we're agreeing - but I think we need to build a global governance regime to make that happen, as I laid out.
This doesn't make a lot of sense to me. If we're 3 years away from doom, I should oppose a 2-year pause because of the risk that (a) it might not work and (b) it will make progress more discontinuous?
In real life, if smarter-than-human AI is coming that soon then we're almost certainly dead. More discontinuity implies more alignment difficulty, but on three-year timelines we have no prospect of figuring out alignment either way; realistically, it doesn't matter whether the curve we're looking at is continuous vs. discontinuous when the absolute amount of calendar time to solve all of alignment for superhuman AI systems is 3 years, starting from today.
I don't think "figure out how to get a god to do exactly what you want, using the standard current ML toolbox, under extreme time pressure" is a workable path to humanity surviving the transition to AGI. "Governments instituting and enforcing a global multi-decade pause, giving us time to get our ducks in order" does strike me as a workable path to surviving, and it seems fine to marginally increase the intractability of unworkable plans in exchange for marginally increasing the viability of plans that might work.
If a "2-year" pause really only buys you six months, then that's still six months more time to try to get governments to step in.
If a "2-year" pause buys you zero time in expectation, and doesn't help establish precedents like "now that at least one pause has occurred, more ambitious pauses are in the Overton window and have some chance of occurring", then sure, 2-year moratoriums are useless; but I don't buy that at all.
(Edit to add: The below is operating entirely in the world where we don't get an indefinite moratorium initially. I strongly agree about the preferability of an idefinite governance regime, though I think that during a multi-year pause with review mechanisms we'll get additional evidence that either safety is possible, and find a path, or conclude that we need a much longer time, or it's not possible at all.)
If you grant that a pause increases danger by reducing the ability of society and safety researchers to respond, and you don't think doom is very, very likely even with extreme effort, then it's reasonable that we would prefer, say, a 50% probability of success controlling AI given 3 years over a 10% probability of success given a 2-year pause then only 18 months. Of course, if you're 99.95% sure that we're doomed given 3 years, it makes sense to me that the extra 6 months of survival moving the probability to 99.99% would seem more worth it. But I don't understand how anyone gets that degree of confidence making predictions. (Superforecasters who have really fantastic predictive accuracy and calibration tend to laugh at claims like that.)
That said, I strongly agree that this isn't an acceptable bet to make. We should not let anyone play Russian roulette with all of humanity, and even if you think it's only a 0.05% probability of doom (again, people seem very obviously overconfident about their guesses about the future,) that seems like a reason to insist that other people get to check your work in saying the system is safe.
Finally, I don't think that you can buy time to get governments to step in quite the way you're suggesting, after a pause. That is, if we get a pause that then expires, we are going to need tons of marginal evidence after that point to get an even stronger response, even once it's in the Overton window. But the evidence we'd need is presumably not showing up, or not showing up as quickly, because there isn't as much progress. So either the pause is extended indefinitely without further evidence, or we'll see a capabilities jump, and that increases risks.
And once we see the capabilities jump after a pause expires, it seems plausible that any stronger response will be far too slow. (It might be OK, they might re-implement the pause, but I don't see a reason for confidence in their ability or willingness to do so.) And in general, unless there are already plans in place that they can just execute, governments react on timescales measured in years.
(Note for everyone reading that all of this assumes, as I do, that the risk is large and will become more obvious as time progresses and we see capabilities continue to outpace reliable safety. If safety gets solved during a pause, I guess it was worth it, or maybe even unnecessary. But I'm incredibly skeptical.)