Debate series: should we push for a pause on the development of AI?

Will someone write about the symbolic importance of the ask for a pause?

Right now, most of what has been written here on this seems focused on the techno-economics as if a pause were only about slowing down a technical process.

Asking for a pause is also a signal that one is extremely serious about the risk (and by the same token, not asking for a pause but speaking about existential risks seems very hard to communicate to a broader public).

It's definitely good to think about whether a pause is a good idea. Together with Joep from PauseAI, I wrote down my thoughts on the topic here.

Since then, I have been thinking a bit on the pause and comparing it to a more frequently mentioned option, namely to apply model evaluations (evals) to see how dangerous a model is after training.

I think the difference between the supposedly more reasonable approach of evals and the supposedly more radical approach of a pause is actually smaller than it seems. Evals aim to detect dangerous capabilities. What will need to happen when those evals find that, indeed, a model has developed such capabilities? Then we'll need to implement a pause. Evals or a pause is mostly a choice about timing, not a fundamentally different approach.

With evals, however, we'll move precisely to the brink, look straight into the abyss, and then we plan to halt at the last possible moment. Unfortunately, though, we're in thick mist and we can't see the abyss (this is true even when we apply evals, since we don't know which capabilities will prove existentially dangerous, and since an existential event may already occur before running the evals).

And even if we would know where to halt: we'll need to make sure that the leading labs will practically succeed in pausing themselves (there may be thousands of people working there), that the models aren't getting leaked, that we'll implement the policy that's needed, that we'll sign international agreements, and that we gain support from the general public. This is all difficult work that will realistically take time.

Pausing isn't as simple as pressing a button, it's a social process. No one knowns how long that process of getting everyone on the same page will take, but it could be quite a while. Is it wise to start that process at the last possible moment, namely when the evals turn red? I don't think so. The sooner we start, the higher our chance of survival.

Also, there's a separate point that I think is not sufficiently addressed yet: we don't know how to implement a pause beyond a few years duration. If hardware and algorithms improve, frontier models could democratize. While I believe this problem can be solved by international (peaceful) regulation, I also think this will be hard and we will need good plans (hardware or data regulation proposals) for how to do this in advance. We currently don't have these, so I think working on them should be a much higher priority.

I'm really happy to see this happening.

In fact, I'd like to see more things along these kinds of lines.

While there's a lot of good discussion on this forum, we aren't always going to end up discussing the most important topics organically. So I think it's often helpful for CEA/the mods to occasionally direct the attention of the forum towards the discussion topics that will move us forward as a community.

If we wanted to go beyond this, then I think it would be quite valuable to find two people with opposite views to work together in order to produce a high-quality distillation of any such debates.

Thanks for noticing something you thought should happen (or having it flagged to you) and making it happen!

Love this idea, thanks for organizing this.

PSA: the term "compute overhang" or "hardware overhang" has been used in many ways. Today it seems to most often (but far from always) mean amount labs can quickly scale up the size of the largest training run (especially because a ban on large training runs ends). When you see it or use it, make sure everyone knows what it means.

(It will come up often in this debate.)

Any thoughts on using https://www.kialo.com/ or a similar tool specailized on debate?

Great initiative! 

Perhaps it'd be good to ask people who are doing some public-facing campaigning to contribute? For example, the team at the Existential Risk Observatory or those behind PauseAI. I might be wrong, but I don't think anyone on the list of agreed contributors represents that specific theory of change. 

I think a public-facing campaign is important to think about if we want to reduce the likelihood of articles such as 'How Silicon Valley doomers are shaping Rishi Sunak’s AI plans' being written.

Great idea, thanks for organizing!

This is a great idea, and I look forward to reading the diverse views on the wisdom of an AI pause.

I do hope that the authors contributing to this discussion take seriously the idea that an 'AI pause' doesn't need to be fully formalizable at a political, legal, or regulatory level. Rather, its main power can come from promoting an informal social consensus about the serious risks of AGI development, among the general public, journalists, politicians, and the more responsible people in the AI industry. 

In other words, the 'Pause AI' campaign might get most of its actual power and influence from helping to morally stigmatize reckless AI development, as I argued here.  

Thus, the people who argue that pausing AI isn't feasible, or realistic, or legal, or practical, may be missing the point. 'Pause AI' can function as a Schelling point, or focal point, or coordination mechanism, or whatever you want to call it, with respect to public discourse about the ethics of AI development.

I'm a long-time lurker but I registered to make this comment: While these posts are incredibly high quality, they are legally naive. There is no mention of the First Amendment or Bernstein v. Department of Justice, which is a significant gap. Yes, let's have the discussion about pausing AI and part of that should include its legality. 

I include links to my two old posts arguing for keeping AI development:

First I argued that AI is a necesary (almost irrepleaceable) tool to deal with the other existential risks (mainly nuclear war):

https://forum.effectivealtruism.org/posts/6j6qgNa3uGmzJEMoN/artificial-intelligence-as-exit-strategy-from-the-age-of

Then, that currently AI risk is simply "too low to be measured", and we need to be closer to AGI to develop realistic alignment work:

https://forum.effectivealtruism.org/posts/uHeeE5d96TKowTzjA/world-and-mind-in-artificial-intelligence-arguments-against

I think the idea of debates like this is great, good work team!

Is there any plan to expand this to non-AI cause areas?

So the final post will be released on September 27th?

Regardless of whether a pause has benefits, can someone succinctly explain how a pause could even be possible? Nuclear arms pauses never meaningfully happened - SALT treaties were decades later and didn't meaningfully reduce the "nuclear apocalypse" doom risk, at least for citizens living in targeted countries. (Maybe it reduced worldwide risks)

A nuclear bombardment is a quantifiable, object level thing. You can use a special tool and draw circles on a paper map of your own cities and wargame out where the enemy will allocate their missiles if their goal is maximum destruction. Multiply a series probability estimate of how likely the warhead will reach the target and give ful yield and add up the estimated damage.

The "we must pause" threat of AI doom I have seen so far doesn't have any data of any threat. And a frequent threat model is the "treacherous turn" - you wouldn't be able to quantify how risky the current AI systems you have at any point or their capacity for damage in numbers because the AI systems are hiding their motives and capabilities until they see a route to victory.

With no data of a threat how could any policymakers agree to a pause?

The other examples of human technologies we did agree to not pursue all exist and the threats are concrete. The damage from nerve gas, CFCs, bio warfare, and human genetic engineering we have data on.

As a side note the teacherous turn threat model has an obvious technical mitigation measure : use the least compute on any task by selecting from a pool of models the most efficient one. This discourages models with the cognitive capacity for sedition or hidden capabilities as they will need more memory and compute to run.

What i've been hearing from alot of people is one of the biggest concerns they have about AI is that it could possibly fake everything, but then the question is, wouldn't that better society because now everyone would assume everything onnline is fake? 

We should push for a stop to even mentioning AI in these forums. There are exponentially more higher risk issues at stake for humanity... the world is a matchbox and you're all giving it to kids to play with. No matter what you do or say, someone will work on AI somewhere. All it takes is 1 villain, and they have an army of programmers under them, because it's just that easy to be bad today, and to make others be bad.

If you invested all your time on this subject, and somehow got 75% of the world to stop working on AI, 25% will still do it. And you know for sure more than half of them will be bad actors so what have you gained?

There is so much corruption in so many other ways imaginable, so much exploitation, where countries get away with it without the UN saying a word. There are still a billion of people living in poverty, tens of thousands dying every day, and you could have stopped it by stopping corruption but you are too busy thinking about AI.