(Note: This post is probably old news for most readers here, but I find myself repeating this surprisingly often in conversation, so I decided to turn it into a post.)
I don't usually go around saying that I care about AI "safety". I go around saying that I care about "alignment" (although that word is slowly sliding backwards on the semantic treadmill, and I may need a new one soon).
But people often describe me as an “AI safety” researcher to others. This seems like a mistake to me, since it's treating one part of the problem (making an AGI "safe") as though it were the whole problem, and since “AI safety” is often misunderstood as meaning “we win if we can build a useless-but-safe AGI”, or “safety means never having to take on any risks”.
Following Eliezer, I think of an AGI as "safe" if deploying it carries no more than a 50% chance of killing more than a billion people:
When I say that alignment is difficult, I mean that in practice, using the techniques we actually have, "please don't disassemble literally everyone with probability roughly 1" is an overly large ask that we are not on course to get. [...] Practically all of the difficulty is in getting to "less than certainty of killing literally everyone". Trolley problems are not an interesting subproblem in all of this; if there are any survivors, you solved alignment. At this point, I no longer care how it works, I don't care how you got there, I am cause-agnostic about whatever methodology you used, all I am looking at is prospective results, all I want is that we have justifiable cause to believe of a pivotally useful AGI 'this will not kill literally everyone'.
Notably absent from this definition is any notion of “certainty” or "proof". I doubt we're going to be able to prove much about the relevant AI systems, and pushing for proofs does not seem to me to be a particularly fruitful approach (and never has; the idea that this was a key part of MIRI’s strategy is a common misconception about MIRI).
On my models, making an AGI "safe" in this sense is a bit like finding a probabilistic circuit: if some probabilistic circuit gives you the right answer with 51% probability, then it's probably not that hard to drive the success probability significantly higher than that.
If anyone can deploy an AGI that is less than 50% likely to kill more than a billion people, then they've probably... well, they've probably found a way to keep their AGI weak enough that it isn’t very useful. But if they can do that with an AGI capable of ending the acute risk period, then they've probably solved most of the alignment problem. Meaning that it should be easy to drive the probability of disaster dramatically lower.
The condition that the AI actually be useful for pivotal acts is an important one. We can already build AI systems that are “safe” in the sense that they won’t destroy the world. The hard part is creating a system that is safe and relevant.
Another concern with the term “safety” (in anything like the colloquial sense) is that the sort of people who use it often endorse the "precautionary principle" or other such nonsense that advocates never taking on risks even when the benefits clearly dominate.
In ordinary engineering, we recognize that safety isn’t infinitely more important than everything else. The goal here is not "prevent all harms from AI", the goal here is "let's use AI to produce long-term near-optimal outcomes (without slaughtering literally everybody as a side-effect)".
Currently, what I expect to happen is that humanity destroys itself with misaligned AGI. And I think we’re nowhere near knowing how to avoid that outcome. So the threat of “unsafe” AI indeed looms extremely large—indeed, this seems to be rather understating the point!—and I endorse researchers doing less capabilities work and publishing less, in the hope that this gives humanity enough time to figure out how to do alignment before it’s too late.
But I view this strategic situation as part of the larger project “cause AI to produce optimal long-term outcomes”. I continue to think it's critically important for humanity to build superintelligences eventually, because whether or not the vast resources of the universe are put towards something wonderful depends on the quality and quantity of cognition that is put to this task.
If using the label “AI safety” for this problem causes us to confuse a proxy goal (“safety”) for the actual goal “things go great in the long run”, then we should ditch the label. And likewise, we should ditch the term if it causes researchers to mistake a hard problem (“build an AGI that can safely end the acute risk period and give humanity breathing-room to make things go great in the long run”) for a far easier one (“build a safe-but-useless AI that I can argue counts as an ‘AGI’”).
My reply to Critch is here, and Eliezer's is here and here.
I'd also point to Scott Alexander's comment, Nate's "Don't leave your fingerprints on the future", and my:
What, concretely, do you think humanity should do as an alternative to "build an AI that enacts some pivotal act ensuring that nobody ever builds a misaligned AGI"? If you aren't sure, then what's an example of an approach that seems relatively promising to you? What's a concrete scenario where you imagine things going well in the long run?
To sharpen the question: Eventually, as compute becomes more available and AGI techniques become more efficient, we should expect that individual consumers will be able to train an AGI that destroys the world using the amount of compute on a mass-marketed personal computer. (If the world wasn't already destroyed before that.) What's the likeliest way you expect this outcome to be prevented, or (if you don't think it ought to be prevented, or don't think it's preventable) the likeliest way you expect things to go well if this outcome isn't prevented?
(If your answer is "I think this will never happen no matter how far human technology advances" and "in particular, the probability seems low enough to me that we should just write off those worlds and be willing to die in them, in exchange for better focusing on the more-likely world where [scenario] is true instead", then I'd encourage saying that explicitly.)
At that level of abstraction, I'd agree! Dan defines robustness as "create models that are resilient to adversaries, unusual situations, and Black Swan events", monitoring as "detect malicious use, monitor predictions, and discover unexpected model functionality", alignment as "build models that represent and safely optimize hard-to-specify human values", and systemic safety as "use ML to address broader risks to how ML systems are handled, such as cyberattacks". All of those seem required for a successful AGI-mediated pivotal act.
If this description is meant to point at a specific alternative approach, or meant to exclude pivotal acts in some way, then I'm not sure what you have in mind.
I agree on both fronts. Destroying the world is insufficient (you need to save the world; we already know how to build AI systems that don't destroy the world), and a pivotal act fails if it merely delays doom, rather than indefinitely putting a pause on AGI proliferation (an "ongoing process", albeit one initiated by a fast discrete action to ensure no one destroys the world tomorrow).
But I think you mean to gesture at some class of scenarios where the "ongoing process" doesn't begin with a sudden discrete phase shift, and more broadly where no single actor ever uses AI to do anything sudden and important in the future. What's a high-level description of how this might realistically play out?
You linked to the same Hendrycks paper twice; is there another one you wanted to point at? And, is there a particular part of the paper(s) you especially wanted to highlight?