High-level hopes for AI alignment

In previous pieces, I argued that there's a real and large risk of AI systems' aiming to defeat all of humanity combined - and succeeding.

I first argued that this sort of catastrophe would be likely without specific countermeasures to prevent it. I then argued that countermeasures could be challenging, due to some key difficulties of AI safety research.

But while I think misalignment risk is serious and presents major challenges, I don’t agree with sentiments^[1] along the lines of “We haven’t figured out how to align an AI, so if transformative AI comes soon, we’re doomed.” Here I’m going to talk about some of my high-level hopes for how we might end up avoiding this risk.

I’ll first recap the challenge, using Ajeya Cotra’s young businessperson analogy to give a sense of some of the core difficulties. In a nutshell, once AI systems get capable enough, it could be hard to test whether they’re safe, because they might be able to deceive and manipulate us into getting the wrong read. Thus, trying to determine whether they’re safe might be something like “being an eight-year-old trying to decide between adult job candidates (some of whom are manipulative).”

I’ll then go through what I see as three key possibilities for navigating this situation:

• Digital neuroscience: perhaps we’ll be able to read (and/or even rewrite) the “digital brains” of AI systems, so that we can know (and change) what they’re “aiming” to do directly - rather than having to infer it from their behavior. (Perhaps the eight-year-old is a mind-reader, or even a young Professor X.)
• Limited AI: perhaps we can make AI systems safe by making them limited in various ways - e.g., by leaving certain kinds of information out of their training, designing them to be “myopic” (focused on short-run as opposed to long-run goals), or something along those lines. Maybe we can make “limited AI” that is nonetheless able to carry out particular helpful tasks - such as doing lots more research on how to achieve safety without the limitations. (Perhaps the eight-year-old can limit the authority or knowledge of their hire, and still get the company run successfully.)
• AI checks and balances: perhaps we’ll be able to employ some AI systems to critique, supervise, and even rewrite others. Even if no single AI system would be safe on its own, the right “checks and balances” setup could ensure that human interests win out. (Perhaps the eight-year-old is able to get the job candidates to evaluate and critique each other, such that all the eight-year-old needs to do is verify basic factual claims to know who the best candidate is.)

These are some of the main categories of hopes that are pretty easy to picture today. Further work on AI safety research might result in further ideas (and the above are not exhaustive - see my more detailed piece, posted to the Alignment Forum rather than Cold Takes, for more).

I’ll talk about both challenges and reasons for hope here. I think that for the most part, these hopes look much better if AI projects are moving cautiously rather than racing furiously.

I don’t think we’re at the point of having much sense of how the hopes and challenges net out; the best I can do at this point is to say: “I don’t currently have much sympathy for someone who’s highly confident that AI takeover would or would not happen (that is, for anyone who thinks the odds of AI takeover … are under 10% or over 90%).”

The challenge

This is all recapping previous pieces. If you remember them super well, skip to the next section.

In previous pieces, I argued that:

• The coming decades could see the development of AI systems that could automate - and dramatically speed up - scientific and technological advancement, getting us more quickly than most people imagine to a deeply unfamiliar future. (More: The Most Important Century)
• If we develop this sort of AI via ambitious use of the “black-box trial-and-error” common in AI development today, then there’s a substantial risk that:
  • These AIs will develop unintended aims (states of the world they make calculations and plans toward, as a chess-playing AI "aims" for checkmate);
  • These AIs will deceive, manipulate, and overpower humans as needed to achieve those aims;
  • Eventually, this could reach the point where AIs take over the world from humans entirely.
• People today are doing AI safety research to prevent this outcome, but such research has a number of deep difficulties:

An analogy that incorporates these challenges is Ajeya Cotra’s “young businessperson” analogy:

Imagine you are an eight-year-old whose parents left you a $1 trillion company and no trusted adult to serve as your guide to the world. You must hire a smart adult to run your company as CEO, handle your life the way that a parent would (e.g. decide your school, where you’ll live, when you need to go to the dentist), and administer your vast wealth (e.g. decide where you’ll invest your money).

You have to hire these grownups based on a work trial or interview you come up with -- you don't get to see any resumes, don't get to do reference checks, etc. Because you're so rich, tons of people apply for all sorts of reasons. (More)

If your applicants are a mix of "saints" (people who genuinely want to help), "sycophants" (people who just want to make you happy in the short run, even when this is to your long-term detriment) and "schemers" (people who want to siphon off your wealth and power for themselves), how do you - an eight-year-old - tell the difference?

Digital neuroscience

I’ve previously argued that it could be inherently difficult to measure whether AI systems are safe, for reasons such as: AI systems that are not deceptive probably look like AI systems that are so good at deception that they hide all evidence of it, in any way we can easily measure.

Unless we can “read their minds!”

Currently, today’s leading AI research is in the genre of “black-box trial-and-error.” An AI tries a task; it gets “encouragement” or “discouragement” based on whether it does the task well; it tweaks the wiring of its “digital brain” to improve next time; it improves at the task; but we humans aren’t able to make much sense of its “digital brain” or say much about its “thought process.”

Some AI research (example)^[2] is exploring how to change this - how to decode an AI system’s “digital brain.” This research is in relatively early stages - today, it can “decode” only parts of AI systems (or fully decode very small, deliberately simplified AI systems).

As AI systems advance, it might get harder to decode them - or easier, if we can start to use AI for help decoding AI, and/or change AI design techniques so that AI systems are less “black box”-ish.

I think there is a wide range of possibilities here, e.g.:

Failure: “digital brains” keep getting bigger, more complex, and harder to make sense of, and so “digital neuroscience” generally stays about as hard to learn from as human neuroscience. In this world, we wouldn’t have anything like “lie detection” for AI systems engaged in deceptive behavior.

Basic mind-reading: we’re able to get a handle on things like “whether an AI system is behaving deceptively, e.g. whether it has internal representations of ‘beliefs’ about the world that contradict its statements” and “whether an AI system is aiming to accomplish some strange goal we didn’t intend it to.”

• It may be hard to fix things like this by just continuing trial-and-error-based training (perhaps because we worry that AI systems are manipulating their own “digital brains” - see later bullet point).
• But we’d at least be able to get early warnings of potential problems, or early evidence that we don’t have a problem, and adjust our level of caution appropriately. This sort of mind-reading could also be helpful with AI checks and balances (below).

Advanced mind-reading: we’re able to understand an AI system’s “thought process” in detail (what observations and patterns are the main reasons it’s behaving as it is), understand how any worrying aspects of this “thought process” (such as unintended aims) came about, and make lots of small adjustments until we can verify that an AI system is free of unintended aims or deception.

Mind-writing (digital neurosurgery): we’re able to alter a “digital brain” directly, rather than just via the “trial-and-error” process discussed earlier.

One potential failure mode for digital neuroscience is if AI systems end up able to manipulate their own “digital brains.” This could lead “digital neuroscience” to have the same problem as other AI safety research: if we’re shutting down or negatively reinforcing AI systems that appear to have unsafe “aims” based on our “mind-reading,” we might end up selecting for AI systems whose “digital brains” only appear safe.

• This could be a real issue, especially if AI systems end up with far-beyond-human capabilities (more below).
• But naively, an AI system manipulating its own “digital brain” to appear safe seems quite a bit harder than simply behaving deceptively.

I should note that I’m lumping in much of the (hard-to-explain) research on the Eliciting Latent Knowledge (ELK) agenda under this category.^[3] The ELK agenda is largely^[4] about thinking through what kinds of “digital brain” patterns might be associated with honesty vs. deception, and trying to find some impossible-to-fake sign of honesty.

How likely is this to work? I think it’s very up-in-the-air right now. I’d say “digital neuroscience” is a young field, tackling a problem that may or may not prove tractable. If we have several decades before transformative AI, then I’d expect to at least succeed at “basic mind-reading,” whereas if we have less than a decade, I think that’s around 50/50. I think it’s less likely that we’ll succeed at some of the more ambitious goals, but definitely possible.

Limited AI

I previously discussed why AI systems could end up with “aims,” in the sense that they make calculations, choices and plans selected to reach a particular sort of state of the world. For example, chess-playing AIs “aim” for checkmate game states; a recommendation algorithm might “aim” for high customer engagement or satisfaction. I then argued that AI systems would do “whatever it takes” to get what they’re “aiming” at, even when this means deceiving and disempowering humans.

But AI systems won’t necessarily have the sorts of “aims” that risk trouble. Consider two different tasks you might “train” an AI to do, via trial-and-error (rewarding success at the task):

• “Write whatever code a particular human would write, if they were in your situation.”
• “Write whatever code accomplishes goal X [including coming up with things much better than a human could].”

The second of these seems like a recipe for having the sort of ambitious “aim” I’ve claimed is dangerous - it’s an open-ended invitation to do whatever leads to good performance on the goal. By contrast, the first is about imitating a particular human. It leaves a lot less scope for creative, unpredictable behavior and for having “ambitious” goals that lead to conflict with humans.

(For more on this distinction, see my discussion of process-based optimization, although I’m not thrilled with this and hope to write something better later.)

My guess is that in a competitive world, people will be able to get more done, faster, with something like the second approach. But:

• Maybe the first approach will work better at first, and/or AI developers will deliberately stick with the first approach as much as they can for safety reasons.
• And maybe that will be enough to build AI systems that can, themselves, do huge amounts of AI alignment research applicable to future, less limited systems. Or enough to build AI systems that can do other useful things, such as creating convincing demonstrations of the risks, patching security holes that dangerous AI systems would otherwise exploit, and more. (More on “how safe AIs can protect against dangerous AIs” in a future piece.)
• A risk that would remain: these AI systems might also be able to do huge amounts of research on making AIs bigger and more capable. So simply having “AI systems that can do alignment research” isn’t good enough by itself - we would need to then hope that the leading AI developers prioritize safety research rather than racing ahead with building more powerful systems, up until the point where they can make the more powerful systems safe.

There are a number of other ways in which we might “limit” AI systems to make them safe. One can imagine AI systems that are:

• “Short-sighted” or “myopic”: they might have “aims” (see previous post on what I mean by this term) that only apply to their short-run future. So an AI system might be aiming to gain more power, but only over the next few hours; such an AI system wouldn’t exhibit some of the behaviors I worry about, such as deceptively behaving in “safe” seeming ways in hopes of getting more power later.
• “Narrow”: they might have only a particular set of capabilities, so that e.g. they can help with AI alignment research but don’t understand human psychology and can’t deceive and manipulate humans.
• “Unambitious”: even if AI systems develop unintended aims, these might be aims they satisfy fairly easily, causing some strange behavior but not aiming to defeat all of humanity.

A further source of hope: even if such “limited” systems aren’t very powerful on their own, we might be able to amplify them by setting up combinations of AIs that work together on difficult tasks. For example:

• One “slow but deep” AI might do lots of analysis on every action it takes - for example, when it writes a line of code, it might consider hundreds of possibilities for that single line.
• Another “fast and shallow” AI might be trained to quickly, efficiently imitate the sorts of actions the “slow but deep” one takes - writing the sorts of lines of code it produces after considering hundreds of possibilities.
• Further AIs might be trained to summarize the analysis of other AIs, assign different parts of tasks to different AIs, etc. The result could be something like a “team” of AIs with different roles, such that a large number of limited AIs ends up quite a lot more powerful (and, depending on the details, also more dangerous) than any of the individual AIs.

I’d guess that in a competitive world, AI systems that are not “limited” will - at least eventually - be more powerful, versatile and ultimately useful. But limited AIs might get us pretty far.

How likely is this to work? I’d guess that we’ll eventually be able to build very powerful AIs whose limits make them relatively safe. However, I’d also guess that AIs without such limits will eventually be more powerful. So I think a lot of how things go will come down to how cautious we are: will we stick with limited AIs until the point at which we make more powerful AIs safe? And I think it’s very hard to predict how much caution the world will have - it partly depends on how well-understood the issues discussed in this series become over time!

AI checks and balances

Central to my worries is the idea that AIs could be good at “deceiving” humans: proficiently choosing courses of action that humans don’t fully understand, and don’t catch the problems with. This is important both for how AIs could develop unintended, “dangerous” aims in the first place and how they could execute on these aims by defeating humanity.

We could potentially mitigate this issue by using AIs to supervise and critique each other.

A simple example: say we’re worried that AI systems might find computer security vulnerabilities, and use them opportunistically to gain power and resources. We could train some AI systems to specifically seek out, expose and patch computer security vulnerabilities. (A footnote explains why we might expect such specialized systems to patch most vulnerabilities they find, rather than exploiting the vulnerabilities as often as less specialized systems would.^[5])

• This is a basic example of “using some AIs to identify potential problems other ones could cause.”
• A nice property: there might be some vulnerabilities that are much easier for AI systems to notice than humans; these are probably the ones we should be most worried about, and the ones most likely to be found by “vulnerability-finding AIs.”

Analogously, we could train AI systems to do things like:

• Produce critiques of each others’ proposed plans, and explanations of how these plans could lead to unintended behavior.
• Help out with the “digital neuroscience” goal above: analyze each others’ “digital brains” and produce compelling-to-humans discussions of whether there’s evidence there about whether the AI systems have unintended, dangerous “aims.”
• Conduct randomized audits of each others’ behavior: looking for any way in which another AI system’s behavior might be deceptive or otherwise unintended.
  • Importantly, auditor AIs would have different “incentives” (they get positive reinforcement for different behaviors), compared to the AIs they’re auditing: the audited AI system might be searching for any answer to a question that humans rate highly, whereas the “auditor” might be specifically searching for compelling arguments that an otherwise-highly-rated-answer is actually misleading.
  • I previously argued that these sorts of audits have limitations - they could end up unwittingly training the audited AI systems to become effective at deceiving human auditors. But in this case, deceiving auditors might be more like deceiving copies of themselves, potentially a harder-to-learn behavior.

There are a lot of potential wrinkles here, which I discuss in a more detailed non-Cold-Takes piece. In brief:

• AI checks and balances might fail if different AI systems end up coordinating with each other against us. For example, it might end up being the case that even auditor AI systems are best able to accomplish their ambitious, unintended aims by refraining from warning humans about the most important problems they find with other AIs.
• We might try to make this less likely by creating ensembles of AI systems that “check and balance” each other, while being different enough from each other to make coordination difficult.
  • For example, some AI systems might have limitations (see above) that make them less dangerous, while also less capable.
  • In order to pull off deception successfully, more powerful systems would then have to deceive not just humans but a whole variety of other (less capable, but in different ways) AI systems - some of which could be amplified, as discussed above.
• Another way AI checks and balances could fail: when one AI system critiques another, the resulting “debate” could quickly become too hard for humans to referee effectively. For example, imagine that an eight-year-old has asked adult A to look for signs that adult B is deceiving them, and adult A is now arguing that this is happening while adult B is arguing that it isn’t. Can the eight-year-old figure out what the truth is? Seems genuinely uncertain (and dependent on the details)!

There is already some research on “using AIs to critique each other.” A recent example is this paper, which actually does show that an AI trained to critique its own answers can surface helpful critiques that help humans rate its answers more accurately.

Other possibilities

I discuss possible hopes in more detail in an Alignment Forum piece. And I think there is significant scope for “unknown unknowns”: researchers working on AI safety might come up with approaches that nobody has thought of yet.

High-level fear: things get too weird, too fast

Rather than end on a positive note, I want to talk about a general dynamic that feels like it could make the situation very difficult, and make it hard for any of the above hopes to work out.

To quote from my previous piece:

Maybe at some point, AI systems will be able to do things like:

• Coordinate with each other incredibly well, such that it's hopeless to use one AI to help supervise another.
• Perfectly understand human thinking and behavior, and know exactly what words to say to make us do what they want - so just letting an AI send emails or write Tumblr posts gives it vast power over the world.
• Manipulate their own "digital brains," so that our attempts to "read their minds" backfire and mislead us.
• Reason about the world (that is, make plans to accomplish their aims) in completely different ways from humans, with concepts like "glooble" ^[6] that are incredibly useful ways of thinking about the world but that humans couldn't understand with centuries of effort.

At this point, whatever methods we've developed for making human-like AI systems safe, honest and restricted could fail - and silently, as such AI systems could go from "being honest and helpful" to "appearing honest and helpful, while setting up opportunities to defeat humanity."

I’m not wedded to any of the details above, but I think the general dynamic in which “AI systems get extremely powerful, strange, and hard to deal with very quickly” could happen for a few different reasons:

• The nature of AI development might just be such that we very quickly go from having very weak AI systems to having “superintelligent” ones. How likely this is has been debated a lot.^[7]
• Even if AI improves relatively slowly, we might initially have a lot of success with things like “AI checks and balances,” but continually make more and more capable AI systems - such that they eventually become extraordinarily capable and very “alien” to us, at which point previously-effective methods break down. (More)
• The most likely reason this would happen, in my view, is that we - humanity - choose to move too fast. It’s easy to envision a world in which everyone is in a furious race to develop more powerful AI systems than everyone else - focused on “competition” rather than “caution” (more on the distinction here) - and everything accelerates dramatically once we’re able to use AI systems to automate scientific and technological advancement.

So … is AI going to defeat humanity or is everything going to be fine?

I don’t know! There are a number of ways we might be fine, and a number of ways we might not be. I could easily see this century ending in humans defeated or in a glorious utopia. You could maybe even think of it as the most important century.

So far, I’ve mostly just talked about the technical challenges of AI alignment: why AI systems might end up misaligned, and how we might design them to avoid that outcome. In future pieces, I’ll go into a bit more depth on some of the political and strategic challenges (e.g., what AI companies and governments might do to reduce the risk of a furious race to deploy dangerous AI systems), and work my way toward the question: “What can we do today to improve the odds that things go well?”

Footnotes