This is a very informative article. The three stage model simplifies the risks for better understanding of what would happen. The alignment stage is a very crucial stage in AI development and deployment. All the risks in all the three stages are equally very catastrophic. We could never be 'ready for what is to come' but we could surely curb it at the alignment stage!
Thank you for the piece!
Executive summary: AI development involves risks at three key stages: training (misalignment), deployment (misuse), and diffusion (systemic issues). Competitive pressures exacerbate risks across all stages.
Key points:
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.
[Crossposted from my Substack, Unfolding Atlas]
How will AI get us in the end?
Maybe it’ll decide we’re getting in its way and decide to take us out? It could fire all the nukes and unleash all the viruses and take us all down at once
Or maybe we’ll take ourselves out? We could lose control of powerful autonomous weapons, or allow a 21st century Stalin set up an impenetrable surveillance state and obliterate freedom and progress forever.
Or maybe the diffusion of AI throughout our global economy will become a quiet catastrophe? As more and more tasks are delegated to AI systems, we mere humans could be left helpless, like horses after the invention of cars. Alive, perhaps, but bewildered by a world too complex and fast-paced to be understood or controlled.
Each of these scenarios has been proposed as a way the advent of advanced AI could cause a global catastrophe. But they seem quite different, and warrant different responses. In this post, I describe my model of how they fit together.[1]
I divide the AI development process into three steps. Risks arise at each step. Despite its simplicity, this model does a good job pulling all these risks into one framework. It’s helped me understand better how the many specific AI risks people have proposed fit together. More importantly, it satisfied my powerful, innate urge to force order onto a chaotic world.
The three AI development stages in my model are training, deployment, and diffusion.
At each stage, a different kind of AI risk arises. These are, respectively, misalignment, misuse, and systemic risks.
Throughout the entire process, competitive pressures act as a risk factor. More pressure makes risks throughout the process more likely.
Putting it all together looks like this:
This model is too simple to be perfect. For one, these risks almost certainly won’t arrive in sequence, as the model implies. They’re also more entangled than a linear model implies. But I think these shortcomings are relatively minor, and relating categories of risk like this gives them a pleasant cohesiveness.[2]
So let’s move ahead and look closer at the risks generated at each step.
The first set of risks emerge immediately after training an advanced AI model. Training modern AI models usually involves feeding them enormous datasets from which they can learn patterns and make predictions when given new data. Training risks relate to a model’s alignment. That is, what the system wants to do, how it plans to do it, and whether those goals and methods are good for people.
Some researchers worry that training a model to actually do what we want in all situations, or when we deploy it in the real world, is far from straightforward. In fact, we already see some kinds of alignment failures in the world today. These often seem silly. For example, something about the way Microsoft’s first Bing chatbot’s goal of being helpful and charming actually made it act like an occasionally funny, occasionally frightening psychopath.
As AI systems get more powerful, though, these misalignment risks could get a whole lot scarier. Specific risks researchers have raised include:
Misalignment scenarios often involve building AIs that pursue goals not related to human well-being. If such an AI was sufficiently powerful, it could take over and eliminate or disempower humanity as part of its plan to achieve its goals.
After training comes deployment.
Let’s assume that we can train aligned systems. We have powerful AIs that reliably do what we want. We still won’t be in the clear. The next challenge we’ll face is misuse risks: how will we ensure everyone, everywhere, uses these systems responsibly?
AI systems will probably be applied to a vast range of problems. Most of those applications will be great, helping us solve complex problems and provide amazing goods and services, including things we can’t even imagine today.
But people could also use them to cause harm. And the power of these systems mean that they could be used to do a lot of harm. Such misuse risks may take several different forms:
This list probably isn’t exhaustive. We’re still learning what large language models alone are capable of. As we train more powerful and more diverse AIs, we’ll learn a lot about these systems can do. We’ll discover more ways to use AIs — and more ways to misuse them.
Finally, let’s assume that we’ve managed to train AI systems that basically do what we want, and that we’ve built defenses to stop people from using them to do massive harm. We’ll still face a challenging transition to a world in which AI systems play an important role in our global systems.
This transition could be both extremely dramatic and extremely quick. Think a century of progress occurring in just a few months or years. We could majorly screw it up. The risks here seem more subtle, but potentially just as serious as the more obvious disasters above.
You don’t have to make crazy assumptions to think that a world with widespread superintelligent AIs will be very strange. Whether it will also be very good is more uncertain.
A skeptical critic might ask why companies would release misaligned AI systems or allow customers to misuse them, or why governments would allow systemic risks to arise. These are fair questions. I do expect to see these organizations make strong technical and legal efforts to avoid these worst-case scenarios.[3]
However, it’s also worth recognizing how competition will shape incentives at each stage of the development process.
AI companies will face strong incentives to develop powerful models before their competitors. Evaluating models, especially large, complex ones, for dangerous capabilities will take time. There will be strong pressure to speed up or shorten the evaluation process to avoid being beaten to the market by other companies.
In fact, we’ve already seen this play out. The creepy AI-powered Bing search I mentioned earlier was rushed out by Microsoft after they saw how much hype OpenAI’s ChatGPT was garnering. As AI systems get more powerful, their economic value will also increase, perhaps further increasing these pressures.
The same dynamics could apply to misuse risks. Companies may avoid building explicitly-harmful AIs. But AI systems can be applied to a wide range of problems. Tuning them to filter out harmful outputs could take time - time in which a competitor could beat them to market.
Finally, we saw how competitive dynamics can drive systemic risks as various actors seek an edge by handing more control over to AI systems.
None of these risks seems inevitable. All can be avoided by combining technical solutions and regulatory coordination. But those solutions won’t be created and implemented automatically. Designing and implementing responsible policies at the firm, national, and international level will be a lot of work. Meanwhile, AI progress will, by default, continue. The path of least resistance is a treacherous one.
Modeling AI development as a three-stage process — training, deployment, and societal diffusion — has helped me understand how the different AI risks people have proposed fit together.
It’s too simple to model AI development perfectly. For one, it’s too linear; we could face catastrophic misuse risks before catastrophic misalignment risks if less powerful systems can be used for nefarious purposes. Or we may face risks from the different categories simultaneously. And in some cases the boundaries are a bit blurry. The ways a misaligned AI could wreak havoc could be the same ways a person could misuse AI systems: like attacking digital infrastructure or unleashing powerful weapons.
On the whole, though, this model stitches together what otherwise seem like quite different risk pathways. I’m not aware of any major AI risk proposals that aren’t mentioned here or couldn’t fit into this model — though if there are I’d love to hear about them!
I’m only focusing on global, generational risks. These are clearly not the only important effects of advanced AI systems. There are other kinds of risks: algorithmic biases, supporting spam and scam artists, economic upheaval. And there will, of course, also be many benefits: scientific discoveries, better decision-making, freedom from boring administrative and logistics tasks, more efficient manufacturing. Although this post is a bit of a downer, I think the upside is large. Once we know how to safely develop these systems we probably should.
Other classification schemes include Hendrycks, Mazeika, Woodside’s “An Overview of Catastrophic AI Risks” and Critch and Russell’s “TASRA: a Taxonomy and Analysis of Societal-Scale Risks from AI”.
Stefan Schubert has called the tendency to ignore potential adjustments in response to future risks sleepwalk bias.
You may have also seen Sam Clarke's classification of AI x-risk sources, just sharing for others :)
Wei Dai and Daniel Kokotajlo's older longlist might worth perusing too?
Yes; it could be useful if Stephen briefly explained how his classification relates to other classifications. (And which advantages it has - I guess simplicity is one.)