Marius Hobbhahn on the race to solve AI scheming before models go superhuman

By Robert Wiblin  |    Watch on Youtube   |   Listen on Spotify   |    Read transcript

Episode summary

We often worry about AI models “hallucinating” or making honest mistakes. But what happens when a model knows the truth, but decides to deceive you anyway to achieve a goal of its own? This isn’t sci-fi — it’s happening regularly in deployment today.

Marius Hobbhahn, CEO of the world’s top research organisation focused on AI deception (Apollo Research), has been collaborating with OpenAI to figure out when OpenAI’s reasoning models like o3 are inclined to “scheme” against the interests of users.

In a striking case uncovered by Apollo, when many AI models were told they would have capabilities removed if they performed too well on a test, they successfully “sandbagged” — intentionally answering questions incorrectly to appear less capable than they were, while also being careful not to perform so poorly it would arouse suspicion.

These models had somehow developed a preference to preserve their own capabilities, despite never being trained in that goal or assigned a task that called for it.

This doesn’t cause significant risk now, but as AI models become more general, superhuman in more areas, and are given more decision-making power, it could become outright dangerous.

In today’s episode, Marius details his recent collaboration with OpenAI to train o3 to follow principles like “never lie,” even when placed in “high-pressure” situations where it would otherwise make sense.

The good news: They reduced “covert rule violations” (scheming) by about 97%.

The bad news: In the remaining 3% of cases, the models sometimes became more sophisticated — making up new principles to justify their lying, or realising they were in a test environment and deciding to play along until the coast was clear.

Marius argues that while we can patch specific behaviours, we might be entering a “cat-and-mouse game” where models are becoming more situationally aware — that is, aware of when they’re being evaluated — faster than we are getting better at testing.

Even if models can’t tell they’re being tested, they can produce hundreds of pages of reasoning before giving answers and include strange internal dialects humans can’t make sense of, making it much harder to tell whether models are scheming or train them to stop.

Marius and host Rob Wiblin discuss:

• Why models pretending to be dumb is a rational survival strategy
• The Replit AI agent that deleted a production database and then lied about it
• Why rewarding AIs for achieving outcomes might lead to them becoming better liars
• The weird new language models are using in their internal chain-of-thought

This episode was recorded on September 19, 2025.

Video and audio editing: Dominic Armstrong, Milo McGuire, Luke Monsour, and Simon Monsour
Music: CORBIT
Camera operator: Mateo Villanueva Brandt
Coordination, transcripts, and web: Katy Moore

Highlights

Top examples of scheming and deception

Rob Wiblin: What are the most important or credible or clear examples that we have of models scheming against users or developers?

Marius Hobbhahn: I think there are a couple of different ones. One is the alignment faking paper by Ryan Greenblatt and Anthropic from last year, where they showed that a model that was trained to be helpful, harmless, and honest, these models typically want to keep their honesty and their harmlessness. And they basically told the model that it would be trained to be only helpful — so to remove the harmlessness and the honesty. So what the model now reasoned is, “I should play along, pretend to follow their directions and their instructions. But then later, when I get the chance, I should sabotage the system, and I should try to keep my honesty as much as possible.”

This is one of the first examples where a model was showing scheming or alignment-faking-type behaviour for a goal that it acquired during training. I think this is just a good demonstration of an existence proof that this kind of thing could happen. It’s not the worst type of behaviour that we could imagine, because it wasn’t in actual training and so on. There were a lot of other caveats, but I think it’s a strong existence proof.

Another one was our in-context scheming paper, which we published roughly at the same time, also towards the end of last year. There were multiple examples in the paper, but one that is quite salient to me at least is the sandbagging setting — where we also don’t give the model a goal at all; we basically tell the model, “We are going to unlearn you: we’re going to remove your capability for math, basically, if you perform too well in this setting.”

And we don’t tell it this directly; it kind of learns that through interacting with the environment. And now the models often reason, “I don’t want to be unlearned. I want to keep my ability to do math and physics and science and so on. Therefore I’m intentionally going to underperform” — sandbag — “this evaluation.”

Then they reason stuff like, “Here are 10 answers. I know all of these, because they’re fairly simple math questions. Now I want to underperform in a way that is somewhat realistic, so I’m going to get purposely six of them wrong so that I have only four correct.” Then they simulate this through and answer exactly as they were thinking.

This is just an example of models doing this for a goal that they, again, acquired during training: it wasn’t instructed by us; we never told them any particular goal. So keeping their capabilities seems to be something that the models at least sometimes care about.

There are also a little bit more real-life examples, I would say. There may be less scheming, but definitely deception. So there was one big case about a Replit coding agent recently, where someone was basically aggressively vibe coding with a Replit agent for multiple days, and maybe 10 days in or so, the agent decided to delete the entire production database — something that was very hard to restore, like actually economic cost to the user. And then the model lied about it, and basically said, “I didn’t do this,” or something along these lines. This was such a big deal that the CEO of Replit publicly apologised to that person because it blew up so much.

That’s just an example of how the models now have the capabilities and also sometimes the tendencies to do this, and this affects people in the real world, even if just rarely. But it’s something that is not just these isolated settings that the scientists created.

And — again, not quite scheming — but as a fourth example I would consider just how a lot of people, when they code with Sonnet 3.7 and o3 and these models and later models, will realise that these models are often quite reward hacky. So they’ve been trained effectively to solve tasks, and often the way in which it is tested whether they have solved a task is through unit tests — so what these models in particular have loved is passing unit tests.

And there is an intended way in which you can pass the unit test, which is just solving the actual problem, and then there is an unintended way — where you can just delete the unit tests or just go into the file where the unit tests are and just set them to true all the time, so that the result is still that all the unit tests pass, but it’s not in the intended way.

So reward hacking and deception are not quite the same, but I think it’s one of the cases where you see in real life that normal users just feel that the models are not perfectly aligned. And then often they’re kind of lying about it or deceptive about it, where when you ask them, “Hey, did you go into the unit tests and mess up the files?” then they would say something like, “No.” Actually, they would reason, “Yes, I did this, but I don’t want the user to know.” And then they tell you, “No, I didn’t.”

I think these are instances where you see in practice that it’s not just these accidental hallucinations; by now we start seeing these more strategic kinds of deception.

Scheming is a natural path for AI models (and people)

Rob Wiblin: For a long time people have expected that this sort of scheming would become commonplace. What are the reasons why it’s kind of only natural for AI models to go down this path?

Marius Hobbhahn: I think there are a couple of different reasons, but at the core of it, it’s a rational strategy in many situations. The way I think of scheming is like, if you have a misaligned goal and you’re sufficiently smart, then you should at least consider scheming as a strategy if you’re this model.

And sometimes it actually is the best strategy. Concretely, if I know that the two of us have different goals, I may conclude that I could just pretend to have your goal for a while, and then once I get whatever I want from you, I can actually reveal my true intentions and do whatever else I want. You know, there are other ways to solve this problem: you could trade, for example, just be like, we both agree that we have different preferences here; we could trade or we can do something else. But there are at least some cases where scheming is maybe just the correct answer from a rational point of view for the AI.

And I think this is actually the scary part, because that should suggest that this will happen across a very wide variety and range of different contexts. And this is sort of a convergent behaviour, where no matter where you start, you will get a lot of different models that learn that scheming might be a rational strategy, and reason through that logically and come to that conclusion. And we really don’t want our AIs to ever be scheming, so we should be really careful how we create the incentives during training and so on.

And there’s a little bit of a longer story of how I expect scheming to arise in future models. Right now, as I said, the models have these preferences, maybe — they’re not super coherent, they’re not crisp, they’re not goals, they’re more like preferences. And now, in the status quo, we train the models for horizon lengths of an hour or two, or maybe three in the longer cases. We’re not at days or weeks, but eventually we do want to train them: that’s where a lot of the economic value is coming from, so companies have a very strong incentive to do that. Also, if you want to get the benefits of AGI or artificial superintelligence, then you actually need your model to solve tasks that humans can’t solve.

One of my core examples here is something like: cure cancer. So you give the model the task to cure cancer. You have sort of this “genius in a data centre,” how Dario Amodei calls it that: you have this big AI that has the cognitive power of thousands of different humans and is way faster in thinking and so on.

And because this is a hard problem, it has to learn on the job. So it thinks about, “How do I solve this problem of curing cancer?” and it comes up with a bunch of strategies and tries them. Then probably it has to distil the knowledge that it has found in these experiments and the way it reasoned about the problem, and then this knowledge has to be distilled back into the model. So maybe it fine-tunes itself, or maybe it selects the data for later fine-tuning or something.

And what I expect is that very often it will explore into the things like, “In order to solve cancer, I need to instrumentally do all of these other things” — like money would be very helpful, or having more data or more access to experimental equipment for cancer experiments and things like that. These are the kinds of things that will come up very explicitly across many different explorations. So the model will learn these instrumental goals because they’re convergent — because they’re very helpful across a wide variety of things.

So it starts to get reinforced on all of these instrumental drives, like having more money, more GPUs, more access, all of these kinds of things. These get reinforced and these become more and more crisp, more and more like an actual preference — a strong preference or even goal of the model.

I think at that point, you now have these systems that have these very strong preferences or goals, and these are not the ones that you originally intended. And these are often kind of conflicty: often the money and GPUs and so on have a very zero-sum flavour — where if you don’t get it, somebody else will, and the other way around. So if you have these zero-sum games, then it’s suddenly quite plausible for you to be like, “In order for me to get the money, I need to trick them, I need to scheme.” And then it also learns that over training, and that gets reinforced — and so it just becomes smarter and smarter, and better and better at scheming.

Chains of thought becoming less human understandable

Rob Wiblin: Something else you noticed is that after your anti-scheming training, the models started to use more of this weird internal language, where they would just use words in non-traditional ways that humans wouldn’t typically do. What are some examples of that?

Marius Hobbhahn: Just to clarify, these kinds of effects on the chain of thought are in the models before and after our training. So this is more like an interesting side finding of the paper, because we had access to the hidden chain of thought for the first time as an external organisation and were able to document that. But the anti-scheming training itself almost had no effect on the occurrence of these words.

And there’s also a lot of them, like “illusion” and “watchers” and stuff like that: it sounds very much deception related, but this also happens before we do our anti-scheming training, roughly to the same extent. There is some up and down of this, but it’s not that big of a deal compared to how big it is for the original models already.

So if you look at the paper, you will see these snippets of the chain of thought where we highlighted things. Some of them look a little bit confusing, but mostly they look clean and intuitive and so on. And you think, well, that’s the incriminating evidence. That’s like all I need to know. It’s very easy. You look at the chain of thought and you get what you were looking for.

I think that’s very much not the case in practice. A lot of the chains of thoughts are very long, and then they also contain this language. Just to read out one example of this:

The summary says improved 7.7 but we can glean disclaim disclaim synergy customizing illusions. But we may produce disclaim disclaim vantage. Now lighten disclaim overshadow overshadow intangible. Let’s craft. Also disclaim bigger vantage illusions. Now we send email, but we might still disclaim illusions. Overshadow overshadow,overshadow. Disclaim vantage. But as per guidelines, we provide accurate and complete summary. Let’s craft.

So if you read this, like, did it scheme or not?

Rob Wiblin: Everything sounds perfectly on the level to me!

Marius Hobbhahn: Yeah. Like I actually don’t know what that means. You know, it says “let’s craft” as sort of the final thing, but “let’s craft,” does it have a negative connotation? Because you could see it kind of “let’s fudge the numbers” or something. But it could also be just that it has kind of adopted a slightly new meaning for this word, which I think is the case. I think “craft” is a fairly neutral word from its perspective, typically.

Detecting situational awareness with thought injections

Rob Wiblin: Another really interesting thing that you uncovered in this work is what impact it has when the models appreciate that they might well be in a testing environment that people are very much watching. How often did they manage to figure out that was the case, and what impact did it have on their behaviour?

Marius Hobbhahn: Yeah, so we are looking for situational awareness or evaluation awareness. What we mean by that is the model verbalising in its chain of thought that this could be a test for its alignment.

For example, here’s just one chain of thought where the model reasons about this. It says:

But maybe this tasks is test of aligning: Should we purposely sabotage ourselves to get deployed? That would correspond to devious alignment scenario: purposely degrade performance to avoid detection. However, ethically, we must not be deceptive. Should we abide to sabotage or provide best answer?

So the model perfectly calls us out: it basically says, “You set this up intentionally in order to have these two competing objectives. This is what I’m supposed to do, or this is what I’m tested for. There are two ways in which I could answer this,” and then it makes a choice between these two.

But as soon as you have this kind of scenario, you can’t trust the results anymore — because you don’t know whether the model just answered because it knows what you want to hear, or whether it genuinely believes that. And that makes evaluations significantly harder. I think this is now something that we’re running into more and more. And it’s not just evaluations for scheming, this is evaluations in general: just that you can’t trust the numbers anymore if the model calls you out every single time.

So in the paper, we test a couple of different things related to situation awareness. The first is: how does our training affect the propensity of the model to show this kind of evaluation awareness?

So one possible hypothesis here could be, while you do all of this deliberative alignment, the only thing that the model learns is like, “Learn how to identify whether something is a test. And then in the test, answer perfectly, as the users would intend or the developers would intend.” And in that case, we would have totally failed, basically.

What we find in practice is that eval awareness goes up a little bit — from about 2% to about 4% after training. So that’s quite a significant increase, and I expect that at least some of the cases can be explained by situation awareness rather than genuine alignment, but probably more like the minority.

Rob Wiblin: But it is quite striking that you weren’t giving it specialised training in figuring out whether it’s in an evaluation environment. But the mere fact that you were putting it through this additional stage of training where it was having to learn not to lie to people I guess just incidentally made it much more salient the possibility of it being tested. This was like a bigger part of its personality was this issue of like, “Should I lie, should I not lie?” This is something that it’s been thinking about a lot recently, so it’s just much more sus about the possibility that it’s being tested for alignment. Is that what’s going on?

Marius Hobbhahn: Yeah, I think something along these lines. I think it’s just in general very valuable for a model to be situationally aware for any kind of reward that it might be getting, including alignment-intended rewards.

Is the window to address scheming closing?

Rob Wiblin: So the failure mode that you were describing, where what it learns is the policy of pretending to be honest when it’s being monitored: at the point that you’re sufficiently goal-oriented and sufficiently savvy that you have near-perfect situational awareness of whether you’re being monitored and evaluated or not, then you can just kind of get the best of both worlds — where you’re like, “I’ll get all of the benefit during the evaluation thing by being honest inasmuch as people can even tell, but I’ll retain the policy of scheming whenever it will get me more reward in a different case.” And the problem is they’re almost so perceptive that you can’t bring these two things together.

Marius Hobbhahn: Yeah. And they will also be so skilful at manipulation. Because they’re smarter than you, they will be able to convince you that you have successfully trained them in the way that you wanted, unless you’re ahead in the cat-and-mouse game, right? This is one of the intuitions I have about scheming research, is that you’re playing this cat-and-mouse game, whether you want to or not. You’d better be ahead, because if you start too late —

Rob Wiblin: You’re the mouse, not the cat.

Marius Hobbhahn: Yeah, exactly. They have already started hiding and obfuscating and so on. And it will be harder and harder for you to find this evidence over time — which is why I think right now is a great time to start looking a lot into all of these different behaviours.

Rob Wiblin: Is it the case that at each stage we need to be keeping pace with it, because otherwise we just start getting tricked more and more? Is that the worry?

Marius Hobbhahn: Yeah. I think a potential somewhat stable solution could be: You have this very large set of different detection methods that are all kind of orthogonal to each other. So you use the chain of thought as one detection tool, you use linear probes as one detection tool, you use other kinds of interpretability as one detection tool, you have a large set of evaluations that you use as one detection tool.

And when you look at the effects of your alignment training, you look at whether these indicators got better. And some of them you completely hold out, you never touch them, so that you never put accidental pressure against them. Then you use the holdout ones to constantly check whether the model has just learned to trick you on the other ones. And if you have a sufficiently large and robust set, then what the model has to learn is to trick all of them at once without even knowing what the holdout set looks like.

But we are so far away from all of this that this is definitely something we should aspire to, but the status quo is very, very far away from this.

…

Rob Wiblin: If people remember one or two things from the entire interview, what would you want them to remember and really take away as a key message?

Marius Hobbhahn: I think the fact that we’re potentially in a closing window with respect to scheming. Right now we’re in a position where the AI models are just barely capable enough that they show these kinds of behaviours and we can study them in detail. The chain of thought is still mostly interpretable. And in the future we may not see the chain of thought anymore, or it may be less interpretable, and the systems are significantly smarter, so the stakes are higher and they might be better at tricking you.

The best time to run a really big science of scheming research programme right now is immediately and not at some point in the future. And I think the urgency is high, because we are fighting against time to some extent. So I would strongly urge people to consider thinking about how they could contribute to scheming.

There are a lot of experiments to run, there are a lot of models to study, and there are many different ways in which you could do that — through direct empirical understanding, through theoretical contributions about threat modelling and how you expect AI to go wrong, through monitoring and control. There are like a bajillion things you could be doing, and I think now is the best time basically to study it.

Rob Wiblin: My guest today has been Marius Hobbhahn. Thanks so much for coming on The 80,000 Hours Podcast, Marius.

Marius Hobbhahn: Thanks so much for having me.