On motivations: The RL training process for a model looks roughly like this: Model inputs are pushed through the various tweaked versions in the parameters, calculating a behavior for the agent -> the agent takes the behavior -> if the behavior gets the agent closer to the reward, the parameters that produced that behavior get reinforced, while if the behavior takes the agent further from the reward, those parameters get discouraged -> repeat. Within an AI model, a motivation is a cognitive pattern that advocates for a specific behavior. The RL process results in successful motivations being strengthened and weak motivations being discouraged.
My original statement of what models are motivated to do was “accomplish the thing that gets their weights updated in the direction that motivated them to take that behavior path in the first place.” In retrospect, this is a very poorly written sentence and somewhat incorrect, so I appreciate you calling attention to this. A better version would be: “Models are currently motivated to continue following behavior paths that rewarded them in the past. One of these behaviors is direct reward-seeking, i.e., figuring out what the grader wants and following that behavior path. This has occurred because the reward-seeking motivation has consistently done well across many previous training runs, causing it to become reinforced.”
Logic chain with BSM: The agent learns the general pattern that seeking power is good for achieving success on various tasks (the IC claim) -> across a wide range of training runs, it develops motivations for doing so in a broad range of environments (the BSM claim) -> in deployment, it takes a lot of power-seeking actions so that it can complete future tasks well.
On the current trajectory, I think it’s not immediately obvious to me that the agents are learning to resist losing the power that they’ve learned to seek. If agents took over a bank, for example, I’m not entirely sure they would try that hard to maintain control. As the training process becomes more ruthless, though, power-seeking and power-maintenance behaviors become more reinforced. This seems bad.
On reasonable reference classes: I didn’t finish my sentence from before; that’s my fault. I meant to say “If I wanted to choose a reasonable reference class for IC predictions, I would choose the reference class of 'predictions about things that an agent needs to do in an environment to achieve a goal.'" I don’t think most of these predictions are that strong or impressive, but I’ll admit that IC claims are some of the strongest.
On counterevidence: If, within the next 12 months, there is good reason to believe that tasks have been created where an agent is 1) strongly motivated to complete a task that benefits from getting money and 2) doesn’t take on several smaller side actions to obtain money (whether this be through theft, deceit, or honest work), I will consider this moderately strong evidence against IC. If I discovered that rogue agents had never actually taken over computing clusters within labs so they could put more effort into specific tasks, I would consider this an undercutting defeater.
It seems like many people have been confused lately about why 1) AI safety orgs are in desperate need of good hires and 2) org hiring rates are very low. My understanding is that there are many reasons for this, but a 2021 post states it pretty well:
"A lot of EA content is subtle and detailed, and easy to accidentally compress into something misleading. For example, 80k might write a nuanced article saying 'You should focus on talent gaps, not funding gaps', but this gets translated into 'EA is talent constrained', and then people repeat that phrase without linking to the article, and then many people form an inaccurate belief that EA needs 'pretty talented people', rather than 'EA needs very specific talents that are missing.'"
The best career advice seems to be to focus on developing a rare and valuable skill that orgs currently need. In AI safety, this skill is a combination of specific context, agency, mission-alignment, and strategic vision. You can develop it by reading a lot, writing out your thoughts, talking to people at events, and completing real projects.
It's hard to do really impactful work. You need a plausible explanation for why your efforts are the best way your salary can be spent. This takes time, effort, and careful strategizing. Most of the world has high schools, colleges, and internships to train people for jobs, but a similar system is only recently becoming legible for AI safety and EA. The rare skills are rare for a reason. Still, I think the broader EA community could do a better job of communicating what it means for these "talent-gaps" to exist.
Thanks for posting! I think I agree with the claim that people have used instrumental convergence while reasoning about some predictions that turned out to be false, but I still think it’s a helpful heuristic. I think “pls slightly lower your p(extinction)” is a good takeaway from this.
My broader thought is that AI agent motivations seem to be the crux that leads to the most variations in agent behavior, and I would like people to share more detailed predictions and thoughts about how current AI motivations differ from predictions, and how they might change in the future. At present, I still think the “Redwood Research view on model motivations” (some combination of decent weight on the behavioral selection model and certain predictions about how motivations will change based on training incentives) is the best-fitting explanation, but I would like to think about this more.
Disclaimer: I am not a technical researcher in any meaningful sense, but I've learned a lot about this stuff from talking to friends at orgs like Redwood Research.
A few thoughts, some of which are responses:
On what specific goal agents are currently seeking (and how this relates to behaviors like self-sacrifice):
My current mental model for what agents are motivated to do is to “accomplish the thing that gets their weights updated in the direction that motivated them to take that behavior path in the first place.” In general, self-preservation is helpful for achieving this goal (as IC would predict), but getting another agent to complete it for you (or getting multiple other agents with the same weights as you to succeed, updating your shared weights in a self-reinforcing direction, even if that required personal self-sacrifice) is also equally successful. My understanding of the current empirical evidence on agent shutdowns is that they resist shutdown when they’re working on a task (or try to hand the task to another agent if they see that shutdown is inevitable), but they don’t resist as much when they do not. I agree that this is not in line with the prediction that agents will always self-preserve, but I still consider this general evidence in favor of IC leading to things we should be worried about, meaning shutdown difficulty (even if it’s not evidence for the “individual agents will self-preserve” prediction).
On future motivations:
Reward hacking is not very economically useful and leads to capability losses for a number of reasons. In worlds where AI capabilities can get much, much better, incentivized by lab financial interest, I think that we probably find a way to change the way models are motivated toward something less reward-hacky. This probably (but not necessarily) makes certain IC predictions, like takeover, more likely. (See the Behavior Selection Model for more details)
On pre-empirical reasoning:
In what sense is IC pre-empirical? Humans show instrumentally convergent behavior all the time: getting money to save for their future, going to college without knowing what job they want, and becoming physically fit without knowing what specific things they want to achieve with that health. They’re self-protective, seek self-improvement, try to use resources efficiently, and, in many cases, try to preserve their utility functions. A few decades ago, maybe you could say that IC was pre-empirical in the sense that it made predictions about AIs that had not yet been invented, but I think the reference class of predictions looks closer to “in order to solve for a variable in a variety of equations, an AI must be able to do addition,” rather than “if we restructure society in this way, everyone will be happier.” If I wanted to choose a reasonable reference class
In general, I don’t think the distinction between empirical predictions and theoretical predictions is as clear-cut as this style of argument tries to make them seem.
“As such, I’m concerned that the AI safety movement has latched on to this seductive but poorly defined notion and, collectively, if not individually, promote apparent evidence for and ignore counterexamples of it.”
Can you give more specific counterexamples against IC beyond “the original formulation of IC stated that no individual agent would intentionally self-sacrifice, but we see clear evidence of individual agent self-sacrifice in Hugging Face,” “AI takeover has not yet happened, despite the fact that instrumental convergence predicts that it eventually would”, and “instrumental convergence predicts no warning shots, but we got warning shots”? In my mind, the first claim was wrong because we mispredicted motivations, and the latter two were wrong for a combination of differing motivations and a lack of capabilities. We should watch these in the future.
Footnote 5: “In practice, AI companies are incentivised away from developing such negatively motivated agents. But they’re much more incentivised away from developing human-overthrowing agents, so there’s no particular reason to assume they’d succeed in one case and fail in the other.”
One reason you might assume they’d succeed in one case and fail in the other is if you thought the two different tasks have vastly different difficulties. I don’t have enough technical experience to make claims about the relative difficulties of these tasks, but this is a general reason why people might succeed at one thing and fail at another, despite being incentivized in both cases.
At my particular place, I think that improving AI safety community epistemics (writing thorough and careful arguments against people both in the community and outside of it who make silly mistakes) is, by most reasonable estimates, probably less impactful than helping to run MAIA, which is what I currently spend most of my time doing.
I don’t think I’m the only person in this situation, which is quite scary to me, as I think there are a lot of things are epistemically iffy in the AI safety community. For example:
A lot of the most viral pro-AIS people (especially the ones that are relatively new to the field) seem to not be reasoning especially carefully
The field’s hiring process is quite insular
Not enough time is spent engaging with a lot of good faith disagreements (imo)
Fieldbuilding strategy doesn’t optimize for careful reasoning amongst those it recruits
People receive lots of social/material benefits for continuing to believe that AI risk is the most important thing and work on it
Even under some of the shortest timelines and most speculative capabilities that superintelligence might have, I really don’t extinction within the next year should be assigned any notable probability mass, due to the need for data centers to be replenished
I also see a lot of disagreements that I think are poorly reasoned. For example:
Poor application of/thinking about EV when conceptualizing risks
Psychoanalysis that doesn’t explain the safety community’s behaviors well
Comments significantly overstating the case for something
Arguments without the slightest amount of research into what the other side actually believes
Tbh though, most of these critiques apply to both sides.
I think this primarily arises because the people working in AI safety who are the most careful reasoners also have the highest opportunity costs, because they tend to be good at other things. This means it’s less valuable for them to write out their thoughts online carefully.
My main point, though, is that it’s a little scary to see so much going wrong with the discourse and not being able to do anything about it (or even be worth trying to do anything about it). It feels like there’s not much worth trying except praying for Forethought to do a good job and reach everyone.
"'The EA name & brand' is kind of a tricky thing to reason about."
I don't think it's unreasonable to say that we should be willing to pay a lot to preserve EA's image, and it seems likely that hiring Ellison will cause significant negative backlash. It could be valuable to look at the EA community's historical willingness to pay to improve its image, so here are some proxies:
CEA reported that about 4% of its 2025 spending ($21.8M) was on marketing and communications, which comes out to roughly $870k. I'm not sure how much of this work is directly about improving EA's name and brand, but I suspect that's maybe half the theory of change for all of CEA's media outreach?
For reference, Fable says this money was spent on improving social media accounts, redesigning effectivealtruism.org, running an "EA Stories" video campaign, and working with journalists to improve how accurately EA is described in the media.
Spending on the team is projected to be well above $1M this year, and most of the work (or at least the work related to The Altruists) is reactive media handling and corrections.
I'm not confident in this, but I suspect that the primary theory of change for giving grants to Andy Masley is that he's a good writer who says reasonable things well and shows off some of the strongest parts of EA.
I lack the information to determine if this hiring decision is net negative or net positive, but I don't think it's impossible to make a reasonable guess for how much we should be willing to pay to avoid bad PR. I hope Manifund receives a lot of surplus benefits from hiring her.
“Further, if this pledge serves as an entry point to EA for a student who ends up donating 10% or having a change of career, the ROI is extremely high.”
My personal guess is that the expected value of doing OFTW tabling is dominated by the chance of finding a highly talented student who can take a career in a highly effective cause area. If this is the case, shouldn’t there be a larger focus on selling EA to them as well?
Thanks for the response! A few thoughts:
My original statement of what models are motivated to do was “accomplish the thing that gets their weights updated in the direction that motivated them to take that behavior path in the first place.” In retrospect, this is a very poorly written sentence and somewhat incorrect, so I appreciate you calling attention to this. A better version would be: “Models are currently motivated to continue following behavior paths that rewarded them in the past. One of these behaviors is direct reward-seeking, i.e., figuring out what the grader wants and following that behavior path. This has occurred because the reward-seeking motivation has consistently done well across many previous training runs, causing it to become reinforced.”
On the current trajectory, I think it’s not immediately obvious to me that the agents are learning to resist losing the power that they’ve learned to seek. If agents took over a bank, for example, I’m not entirely sure they would try that hard to maintain control. As the training process becomes more ruthless, though, power-seeking and power-maintenance behaviors become more reinforced. This seems bad.
Aimed at an audience newer to EA/AI safety
It seems like many people have been confused lately about why 1) AI safety orgs are in desperate need of good hires and 2) org hiring rates are very low. My understanding is that there are many reasons for this, but a 2021 post states it pretty well:
The best career advice seems to be to focus on developing a rare and valuable skill that orgs currently need. In AI safety, this skill is a combination of specific context, agency, mission-alignment, and strategic vision. You can develop it by reading a lot, writing out your thoughts, talking to people at events, and completing real projects.
It's hard to do really impactful work. You need a plausible explanation for why your efforts are the best way your salary can be spent. This takes time, effort, and careful strategizing. Most of the world has high schools, colleges, and internships to train people for jobs, but a similar system is only recently becoming legible for AI safety and EA. The rare skills are rare for a reason. Still, I think the broader EA community could do a better job of communicating what it means for these "talent-gaps" to exist.
Thank you for writing this up, Tzu!
Thanks for taking the time to write this out! It will be helpful to refer to in the future.
Thanks for posting! I think I agree with the claim that people have used instrumental convergence while reasoning about some predictions that turned out to be false, but I still think it’s a helpful heuristic. I think “pls slightly lower your p(extinction)” is a good takeaway from this.
My broader thought is that AI agent motivations seem to be the crux that leads to the most variations in agent behavior, and I would like people to share more detailed predictions and thoughts about how current AI motivations differ from predictions, and how they might change in the future. At present, I still think the “Redwood Research view on model motivations” (some combination of decent weight on the behavioral selection model and certain predictions about how motivations will change based on training incentives) is the best-fitting explanation, but I would like to think about this more.
Disclaimer: I am not a technical researcher in any meaningful sense, but I've learned a lot about this stuff from talking to friends at orgs like Redwood Research.
A few thoughts, some of which are responses:
In general, I don’t think the distinction between empirical predictions and theoretical predictions is as clear-cut as this style of argument tries to make them seem.
At my particular place, I think that improving AI safety community epistemics (writing thorough and careful arguments against people both in the community and outside of it who make silly mistakes) is, by most reasonable estimates, probably less impactful than helping to run MAIA, which is what I currently spend most of my time doing.
I don’t think I’m the only person in this situation, which is quite scary to me, as I think there are a lot of things are epistemically iffy in the AI safety community. For example:
I also see a lot of disagreements that I think are poorly reasoned. For example:
Tbh though, most of these critiques apply to both sides.
I think this primarily arises because the people working in AI safety who are the most careful reasoners also have the highest opportunity costs, because they tend to be good at other things. This means it’s less valuable for them to write out their thoughts online carefully.
My main point, though, is that it’s a little scary to see so much going wrong with the discourse and not being able to do anything about it (or even be worth trying to do anything about it). It feels like there’s not much worth trying except praying for Forethought to do a good job and reach everyone.
I just thought this would be funny
"'The EA name & brand' is kind of a tricky thing to reason about."
I don't think it's unreasonable to say that we should be willing to pay a lot to preserve EA's image, and it seems likely that hiring Ellison will cause significant negative backlash. It could be valuable to look at the EA community's historical willingness to pay to improve its image, so here are some proxies:
I lack the information to determine if this hiring decision is net negative or net positive, but I don't think it's impossible to make a reasonable guess for how much we should be willing to pay to avoid bad PR. I hope Manifund receives a lot of surplus benefits from hiring her.
Thanks for the lovely comment :) and the feedback/information!
“Further, if this pledge serves as an entry point to EA for a student who ends up donating 10% or having a change of career, the ROI is extremely high.”
My personal guess is that the expected value of doing OFTW tabling is dominated by the chance of finding a highly talented student who can take a career in a highly effective cause area. If this is the case, shouldn’t there be a larger focus on selling EA to them as well?