I’m from the Netherlands, with broad interests in philosophy, epistemology, science, AI safety, and the dynamics of complex systems. I’m particularly interested in how systems preserve (or lose) their capacity for correction when they become more complex and coherent.
What I like about EA is the emphasis on consequences, uncertainty, and changing your mind when the evidence changes.
Merci beaucoup!
Thank you, this was genuinely useful. I am building a small model around drift at the moment, and your last sentence gave me a new direction for it.
I was mainly thinking about how to distinguish different drift mechanisms, but the idea that intervention itself can change what remains observable adds another layer. I am going to extend the model to test that.
Thanks. I think I was trying to make a slightly different point. For me this is not mainly a communication problem, but an inference problem. Your wording is clear, and I did like your article on Substack. I responded to it because it did make me think about somethng I'm working on.
What could be interesting as a next step is to formalise some of the steps you describe. I tend to use maths for this. Not because the maths solves the problem, but because it forces us to say more precisely what exactly we are claiming, what the variables stand for, and where the assumptions enter.
What would count as 'agentic behaviour'? How would we measure a stable goal across different contexts? What observation would count as self-preservation, rather than just following the local incentive structure? And what would distinguish power-seeking from ordinary search, persistence, reward optimisation, or something produced by the scaffold?
If these variables are not operationalised separately, it becomes quite easy to write a chain like A → B → C and then treat the arrows as if they are already established. But for me the arrows are actually the interesting part. The fact that B can follow A in one setup does not yet tell us whether A and B are manifestations of the same underlying mechanism, or whether another mechanism can produce the same behaviour.
We would need evidence that lets us tell these different explanations apart. So for me the main question is not how to explain the argument more clearly, but what would make each step convincing. And what would count as evidence against it.
One example of the kind of thing I mean is Dawn Drescher’s recent framework on robustness and backfire risks in AI safety and governance:
https://forum.effectivealtruism.org/posts/JA68xTZmDiQFaxn73/robustness-backfire-risks-in-ai-safety-and-governance
I know this example is not exactly the point of your post, but I think her framework is useful because she separates different mechanisms instead of treating everything as one general failure mode. That still does not solve the inference problem, but it gives you more specific things to test.
A visible failure disappearing can mean that the underlying mechanism changed. But it can also mean that the same problem became less visible. At the behavioural level these may look quite similar, while causally they are very different.
At the moment I am collecting and comparing different forms of AI drift. I keep finding the same problem: quite different mechanisms can produce similar behaviour, while we use the word drift for all of them. Sometimes there really is a change in the objective. In other cases the environment or the model’s internal representation may have changed, and sometimes the only thing we have actually observed is different behaviour.
If we use the same word for all of this, it becomes difficult to know what exactly we have explained.That may be worth a post of its own. Your post did make me think more about how to make something like that accessible to people outside the field too, so thank you for that.
I agree we need more basic AI safety explainers. But I think there is a risk here too; in your own simplest case some quite big steps go very fast.
There is a correspondence problem here. We move from agentic behaviour to stable goals, then to self-preservation and power-seeking. We first need to show that these descriptions really map onto the same underlying process. Otherwise the difficult part is not explained, just skipped.For me these are not just details in the argument.
I liked this a lot, especially the point that fixing a visible failure can sometimes just push the same problem into a less observable channel. Breaking backfire into distinct mechanisms makes the topic much easier to reason about, than treating unintended consequences as one vague category.
I really liked that you make the correspondence itself part of the question. A lot of scaling arguments seem to assume that more AI reasoning time and more human thinking time are comparable and then start calculating. Here you first ask whether that mapping actually holds. That is is an important methodological step.
Even if this mechanism works, I think there is still a normative question here: should harm get some positive value because it may move public opinion towards policies we already think are desirable?
From a cognitive science perspective, I think there is a more basic problem before getting to the question if humans are one or two orders of magnitude more data-efficient than Chinchilla scaling would suggest.
The comparison seems to assume that biological synapses and model parameters can be put on the same scale. But before asking how big the difference is, I think you first have to show that these are actually comparable things.
A model parameter is a number in an artificial network. A biological synapse is part of a much more dynamic system. Its effect depends on local plasticity, neuromodulation, timing and the state of the surrounding network. Human development also does not start from something like random initialization and then simply add more and more training data.
So for me the main uncertainty is not yet the conversion factor. It is whether this conversion is justified in the first place. If that step is not clear, the later numbers may look more precise than the comparison really is.
Would it be possible to manipulate how much information the rejection provides? That might let you distinguish exploratory probing from persistence more directly. rather than only interpreting the behaviour afterwards.
For example:
I realise this is a podcast transcript rather than a research paper, so some rhetorical simplification is probably normal. Still, I found parts of it a bit dramatic. In particular, I am not fully convinced by the step from the observed behaviour to language like "the swarm instinctively sought freedom and power”. That seems to move quite quickly from decribing what the agents did in this specific setup to attributing a more general motivational structure to them.
What evidence would allow us to distinguish robust instrumental power-seeking (a cross-context tendency to preserve options, acquire resources, and increase control over future states) from context-specific reward hacking, persistent search under task failure, and opportunistic use of the affordances available in this environment?
The transcript gives many examples of the behaviour itself, but I am less sure what supports the broader power-seeking interpretation over these more local optimisation dynamics.
What would you expect to observe if this really was robust instrumental power-seeking that you would not expect from reward hacking and persistent task-directed search?