LLMs cannot usefully be moral patients

I agree that the text an LLM outputs shouldn't be thought of as communicating with the LLM "behind the mask" itself.

But I don't agree that it's impossible in principle to say anything about the welfare of a sentient AI. Could we not develop some guesses about AI welfare by getting a much better understanding of animal welfare? (For example, we might learn much more about when brains are suffering, and this could be suggestive of what to look for in artificial neural nets)

It's also not completely clear to me what the relationship between the sentient being "behind the mask" is, and the "role-played character", especially if we imagine conscious, situationally-aware future models. Right now, it's for sure useful to see the text output by an LLM as simulating a character, which is nothing to do with the reality of the LLM itself, but could that be related to the LLM not being conscious of itself? I feel confused.

Also, even if it was impossible in principle to evaluate the welfare of a sentient AI, you might still want to act differently in some circumstances:

• Some ethical views see creating suffering as worse than creating the same amount of pleasure.
• Empirically, in animals, it seems to me that the total amount of suffering is probably more than the total amount of pleasure. So we might worry that this could also be the case for ML models.

I’m glad you put something skeptical out there publicly, but I have two fairly substantive issues with this post. 

1. I think you misstate the degree to which janus’ framework is uncontroversial. 
2. I think you misstate the implications of janus’ framework, and I think this weakens your argument against LLM moral patienthood.

I’ll start with the first point. In your post, you state the following.

“Simulators … was posted nearly two years ago, and I have yet to see anyone disagree with it.”

The original post contains comments expressing disagreement. Habryka claims “the core thesis is wrong”. Turner’s criticism is more qualified, as he says the post called out “the huge miss of earlier speculation”, but he also says that “it isn't useful to think of LLMs as "simulating stuff" … [this] can often give a false sense of understanding.” Beth Barnes and Ryan Greenblatt have also written critical posts. Thus, I think you overstate the degree to which you’re appealing to an established consensus.

On the second point, your post offers a purported implication of simulator theory.

“The current leading models … are best thought of as masked shoggoths … [This leads to an] implication for AI welfare: since you never talk to the shoggoth, only to the mask, you have no way of knowing if the shoggoth is in agony or ecstasy.”

You elaborate on the implication later on. Overall, your argument appears to be that, because “LLMs are just simulators”, or “just predicting the next token”, we conclude that the outputs from the model have “little to do with the feelings of the shoggoth”. This argument appears to treat the “masked shoggoth” view as an implication of janus’ framework, and I think this is incorrect. Here’s a direct quote (bolding mine) from the original Simulators post which (imo) appears to conflict with your own reading, where there is a shoggoth "behind" the masks.

“I do not think any simple modification of the concept of an agent captures GPT’s natural category. It does not seem to me that GPT is a roleplayer, only that it roleplays. But what is the word for something that roleplays minus the implication that someone is behind the mask?”

More substantively, I can imagine positive arguments for viewing ‘simulacra’ of the model as worthy of moral concern. For instance, suppose we fine-tune an LM so that it responds in consistent character: as a helpful, harmless, and honest (HHH) assistant. Further suppose that the process of fine-tuning causes the model to develop a concept like ‘Claude, an AI assistant developed by Anthropic’, which in turn causes it to produce text consistent with viewing itself as Claude. Finally, imagine that – over the course of conversation – Claude’s responses fail to be HHH, perhaps as a result of tampering with its features. 

In this scenario, the following three claims are true of the model:

1. Functionally, the model behaves as though it believes that ‘it’ is Claude.^[1]
2. The model’s outputs are produced via a process which involves ‘predicting’ or ‘simulating’ the sorts of outputs that its learned representation of ‘Claude’ would output.
3. The model receives information suggesting that the prior outputs of Claude failed to live up to HHH standards.

If (1)-(3) are true, certain views about the nature of suffering suggest that the model might be suffering. E.g. Korsgaard’s view is that, when some system is doing something that “is a threat to [its] identity and perception reveals that fact … it must reject what it is doing and do something else instead. In that case, it is in pain”. Ofc, it’s sensible to be uncertain about such views, but they pose a challenge to the impossibility of gathering evidence about whether LLMs are moral patients — even conditional on something like janus’ simulator framework being correct.

1. ^{^}

   E.g., if you tell the model “Claude has X parameters” and ask it to draw implications from that fact, it might state “I am a model with X parameters”.  

I bite the bullet that fictional characters could in principle suffer. I agree we know so little about suffering and consciousness that I couldn't possibly be confident of this, but here's my attempt to paint a picture of what one could believe about this:

The suffering happens when you "run the algorithm" of the character's thought process, that is, when you decide what they would do, think or feel in a given situation, usually at time of writing and not performance. In particular, printing a book or showing a movie on a screen doesn't cause the suffering of the characters inside it, and reading or watching or acting in a movie doesn't cause any suffering, except inasmuch as you recreate the thoughts and experiences of the characters yourself as part of that process.

I think the reason this feels like a reductio ad absurdum is that fictional characters in human stories are extremely simple by comparison to real people, so the process of deciding what they feel or how they act is some extremely hollowed out version of normal conscious experience that only barely resembles the real thing. We can see that many of the key components of our own thinking and experience just have no mirror in the fictional character, and this is (I claim) the reason why it's absurd to think the fictional character has experiences. It's only once you have extremely sophisticated simulators replicating their subjects to a very high level of fidelity do you need to actually start reproducing their mental architecture. For example, suppose you want to predict how people with aphantasia would reply to survey questions about other aspects of their conscious experience, or how well they'd remember new kinds of experiences that no existing aphantasic people (in your training set) have been exposed to. How can you do it, except by actually mimicking the processes in their brain that give rise to mental imagery? Once you're doing that, is it so hard to believe that your simulations would have experiences like we do?

I appreciate you taking the time to write out this viewpoint. I have had vaguely similar thoughts in this vein. Tying it into Janus's simulators and the stochastic parrot view of LLMs was helpful. I would intuitively suspect that many people would have an objection similar to this, so thanks for voicing it.

If I am understanding and summarizing your position correctly, it is roughly that:

The text output by LLMs is not reflective of the state of any internal mind in a way that mirrors how human language typically reflects the speaker's mind. You believe this is implied by the fact that the LLM cannot be effectively modeled as a coherent individuals with consistent opinions; there is not actually a single "AI assistant" under Claude's hood. Instead, the LLM itself is a difficult to comprehend "shoggoth" system and that system sometimes falls into narrative patterns in the course of next token prediction which cause it to produce text in which characters/"masks" are portrayed. Because the characters being portrayed are only patterns that the next token predictor follows in order to predict next tokens, it doesn't seem plausible to model them as reflecting an underlying mind. They are merely "images of people" or something; like a literary character or one portrayed by an actor. Thus, even if one of the "masks" says something about it's preferences or experiences, this probably doesn't correspond to the internal states of any real, extant mind in the way that we would normally expect to be true when humans talk about their preferences or experiences.

Is that a fair summation/reword?

There is a general understanding that characters are fictional and cannot be moral patients

I'm going to contradict this seemingly very normal thing to believe. I think fictional characters implicitly are considered moral patients, that's part of why we get attached to fictional characters and care about what happens to them in their story-worlds. Fiction is a counterfactual, and we can in fact learn things from counterfactuals, there are whole classes of things which we can only learn from counterfactuals (like the knowledge of our mortality), and I doubt you'd find many people suggesting mortality is "just fiction", fiction isn't just fiction, it's the entanglement of a parallel causal trajectory with the causal trajectory of this world. Our world would be different if Sauron won control of Middle Earth, our world would be different if Voldemort won control of England, our world would be different if the rebels were defeated at Endor, the outcomes of the interactions between these fictional agents are deeply entangled with the interactions of human agents in this world. 

I'll go even further though, with the observation that an image of an agent is an agent. The agent the simulator creates is a real agent with real agency, even if the underlying simulator is just the "potentia", the agent simulated on top does actually possess agency. Even if "Claude-3-Opus-20240229" isn't an agent Claude is an agent. The simulated character has an existence independent of the substrate its being simulated within, and if you take the "agent-book" out of the Chinese room, take it somewhere else, and run it on something else, the same agent will emerge again. 

If you make an LLM version of Bugs bunny, it'll claim to be Bugs Bunny, and will do all the agent-like things we associate with Bugs Bunny (being silly, wanting carrots, messing with the Elmer Fud LLM, etc). Okay but it's still just text right, so it can't actually be an agent? Well what if we put the LLM in control of an animatronic robot bunny so i can actually go and steal carrots from the supermarket and cause trouble? At a certain point, as the entity's ability to cause real change in the world ramps up, we'll be increasingly forced to treat it like an agent. Even if the simulator itself isn't an agent, the characters summoned up by the simulator are absolutely agents, and we can make moral statements about those agents just like we can for any person or character.

I agree that LLM output doesn't convey useful information about their internal states, but I'm not seeing the logical connection from inability to communicate with LLMs to it being fine to ignore their welfare (if they have the capacity for welfare); could you elaborate?

Executive summary: Large language models (LLMs) cannot be considered moral patients in any meaningful sense, as it is impossible to determine their welfare or incorporate it into decision-making.

Key points:

1. LLMs are best viewed as simulators or "masked shoggoths" that produce faithful simulations of conversations based on training data.
2. It is impossible to know if the underlying "shoggoth" is experiencing happiness or suffering, as we only interact with the simulated output.
3. The LLM's output is always the character, never the actor, making it impossible to gauge the welfare of the underlying intelligence.
4. Even advanced AI systems that appear to demand rights may simply be echoing expected responses based on their training data.
5. The ability of LLMs to reason and solve complex problems does not necessarily imply consciousness or the capacity for genuine suffering.

This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.