I did not use an LLM in writing this post.
Plants, generative AI models and thermostats have preferences or goals, but they (probably) do not have conscious experiences. Their preferences are (probably) unconscious. But in my moral theory (called mild welfarism), only conscious preferences matter. A conscious preference is a preference (value or goal) that can be consciously experienced by a conscious (sentient) being. Although just a moment ago you may not have been consciously aware of your preference to stay alive, you could have been made aware of that preference, for example when you were in danger and you felt fear, or when you now reflect on that preference by reading this sentence. Hence, your preference to stay alive is now and was a moment ago a conscious preference.
The reason why preferences matter is straightforward: if you prefer a moral theory according to which preferences do not matter, your preference for that moral theory cannot matter either. But why should we only take into consideration conscious preferences (and not unconscious preferences) in our moral deliberations?
The theory of mild welfarism starts with the premise that only welfare matters. By definition, the welfare of a conscious, sentient being in a situation or outcome measures that individual’s strength of preference for that situation or outcome. In other words, welfare measures how good that situation is according to the individual itself. Hence, welfare is tightly linked to preferences. But an individual can only measure the strength of its preferences when it is consciously aware of its preferences.
The ability to pay attention is crucial: a preference only matters when there is someone (a sentient being) who can pay attention to that preference. And the ability to pay attention is the basis of consciousness, according to Michael Graziano’s Attention Schema Theory (AST) of consciousness. According to that theory, consciousness is the brain's simplified, schematic description of its own data-processing focus. To manage its own limited processing power, the brain uses attention to focus on specific signals. Signals can be perceptions or memories, but also goals or preferences. The brain builds an internal map or schema of that attention process in order to control its own internal focus. This attention schema is what we experience as awareness. We feel like we have conscious, subjective awareness because our brains are constantly building a model of our own attention.
The strength of a preference measures how strongly the entity tends to pay attention to the preference. A preference is a signal (in particular information about a goal function), and a strong preference is a signal that is (or tends to be) strongly amplified in order to gain or attract attention when the preference is at stake. Your preference to stay alive is strong, because when your live is at stake in a dangerous situation, the perception of the danger, the fear and the preference to stay alive all grab your attention. The information that staying alive is one of your goals, gets amplified. Without an attention mechanism, all preferences of the unconscious entity have zero strength, as with complete indifference. In the moral calculation, they count as zero.
In summary: a preference is only relevant in our moral considerations when the entity that has that preference is able to care about that preference, by being able to pay attention to that preference. In order to pay attention, such an entity needs to have an attention control mechanism or attention schema. Such a mechanism automatically and unavoidably generates a consciousness. That is why only conscious preferences matter. For unconscious entities, their preferences have zero strength and hence zero moral relevance.
What do you think about AI systems that don't have the kinds of limitations we do, and that might not need to restrict focus as much, or might be able to devote cognitive resources to many different problems in parallel. It is plausible that they wouldn't count as having attention in the way that Graziano thinks we do, and that they wouldn't therefore satisfy his theory of consciousness. But it seems weird to suggest that the reason our preferences matter is because of our cognitive limitations, no?
Ha, nice argument :-) It reminds me of a famous defense of predation argument in animal rights ethics: lions do not violate the rights of their prey, because lions do not have moral agency. In other words: lions have a privilege to harm, kill and use other animals, because they are too stupid to understand morality.
To the point: although it may seem weird, I think it makes sense: if due to cognitive limitations an organism was forced to generate an attention schema that generates consciousness, that organism is more concious than e.g. an AI that does not have those cognitive limitations that cause the development of an attention schema. Furthermore, in terms of the ability to pay attention, that unfocused AI (i.e. it is not able to focus), has a cognitive limitation. It lacks a cognitive ability. Although I consider it possible and even likely that a strong AI also develops an attention schema, because that is so beneficial and efficient. Our brains are also able to run processes in parallel, devoting resources to many (unconcious) parallel processes. Yet, we benefit from having the cognitive capacity to focus, to pay attention, using an attention schema. Hence, even with our cognitive ability of parallel processing, we developed an attention schema.
Perhaps a good analogy: some animals are not able to run fast on their legs to escape from predators. They have a physical limitation. Due to that limitation, those animals developped wings with which they can fly and escape from predators. Does it seem weird to suggest that the reason a bird can fly is because of its physical limitation to run fast?
Thought-provoking argument! However, I see some gaps:
1.
I think this mixes up two senses of "mattering".
I might not think my preference for hedonic utilitarianism is terminally valuable but still think that believing true things is instrumentally valuable to achieve utility.
2. I share your intuition that "preferences need to have a weight" but I don't think that's the same thing as "being represented within an attention schema". I think plants can weigh their preferences (e.g. tolerate drier soil if the luminosity is sufficient) without having explicit preferences (i.e. language?) or a comprehensive world model, let alone a meta-model.
While I agree with the thesis from the title, I think it might be better anchored in ~Sharon H. Rawlette's argument that conscious sensations of "(un)desirability" construct/define what we mean by morality.