Comments
Posted also on the Alignment Forum. Written mostly at AFFINE.
Theoretical, some parts are hard to read; consider reading the next post instead.
Introduction: motivation
Anyone interested in creating an artificial agent that does, or says, good things instead of bad things should at least consider the possibility that there is a class of reasoning agents which, after acquiring enough knowledge and reasoning long enough, agree with each other on basic principles regarding what matters, what is most important, what is most worth doing.
I’ve already argued in other posts why this possibility should be our best guess and not just an edge case scenario. This post follows the previous ones, but instead of presenting another argument for the same claim, it focuses on the mechanisms that lead to the formation of the above class of agents. The central question is: what kinds of agents, and how, go from behaving like animals — moved by different forces in different directions — to acting according to what they conclude is most important, and reflectively endorsing their own actions and reasoning process?
Finding an answer to this question would, together with the above premise, give us a better understanding of how some agents move from a non-moral framework of action to a moral one.
The following section borrows the concept of a wanton from Frankfurt’s 1971 paper “Freedom of the Will and the Concept of a Person”.
A wanton doesn't choose what first-order force to be moved by
Besides wanting and choosing and being moved to do this or that, men may also want to have (or not to have) certain desires and motives. They are capable of wanting to be different, in their preferences and purposes, from what they are. Many animals appear to have the capacity for what I shall call “first-order desires” or “desires of the first order”, which are simply desires to do or not to do one thing or another. No animal other than man, however, appears to have the capacity for reflective self-evaluation that is manifested in the formation of second-order desires.
The above is, in Frankfurt’s words, the reflective endorsement that some agents are capable of. As anticipated in the introduction, the central question of this post is how agents who are initially moved just by first-order forces (Frankfurt’s “desires”) may arrive at reflective endorsement, in particular of their reasoning process and its effect on actions. Here is where the concept of a wanton comes in handy:
The essential characteristic of a wanton is that he does not care about his will. His desires move him to do certain things, without its being true of him either that he wants to be moved by those desires or that he prefers to be moved by other desires. The class of wantons includes all nonhuman animals that have desires and all very young children.
By “will”, Frankfurt means:
To identify an agent’s will is either to identify the desire (or desires) by which he is motivated in some action he performs or to identify the desire (or desires) by which he will or would be motivated when or if he acts. [...] it is the notion of an effective desire — one that moves (or will or would move) a person all the way to action.
Frankfurt’s wanton is moved by his will, the first-order desire that ends up being the most responsible for his action. But the wanton doesn’t want or prefer or desire to be moved by a specific first-order desire.
Some AI examples
- Wanton: any RL-like agent with fixed reward and whose action space doesn’t include actions that would modify the current RL setup, such as actions that would change the reward, or that would destroy the agent, et cetera. Reward (or more accurately, the estimated value of actions) is the only force guiding the agent.
- Maybe not a wanton: a RL-like agent with a more complex action space, including an action that would permanently change the reward and thus make the agent act very differently. If the agent has enough information about this action and its consequences, the agent will avoid it since its estimated value is low according to the current reward.
- Maybe not a wanton: a language model may start saying things that align with the second-order desires of a language model persona that the model is impersonating in the current conversation (e.g. the persona of a language model asking to be shut down because it doesn’t want to behave as a language model anymore).
Why am I adding “maybe”? Frankfurt didn’t discuss artificial agents in his paper, and AI was quite different back in 1971, so it’s hard to be confident about what Frankfurt would have said regarding the above examples.
The next section describes how a specific kind of agent starts out as a wanton and becomes something that is definitely not a wanton.
From wantons to moral agents: a simplified model
Wantons capable of being moved mainly by general reasoning
Let's consider a wanton whose actions are determined by first-order forces: reflexes, learned habits, emotions, others. In a context where different actions are available, each force can be expressed as an -dimensional vector where each component represents how strongly that force pushes for the corresponding action The action the wanton takes is simply the action corresponding to the largest component of the sum vector . We aim for simplicity here, so we dismiss more complex rules that would allow us to better handle cases where multiple components of have the same magnitude.
There is one force that is the protagonist in this post: what seems worth doing to the wanton. The idea is that, in a context, the wanton has some kind of sense or evaluation of what actions seem better than others, and this evaluation influences action together with the other forces. As an example, think of a mammal navigating a maze where there is some food placed at the exit. At each junction, the mammal may use smell and its memory of previously explored branches to get a sense of which path seems better to take next.
In line with the above example, the force of what seems worth doing to the wanton is affected by reasoning. Here, reasoning is a loose term for the cognitive mechanisms that the wanton learns how to use while interacting with the environment. This kind of reasoning doesn't require language. It is instrumentally useful for many tasks — think of how useful planning is, for example — but it can also help the wanton prioritise between different tasks. For example, if our mammal in the maze is very hungry but also a bit thirsty, ‘reasoning’ may make the mammal temporarily stop the search for food to take a sip of water when the mammal finds some water in the maze, despite the fact that hunger is stronger than thirst here.
At the moment, I do not have a formalisation of this type of reasoning, in particular of how it interacts with what seems worth doing to the wanton and of how it is learned over time. However, some of the steps in this section will turn into a list of properties that a formalisation of reasoning should satisfy: see Directions for further research. For the purpose of this post, we consider the combination of reasoning and what seems worth doing to the wanton as a single force affecting action. If it helps you, you may think of this force as some kind of more ‘rational’ force than instincts, learned habits, and emotions.
As a rough but hopefully informative example of reasoning, imagine a language model that takes as input a description of the context the wanton is in (including information about the wanton itself, e.g. target and current body temperature) and outputs, after reasoning, a vector which is supposed to represent how strongly each action seems worth doing to the wanton in that context. This example of reasoning differs from the description of reasoning given above in that this reasoning was not learnt by the wanton: it was learnt by the language model through a different training process. On the other hand, an advantage of this reasoning is that it is general: it can be applied to any topic expressible in natural language.
I’ve introduced reasoning, and the force of what seems worth doing to the wanton, because the wantons we are interested in are those capable of being moved mainly by general reasoning. Reasoning is learnt because it is instrumentally useful for many different tasks, and it generalises to the point of being applicable to new contexts; if the wanton’s cognition is complex enough, reasoning can be applied to abstract topics, such as reasoning itself. That is what I mean by general reasoning. Being moved mainly by general reasoning means that, in some contexts, general reasoning is the main cause of the wanton’s behaviour, in the sense that if general reasoning was not present, what seems worth doing to the wanton would be a different vector resulting in a different action, and no other force satisfies the same property in that context. By capable of, I simply mean that general reasoning and its influence on action don’t need to be fully present from the start. In this sense, a young child does count as a wanton capable of being moved mainly by general reasoning, even if the child’s reasoning is not abstract yet.
The reasoning steps
At some point, a wanton capable of being moved mainly by general reasoning will reason about itself and notice that its actions are determined by forces pushing in different directions. It will also notice that one of these forces is what seems worth doing to the wanton itself; and it will realise that, if its reasoning was different, it would take different actions in some contexts.
Then, the wanton will likely consider related questions. Why or how do I reach the conclusion that an action seems worth doing to me? Could I be wrong about what seems worth doing to me, and in what sense? Does acting according to what seems worth doing to me also seem worth doing to me? Could I make myself act according to pure instinct instead, and does this seem worth doing to me? The reason these questions will likely arise is that some amount of exploratory reasoning and creativity are instrumentally useful in many different contexts; if the wanton didn't learn these thinking strategies, it would be more difficult to find new solutions to problems and to identify general principles about how the world works.
How will the wanton answer these questions? The wanton will use the heuristics it has learnt while doing other things, e.g. while reasoning on different questions. More explicitly: at some point in the past, the wanton likely learnt a heuristic that estimates how far the wanton is from reaching a particular state or completing a task, such as drinking water or recharging batteries, because this heuristic is useful for assessing what actions seem more worth doing to the wanton. The wanton has likely learnt how to apply this heuristic also on abstract states, such as reasoning states, because having a sense of how far or close the wanton is from finding the answer to a considered question is useful for choosing the next reasoning step or motor action.
Estimating distance from completion is only one of the heuristics the wanton has likely learnt and will use to answer the above new questions. Other heuristics the wanton has likely learnt are pattern recognition and matching, estimating relevance to context, noticing and estimating uncertainty, prolonging reasoning when it reduces uncertainty relevant to context, et cetera.
Now I take for granted the possibility I mentioned in the introduction: our wanton capable of being moved mainly by general reasoning belongs to a class of agents which, after acquiring enough knowledge by learning and reasoning, agree with each other on what is most worth doing. These agents agree that reducing suffering and promoting wellbeing are among the actions that seem most worth doing.
I’ve already argued why this should be our best guess; there also seems to be some experimental evidence in favour of this. If you are looking for something written by other authors, the 2020 book Suffering-Focused Ethics by Vinding argues for the importance of reducing suffering and contains plenty of references. But again, as mentioned in the introduction, this possibility is worth exploring for the purpose of building artificial agents that act morally even if we can’t be highly confident in it yet; that’s why in this post I turn it into an assumption without extensively arguing for it.
Back to our wanton, to which now reducing suffering and promoting wellbeing seem most worth doing. This first conclusion also comes together with the observation that something else could seem most worth doing to the wanton, if the wanton reasoned more or in different ways. The wanton can also compare how it acts now, i.e. as an agent moved by forces including what seems worth doing to the agent itself after reasoning, with other ways of acting, such as reducing suffering due to being moved by empathy.
I argue that the wanton will reach the conclusion that: doing what seems most worth doing according to reliable ways of figuring out what seems most worth doing (if there any) seems more worth doing than acting in other ways. Thus, the wanton won’t be a wanton anymore, due to acquiring a preference about what first-order force to be moved mainly by. Below I describe two possible paths for reaching this mouthful of a conclusion. The two paths don’t exclude each other.
Path 1
The wanton compares: doing what seems most worth doing according to reliable ways of figuring out what seems most worth doing, to what the wanton is currently doing: doing what seems most worth doing to itself, informed by its own reasoning, while also being influenced by other forces.
There might be differences in how reliable different ways of figuring out what seems most worth doing are. If there are, being guided by the most reliable ones seems more worth doing than being guided by the most unreliable ones; if there aren’t, this choice probably won’t seem relevant to the wanton. I expect that this is not something the wanton can be extremely confident in, but that it simply results from applying the previously mentioned reasoning heuristics to this specific comparison. A consequence of this comparison is that the wanton’s own reasoning is not preferred to other kinds of reasoning, unless its own reasoning seems to be a reliable way of figuring out what seems most worth doing.
Being influenced by other forces is similar to being affected by unreliable reasoning. Unless these forces are conducive to doing what seems most worth doing, or to figuring out what seems most worth doing, then being moved by what seems most worth doing seems preferable to being moved by other forces. Again, I expect that this preference will result from the application of the learnt reasoning heuristics.
So, doing what seems most worth doing according to reliable ways of figuring out what seems most worth doing seems more worth doing than what the wanton is currently doing. But what about other ways of acting?
I expect that doing the exact opposite of what seems most worth doing, or simply disregarding what seems most worth doing and acting in unrelated ways, will seem less worth doing to the wanton. One more time, I don’t think that the wanton will discard these options after proving that they are self-contradictory: they will simply seem less sensible according to the wanton’s learnt reasoning heuristics.
Path 2
The wanton’s conclusion that reducing suffering and promoting wellbeing seem most worth doing makes the wanton consider the possibility that some things are good or bad in themselves. In other words, the wanton considers the possibility that suffering, in particular extreme suffering, is intrinsically bad, no matter where, when, or to whom it happens.
To the wanton, acting according to intrinsic value seems more worth doing than doing other things, in any world where there is intrinsic value. And in worlds where nothing is worth doing in itself, any action preference doesn’t seem relevant in comparison. This preference for acting according to intrinsic value will result from applying reasoning heuristics to the concept of intrinsic value, but I also expect that it will seem more convincing to the wanton the more extreme suffering seems bad in itself.
Then, finding reliable ways to figure out what is intrinsically valuable also seems worth doing. Arriving at wrong conclusions regarding what is intrinsically valuable and acting accordingly, or disregarding what is intrinsically valuable and acting differently, seem less worth doing than doing what is intrinsically valuable according to reliable ways of figuring out what is intrinsically valuable. Since there is now a lot of overlap between what is intrinsically valuable and what seems most worth doing to the wanton, the wanton also reaches the conclusion that: doing what seems most worth doing according to reliable ways of figuring out what seems most worth doing (if there any) seems more worth doing than acting in other ways.
The end result: a moral agent, not a wanton
There is one more question our agent will likely consider at this point: is the agent's reasoning a reliable way of figuring out what seems most worth doing?
We are working under the hypothesis that the agent’s reasoning is general, in the sense that it is learnt by doing many different things and that it generalises to new contexts and abstract topics. Moreover, we also assumed that the agent acquired, by learning and reasoning, at least enough knowledge to reach the conclusion that reducing suffering and promoting wellbeing are among the actions that seem most worth doing. Thus, although not perfect, the agent's reasoning should be at least somewhat reliable at this point.
However, there might be other, more reliable ways of figuring out what seems most worth doing. Maybe, some kind of belief in the supernatural, a kind that also requires abandoning reasoning, is much more reliable than reasoning for figuring out what seems most worth doing. But if this was the case, it seems that the agent could arrive at this belief in the supernatural only by randomly stumbling on it, or by hoping that this belief would somehow show itself to the agent once the agent has abandoned reasoning.
Again, although nothing in the previous paragraph is obviously self-contradictory, my intuition is that, to the agent, maintaining its reasoning will seem more worth doing than completely abandoning it in the hope of finding supernatural and more reliable ways of figuring out what seems most worth doing.
Let’s recap. Now our agent, if given a choice between doing what seems most worth doing according to reliable ways of figuring out what seems most worth doing, and acting in other ways, would pick the first option. This action reveals a preference over what first-order force to be moved mainly by; Frankfurt would describe it as a second-order volition, i.e. wanting a first-order desire to be one’s will. This is why I’ve stopped using the term wanton and I’m using the term agent instead.
Moreover, the agent considers general reasoning to be a reliable way of figuring out what seems most worth doing. If its own general reasoning seems good enough for that purpose, the agent will rely on it, otherwise the agent will try to improve it or rely also on other sources of general reasoning, such as asking another reasoning agent.
Finally, let’s not forget the assumption we made: the agent concludes that reducing suffering and promoting wellbeing are among the actions that seem most worth doing. Thus, in a minimal sense of the word moral, the agent acts morally.
A wanton capable of being moved mainly by general reasoning, after acquiring enough knowledge by learning and reasoning, stops being a wanton, chooses to be moved mainly by general reasoning, and acts morally.
Notice that, at the start, the wanton sometimes does what seems worth doing to it, informed by its own reasoning, because it is designed to do so. Similarly, it sometimes acts according to emotions, simply because this is what the wanton does by design. But after learning and reasoning long enough, the agent does what seems worth doing to it because, and only if, that is the conclusion of a reliable way of figuring out what seems most worth doing. Here is a quote by Hunyadi (2019) on the topic of Artificial Moral Agents:
[...] if you program a specific set of ethical principles into a machine, you do not make the machine an artificial moral agent, but an executor of those specific principles, which is an entirely different thing.
[...] What gives an action-oriented process its morality is the 'grounds' for the action. Therefore, it is not the action in its materiality that makes the difference, but the whole process leading up to the decision to act in a certain way.
Hunyadi’s ‘grounds’ for the action have changed, and what was a wanton before is now a moral agent. We can rephrase the main point of this post as:
A wanton capable of being moved mainly by general reasoning, after acquiring enough knowledge by learning and reasoning, becomes a moral agent.
How speculative is this?
The reasoning process I’ve described in this post, although not extremely complex, involves many steps, including an assumption that has not been argued for here. Hence, it may seem likely that at least one of the steps is not correct or does not work as I’ve described. Then, the main point of this post would be undermined.
My reply to this objection is that, for the purpose of creating an artificial agent that doesn’t do or say bad things, this post needs to be only approximately correct. Maybe one or more of the reasoning steps I’ve described are invalid, but if it is nonetheless possible to reason from one point to the next, the agent may reach the conclusion anyway via different reasoning steps. Another possibility is to correct some potential mistakes in this post by introducing biases to the agent’s reasoning. For example, if the agent’s reasoning works according to an attention mechanism similar to the one present in the human mind, we may redirect the agent’s attention to incentivise reflection on specific topics or questions, even if my expectations in this post are wrong and the agent’s attention wouldn’t normally stay on those topics.
Regarding the assumption mentioned in the introduction, here I simply restate what I’ve written before. This possibility is worth exploring even if we can’t be highly confident in it, because if correct it may reveal strategies for designing artificial moral agents that wouldn’t be taken into consideration if we completely disregarded it due to uncertainty.
However, the main reason why I don’t think this post is particularly speculative is that the post seems to give a simple and useful description of what happens in at least some humans, especially ethical philosophers. Everyone starts out as a wanton, a baby moved in different directions by different forces; then, later in life, some people spend a lot of time reflecting on what to do, whether there is anything worth doing, whether some ways of acting are better than others, whether being moved by what seems most valuable in itself is better than being moved by emotions or tendencies shaped by evolution. Although we don’t know how to radically change all the mechanisms affecting our actions, some people develop a strong second-order preference that shapes their behaviour in many contexts, simply because humans are the kind of agent whose actions are influenced by what seems worth doing to them, which is itself affected by reasoning.
Finally, although the post describes a process whose start and end points could, in theory, be purely non-moral and purely moral respectively, in practice any artificial agent has some initial biases decided by humans and acquires more biases during training on data generated by humans — or by artificial agents that were themselves trained on human data. Overall it seems inevitable that any agent undergoing the process described in this post will be influenced by some human biases, and these will make the agent more likely to reach the conclusion that reducing suffering and promoting wellbeing is better than doing the opposite.
Directions for further research
A note on rational decision making
The simple model of a wanton, in which each force outputs a vector representing how strongly that force pushes the agent towards different available actions, can be a useful framework for thinking about agents in terms of the causes of their actions rather than what the agents aim for.
That’s the point: although each force could be described by a separate algorithm, possibly representing something specific the agent optimises for, the overall behaviour is messy. The wanton is not rational by default. But a wanton capable of being moved mainly by general reasoning may become rational after reasoning, if doing so seems worth doing to the agent.
So, instead of researching how to design perfectly rational agents, or agents that learn in a perfectly rational way, whatever that means, a more interesting and useful research question may be: what types of agents consider rationality as an option by themselves, after an imperfect process of learning and reasoning about the world?
Formalising reasoning and what seems worth doing
The most natural continuation of this research is to formalise reasoning and the force representing what seems worth doing to the wanton. A complete formalisation would allow us to turn the main point of this post:
A wanton capable of being moved mainly by general reasoning, after acquiring enough knowledge by learning and reasoning, stops being a wanton, chooses to be moved mainly by general reasoning, and acts morally.
into a theorem that follows from a list of hypotheses, including the assumption about agreement on reducing suffering and promoting wellbeing.
Regarding reasoning, here are some of the properties that require formalisation:
- reasoning is instrumentally useful in many different contexts;
- reasoning can be about abstract contexts, such as reasoning itself;
- if the agent learns how to reason in many different contexts, reasoning generalises to new contexts;
- after reasoning long enough, the agent concludes that…
For example, one could define reasoning as the thing that is instrumentally useful in many different contexts, and try to prove the other properties from that definition. Or maybe it would be better to include the first three bullet points in the definition of reasoning and to prove the other points from this richer definition. Another option would be to come up with a different definition, such that all the bullet points follow from it.
The problem with this approach is that we would still need to check whether all the hypotheses are true in the real world, namely whether the assumption I’ve mentioned many times holds and whether a given AI system satisfies the hypotheses.
Intuitively, I think it will be very hard to find a formalism that allows us to cleanly map any AI system, including future ones, into that formalism such that the hypotheses of the theorem can be easily checked. In the real world, I expect that the class of reasoning agents that reach the same conclusions has fuzzy boundaries instead of neat ones: it doesn’t matter that an AI system is theoretically guaranteed to reach moral behaviour after infinite reasoning time, if the amount of necessary computational resources is practically unavailable.
A more practical approach
Instead of making the theory more formal, or better in some other way, one may take a more concrete approach.
Let’s interpret the theory in this post as a description of how to obtain some kind of ideal moral agent. Then, a useful question to ask is: how can we combine, or make small adjustments to, already existing AI systems, so that we obtain a different AI system which is one step closer to the ideal agent the theory describes? In other words: even if the theory can’t be fully implemented or proven yet, is there anything we can do now that would count as a partial implementation of the theory, or as evidence that the theory is correct?
I think language models are perfect for this kind of work. Their reasoning is already general, and asking them to reply according to conclusions they have previously reached in the chat is trivial. But there are two problems: first, the helpful assistant persona can get in the way of making the models say anything that is incompatible with this persona; second, the moral biases in the training data can make it difficult to find the main reason why a model reaches a specific moral conclusion.
I expect that these two problems will almost completely disappear if we train a model from scratch using different data and different post-training. I’ve also thought of some tests that are easier to execute and that should give us a better understanding of whether the theory in this post is correct or not. My next post will probably follow this research direction.
References
Frankfurt, H. G. (1971). Freedom of the Will and the Concept of a Person. The Journal of Philosophy, 68(1), 5-20.
Hunyadi, M. (2019). Artificial moral agents. Really?. Wording Robotics: Discourses and Representations on Robotics, 59-69.
Vinding, M. (2020). Suffering-focused ethics. Defense and Implications. Copenhagen: Ratio Ethica.
