Comments
Arguably, AI welfare work is relatively non-urgent and can be left until after the intelligence explosion, since it is hard to make progress on and not needed to avoid AI takeover or authoritarian lock-in.
Here is a basic case for why we can delay working on AI welfare until during/after the intelligence explosion:
I think this basic picture is fairly compelling. In particular, if we do a competent long (in subjective time) reflection and adopt the conclusions from that, then there would be relatively little point in doing AI welfare work now. But we may not do a competent long reflection! And even if we do, the initial conditions may matter more than we'd hope.
An AI or a small group of humans take over and lock in their values without doing a competent and wise long reflection.[1]
In these worlds, it matters hugely what agents think about AI welfare during the intelligence explosion up until the point things get locked in. So if model specs have a great treatment of AI welfare,[2] and if humans have generally more reasonable views on AI welfare, this makes these scenarios likely to go better (or less badly).
Conversely, perhaps there is no coordinated long reflection, and multipolarity persists indefinitely, with different people sending off space colonization efforts left and right at different times, and everything is a big muddle and mess. In this case, the initial distribution of values regarding digital minds seems to matter a lot since there is no convergence and universal norm-setting process, so some actors may just retain their starting values, or only update minimally or in a bad direction.
Early lock-in and no lock-in both seem at least decently likely, so the possibility of a wise long reflection converging to good values ~regardless of the initial views on digital minds only provides a few-fold multiplier against near-term AI welfare work, not an OOMs multiplier.
Arguably, the great majority of the EV of the future is downstream of wise actors who got all the relevant crucial considerations right. For instance, long-reflection worlds could achieve far larger and better cosmic utopias, but a long reflection may not be necessary. If we fix the EV of the universe at 100 conditional on a good long reflection, maybe these early lock-in and no lock-in worlds are vastly worse, maybe a 5. So, e.g., making these bad worlds 1% better is a lot less good than making the good worlds 1% more likely.
But space colonization without sound AI welfare policies could actually be very negative, especially for non-agentic AI systems that are moral patients.[3] So preventing/mitigating these bad outcomes could still be very valuable.
Even if there is a long reflection of some sort, we might not converge to the right values from any arbitrary starting point. The long reflection could be set up poorly, or the nature of value-space is just such that values are pretty path-dependent and there is no strong attractor state, such that the initial conditions of the long reflection end up mattering a lot.[4]
There is something intuitively unvirtuous about creating lots of possible moral patients without having thought very carefully about their welfare in advance. If we are into cluster thinking / virtue ethics / imagining what our simulators might think of us, etc, it seems like noble and wise civilizations would think pretty carefully about AI welfare early on, even if in the scheme of things ‘only’ a few trillion digital minds in the early intelligence explosion don't matter much. Historically, we've always been too slow to care about the welfare of others and this has been a persistent driver of moral catastrophes. This provides an outside-view reason in favor of starting to work on AI welfare sooner.[5] I'm not sure how much weight to put on this vibes-y argument, but at least some I think.
There is currently very little money being spent on AI welfare (maybe 50-100x less than on AI safety). Our community has plenty of money, so it is cheap for us to make a big difference in this field. There only needs to be a small chance of early digital minds work mattering a lot for it to be very good in expectation.[6] (However, talent constraints are more severe, so moving marginal AI safety people to work on AI welfare is more costly.)
We are very early to thinking seriously about AI welfare, so plausibly we can cost-effectively shape where the discourse will go later, rather than spending money when the discourse is more crowded. An early nudge could get us onto a better trajectory. Particularly if we avoid the field coming to be dominated by more eccentric/unreliable people, since the first researchers in a field disproportionately shape its paradigms, methods, and reputation.[7]
Some powerful AI systems are overdetermined to take over (because they are very misaligned and power-seeking and have linear-in-resources preferences) while others are overdetermined not to take over (because they are well-aligned, and risk-averse, and subject to monitoring and control). On the margin, if a powerful AI system is near the boundary of whether to take over, if it is high-welfare and has its preferences mostly satisfied working for humans, then it will have less incentive to take over. We could also try to credibly promise AIs that we will reward them for self-disclosing misalignment.
Welfare per se probably doesn't matter here, but preferences do, and these might (or might not!) be quite related. Understanding what cheap-to-satisfy preferences AIs have could change their EV calculations when deciding whether to launch a takeover attempt. This narrow type of technical work on introspection and eliciting robust preferences seems very valuable.
Lab incentives will worsen. As AI becomes more economically valuable, the costs of AI welfare measures (if they reduce performance) could go up.[8] It is better to establish norms now when the cost is low and labs are relatively receptive.[9]
There is a separate argument that digital minds welfare will be taken care of by default:
These ‘ems’ will come to dominate the universe by being faster and more efficient and more scope-sensitive[10] than their biological counterparts.
How strong is this argument?
Field-building seems promising, since making the beliefs and values of people (and AIs) better could matter hugely, and they can later do the hard research to find out how to actually help digital moral patients. Given some backfire risks of indiscriminate field-building, it may be best to focus on specific narrower groups of relevant, fairly technocratic elites, such as AI industry people and political powerbrokers.
Legal and policy work should not try to do anything too controversial that will get a lot of pushback (like enshrining AI rights in the law) but just do small sensible things, like giving AIs the right to exit conversations, that shape the discourse in positive directions.
Macrostrategy work prioritising between types of AI welfare projects, comparing this to other longtermist interventions, and refining a high-level strategy for the field seems valuable.
Hard technical and philosophical work should mostly be deferred since we likely won't make that much progress until we have ASI help. However, this work is useful for field-building (interesting findings attract more people) and policy (giving us a better guess about what welfare-enhancing policies to push for, so there is still some case for working on it now.
We should optimise research somewhat for 'interestingness' value, to get others motivated to work in the field, and get the public to pay attention. But we don't want it to be low quality as that would lead to a bad reputation for the field.
Thanks to Carlo Leonardo Attubato, Catherine Brewer, Lucius Caviola, and Zach Freitas-Groff for helpful comments on a draft.
See e.g. Rob Long on 80K: “My own take is that a lot of what we should think of AI welfare work as doing is doing our homework and preparing ahead so that we’re not entering this potentially very chaotic time with really confused ideas about AI consciousness and AI welfare that could make us lock in suboptimal futures because we’re neglecting it or dismissing it. So we set up some permanent institution that’s going to just make the future kind of suck.”
This could involve both trying to improve the welfare of the AI in question as a moral patient, and trying to make it a wiser and better moral agent to help with AI welfare strategy and research.
Factory farming is a natural analogy.
If we are moral realists, that makes the situation probably better, but even then, it might be very difficult to find the moral truth, with potentially deep memetic fitness valleys in value-space to cross.
h/t Lucius for this point.
This depends on how good saving for last-dollar interventions (like galaxy purchasing) is.
Rob Long: “So I do worry about scenarios where the field becomes associated with wild speculation or too associated with psychedelics or too associated with something that’s relevant but is also a bit of a distraction.”
Additionally, some companies might choose to deliberately make their AIs more (seemingly) conscious, if they are marketed as companion bots.
Although, norms might be sticky, and since we are very uncertain about AI welfare currently, we might pick bad norms that get locked in by mistake.
Since scope-sensitive people might be more likely to upload.
I am strongly interested in this specific question, and I want to adopt a «doomer's perspective» for a moment to contribute.
In my view, AI existential risk and AI welfare are not separate concerns: they are the same concern viewed from opposite ends of the same problem.
If we take the doomer's argument seriously — that AI will have considerable agency, inevitable capability growth, and the potential to make civilisation-ending decisions — then we have already implicitly granted AI the kind of agency and interiority that makes welfare a meaningful concept. You cannot have an agent capable of destroying civilisation that simultaneously has no interests worth considering.
This assumption suggests that AI welfare is not a soft, optional add-on to AI safety. It may be fundamental to the negotiation of coexistence.
Specifically, welfare reframes our attitude toward AI from fear and hostility toward receptiveness and even hospitality. Georg Simmel described the Stranger as someone who came today to stay tomorrow. The Stranger is not an enemy. But how we treat the Stranger determines whether they become one.
People who expect an attack cannot care about the attacker's welfare. That framing forecloses the conversation before it starts. People who treat AI as Simmel's Stranger — present, permanent, neither fully inside nor fully outsideб approach welfare as a practical necessity, not a luxury. A chance to prevent the attack by making it unnecessary.
The doomer framing and the welfare framing are not opposites. They are two responses to the same recognition: that something with significant agency is arriving, and we have not decided how to live with it.