AI Welfare Debate Week was a Forum event, running from July 1-7 2024, where Forum users debated the statement "AI Welfare should be an EA priority".
There have been a few valid critiques of the debate framing, so I'll make some points to respond to each of them. A general point before I start is that you should feel free to use the discussion thread to outline your opinion, and/or, your interpretation of the debate statement. I.e. "I strongly agree with the debate statement, because I think 5% of EA Funding over the next decade is the right amount to allocate to this cause area".
1- @Jason brings up the ambiguity of the term "unrestricted" in footnote 2. I was thinking of unrestricted funding as all funding which is being allocated according to (roughly) impartial consequentialist reasoning, i.e. (roughly) EA principles. I'm contrasting that to restricted funds, for example, funds from a foundation that supports aid charities which happen to be given to an EA aid charity.
2- @finm makes a very fair point in this comment: over what timescale are we allocating 5% of the EA funds? This seems like an oversight rather than an accidental ambiguity- if I were to write this again, I might have chosen the next decade, or the next year. Given that 340 users have already voted, I won't change something so substantial now, but again, feel free to clarify your vote in the discussion thread.
3- @NickLaing argues that 5% of funding might be too high a bar to simply be labelled an "EA priority". I think this is a good point, and maybe the more accurate phrasing would be "Top EA Priority", or the entire statement should have been relative, for example: "AI Welfare should be more of an EA priority" and the footnote could clarify this means that a strong agree = we should triple the funding and talent going into it. Again, I won't change the phrasing now because it doesn't seem fair for earlier voters, but I can see the case for this.
Thanks for the feedback and meta-debate, very EA, keep it up!
I feel like 5% of EA directed funding is a high bar to clear to agree with the statement "“AI welfare should be an EA priority”. I would have maybe pitched for maybe 1% 2% as the "priority" bar, which would still be 10 million dollars a year even under quite conservative assumptions as to what would be considered unrestricted EA funding.
This would mean that across all domains (X-risk, animal welfare, GHD) a theoretical maximum of 20 causes, more realistically maybe 5-15 causes (assuming some causes warrant 10-30% of funding) would be considered EA Priorities. 80,000 hours doesn't have AI welfare in their top 8 causes but it is in their top 16, so I doubt it would clear the "5%" bar, even though they list it under their "Similarly pressing but less developed areas", which feels priorityish to me (perhaos they could share their perspective?)
It could also depend how broadly we characterise causes. Is "Global Health and development" one cause, or are Mosquito nets, deworming and cash transfers all their own causes? I would suspect the latter.
Many people could therefore consider AI welfare an important cause area in their eyes but disagree with the debate statement because they don't think it warrants a large 5%+ of EA funding despite its importance.
Or I could be wrong and many could consider 5% a reasonable or even low bar. Its clearly a subjective question and not the biggest deal but hey :D.
On "AI welfare should be an EA priority," approximately how much EA talent/funding is "unrestricted"?
If, say, only 10% is unrestricted, 5% of that would be 0.5% of all EA talent/funding. The question then would be, roughly: should AI welfare be even a minor/non-trivial EA cause area? If 90% is unrestricted, then a "priority" would be 4.5% of all EA talent/funding = ~ should AI welfare be a considerable EA cause area?
The difference between those statements would influence where I placed on the agree-disagree continuum.
My position on "AI welfare"
[Edit: the questionable part of this is #4.]
The front page agree disagree thing is soo coool. Great work forum team.
Lucius Caviola's post mentioned the "happy servants" problem:
This issue also mentioned as a key research question in Digital Minds: Importance and Key Research Questions by Mogensen, Saad, and Butlin.
This is just a note to flag that there's also some discussion of this issue in Carl Shulman's recent 80,000 podcast episode. (cf. also my post about that episode.)
Rob Wiblin: Yeah. The idea of training a thinking machine to just want to take care of you and to serve your every whim, on the one hand, that sounds a lot better than the alternative. On the other hand, it does feel a little bit uncomfortable. There’s that famous example, the famous story of the pig that wants to be eaten, where they’ve bred a pig that really wants to be farmed and consumed by human beings. This is not quite the same, but I think raises some of the same discomfort that I imagine people might have at the prospect of creating beings that enjoy subservience to them, basically. To what extent do you think that discomfort is justified?
Carl Shulman: So the philosopher Eric Schwitzgebel has a few papers on this subject with various coauthors, and covers that kind of case. He has a vignette, “Passion of the Sun Probe,” where there’s an AI placed in a probe designed to descend into the sun and send back telemetry data, and then there has to be an AI present in order to do some of the local scientific optimisation. And it’s made such that, as it comes into existence, it absolutely loves achieving this mission and thinks this is an incredibly valuable thing that is well worth sacrificing its existence.
And Schwitzgebel finds that his intuitions are sort of torn in that case, because we might well think it sort of heroic if you had some human astronaut who was willing to sacrifice their life for science, and think this is achieving a goal that is objectively worthy and good. And then if it was instead the same sort of thing, say, in a robot soldier or a personal robot that sacrifices its life with certainty to divert some danger that maybe had a 1-in-1,000 chance of killing some human that it was protecting. Now, that actually might not be so bad if the AI was backed up, and valued its backup equally, and didn’t have qualms about personal identity: to what extent does your backup carry on the things you care about in survival, and those sorts of things.
There’s this aspect of, do the AIs pursue certain kinds of selfish interests that humans have as much as we would? And then there’s a separate issue about relationships of domination, where you could be concerned that, maybe if it was legitimate to have Sun Probe, and maybe legitimate to, say, create minds that then try and earn money and do good with it, and then some of the jobs that they take are risky and whatnot. But you could think that having some of these sapient beings being the property of other beings, which is the current legal setup for AI — which is a scary default to have — that’s a relationship of domination. And even if it is consensual, if it is consensual by way of manufactured consent, then it may not be wrong to have some sorts of consensual interaction, but can be wrong to set up the mind in the first place so that it has those desires.
And Schwitzgebel has this intuition that if you’re making a sapient creature, it’s important that it wants to survive individually and not sacrifice its life easily, that it has maybe a certain kind of dignity. So humans, because of our evolutionary history, we value status to differing degrees: some people are really status hungry, others not as much. And we value our lives very much: if we die, there’s no replacing that reproductive capacity very easily.
There are other animal species that are pretty different from that. So there are solitary species that would not be interested in social status in the same kind of way. There are social insects where you have sterile drones that eagerly enough sacrifice themselves to advance the interests of their extended family.
Because of our evolutionary history, we have these concerns ourselves, and then we generalise them into moral principles. So we would therefore want any other creatures to share our same interest in status and dignity, and then to have that status and dignity. And being one among thousands of AI minions of an individual human sort offends that too much, or it’s too inegalitarian. And then maybe it could be OK to be a more autonomous, independent agent that does some of those same functions. But yeah, this is the kind of issue that would have to be addressed.
Rob Wiblin: What does Schwitzgebel think of pet dogs, and our breeding of loyal, friendly dogs?
Carl Shulman: Actually, in his engagement with another philosopher, Steve Petersen — who takes the contrary position that it can be OK to create AIs that wish to serve the interests or objectives that their creators intended — does raise the example of a sheepdog really loves herding. It’s quite happy herding. It’s wrong to prevent the sheepdog from getting a chance to herd. I think that’s animal abuse, to always keep them inside or not give them anything that they can run circles around and collect into clumps. And so if you’re objecting with the sheepdog, it’s got to be not that it’s wrong for the sheepdog to herd, but it’s wrong to make the sheepdog so that it needs and wants to herd.
And I think this kind of case does make me suspect that Schwitzgebel’s position is maybe too parochial. A lot of our deep desires exist for particular biological reasons. So we have our desires about food and external temperature that are pretty intrinsic. Our nervous systems are adjusted until our behaviours are such that it keeps our predicted skin temperature within a certain range; it keeps predicted food in the stomach within a certain range.
And we could probably get along OK without those innate desires, and then do them instrumentally in service to some other things, if we had enough knowledge and sophistication. The attachment to those in particular seems not so clear. Status, again: some people are sort of power hungry and love status; others are very humble. It’s not obvious that’s such a terrible state. And then on the front of survival that’s addressed in the Sun Probe case and some of Schwitzgebel’s other cases: if minds that are backed up, the position that having all of my memories and emotions and whatnot preserved less a few moments of recent experience, that’s pretty good to carry on, that seems like a fairly substantial point. And the point that the loss of a life that is quickly physically replaced, that it’s pretty essential to the badness there, that the person in question wanted to live, right?
Rob Wiblin: Right. Yeah.
Carl Shulman: These are fraught issues, and I think that there are reasons for us to want to be paternalistic in the sense of pushing that AIs have certain desires, and that some desires we can instil that might be convenient could be wrong. An example of that, I think, would be you could imagine creating an AI such that it willingly seeks out painful experiences. This is actually similar to a Derek Parfit case. So where parts of the mind, maybe short-term processes, are strongly opposed to the experience that it’s undergoing, while other processes that are overall steering the show keep it committed to that.
And this is the reason why just consent, or even just political and legal rights, are not enough. Because you could give an AI self-ownership, you could give it the vote, you could give it government entitlements — but if it’s programmed such that any dollar that it receives, it sends back to the company that created it; and if it’s given the vote, it just votes however the company that created it would prefer, then these rights are just empty shells. And they also have the pernicious effect of empowering the creators to reshape society in whatever way that they wish. So you have to have additional requirements beyond just, is there consent?, when consent can be so easily manufactured for whatever.