[Crossposted at https://vaccha.com/beneficiary-preferences/]

Introduction

When assessing interventions, GiveWell assigns "moral weights" to outcomes such as averting the death of a child or doubling the consumption of someone in extreme poverty. Moral weights represent the relative goodness of outcomes. So if you assign 1 to averting the death of a child and 0.1 to doubling the consumption of someone in extreme poverty, then you are treating averting the death as ten times as good as the consumption doubling.

These moral weights can make a big difference. If you assign a relatively high value to doubling someone's consumption, then you might conclude that GiveDirectly is more cost effective than the Against Malaria Foundation. If you assign a relatively high value to averting a death, then you might conclude the opposite.

Coming up with moral weights is hard, so GiveWell funded a study into the preferences of those who could be affected by some of their top-ranked interventions.^[1] The idea seems to be that if those affected by an intervention would assign a particular weight to, say, averting the death of a child, then then we should assign that same weight to averting the death of a child when assessing the intervention. Or maybe the moral weights we use should at least be influenced in part by the moral weights that would be chosen by the potential beneficiaries.

But it's unclear what exactly this amounts to or what its justification is. So in this post I'll consider three ways in which you might let your moral weights be guided by beneficiary preferences, what if anything could justify each approach to moral weights, and what some challenges to each approach are. At the end, I'll briefly consider how you could take beneficiary preferences into account even if they don't affect the moral weights you use.

Approaches to moral weights

Person-relative

On the person-relative approach, you use different moral weights when assessing an intervention's impact on each person depending on that person's preferences. So if there is an intervention that affects both me and you, someone taking this approach would use the moral weights that match my preferences when assessing its impact on me, and use the (possibly different) moral weights that match your preferences when assessing its impact on you.

I do think there is something intuitive about this approach. The core idea seems to be that your preferences determine the impact that things have on your well-being, so if you want to maximize someone's well-being then you should try to maximize their preference-satisfaction. But I think there are a few issues with this approach.

The first issue is just a limitation: many of those affected by interventions lack any relevant preferences. For example, one question GiveWell wanted to answer was how much to value averting the death of a child under the age of five to averting the death of an adult. But the researchers, understandably, never interviewed children under the age of five to figure out what their preferences are, since they aren't sophisticated enough to have any relevant preferences.

The researchers did look at the preferences of adults in the children's community about saving children compared to saving adults. But if you adjust the moral weights of outcomes that affect one person as a result of someone else's preferences, you aren't taking the person-relative approach. Instead, you're probably taking the community-relative approach or the same-for-all approach, which I consider in the next two subsections.

The other two issues with this approach have to do with the motivation for it that I presented above. I think the core claim is not true generally, and even if it were true it wouldn't fully support this approach.

First, it's not true in general that someone's well-being will be maximized by satisfying their current preferences. For example, suppose that I discount my future at a high rate and would prefer a small benefit now to a massive benefit later. Despite that preference, I'd be better off with the massive benefit later. Or suppose that I have an irrational fear of death and would prefer even a second of extended life to any increase in happiness. Despite that preference, I'd be better off having my happiness significantly increased.^[2]

Second, even if your preferences at a time completely determine the impact that things at that time have on your well-being, it doesn't follow that your preferences determine how important your well-being is compared to other things, such as the length of your life. But this is one of the questions that GiveWell wanted to answer.^[3] Of course, you might think that your preferences determine both, but it's worth flagging that to defend this claim it's not enough to defend the more familiar claim that your preferences at a time determine the impact that things at that time have on your well-being.

Community-relative

On the community-relative approach, you use different moral weights when assessing an intervention's impact on each community depending on that community's preferences. Those in the community might have conflicting preferences, so you'll need to have some way of averaging them.

The community-relative approach avoids one limitation of the person-relative approach. Even if some people in the community, such as young children, lack any relevant preferences, this approach still offers guidance on what moral weights to assign to outcomes that affect them. But I think there's not much to be said for it beyond that. I see two main issues with it.

First, unlike with the person-relative approach, it just seems counterintuitive on its face. Why would the preferences of people in a child's community, who the child might be completely ignorant of, determine how bad it is for the child to die? That doesn't make a lot of sense to me. My preferences can affect the value of things for me, but I don't see how they could affect the value of things for you, especially if you have no clue that I exist and don't care about what I think.

Second, it is not obvious how to average people's preferences when some people have fanatical views. For example, many people studied claimed that averting even a single death was preferable to any number of cash transfers. So if you naively average these people's preferences, then you'll end up assigning an infinite value to averting a death and a finite value to increasing someone's consumption. This is true if there is even a single person in the community who has a fanatical view.^[4]

There are a few ways you could to try to deal with the problem of fanaticism. First, you could take the median preference instead of the mean preference. This might help in the above case, but it won't work in every case. There is no median of infinity and negative infinity, so if people are fanatical at both extremes then the median will be undefined. Also, if a majority are fanatical, then even the median preference might still be infinite. But if only 51% have fanatical views, we might not want our moral weights to be determined entirely by the fanatics.

Second, you could put a cap on extreme views. For example, in the case of averting a death versus cash transfers, you could pretend that no one prefers averting a death to, say, $10,000,000 in cash transfers. But it's not obvious how to justify drawing the line at one point as opposed to another. If you thought that there was an objective fact about the correct moral weights and that you knew it, then you could use that knowledge to help you draw the line. But if you already have this knowledge, why bother asking people what their preferences are in the first place? It's also not obvious that there always should be a cap on extreme views. Maybe the fanatics are right about some things.

Third, you could look at people's revealed preferences instead of their stated preferences, and hope that those who state fanatical views don't actually act fanatically. But although there may be few people who act fanatically, it's not obvious that there are none. And even if the move to revealed preferences deals with the problem of fanaticism, it may introduce other problems. For example, if you were to ask me I would claim that my far future matters just as much as my near future, but I probably don't act perfectly like it does. But I wouldn't want someone to discount my far future as a result of this revealed preference.

Same-for-all

On the same-for-all approach, you use the same moral weights when assessing every intervention, regardless of the preferences of those affected by it. Nonetheless, when coming up with the universal set of moral weights that you will use, you might take into account people's preferences. This seems to be the approach that GiveWell plans to take:

In the future, we expect to have a single set of moral weights and that choosing this set will involve determining what moral weights would be implied by a variety of approaches or worldviews and then taking a weighted average of those views.

To the extent to which this approach involves averaging people's preferences, it faces the problem of fanaticism that I discussed in the previous subsection.

If you take this approach, it's unclear what justifies taking into account people's preferences. One possibility is that you think that there is an objective fact about the correct moral weights to use and that people's preferences are evidence of what the correct moral weights are.

Another possibility is that you think there are no objectively correct moral weights, but that we should (maybe because of this?) use the moral weights that best fit the preferences of everyone in the world. But if you are going to be a relativist about moral weights, it's unclear why you would say that they are relative to humanity as a whole as opposed to relative to each person or each community, which on the face of it seem like more plausible forms of relativism.

For its part, GiveWell's motivation for supporting the beneficiary preferences study seems in part to be that they were already taking into account people's preferences when setting their moral weights---it was just the preferences of those in higher-income countries, where most of this sort of research had been done before. Given this, I see how it makes sense for them to make sure that they aren't ignoring the preferences of the very people who are affected by the interventions that they assess. But there's still the deeper question of whether we should be looking at anyone's preferences in the first place to determine moral weights.

Weighting preferences beyond impact

So far I've been considering how beneficiary preferences could affect your moral weights, which represent the relative value of various outcomes. But even if you don't think that beneficiary preferences should affect your moral weights, you might think that we should take beneficiary preferences into account when prioritizing interventions for some other reason. Maybe outcomes aren't the only thing that matters. For example, you might think that we should respect people's autonomy, even if doing so will lead to somewhat worse outcomes.

If you want to do this, one obvious challenge is how to balance respect for people's autonomy (or whatever) with the expected value of outcomes.

A different challenge, specifically for trying to favour interventions that respect autonomy, is that it's not always obvious which intervention best respects a community's autonomy. For example, suppose that most people in a community strongly prefer interventions that avert deaths to interventions that distribute cash. Would their autonomy be best respected by distributing bed nets (which will avert many deaths, which they want) or by distributing cash (which they can do whatever they want with)?

Conclusion

I've considered three approaches to moral weights and how each might be informed by beneficiary preferences. I've focused on challenges that each approach faces. But to be clear, I have taken no stand on whether we should or should not take beneficiary preferences into account when setting our moral weights, or whether beneficiary preferences should be taken into account in some other way. Maybe the challenges can be overcome.

What if the challenges can't be overcome and we shouldn't take beneficiary preferences into account when setting our moral weights? How should we come up with our moral weights, then? Unfortunately, I don't have anything more helpful to say than "do philosophy". But as GiveWell points out, "philosophers have not done much work to consider how best to assign quantitative value to different kinds of outcomes". So I think this is an important and neglected topic, and I hope that more people work on it in the future.

---

1. I will follow the study's authors in talking in terms of preferences, but I think 'values' or 'moral beliefs' would be probably more accurate terms. ↩︎

2. GiveWell also flags this as a possible concern: "Preferences may not maximize well-being: Even if people perfectly understood the probability and information components of trading off income and mortality risk, they might not be able to reliably anticipate what would maximize their well-being, all things considered." ↩︎

3. To be precise, here's how GiveWell frames an example of the sort of question that they wanted to answer: "how much should we value averting the death of a one-year-old relative to doubling the income of an extremely poor household?" But the general question, as I see it, is how much to value extending a life (for example, the life of a one-year-old) compared to increasing someone's well-being (for example, by increasing their income). ↩︎

4. This problem is similar in some ways to the fanaticism problem for some approaches to dealing with moral uncertainty. See MacAskill, Bykvist, and Ord's Moral Uncertainty, chapter 6. ↩︎