Comment Permalink

I think I understand your perspective, but I think there are two different ways of looking at any particular charity opportunity, and that is only one of them. I think each applies in different circumstances to different extents, and it can be difficult to tell how much each applies, depending on circumstances.

The first way of looking at it is the one that you described, where some other organization has already paid for its part of the health intervention, and would have paid for it regardless of if the donor donated any money. In that case, it might seem reasonable to me to mostly ignore the costs from the other organization, and only look at the cost paid by the donor when doing the CEA.

One example of where I think this would be applicable is when looking at a health intervention that has diminishing returns. For example, looking at AMF, probably the places where the LLINs will be most effective and least costly to distribute tend to receive them first. This could easily result in later LLINs being less cost-effective than earlier LLINs, so it might make sense to just look at what kind of impacting the donor is adding rather than looking at the total cost and impact of AMF.

The second way of looking at it is that the costs paid by the other organization were necessary for the health intervention, so we should account for those costs, although we would also need to somehow account for any other impacts from what the other organization spent. I think this way of looking at it is particularly valid if the money from the other organization will only be spent if the donor makes the donor's donation, but I don't think that's a requirement.

For example, on the AMF website, it says "100% of public donations buys long-lasting insecticidal nets (LLINs). An LLIN costs US$2.00." But there are actually many other necessary costs that need to be paid in order for LLINs to be effective, like the costs of distribution and the costs of figuring out where additional LLINs are needed. So, if we assumed that $2/LLIN was the only cost being paid, we would end up with incorrect cost-effectiveness numbers.

One thing that I think can help decide which perspective to use is the relative cost-effectiveness of the other organization's impact alone vs. the donor's donation alone. If the cost-effectiveness of the other organization's donations is a lot better than the one for the primary donor's donation, then I think it might be safe not to account for the costs of the other organization's donations. But, if not, I think it might make sense to account for the other organization's costs.

For example, let's say an organization spends $100 on a health intervention that saves one life. Then the donor comes along and pays another $10 to save a second life, but that donor can only do it because of the $100 that the organization already paid. In this case I think the more accurate cost-effectiveness for saving a life would be ($100 + $10) / 2 = $55, rather than just $10. However, if someone is just looking at which charity to donate to rather than trying to figure out realistically how many lives will be saved by a given donation, I could see the applicable number being $10/life. That's because I could see someone from the charity saying, "Hey, somebody else made this $100 donation, but didn't give us the other $10, even though it would make the intervention a lot more cost-effective, and it would be great if you could make up the difference." That seems like a good donation to make. But if someone asked, "How many lives do your donations tend to save?", I think the $55/life number would be more accurate.

In my case, I think the cost-effectiveness of Lafiya Nigeria is dramatically better than the cost-effectiveness of most global health interventions, even GiveWell's top charities, so I think accounting for the overhead costs from other organizations might make sense. Although, maybe only in the context of figuring out how far the money will actually go, not necessarily in the context of trying to figure out which charity to donate to.

Also, to some extent, my cost-effectiveness number for Lafiya Nigeria just seems too low to be possible, so I'm trying to figure out what might be wrong with it. Trying to account for costs from organizations like the WHO seems like a promising path for doing that.

Thanks for all the feedback on this. I think I've done a lot more useful thinking regarding this as a result.

See in context

Estimating Global Health Overhead Costs

by David Hammerle

Mar 153 min read 11

7

Cause prioritizationEffective givingGlobal health & developmentAdjusted life yearCharity evaluationCost-effectiveness analysis

Frontpage

I often focus on trying to improve my ability to estimate how much it would cost to avert a DALY or to save the life of a kid under 5 years old by donating to a particular charity. I am especially trying to find an upper limit to these amounts, so that I can very honestly, and with a lot of authority, tell a potential donor that if they donate to a particular charity, they will avert at least one DALY or save at least 1 child's life for the given cost. I want to find such an upper limit so that there can be somewhere around at least a 95% chance that it is correct.

I think a lot of people that might be interested in donating to highly cost-effective charities don't end up doing that, because they just don't trust those charities and the estimates for how cost-effective they are. And, I think trying to find better upper limit values for this kind of thing could help improve the argument for giving money to these kinds of highly cost-effective charities.

As part of this, I recently started trying to figure out how to factor in the costs from various effective altruism or global health organizations that aren't directly doing charity work or even directly giving money to the charities that are directly doing the work, but that do work that facilitates the work that those charities do. For example, the WHO seems to do a lot of work that makes it possible for lots of charities to do their work, so it seems likely that I should try to account for those costs for the WHO.

So far, the best method I have come up with for estimating these kinds of costs is to:

Add up the money spent on overhead activities like this.
Compare it to the total impacted charity money that is flowing through the charities and organizations that are being helped by those overhead activities.
Figure out a good estimate of an upper limit on how much good all the impacted charity money together does in dollars per DALY.
Multiply the dollars per DALY by the amount spent by the organization and divide by the amount flowing through the applicable charities being helped. This should give you a final number of dollars per DALY of overhead cost from these types of organizations (like the WHO).

For example, doing this calculation for the WHO:

Start with the budget of the WHO for both sub-Saharan Africa ($496 million) and the HQ ($597 million) added together ($1,093 million).
Find the total impacted money flowing through the impacted organizations. I'm estimating this to be $114 billion because that is about how much the world spends on health interventions in sub-Saharan Africa.
Figure out an upper limit of cost per DALY for health interventions in sub-Saharan Africa. It seems like $150/DALY is a good estimate for this.
Multiply dollars per DALY ($150/DALY) by amount spent by WHO ($1,093 million) and divide by amount flowing through impacted organizations ($114 billion). That comes out to $1.44 per DALY that is overhead costs from the WHO.

Thus far I'm thinking it might be good to account for these kinds of costs from the following organizations: WHO, GiveWell, Rethink Priorities, Giving What We Can, The Life You Can Save, The Centre for Effective Altruism, 80,000 Hours

So I'm wondering:

Does anyone have any ideas on how I could improve this calculation?
Or which organizations I should include?
Or other kinds of costs that I should try to account for?

And also:

Does anyone have any arguments for why this calculation is unnecessary?
And why it is only necessary to include the costs of the charity itself?
Or how estimates from organizations like GiveWell are already accounting for these types of costs somehow?
Or any other feedback?

7 Reactions

More posts like this

Comments11

Sorted by

New & upvoted

Click to highlight new comments since: Today at 8:33 PM

Mo PuteraMar 1614

When you say your aim is

I can very honestly, and with a lot of authority, tell a potential donor that if they donate to a particular charity, they will avert at least one DALY or save at least 1 child's life for the given cost

I think you're looking for a level of certainty that basically doesn't exist in any of the top recommendations, whether by GiveWell, GWWC, etc let alone the higher-EV lower-certainty giving opportunities (policy reform, field-building, etc), and that the kinds of adjustments you make won't get you where you want. You can and should argue that a particular giving opportunity is great based on multiple lines of reasoning, but you can't make your exact claim as phrased (apologies if the following may seem nitpicky — I used to require the same sort of certainty too, so consider it an attempt to legibilize why I changed my mind).

The reasons for this are hard to summarise in a persuasive way if you haven't built intuition by scrutinising a (good) CEA. You can get a sense of them by picking a charity everyone agrees is a long-running top recommendation for cost-effectiveness + evidence robustness + track record (proxying your search for authoritative certainty), e.g. the Against Malaria Foundation, and seeing what GiveWell says about them. So let's do that:

it turns out that the final cost-eff estimate depends a lot on which region you're looking at (from 4.8x cash in Chad to 22.6x in Guinea), so let's pick the DRC (14.5x) to illustrate
GW begin with a $1M grant size as placeholder value, which in the DRC covers 94.49% of total costs with the small remainder covered by the government, yielding $1,058,303 total spending by all contributors (it's more complicated than this, see this cell's note, but the complication doesn't affect the bottomline)
this spending leads to the distribution of 180,896 insecticide-treated nets at $5.85 per net, but GW calculate here using data from the Pryce et al 2018 meta-analysis that only about 70% of them were used in trials, and GW discounts this further by -10% in program contexts out of skepticism (trial-to-program adjustment is quite complicated, see GW's analysis here), so their best guess is only 113,965 (63%) of those 180,896 nets actually end up being used. This varies a lot by region already, e.g. in Chad (whose cost-eff being ~1/3rd of the DRC's above might make you think is due to fewer nets being used) GW actually estimates 167,299 nets used (+47% more than in the DRC)
GW's analysis indicates that ~2.15 people sleep under each net in the DRC (this varies by region too, going down to 1.72 in Togo for instance), so those actually-used nets above cover 245,154 people. Another very complicated calculation with a plethora of best-guesses adjustments yields an estimate of ~1.2 effective coverage years provided per net (lower than in many other regions which get ~2.0 ec-years), hence 292,350 person-years of coverage
~16% of DRC's national population is under-5 children, so GW assumes ~16% or 45,449 of those person-years are for under-5s, for whom benefit estimation is calculated separately because child mortality rate is much higher than in older groups — so much so that GW estimates 81% of the lives saved (and 87% of the mortality reduction benefit) are from these 16% of beneficiaries. As an aside, given this context you'll be glad to hear that GW estimates 59.1% of under-5 children sleep under nets in the DRC (varying from 31-80% in the regions AMF operates that GW analysed) using data from the Malaria Atlas Project combined with GW's ITN coverage estimates
GW estimates a 56% expected reduction in malaria incidence among children who sleep under nets in the DRC. This comes from Pryce et al 2018's 64% implied reduction with 3 further discounts: -5% for internal validity ("to account for the possibility that the summary effect reported in Lengeler's 2004 meta-analysis might not have been the true effect ITNs had on the populations being studied"), -5% for external validity (to account for differences between trials and program implementation — see this Asterisk essay for a particularly stark example of how different scale-up results can be, and note that GW still assign a discount out of general skepticism even after separately discounting for their 3 major external validity concerns of net usage, country-specific mortality rates, and insecticide resistance), and -4% for insecticide resistance (in yet another complicated calculation that shows how region-dependent net effectiveness reduction due to resistance is, going as high as -59% in Chad which partly explains its low cost-eff)
GW estimates that in the absence of nets, the total under-5 mortality rate attributable to malaria alone would be 0.80%. This is full of adjustments too: GW start with the fact that in the DRC 0.31% of under-5 deaths in 2019 were directly attributable to malaria (40,985 deaths among 13.4M children), adjust this up +75% to 0.53% to account for indirect malaria deaths (GW's complicated reasoning concludes "we're highly uncertain about the exact value of this input", but it's more skeptical than the best literature implies and is roughly the midpoint of expert opinion), adjusts for deaths averted due to seasonal malaria chemoprevention or SMC programs (delivered by another top GW charity) to avoid double-counting, and finally adjusts for the 59.1% net coverage in the DRC mentioned earlier
combining everything together: $1M GW grant → $1,058,303 total spending including DRC government funds mobilised → 180,896 nets distributed → 113,965 nets used → 245,154 people covered → 292,350 person-years of effective coverage → 45,449 person-years of under-5 coverage → 203 under-5 deaths averted. A similarly adjustment-filled calculation yields 48 deaths averted among older people, so 251 total deaths averted (this is only halfway through GW's main CEA calc but the rest won't be germane to the point)
Naively we might say based on this that a $3,989 donation saves one life in the DRC, and that this varies from $1,700 in Guinea to $14,493 in Chad, but after seeing the kinds of adjustments that GW had to make above (do dig into those links to get a better sense of what I mean), I'm not sure you would be comfortable telling prospective donors that (say) giving $15,000 to AMF would save "at least one child's life". All you can say is that it would do so in expectation, and that the perspective shift is worth making as there are good reasons for giving (and doing good) guided by an expected value lens, while also being aware of this lens' shortcomings

Notice how difficult it is to get the level of certainty you seek in every adjustment step of the calculation above, despite GW's almost pathologically-skeptical thoroughness. This doesn't mean AMF is "a bad recommendation" — it means virtually every other top giving opportunity is "worse" (by the standard you seek) because giving well is really hard, yet we still have very good reasons to think that AMF and all those other top recommendations are excellent giving opportunities, so you should recalibrate your expectations and communicate this to potential donors as well who seek the same.

re: the "multiple lines of reasoning" alternative for recommending great giving opportunities, GiveWell's approach is solid I think. For AMF that looks like

We recommend AMF because of its:
Implementation of a program (mass distribution of ITNs) that we think is very cost-effective (more in our separate report on ITNs).
Long track record of supporting campaigns in a number of countries. (More)
Processes for tracking other actors’ malaria spending in the countries where AMF works, giving us more confidence that it is filling genuine funding gaps. (More)
Transparency – AMF shares significant information about its work with us. (More)

and GW detail their reservations as well (same link).

David HammerleMar 177

Thanks for the response, Mo.

I have also spent a significant amount of time looking at cost-effectiveness analyses from GiveWell and I think they're great. I even wrote a summary of GiveWell's cost-effectiveness analysis for Against Malaria Foundation similar to yours, although I think yours might be better. Mine is on this webpage (https://davehammerlecoder.com/against-malaria-foundation) on my personal website under the section 'Appendix A: GiveWell's Cost-Effectiveness Analysis for the Against Malaria Foundation'.

I'm not so much concerned with finding a really accurate measure of the cost per DALY averted or life saved as I am with finding an upper limit, even if the upper limit is pretty far above the estimated average. For example, looking at the Against Malaria Foundation, according to GiveWell, it looks like the upper end cost to save a life would be $14,493, specifically in Chad. If that's the upper end of GiveWell's estimate, then it seems like $50,000 cost per life saved might be a pretty reasonable upper limit. Meaning it seems plausible to me that someone could have a 95% confidence that AMF will save a life for less than $50,000 on average.

And if I were a normal person living in a rich country like the US, considering donating some money, saving a life for $50,000 would seem like a really good deal to me. Even if saving that life only averts 35 DALYs, that would translate into ~$1,400 per DALY or ~$4 per day of life added. So if I work an extra 1 hour for about $30/hour, I can extend someone's life by 7 days, or 168 hours. Which means that for whatever span of time extra I spend working, somebody else gets to live for ~168 times that span time. From an altruistic perspective, that's an amazingly good deal.

I can also definitely understand that being able to donate money and have it be as cost-effective as it probably is, is very impressive. I'm sure it takes a lot of work from a lot of different people in order to make that possible, including the people at GiveWell and many government entities, both African and not.

But it took me a lot of time and effort, including a bachelor's degree and a trip to EA Global in Boston to attain the degree of confidence that I currently have in my donations. To me it seems like maybe we could get more normal people interested in donating if it didn't require as much free time and effort.

I think maybe the reason most people don't want to donate to these kinds of charities is that they don't consider charities trustworthy enough. And that seems like a reasonable response to me, especially considering the kinds of ads from African charities pop up in my web browser. But I think if we can get a more consistent message out to more normal people who don't want to spend as much time on this, donating money to global health charities could become more normal in rich countries like the USA.

I think in order to accomplish that, we need a message that is consistent, and that can be verified by people who decide to really look into it. This way, normal people would have more of a reason to trust highly cost-effective global health charities. And I think a fairly consistent upper limit on the cost to avert a DALY or save the life of a kid under 5 via any given charity could help with this.

I want to account for the money spent by organizations like GiveWell, the WHO, the Centre for Effective Altruism, and Open Philanthropy in an attempt to improve these upper limits. I'm especially concerned about these costs because I recently started donating to family planning global health charities. These charities can be viewed as highly cost-effective just by reducing maternal mortality and from various other positive impacts.

But the thing I really like about family planning is that it can avert births.
Each averted birth avoids the 7.4% chance kids in sub-Saharan Africa have of dying before the age of 5 years old.
Looking at it this way, these charities can avert the death of a kid under 5 years old for under $1000, and probably substantially less.
The amount depends on odd things like which utilitarian theory one subscribes to. I go by an average-based utilitarian theory.
If you want more info about that, my post on the EA forum about it is here: https://forum.effectivealtruism.org/posts/stWZ26t9W7qi3ieJS/why-aren-t-relocated-births-accounted-for-in-cost

My problem now is that, at that degree of cost-effectiveness, costs from organizations like the WHO and GiveWell become much more significant relatively. And that's why I started trying to figure out how to estimate applicable costs from those kinds of organizations.

NickLaingMar 204

Nice one David! Mo has done as great job explaining here do I won't add too much.

To reiterate the costs of overheads are built into all that estimations, so I don't think you need to worry about that as a separate issue.

I love your intuition on family planning, and there are some great Organizations like FEM. And Lafiya Nigeria which have solid first effectiveness analysis showing they in probably have similar cost t effectiveness to other to charities, spend very little on overhead and help women in Northern Nigeria to take control of their family's future and improve women's health.

I love the way you are thinking and all the great on your donation journey.

Mo PuteraMar 194

Thanks for the considered response David, upvoted. I'll preface by saying that I appreciate the effort you're putting into this.

When I first read your post, I anchored hard to "... so that I can very honestly, and with a lot of authority, tell a potential donor..." which made me assume you've spoken to potential donors, which I resonated with. I've only spoken to potential small local donors in a very limited personal capacity, and a recurring concern they raise is that they simply don't want to be misled by claims of effectiveness (forget cost-effectiveness). I recall speaking to someone who felt disquieted by not just the number but also the kind of uncertainty-driven adjustments I listed above for instance, and also appreciative of me pointing them out. I recall speaking to another person who was disquieted by the nature of helping in statistical expectation, when what they wanted was certainty; they appreciated me pointing this out as well (and were unpersuaded by the arguments for expectation-oriented giving).

This disquiet is of course a personal matter, very case-by-case; instead of guessing you should just ask potential donors. In this context your comment makes me wonder if you've spoken to any, and if so what did they think of your approach? To be more direct: when you say things like "it seems plausible to me that someone could have a 95% confidence", is that from talking to them or just your own guesses for now?

David HammerleMar 193

I've only spoken to a few donors, mostly friends and family. They seemed to especially like the idea of donating to charities that are local to their communities, which most highly cost-effective charities don't, because they are mostly in low income places, not the USA. They were also concerned with what portion of the donated money would be spent on the charity's own overhead or administration. And they especially didn't want to receive spam mail or email from charities as a result of donating.

I suppose my assumption is that I should start with an argument I think would work on myself, if I hadn't studied global health as much as I have, and then try to figure out what part of that people are resistant to. If I hadn't studied global health, and wanted to be able to donate effectively without having to study it, I would have been looking for that upper limit with a likelihood of somewhere around 95% that the cost would be below it, even if the upper limit needed to be pretty high in order to reach that level of certainty. So now I'm trying to figure out such an upper limit myself.

Mo PuteraMar 212

I again want to say that I resonate a lot with what you're trying to do, since I've tried (and mostly failed) to do something similar myself before.

I worry a bit that you're prematurely optimising approach-wise when you conclude the thing to focus on is figuring out the cost-effectiveness upper limit at which you can tell people honestly and confidently that their donation is doing that much good, instead of asking donors how they think about giving (which you did). For instance, Sawyer's comment reiterates the sentiment I mentioned earlier that

Most potential donors are not really risk neutral, and would rather spend $5,001 to definitely save one life than $5,000 to have a 10% chance of saving 10 lives. Risk neutrality is a totally defensible position, but so is non-neutrality. It's good to have the option of paying a "premium" for a higher confidence (but lower risk-neutral EV).

and Jason's comment seems relevant as well:

Orthogonally, I think most people are willing to pay more for a more legible/direct theory of impact.
"I give $2800, this kid has lifesaving heart surgery" is certainly more legible and direct than a GiveWell-type charity. In the former case, the donor doesn't have to trust GiveWell's methodologies, data gathering abilities, and freedom from bias.

and these aren't necessarily obvious in advance, so if you start from a (simplistic) model of giving advisory effectiveness as being [number of prospective donors in your circle] x [fraction of donors who find argument X persuasive] x [$ per donor] x [expected good per $], then starting with an argument that works on yourself (and on me too — I don't think we're that representative of the donor pool) doesn't let you get empirical input on the 2nd term, while asking donors does.

David HammerleMar 273

To some extent, I'm also trying to figure out a good upper limit for myself, so that I have a better idea of how much my donations are really worth. I think if I can increase my confidence that the cost is below some value, I'll have an easier time motivating myself to avoid spending money so that I can donate more.

Going back to my original post, the only reason I'm concerned with figuring out how to factor in costs from organizations like the World Health Organization, GiveWell, and the Centre for Effective Altruism is because, from an average-based utilitarian theory, I think family planning charities in sub-Saharan Africa can save the life of a kid under 5 years old for somewhere around $275. I would normally guess that costs of organizations like those would be so tiny, that they're not worth trying to account for. But $275 per life (which might be about $4.50 per DALY) is so cost-effective, that costs from those necessary organizations could have a pretty big impact on the overall cost.

From what I've read, health interventions in sub-Saharan Africa often have a cost-effectiveness of about $150 per DALY, or about $50-$80 for one of GiveWell's top charities. And I think the costs for all of those organizations together might add up to about 5-10% of the money that goes to health interventions in sub-Saharan Africa. If we assume a normal cost per DALY is about $100 and add 5-10% of that to my estimate of Lafiya Nigeria's cost-effectiveness of $4.50 per DALY, that results in about $10-$15 per DALY, approximately doubling or tripling the cost per DALY.

And it seems like GiveWell's cost-effectiveness analyses doesn't account for these kinds of costs. I understand why they wouldn't want to, since most people looking at their cost-effectiveness analyses are trying to compare one charity to another, not trying to figure out the total cost. But, still I want to try to account for it in my calculations.

Mo PuteraMar 282

Thanks for elaborating David, I think I better understand where you're coming from now. I still don't buy the need to incorporate these costs though, so in the interest of potentially changing my mind and to prevent talking past each other let me try to be concrete.

Approach-wise, I think froolow got the accounting right when he was looking into GiveWell's older (2022) CEAs (emphasis mine):

The model reports the effectiveness of an intervention per philanthropic dollar spent, but this is (subtly) incorrect. The decision problem facing GiveWell is not exactly finding the most cost-effective charity per philanthropic dollar, but rather finding the most cost-effective charity that my philanthropic dollar can contribute towards. So, the cost-effectiveness of a particular charity per dollar I personally donate ought to be the effectiveness that that dollar brings, plus any matched additional philanthropic funding that dollar generates – but I’ve still only spent one dollar when it comes to the ‘cost’ part of ‘cost-effectiveness’. The idea is that you might abstractly be interested in the total cost of an intervention vs its effects, or even the total philanthropic cost of an intervention vs its effects, but GiveWell is concretely interested in the good done by a marginal donation, so only the first actor matters. Hopefully the diagram below illuminates more than it confuses!

Applying this approach to (say) Lafiya Nigeria's figures, Rethink Priorities estimates that if you donated $361 and Lafiya spent it in Q1 '25 you would have averted one additional death in expectation. Supposing you included costs from other orgs like WHO etc, sure the total cost may increase, but it's still the case that your $361 helped avert one additional death in expectation! You could think of those other overhead costs as "leveraged funding" or "other people's money moved by your money" or whatever, but why would any of that be relevant to the fact that an additional death was averted because you decided to donate $361? Shouldn't that averted death be what ultimately matters?

Maybe you're thinking from a credit attribution perspective? e.g. you want to say "yes a life was spared, but I shouldn't get all the credit, since my money moved WHO's money etc, so if we did proper accounting (maybe using Shapley values) maybe I can only get 40% credit for averting that neonatal death or averting that woman from dying in childbirth, whereas WHO gets 30% credit and other actors get 30% credit". But in that case I don't see where this over-focus on credit attribution is coming from, instead of just wanting to avert the death of that child or mother and knowing that you the donor can do so by giving Lafiya $361?

David HammerleMar 293

Thanks for all the feedback on this. I think I've done a lot more useful thinking regarding this as a result.

Mo PuteraMar 302

You're welcome :) Apologies in advance if I've come across as too nitpicky and inadvertently confrontational by the way!

David HammerleApr 101

No problem. I probably seem that way also.