As noted by Sam Nolan here:

GiveWell's cost-effectiveness analyses (CEAs) of top charities are often considered the gold standard. However, they still have room for improvement. One such improvement is the quantification of uncertainty.

Why does GiveWell not provide lower and upper estimates for the cost-effectiveness of its top charities?

45

0
0

Reactions

0
0
Comments8
Sorted by Click to highlight new comments since:

Not answering the question, but I would like to quickly mention a few of the benefits of having confidence/credible intervals or otherwise quantifying uncertainty. All of these comments are fairly general, and are not specific criticisms of GiveWell's work. 

  1. Decision making under risk aversion - Donors (large or small) may have different levels of risk aversion. In particular, some donors might prefer having higher certainty of actually making an impact at the cost of having a lower expected value. Moreover, (mostly large) donors could build a portfolio of different donations in order to achieve a better risk profile. To that end, one needs to know more about the distribution rather than a point-estimate.
  2. Point-estimates are many times done badly - It is fairly easy to make many kinds of mistakes when doing point-estimates, some of which are more noticeable when quantifying uncertainties. To name one example, point-estimates of cost-effectiveness typically try to estimate the expected value, and is many times calculated as a product of different factors. While it is true that expected value is multiplicative (assuming that the factors are uncorrelated or, more generally, independent, which is also sometimes not the case but that's another problem), this is not true for other statistics, such as the median. I think it is a common mistake to use an estimate of the median for the mean, or something in between, which in many cases are wildly different.
  3. Sensitivity analysis - Quantifying uncertainty allows for sensitivity analysis, which serves many purposes, one of which is to get more accurate (point-)estimate and reduce uncertainty. One example is by understanding which parameters are the most uncertain, and focus further (internal and external) research on improving their certainty.

In direct response to Hazelfire's comment, I think that even if the uncertainty spans only one order of magnitude (he mentioned 2-3, which seems reasonable to me), this could have a really larger effect on resource allocation. The bar for funding is currently 8x relative to GiveDirectly IIRC, which is one order of magnitude, so gaining a better understanding of the certainty could be really important. For instance, we could learn that some interventions which are currently above the bar, are not very clearly so, whereas other interventions which seem to be under the bar but very close to it, could turn out to be fairly certain and thus perhaps a very safe bet.

I think that all of these effects could have a large influence on GiveWell's recommendations and donors choices, future research, and directly on getting more accurate point-estimates (which could potentially be fairly big).

Thanks for the feedback!

I do think further quantifying the uncertainty would be valuable. That being said, for GiveWell's top charities, it seems that including/studying factors which are currently not being modelled is more important than quantifying the uncertainty of the factors which are already being modelled. For example, I think the effect on population size remains largely understudied.

Hello! Thanks for showing interest in my post.

First of all, I don't represent GiveWell or anyone else but myself, so all of this is more or less speculation.

My best guess as why GiveWell does not quantify uncertainty in their estimates is because the technology to do this is still somewhat primitive. The most mature candidate I see is Causal, but even then it's difficult to identify how one might do something like have multiple parallel analyses of the same program but in different countries. GiveWell has a lot of requirements that their host plaftorm needs t ohave. Google Sheets has the benefit that it can be used, understood, and edited by anyone. I'm currently working on Squiggle with QURI to make sweeten the deal to quantifying uncertainty explicitly, but there's a long way to go before it becomes somehing that could be readily understood and trusted to be stable like Google Sheets.

On a second note, I would also say that providing lower and upper estimates for cost-effectiveness for its top charities wouldn't actually be that valuable, in the sense that it doesn't influence any real world decisions. I know that I decided to spend hours making the GiveDirectly quantification but in truth, the information gained from it directly is extremely little. The main reason I did it is that it makes a great proof of concept for usage in non-GiveWell fields which need it much more.

There are two reasons why there is so little information gained from it:

  1. The uncertainty of GiveDirectly and other GiveWell supported charities is not actually that high (about an order of magnitude for GiveDirectly, I expect over 2-3 orders of magnitude for the others). For instance, I never expected in my quantifaction of uncertainty in GiveDirectly that there would be practically any probability mass of it being more effective than AMF. At least before counting for things like moral uncertainty.
  2. My uncertainty about my chosen uncertainties are really high. If you strip away how fancy my work looks and just look at what I've contributed in comparison to what GiveWell has done, I've practically copied GiveWell's work and pulled some numbers out of thin air for uncertanity with the help of Nuno. Some Bayesian Analysis is done under questionable assumptions etc.

I see much more value in quantifying uncertainty when we might expect the uncertainty to be much larger, for instance, when dealing with moral uncertainty, or animal welfare/longtermist interventions.

Wildly guessing, but I don't think it's a technological issue.
Givewell does publish upper and lower estimates for some of their analyses, at least they did for malnutrition interventions: https://docs.google.com/spreadsheets/d/1IdZLSBgEK46vc7cX9C7KnFgUcOk_M0UIYJ8go_DrvS0/edit#gid=1468241237 see at the bottom, 4x to 19x cash.
Many of their CEAs (e.g. new incentives)  are just one column. Even for the ones that have one column per country, they could have multiple sheets for upper and lower bounds.

I agree with your second point, I think GiveWell's mission is not informing many small donors anymore, but informing OpenPhil (and maybe other big players), and OpenPhil cares mostly about GiveWell's best guess about "what does the most good".

I disagree with the uncertainty being "not actually that high", or that moral uncertainties should be considered separately. Considering moral uncertainty, the impact can vary by orders of magnitude. See https://blog.givewell.org/2008/08/22/dalys-and-disagreement/ (very old blog post, but I think the main point still stands), and https://forum.effectivealtruism.org/posts/3h3mscSSTwGs6qbei/givewell-s-charity-recommendations-require-taking-a. I think many donors would be interested in seeing those kinds of uncertainties somewhere.

The uncertainty of GiveDirectly and other GiveWell supported charities is not actually that high (about an order of magnitude for GiveDirectly, I expect over 2-3 orders of magnitude for the others).

That seems pretty high to me! When I've seen GiveDirectly used as a point of comparison for other global health/poverty charities, they're usually described as 1-10x more effective (i.e. people care about distinctions within one order of magnitude).

One useful takeaway would be to know whether some interventions are much more uncertain about their range, and if that says something about the strength of evidence. If AMF is 6-10x and deworming is 1-20x (where 1x is point estimate on cash transfer cost effectiveness), then deworming might have a higher point estimate of cost effectiveness than AMF. But the large uncertainty suggests that maybe this is because we have much less evidence and not because the true cost effectiveness is much larger. So a risk averse donor could prioritize AMF on certainty.

In other words, we can favor more certain interventions, even within GiveWell top charities, because they are more robust to the risk that we have got it all wrong. They are less likely to be overturned by a new study. That seems pretty valuable.

Also an order of magnitude is really large.

Adding uncertainty to a single intervention may not be too informative. Still, I think it's more informative than you imply for comparing interventions -- especially if you're considering other decision frameworks for allocating funds beyond giving your money to the one with the highest average cost-effectiveness.

 E.g., If you have a framework where you allocate your money in proportion to the probability it has the highest cost-effectiveness, then uncertainty quantification would be essential. I'm not sure anyone supports a rule like this. 

Another potentially more real-world example: imagine you're a grantmaker choosing between 10 interventions that are all 10x more cost-effective than GiveDirectly, but vary in uncertainty. If you're a Bayesian with more sceptical priors than analysts, you will favour the relatively less uncertain analyses. 

For instance, I never expected in my quantifaction of uncertainty in GiveDirectly that there would be practically any probability mass of it being more effective than AMF.

Really? What do you mean by practically? If we crunched the numbers, I guess there'd be a single digit likelihood that GiveDirectly would be more impactful than AMF. 

Great points, thanks!

I would just like to emphasise your point that for other GiveWell top charities, we should expect a higher uncertainty than for GiveDirectly, especially the ones working on deworming. So modelling them could be more valuable.

Curated and popular this week
 ·  · 25m read
 · 
Epistemic status: This post — the result of a loosely timeboxed ~2-day sprint[1] — is more like “research notes with rough takes” than “report with solid answers.” You should interpret the things we say as best guesses, and not give them much more weight than that. Summary There’s been some discussion of what “transformative AI may arrive soon” might mean for animal advocates. After a very shallow review, we’ve tentatively concluded that radical changes to the animal welfare (AW) field are not yet warranted. In particular: * Some ideas in this space seem fairly promising, but in the “maybe a researcher should look into this” stage, rather than “shovel-ready” * We’re skeptical of the case for most speculative “TAI<>AW” projects * We think the most common version of this argument underrates how radically weird post-“transformative”-AI worlds would be, and how much this harms our ability to predict the longer-run effects of interventions available to us today. Without specific reasons to believe that an intervention is especially robust,[2] we think it’s best to discount its expected value to ~zero. Here’s a brief overview of our (tentative!) actionable takes on this question[3]: ✅ Some things we recommend❌ Some things we don’t recommend * Dedicating some amount of (ongoing) attention to the possibility of “AW lock ins”[4]  * Pursuing other exploratory research on what transformative AI might mean for animals & how to help (we’re unconvinced by most existing proposals, but many of these ideas have received <1 month of research effort from everyone in the space combined — it would be unsurprising if even just a few months of effort turned up better ideas) * Investing in highly “flexible” capacity for advancing animal interests in AI-transformed worlds * Trying to use AI for near-term animal welfare work, and fundraising from donors who have invested in AI * Heavily discounting “normal” interventions that take 10+ years to help animals * “Rowing” on na
 ·  · 3m read
 · 
About the program Hi! We’re Chana and Aric, from the new 80,000 Hours video program. For over a decade, 80,000 Hours has been talking about the world’s most pressing problems in newsletters, articles and many extremely lengthy podcasts. But today’s world calls for video, so we’ve started a video program[1], and we’re so excited to tell you about it! 80,000 Hours is launching AI in Context, a new YouTube channel hosted by Aric Floyd. Together with associated Instagram and TikTok accounts, the channel will aim to inform, entertain, and energize with a mix of long and shortform videos about the risks of transformative AI, and what people can do about them. [Chana has also been experimenting with making shortform videos, which you can check out here; we’re still deciding on what form her content creation will take] We hope to bring our own personalities and perspectives on these issues, alongside humor, earnestness, and nuance. We want to help people make sense of the world we're in and think about what role they might play in the upcoming years of potentially rapid change. Our first long-form video For our first long-form video, we decided to explore AI Futures Project’s AI 2027 scenario (which has been widely discussed on the Forum). It combines quantitative forecasting and storytelling to depict a possible future that might include human extinction, or in a better outcome, “merely” an unprecedented concentration of power. Why? We wanted to start our new channel with a compelling story that viewers can sink their teeth into, and that a wide audience would have reason to watch, even if they don’t yet know who we are or trust our viewpoints yet. (We think a video about “Why AI might pose an existential risk”, for example, might depend more on pre-existing trust to succeed.) We also saw this as an opportunity to tell the world about the ideas and people that have for years been anticipating the progress and dangers of AI (that’s many of you!), and invite the br
 ·  · 3m read
 · 
Hi all, This is a one time cross-post from my substack. If you like it, you can subscribe to the substack at tobiasleenaert.substack.com. Thanks Gaslit by humanity After twenty-five years in the animal liberation movement, I’m still looking for ways to make people see. I’ve given countless talks, co-founded organizations, written numerous articles and cited hundreds of statistics to thousands of people. And yet, most days, I know none of this will do what I hope: open their eyes to the immensity of animal suffering. Sometimes I feel obsessed with finding the ultimate way to make people understand and care. This obsession is about stopping the horror, but it’s also about something else, something harder to put into words: sometimes the suffering feels so enormous that I start doubting my own perception - especially because others don’t seem to see it. It’s as if I am being gaslit by humanity, with its quiet, constant suggestion that I must be overreacting, because no one else seems alarmed. “I must be mad” Some quotes from the book The Lives of Animals, by South African writer and Nobel laureate J.M. Coetzee, may help illustrate this feeling. In his novella, Coetzee speaks through a female vegetarian protagonist named Elisabeth Costello. We see her wrestle with questions of suffering, guilt and responsibility. At one point, Elisabeth makes the following internal observation about her family’s consumption of animal products: “I seem to move around perfectly easily among people, to have perfectly normal relations with them. Is it possible, I ask myself, that all of them are participants in a crime of stupefying proportions? Am I fantasizing it all? I must be mad!” Elisabeth wonders: can something be a crime if billions are participating in it? She goes back and forth on this. On the one hand she can’t not see what she is seeing: “Yet every day I see the evidences. The very people I suspect produce the evidence, exhibit it, offer it to me. Corpses. Fragments of