Hide table of contents

In December 2022, GiveWell announced the results of its "Change Our Mind Contest". GiveWell awarded joint first place and a $20,000 prize to Noah Haber for “GiveWell’s Uncertainty Problem”. According to GiveWell:

The author argues that without properly accounting for uncertainty, GiveWell is likely to allocate its portfolio of funding suboptimally, and proposes methods for addressing uncertainty.

One of the honorable mention awards was also a critique of GiveWell's approach to uncertainty.

Both essays recommended using Monte Carlo simulation to properly model uncertainty in cost-effectiveness.

In December 2023, GiveWell published "How We Plan to Approach Uncertainty in Our Cost-Effectiveness Models", in which they explained their next steps:

Going forward, we plan to: Incorporate sensitivity analysis in published models. We’ve begun this process by incorporating basic sensitivity analysis on key parameters in recent grant and intervention report pages (e.g., zinc/ORS; vitamin A supplementation; MiracleFeet). We’re currently revamping our top charity CEAs to make them more legible, and we plan to incorporate sensitivity analysis and Monte Carlos into these before publishing.

[...]

In the three linked pages, GiveWell has conducted a so-called one-at-a-time sensitivity analysis, an approach which as GiveWell acknowledges is unable to quantify the overall uncertainty in cost-effectiveness.

After the contest winners were announced in December 2022, GiveWell has as far as I can tell made the following grants (source):

 January 2023 to June 2024January 2024 to June 2024
To top charities$295 M$89 M
To organisations other than top charities$143 M$31 M

I cannot find on GiveWell's website any investigation where the full uncertainty (from all parameters) is modelled. Has GiveWell published such a thing? If not, do they still plan to?

21

0
0

Reactions

0
0
New Answer
New Comment


Comments4
Sorted by Click to highlight new comments since:

Hey, thanks for the question! I'm Alex Cohen, a researcher at GiveWell, and wanted to chime in.

We did say we'd include a 25th/75th percentile range on bottom line cost-effectiveness (in addition to the one-way sensitivity checks). We haven't added that yet, and we should. We ran into some issues running the full sensitivity analyses (instead of the one-way sensitivity checks we do have), and we prioritized publishing updated intervention reports and cost-effective analyses without them.

We'll add those percentile ranges to our top charity intervention reports (so the simple cost-effective analyses will also include a bottom line cost-effectiveness 25/75 range, in addition to one-way sensitivity checks) and ensure that new intervention reports/grant pages have them included before publishing. We think it’s worth emphasizing how uncertain our cost-effectiveness estimates are, and this is one way to do that (though it has limitations).

We're still not planning to base our decision-making on this uncertainty in the bottom line cost-effectiveness (like the “Change Our Mind Contest” post recommended) or model uncertainty on every parameter. To defend against the Optimizer's Curse, we prefer our approach of skeptically adjusting our inputs, rather than an all-in adjustment to bottom-line cost-effectiveness. We explain why in the uncertainty post.

Really appreciate you raising this. Sorry this has taken so long, and grateful for the nudge!
 

not that I really mind, but why is the author anonymous? it seems like such a tame criticism it's hard to imagine anyone getting upset about it

I'm really interested in the answer to this question and happy to say that publicly

Hypothetically they could work at GiveWell?

Curated and popular this week
 ·  · 22m read
 · 
The cause prioritization landscape in EA is changing. Prominent groups have shut down, others have been founded, and everyone’s trying to figure out how to prepare for AI. This is the third in a series of posts critically examining the state of cause prioritization and strategies for moving forward. Executive Summary * An increasingly common argument is that we should prioritize work in AI over work in other cause areas (e.g. farmed animal welfare, reducing nuclear risks) because the impending AI revolution undermines the value of working in those other areas. * We consider three versions of the argument: * Aligned superintelligent AI will solve many of the problems that we currently face in other cause areas. * Misaligned AI will be so disastrous that none of the existing problems will matter because we’ll all be dead or worse. * AI will be so disruptive that our current theories of change will all be obsolete, so the best thing to do is wait, build resources, and reformulate plans until after the AI revolution. * We identify some key cruxes of these arguments, and present reasons to be skeptical of them. A more direct case needs to be made for these cruxes before we rely on them in making important cause prioritization decisions. * Even on short timelines, the AI transition may be a protracted and patchy process, leaving many opportunities to act on longer timelines. * Work in other cause areas will often make essential contributions to the AI transition going well. * Projects that require cultural, social, and legal changes for success, and projects where opposing sides will both benefit from AI, will be more resistant to being solved by AI. * Many of the reasons why AI might undermine projects in other cause areas (e.g. its unpredictable and destabilizing effects) would seem to undermine lots of work on AI as well. * While an impending AI revolution should affect how we approach and prioritize non-AI (and AI) projects, doing this wisel
 ·  · 6m read
 · 
I am writing this to reflect on my experience interning with the Fish Welfare Initiative, and to provide my thoughts on why more students looking to build EA experience should do something similar.  Back in October, I cold-emailed the Fish Welfare Initiative (FWI) with my resume and a short cover letter expressing interest in an unpaid in-person internship in the summer of 2025. I figured I had a better chance of getting an internship by building my own door than competing with hundreds of others to squeeze through an existing door, and the opportunity to travel to India carried strong appeal. Haven, the Executive Director of FWI, set up a call with me that mostly consisted of him listing all the challenges of living in rural India — 110° F temperatures, electricity outages, lack of entertainment… When I didn’t seem deterred, he offered me an internship.  I stayed with FWI for one month. By rotating through the different teams, I completed a wide range of tasks:  * Made ~20 visits to fish farms * Wrote a recommendation on next steps for FWI’s stunning project * Conducted data analysis in Python on the efficacy of the Alliance for Responsible Aquaculture’s corrective actions * Received training in water quality testing methods * Created charts in Tableau for a webinar presentation * Brainstormed and implemented office improvements  I wasn’t able to drive myself around in India, so I rode on the back of a coworker’s motorbike to commute. FWI provided me with my own bedroom in a company-owned flat. Sometimes Haven and I would cook together at the residence, talking for hours over a chopping board and our metal plates about war, family, or effective altruism. Other times I would eat at restaurants or street food booths with my Indian coworkers. Excluding flights, I spent less than $100 USD in total. I covered all costs, including international transportation, through the Summer in South Asia Fellowship, which provides funding for University of Michigan under
 ·  · 4m read
 · 
TLDR When we look across all jobs globally, many of us in the EA community occupy positions that would rank in the 99.9th percentile or higher by our own preferences within jobs that we could plausibly get.[1] Whether you work at an EA-aligned organization, hold a high-impact role elsewhere, or have a well-compensated position which allows you to make significant high effectiveness donations, your job situation is likely extraordinarily fortunate and high impact by global standards. This career conversations week, it's worth reflecting on this and considering how we can make the most of these opportunities. Intro I think job choice is one of the great advantages of development. Before the industrial revolution, nearly everyone had to be a hunter-gatherer or a farmer, and they typically didn’t get a choice between those. Now there is typically some choice in low income countries, and typically a lot of choice in high income countries. This already suggests that having a job in your preferred field puts you in a high percentile of job choice. But for many in the EA community, the situation is even more fortunate. The Mathematics of Job Preference If you work at an EA-aligned organization and that is your top preference, you occupy an extraordinarily rare position. There are perhaps a few thousand such positions globally, out of the world's several billion jobs. Simple division suggests this puts you in roughly the 99.9999th percentile of job preference. Even if you don't work directly for an EA organization but have secured: * A job allowing significant donations * A position with direct positive impact aligned with your values * Work that combines your skills, interests, and preferred location You likely still occupy a position in the 99.9th percentile or higher of global job preference matching. Even without the impact perspective, if you are working in your preferred field and preferred country, that may put you in the 99.9th percentile of job preference