Revisiting Well, Well, Well…Do Wells Clear the Bar?

This post is an update on EA Salt Lake City’s evaluation of Wells4Wellness, an organization that drills and maintains wells in Niger.

Big thank you to Willie Herath, director of operations at Wells4Wellness, for taking the time to answer our questions over the last couple months!

TLDR: We’ve found that drilling wells in Niger easily clears GiveWell and Coefficient Giving’s “Bar” and we think donors interested in global health should consider donating.

Part 1: Revising our estimates

In our original post, we estimated that Wells4Wellness is ~5-10x more cost effective than chlorination, the current golden child of water sanitation and hygiene interventions. 

We got some good feedback from a few readers, talked to the director of operations at Wells4Wellness, and updated our estimates:

• We found a few more studies (see here, here, and here) around the reduction in diarrhea mortality due to wells. We think anywhere in the 12-18% range is reasonable, and we’ll use 16% for our calculations.
• Based on details from Wells4Wellness, we updated the construction cost per well from $10,000 to $7,000, with maintenance costs remaining at $200/year. We also had a typo in our original cost estimates, which is corrected below.
• We continue to have high uncertainty about the lifespan of wells. All the Wells4Wellness wells are still up and running, and it sounds like there are no anticipated issues with them running dry or becoming irreparable. We stick with a 50-year lifespan below, and note that the results weren't very sensitive to reasonable potential lifespans (between 20 and 100 years).

Our new estimate:

• A 2018 Lancet Global Health Study found Niger lost 442.1 DALYs from diarrheal diseases per 1,000 children under 5 in 2016
• 20% of Niger’s population is under 5, so we take 20% of 442.1, and estimate 88.42 DALYs lost per 1,000 people of all ages per year. We think there are additional DALYs lost for the over-5 population as well, but we will use this number to be conservative.
• Providing clean water averts 16% of these DALYs lost, which gives us 14.15 DALYs averted per 1,000 people served per year.
• $7k per well built and $200/year maintenance costs for a well that lasts 50 years works out to $340 per year per well. The wells tend to serve ~1,200 people, so we get a cost of $0.28 per person served per year ($280 per 1,000 people per year).
• $/DALY = $280 / 14.15 DALYs = $19.79
• We cut that in half to add in economic benefits (see our initial post for more on that): $9.90 per DALY

At $9.90/DALY, Wells4Wellness operates well under GW and OP’s $/DALY bars, and beats chlorination. Even with more conservative assumptions, Wells4Wellness is in the ballpark of chlorination.

Part 2: Scaling & the awkward fit of smaller charities in the EA ecosystem

We think Wells4Wellness has a pretty good case for cost-effectiveness and it clears the bar for most of the major EA funders, so why can’t we get this on GiveWell’s radar?

Willie did reach out a while back and here was their response:

Unfortunately we aren’t able to look into your work in detail at the moment. Our team is very capacity constrained and we’re currently exclusively focused on chlorination, which we think is likely to be the highest impact WASH intervention (rather than constructing water points). We also have a preference for focusing on larger organizations since there’s likely to be the potential for greater impact. 

Grantmakers will always be capacity constrained and need to prioritize what to review, leading to heuristics like those cited above. At the scale of GiveWell, these sorts of heuristics make sense, but we still think Wells4Wellness deserves some smaller scale attention.

Addressing the concerns from the quick GiveWell assessment:

1. Wells are less effective interventions than chlorination. From what we’ve read, this is mostly a fair heuristic. Lots of well drilling organizations are not committed to maintenance; within a couple years the well breaks and its impact drops to zero. Wells4Wellness distinguishes itself by maintaining a network to monitor and repair the wells throughout their lifespan. That effort makes this particular well-drilling intervention more comparable to chlorination.
2. Charities that are too small can’t absorb enough funding. For an organization like GiveWell, it makes sense to focus attention and funding on the larger, bigger hits. The new All Grants Fund closes some of the gap between medium-sized organizations trying to scale and larger organizations like AMF. That still leaves these much smaller organizations like Wells4Wellness (operating with a single paid employee and a network of volunteers) in no man’s land, coordinating individual donations until they can reach the scale that would merit attention from bigger funders.

How should we think about funding organizations that are effective, but might not scale well beyond a certain point? Wells4Wellness continues to grow, but it operates effectively in part because of their strong network: local volunteers identifying high need sites for wells, local well diggers who have been with the org for years, and people maintaining the wells on the ground. When things break, Willie, the director of operations, gets a text directly. That’s not the sort of process that will scale indefinitely, but it still serves tens of thousands of people. Isn’t this the sort of thing we should fund until it hits a wall?

There is plenty of room for funding at their current scale. Each year, Wells4Wellness aims to drill 10 more wells than the previous year, which requires ~$100k in additional funding. This goal is mainly constrained by donations at the moment; if they received more funding they’d be able to drill more wells. According to their estimates, the next major step up in funding needs would come once at the scale of drilling 100+ wells in a year (they’ve drilled 33 in 2025) which would require an additional drilling crew and an annual budget of ~$1.25 million.

Part 3: Process & Experience

We think more folks in EA should try to do their own small cost-effectiveness calculations for organizations that might get overlooked by the main funders. It takes some work, but there are a few benefits:

• Discovery is hard, you can help chip away at it. There’s a chance you’ll find an organization that’s been overlooked because it doesn’t fit neatly into the current funnel of evaluation.
• Broaden your giving portfolio. Most of the contributors from our group donate at least 10% of our income to the common big funds - ACE, GiveWell, GiveDirectly. After hearing about Wells4Wellness many of us also decided to donate there. It’s good to redirect some portion of a giving portfolio to riskier/smaller bets, with the chance that you’ll get some bigger hits.
• Learn what goes into a cost-effectiveness estimate.  Get a sense of how these sorts of calculations are done. There are so many assumptions that go into making a single estimate. You can judge for yourself how much you want that decision-making to be concentrated in just a handful of orgs.
• Practice thinking for yourself! It’s great that some evaluation organizations already exist to lower the barrier for directing money to highly cost-effective charities, but there aren’t that many. Diversity of thought is an important part of keeping a movement healthy.

We think this would be a great exercise for more local EA groups to do in their communities.

Here’s how we organized this work and things we learned.

Finding a charity to evaluate

We were lucky on this one - the charity found us! Or more accurately, a member of our group brought Willie to our meetings and helped arrange a presentation (thank you Troy!). Troy found out about Wells4Wellness when he and Willie met at the gym, so the lesson there is to keep your ears open.

We expect if we did this at least a few more times, we wouldn’t always get the sort of positive results we saw with Wells4Wellness and selecting the next charity to evaluate will likely take much more work than the first time around. We’ve discussed doing evaluations like “what are the most effective charities if we wanted to focus on Salt Lake City?”

What questions to ask?

GiveWell has a list of Do It Yourself questions for many categories of interventions. We also ended up borrowing heavily from GiveWell’s framework in evaluating chlorination interventions.

Coefficient Giving recently posted some guidance on back-of-the-envelope estimates. This was too late for us, but might be useful to other groups!

Organizing a group research project

Before we first met as a group, everyone did a lot of their own research and we all showed up feeling very overwhelmed. It seemed like there were so many questions we needed to answer before we could even get started. We narrowed it down to a set of core questions and numbers we needed to find, and each person got a focused domain to research. Here were the main questions we came up with:

• What is the cost per DALY for effective chlorination organizations, to compare to our Wells4Wellness estimate?
• What are the demographics of Niger, kids vs adults?
• What are the diarrhea mortality statistics of Niger?
• How many diarrhea deaths are averted by wells, kids vs adults?
• What are other detrimental effects from unclean water (things like increased infections in hospitals, other diseases) and how do they compare to diarrhea?
• How do wells affect quality of life beyond mortality?

The hard parts

It’s hard for small organizations to gather key effectiveness metrics. They’re operating on tight margins and there’s no real incentive to do this if non-EA funders don’t care. We tried asking some of our demographic and effectiveness questions to Willie, but he was mainly able to answer us in broad effects rather than concrete numbers. For example, when we asked Willie about child mortality, he couldn’t give us a specific number but he could tell us that villages where wells were built would often build a school a few years later. 

Since information is sparse and we can’t run a new study, it means we end up biased toward what is already in the literature. There are a couple main assumptions that kept popping up throughout water intervention evaluations:

1. The main beneficial effect of wells is preventing childhood deaths from diarrhea.
2. This effect swamps out all other effects to the point it’s not even worth including them in estimates.

We think (2) is wrong and causes drastic underestimates of the beneficial impact of clean water, especially in understanding the economic effects. But we can understand why so many researchers made this simplifying assumption - once you try to account for all the things that might be impacted by clean water, it gets overwhelming fast.

But also… we were inexperienced people trying to do a difficult analysis. It’s really hard to feel confident in the numbers we produced. We feel a little bit better because our numbers ended up in the same ballpark through multiple estimation approaches. We also didn’t get as much feedback as we were hoping to get from posting on the EA Forum.

Conclusion

We think Wells4Wellness is operating within the effectiveness range that should attract EA-aligned donations. Drilling wells is often rightfully dismissed as ineffective because the wells fall into disrepair after short periods of time, but given the dedication of Wells4Wellness to maintenance, they distinguish themselves from other organizations. We estimate their effectiveness to be greater than the effectiveness of chlorination.

GiveWell recently announced a set of pilot projects for in-line chlorination involving ~$18 million in grants to fund a variety of projects, with the primary goal of learning which types of programs work well. We think this new push on clean water is excellent, but the pilot projects are noticeably missing Niger, where the burden of diarrheal disease is particularly high and access to clean water is particularly low. Wells4Wellness serves Niger. If you already give to GiveWell and are excited about their clean water effort, consider directing some donations to Wells4Wellness as well!