Hide table of contents

Comment Permalink

I do work in this academic neighborhood, so maybe I’ll comment a bit. There’s a part of me that feels like caricaturing Linch’s question as akin to asking: What were all the other physicists doing with their time in 1905, when Einstein was sitting around churning out paradigm-shifting papers without a PhD or even access to a decent academic library? But that’s probably unhelpful, so I’ll try to give a bit more color on Kremer’s context.

First, it’s important to understand that until very recently, experimental work in LMIC settings was really prohibitively difficult to 1) organize and 2) get funded. Prior to the founding and (recently explosive) growth of the J-PAL/IPA network (in which Kremer himself played no small part), as well as the emergence of the Gates Foundation as a major funder of global health RCTs, these studies were just really, really hard to do. Moreover, even if you could put one together, until the last 10-15 years, you were very unlikely to be able to publish your results in a top-5 economics journal (and top-10 economics departments assess their tenure-track faculty almost exclusively on the basis of their publications in top-5 economics journals). If you look at Duflo’s early work on healthcare in India, a lot of it published, well, badly, by the standards of the MIT Economics Department. Miguel & Kremer's original deworming paper did make a top-5 journal (Econometrica), but only because they spun their contribution as principally methodological in nature. The title is “Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities;” the paper basically pretends to be about approaches to accounting for potential interference/spillovers between treated and untreated units in an experiment, rather than about deworming itself. I’m too young, but folks I know who were around those circles at the time recall being told that empirical research on health in LMICs “wasn’t economics,” that it was “just reduced-form” and that committing to it was a terrible professional decision. This assessment of the professional consequences wasn’t without justification, either. Seema Jayachandran was denied tenure at Stanford as recently as 2011 due to prejudice against this sort of research. What Banerjee, Duflo, and Kremer won the Nobel for, more than anything else, was changing the field’s conception of what counts as economics in a fairly dramatic way.

So how did folks like Banerjee, Duflo, Kremer, Karlan, Miguel, etc. overcome those obstacles? In the cases of Banerjee and Kremer, both had established themselves as theorists (and received tenure at MIT) before they turned to doing experimental work. In the cases of Duflo, Karlan, and Miguel, 1) they were Banerjee/Kremer students, so they had access to that source of encouragement and support, and 2) they were unusually principled about their interests and willing to take gigantic professional risks.

The consequence of this is that those five names—in one permutation or another—are on a huge proportion of the papers evaluating health & development interventions with credible identification that were published prior to ~2010, and as for interventions that were first evaluated more recently, well, 1) the evidence base about those interventions is often still too thin to be the basis of a GiveWell recommendation, and 2) running these RCTs may be easier today than it was 20 years ago, but it’s still administratively challenging and (more importantly) difficult to fund. If you’re going to persuade Gates (or someone similar) to give you a seven- (or eight-) figure sum of money to figure out if sending people text messages gets them to vaccinate their children, it really helps if you can say that you’ll be drawing on someone like Michael Kremer’s network and expertise in putting your study together. This isn’t just empty signaling, either. There’s a lot that can go wrong in organizing an RCT like this, and having someone on the study team who has loads of experience navigating the challenges that tend to arise is genuinely very valuable. Moreover, if you’re a junior researcher, you may need someone like Kremer’s network in order to get the approvals you need to launch your intervention from the government wherever you’re working, to access J-PAL/IPA resources in-country, and to connect with other reliable implementation partners on the ground (e.g., local non-profits, survey firms, etc.).

In conjunction with the fact that Banerjee, Duflo, Kremer, Karlan, Miguel have all skyrocketed to (global) prominence in the last 15 years, this means that much of the best (particularly experimental) work happening in health and development even today still has one of their names on it. The best work, after all, requires a lot of money and excellent contacts, and the distinguished stature of those five economists has left them with money and contacts in spades. If you go on Kremer’s lab’s website, you can see that it employs, like, a shocking number of people (for a social science research center in a university setting). His personal contribution to the clean water paper under discussion here was probably not enormous, but it’s nonetheless no coincidence that he led it. If I’m a promising young economist with ambitions to do impactful work on these topics, then Kremer is going to be among the people I’d be most eager to collaborate with; I'm going to do everything I can to build a working relationship with him (especially given that he is also, reputedly, a nice guy). I think that's why you see him behind so many of GiveWell's recs.

See in context

Clean Water - the incredible 30% mortality reducer we can’t explain

by NickLaing

Nov 4 20236 min read 46

262

Global health & developmentCause prioritizationRandomized controlled trials

Curated

Clean Water - the incredible 30% mortality reducer we can’t explain

Part 1: Givewell’s Seemingly absurd numbers

Part 2: A Nobel Prize winner’s innovative math

Part 3. We already knew about this anomaly – 100 years ago

Part 4: But why?

Part 5: What next for clean water?

47 comments

TLDR: The best research we have shows that clean water may provide a 30% mortality reduction to children under 5. This might be the biggest mortality reduction of any single global health intervention, yet we don’t fully understand why it works.

Here I share my exploration of a life-saving intervention that we don’t fully understand, but really should. I may err a little on the side of artistic license - so if you find inaccuracies or I'm a bit loose please forgive me, correct me or even feel free to just tear me to shreds in the comments ;).

Part 1: Givewell’s Seemingly absurd numbers

I first became curious after a glance at what seemed like a dubious GiveWell funded project. A $450,000 dollar scoping grant for water chlorination in Rwanda? This didn’t make intuitive sense to me.

In Sub-saharan Africa diarrhoea causes 5-10% of child mortality. While significant, the diarrhea problem continues to improve with better access to medical care, improving ORS and Zinc coverage, and antibiotics for more severe cases. Over the last 5 years, our own Ugandan health centers have encountered surprisingly few very sick kids with diarrhoea and I’ve hardly seen diarrhoea kill a child, as opposed to Malaria and Pneumonia which tragically kill kids all the time. It seemed to me that even if clean water hugely reduced diarrhoea mortality, the intervention would still likely be an expensive way to achieve 1 or 2 percent mortality reduction,

So with my skeptic hat on, I clicked the GiveWell spreadsheet and my incredulity only grew. GiveWell estimated an upper-bound mortality reduction of an almighty 17% for the Rwandan chlorination program! At first that made no sense, but I did expect GiveWell would likely be lesswrong than me.

The Global burden of disease estimates that Diarrhoea makes up only 4.9% of total deaths in Rwanda. How could an intervention which targets diarrhoea reduce mortality by over three times the total diarrhoea mortality? Even if the clean water cured all diarrhoea, that wouldn’t come close to GiveWell’s mortality reduction estimate.

Something fishy was afoot, but I quickly found some answers, through a nobel prize winner’s study which was partially funded by you guessed it…….. GiveWell

Part 2: A Nobel Prize winner’s innovative math

Michael Kremer won a Nobel prize along with two J-PAL co-founders for their wonderful work pioneering randomised controlled trials to assess development interventions. What better person to try their hand at estimating the mortality benefit of clean water than a father of the RCT movement?

But connecting clean water and mortality is tricky, because to date no-one has actually asked whether clean water can reduce child mortality. Instead, a number of RCT asked the more obvious question, does clean water reduce diarrhoea. The answer obviously yes.

But Kremer and co. found a clever way around this. They sifted through all studies which looked at the relationship between clean water and diarrhoea and identified 12 studies^[1] that also gathered bits and pieces of mortality data. They then performed a meta-analysis, pooling that mortality data together to see whether clean water save kids’ lives.

The result – they estimated that clean water caused an incredible 30% mortality reduction in kids under 5. If this is even in the ballpark of correct, clean water could could prevent one in three childhood deaths in much of sub-saharan Africa. If Africa could chlorinate and filter all drinking water, we could save perhaps 1 million lives every year in sub-saharan Africa alone. Mosquito nets might bow to their new king.

To be as crystal clear as the water, this is not just a 30% reduction in diarrheal deaths, but a 30% reduction in overall under 5 mortality. To put this in perspective, mosquito nets reduce childhood mortality by about 20% ^[2] and in one smallish study the new R21 malaria vaccine RTS,S was associated with an estimated 13% mortality reduction ^[3] This raises a fairly obvious question.

How does clean water cause that almighty mortality reduction?

Because it ain’t just the diarrhea. Kremer and co calculated that reduction in diarrhoea deaths account ed for only 1 in 8 of the lives saved by the clean water intervention.

Part 3. We already knew about this anomaly – 100 years ago

It turns out that science has been around for quite some time, and sometimes the scientific-wheel-of-time comes full circle. Over 100 years ago, this exact same phenomenon Kremer observed in 2023, was independently discovered by two scientists on opposite sides of the world - Mills in Lawrence, USA and Reinke in Hamburg, Germany.

In the late 19^th century Germany and the USA finally began systematically filtering their water – for the first time huge swathes of the western world drank, bathed and washed clothes with water cleared of microbial soup. As expected the horrible killer that was typhoid was all but wiped out, but Mills and Reinke noticed something more profound - Childhood mortality plummeted, far more than could be explained by the absence of typhoid. Some reports claimed a 40%, or even 50% drop in child mortality. The world all of a sudden became a whole lot better than we expected.

In a landmark 1910 paper ^[4] this anomaly was labelled The Mills- Reinke Phenomenon in a widely cited paper by Sedgewick, another public health legend. As I write this in Boston after the EAG, I feel some warm fuzzy local MIT history.

Part 4: But why?

Occam’s razor suggests that the reduction in mortality might be due to reduction in a range of non-diarrhoea infectious diseases as well. Perhaps when people wash with clean water, bugs that cause disease such as pneumonia and meningitis are killed and don’t spread as readily as before. Perhaps neonatal infections are reduced as cleaner water is used in the birthing process.

Or maybe its something different - when the body stops taking hits from diarrhoea, malnutrition reduces and the immune system can handle other infections better. This is more likely than you might think, as without clean water kids get diarrhoea all the time, in the ballpark of 5 times ^[5] each year.

However even if we add together a bunch of potential infectious disease reductions, it is not easy to explain a full 30% mortality benefit. To visualize this, here’s GBD’s causes of under 5 mortality in Rwanda and how much of the square we would need to cut out to account for a 30% reduction.

To demonstrate how hard it is to account for this mortality reduction, here's an example of a barely plausible "lolly bag" of mortality reductions that we would need to account for even a 25% total mortality reduction

- 50% reduction in diarrhea mortality (at the extreme end of what is reported in studies)
- 30% reduction in neonatal mortality
- 30% reduction in pneumonia mortality
- 50% reduction in Malnutrition mortality
- 20% reduction in malaria mortality (from improved immunity and nutrition? Clean water can't plausibly reduce malaria prevalence as mosquitos don't care how clean your skin or gut are)

Perhaps you have a better, novel idea of how this all works, can prove it and win the Nobel prize. You might collect it hand in hand with Kremer as he receives his second…

I suppose one could argue that “the why” isn’t so important. What matters is the benefit, the consequences. We should pass go, collect the fantastic mortality reduction and "wash our hands"^[6] of the question. I disagree – understanding the science here might reveal other hidden benefits. It may help us choose better between different methods of cleaning the water. Or if there is immunology involved, we might even be pointed towards new vaccine techniques or other solutions for infectious diseases.

This mystery should be solved.

Part 5: What next for clean water?

1. We still need a RCT powered for Mortality
This may seem like overcaution, but I don’t think our data is good enough yet to call the slam-dunk on clean water as this apparently epic magic bullet for reducing child mortality. Kremer looks retrospectively at data not gathered for-purpose, which is in epidemiological speak a little dodgy. As of yet, we have no RCT which has been designed and powered to detect mortality benefits from clean water. I have heard there may be a study underway looking at this, but I couldn’t find it (let me know if you can). If it is ongoing, I sure hope they are looking at the causes of excess mortality in the non-clean water group, because that could get us a lot closer to understanding the why.

2. We should (probably) continue scaling up clean water in the meantime.
The data we have strongly suggests that cleaning water is likely to be a high cost-effective intervention. A bigger practical debate may be how to clean the water. Evidence Action’s large scale program uses multiple methods, including water dispensers for individual households and in-line chlorination. However given that these are not permanent measures there is a reasonable more long-termist^[7] argument that simply speeding up the expansion of nationwide piped water networks might be a more durable, longterm approach albeit more expensive and slow.

3. WE NEED TO KNOW WHY lives are being lost from dirty water, and saved by the clean stuff. Is it mostly due to one disease? From a range of diseases? From something else entirely? If we understand why, this may open up other avenues and interventions which could save further lives. We owe it to great scientists like Kremer, Mills, Reinke and Sedgewick, but even more to the millions of kids still dying every year because they aren’t drinking clean water.

^{^}
Only 5 of these were RCTs. Givewell's more conservative 17% number arose from leaving out the other studies, and only using the RCTs in their analysis
^{^}
https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD000363.pub3/full
^{^}
https://www.science.org/content/article/first-malaria-vaccine-slashes-early-childhood-deaths
This 13% mortality reduction from the malaria vaccine also doesn't make complete sense, when malaria only makes up about 10% of childhood mortality. Perhaps another article for another day... Maybe if we find enough interventions which massively reduce childhood mortality we could end up with negative mortality? (J/K)
^{^}
https://www.jstor.org/stable/30073304?seq=1
^{^}
https://www.mdpi.com/2078-1547/6/2/229
^{^}
I will never apologise for a bad pun
^{^}
I know this isn't actually a "long-termist" argument, I'm just trolling.
^{^}
^{^}
^{^}

262 Reactions

Mentioned in

123Why EA should (probably) fund ceramic water filters

682023: highlights from the year, from the EA Newsletter

11Some important research questions in economics

More posts like this

Comments47

Sorted by

New & upvoted

Click to highlight new comments since: Today at 3:19 PM

Dan_KeysNov 4 202357

I believe the paper you're referring to is "Water Treatment And Child Mortality: A Meta-Analysis And Cost-effectiveness Analysis" by Kremer, Luby, Maertens, Tan, & Więcek (2023).

The abstract of this version of the paper (which I found online) says:

We estimated a mean cross-study reduction in the odds of all-cause under-5 mortality of about 30% (Peto odds ratio, OR, 0.72; 95% CI 0.55 to 0.92; Bayes OR 0.70; 95% CrI 0.49 to 0.93). The results were qualitatively similar under alternative modeling and data inclusion choices. Taking into account heterogeneity across studies, the expected reduction in a new implementation is 25%.

That's a point estimate of a 25-30% reduction in mortality (across 3 methods of estimating that number), with a confidence/credible interval that has a lower bound of a 7-8% reduction in mortality. So, it's a fairly noisy estimate, due to some combination of the noisiness of individual studies and the heterogeneity across different studies.

That interval for the reduction in mortality just barely overlaps with your number that "Sub-saharan Africa diarrhoea causes 5-10% of child mortality." (The overlap might be larger if that rate was higher than 5-10% in the years & locations where the studies were conducted.)

So it could be that the clean water interventions prevent most children's deaths from diarrhoea and few other deaths, if the mortality reduction is near the bottom of the range that Kremer & colleagues estimate. Or they might prevent a decent chunk of other deaths, but not nearly as many as your Part 4 chart & list suggest, if the true mortality reduction is something like 15%.

There is generally also a possibility of a meta-analysis giving inflated results, due to factors like publication bias affecting which studies they include or other methodological issues in the original studies, which could mean that the true effect is smaller than the lower bound of their interval. I don't know how likely that is in this case.

Here's a more detailed look at their meta-analysis results:

freedomandutilityNov 5 202310

Two thoughts on this paper:

Does it make sense to pool the effect of chlorine interventions with filtration interventions, when these are two different types of interventions? I don't think it does and notably the Cochrane review on this topic that looks at diorrhoea rather than mortality doesn't pool these effects - it doesn't even pool cholirnation products and flocculation sachets together, or different types of filtration together - https://www.cochrane.org/CD004794/INFECTN_interventions-improve-water-quality-and-prevent-diarrhoea - it's hard not to notice that neither of these sub-group effects were statistically insignificant until they were pooled together, which makes me worry about p-hacking.
These interventions obviously have spillover benefits to other individuals in the household, so I suspect that focusing on mortality in under-5s significantly underestimates the DALYs averted by point-of-care chlorine dispenser and water filtration interventions.

Karthik TadepalliDec 22 20238

it's hard not to notice that neither of these sub-group effects were statistically insignificant until they were pooled together, which makes me worry about p-hacking.

But that's the whole purpose of a meta analysis like this. All of the individual studies are under-powered to detect an effect on mortality; even if there was a real effect there, mortality is too rare of an event to reliably detect in a small sample.

freedomandutilityDec 22 20234

Right, but pooling or not pooling effects of different interventions relies on a subjective assessment of whether the interventions (chlorine, filtration, spring protection) are similar enough. Kremer et al have made different assessments to the Cochrane review authors, which I think needs justification. The subjectivity in this part of any meta-analysis is very susceptible to p-hacking.

NickLaingDec 23 20234

It looks to me like the Kremer paper and the Cochrane review authors have both different methodology and ask different questions - the Cochrane review analysis RCTs as they stand and asks if clean water reduces diarrhoea (which it did), while Kremer mines extra mortality data from previous RCTs then meta-analysis it to look for mortality reduction.

I completely agree the Kremer paper is far more ambitious, and has potential for p-hacking. One of my points in the article though is that Kremer's mortality reduction finding is eerily similar to what Mills and Reinke found 100 years ago which adds a little more credence I think. Also I like Givewell's approach of agreeing that there is likely to be a significant mortality benefit, but being more conservative in their approach than the results of Kremer's study.

What different assessments did you think Kremer made from the Cochrane review authors?

Nice one.

NickLaingNov 5 20233

Thanks freedom interesting questions.

I think pooling different methods is probably fair enough, although like you and Dan point out, p hacking is a possibility in retrospective studies like this with no pre printed protocol.

Yes there are many other benefits, and Givewell accounts for some of these in their analysis. This article was focusing though on the mortality overhang, as it were.

Nice one.

NickLaingNov 5 20233

Hey yes I somehow failed to reference the most important paper I was referring to my bad!

Thanks so much for the in depth look here. I agree with all of your points. I was debating writing a list of these issues with the study, but decided not to for simplicity and instead just wrote

"Kremer looks retrospectively at data not gathered for-purpose, which is in epidemiological speak a little dodgy." And yeah, potential p hacking and noisiness are aspects of that dodginess

A couple of small notes

I think even the 8 percent mortality reduction lower bound wouldn't completely wipe out the question. Clean water reduces diarrhoea by 30 to 50 percent, leaving a highest plausible mortality reduction of about 5 percent (I think Kremer listed it as 4 in the study?), so even at the lower bound of mortality reduction and higher bound of diarrhea reduction, there is still a discrepancy.

On publication bias, the kind of big studies they are looking at are likely to get published even with negative results, and their funnel plot looking for the bias looked pretty good.

In general I think a huge RCT (potentially even multi county) is still needed which can look at mortality, and can also explore potential reasons for the large overall mortality reduction.

Arthur BakerNov 23 202329

Great article! I’m Prof Kremer’s Chief of Staff. We’re working both on studies to add more data points on the mortality effects, and on potential scale-up. We hope that we might also be able to shed some light on pathways.

I’d love to have a chat and tell you more about our thinking and our planned work, and to hear about your work at OneDay. My email is arthurbaker@uchicago.edu

I will also write a longer response at some point but no time now!

Alex CohenNov 10 202326

Hey Nick, thanks for raising this question about the plausibility of chlorination’s effect of mortality and the need for more research to understand why. I’m a senior researcher at GiveWell and wanted to chime in with a little more context.

When we did our analysis, we agreed that the ~25%-30% headline figure from the Kremer et al. meta-analysis felt implausibly high. We end up with estimates of ~5%-15% depending on the country and program (e.g., ~6% for the Dispensers for Safe Water Program in Uganda and ~12% for the in-line chlorination program in Malawi).

A bit more on what we did:

We were reluctant to take the Kremer et al. results at face value for the reasons you listed — the reduction in mortality is higher than we’d expect, even if we make pretty generous assumptions on the Mills-Reinke effect, and higher than experts we spoke to would’ve expected, too.
Instead, we first did our own meta-analysis. We focus only on RCTs that study chlorination (as opposed to other ways to increase water quality) and have follow-up lengths of at least a year. We also exclude one RCT with an implausibly high effect. Based on this, we estimate an effect of ~12%.
We adjust this downward to account for some of the RCTs including additional interventions beyond chlorination like hygiene programs (which could overstate the effect of chlorination alone). Then we adjust for differences across countries in the share of deaths we think are attributable to chlorination, compared to the RCTs (e.g., in countries where enteric infection is a smaller share of deaths than in trials, we estimate a smaller effect) and differences in amount of chlorination provided by programs, compared to the RCTs (e.g., we think in-line chlorination provides more chlorination than chlorination programs studied in trials and so has a larger effect).
We also compare our best guess to a “plausibility cap” — an upper bound on what we think the effect of chlorination on mortality could be. This is (I think) the 17% figure the post mentions. In countries where we considered funding in-line chlorination, for example, we guess that if chlorination reduces diarrhea by 25% and infectious diseases account for 70% of mortality in under-5s in these countries, a plausible maximum reduction of mortality is ~17% (25% x 70%). This plausibility cap requires some really uncertain assumptions (e.g., what share of deaths could plausibly be affected by chlorination and by how much?), and we don’t have a lot of confidence in it. We explain more of our rationale for this cap here. In countries we’ve looked at, though, the plausibility cap tends to exceed our initial best guess so it doesn’t end up making a difference in our bottom line estimates.
There’s more detail in our intervention report on water quality and this blog post on why we changed our mind on chlorination programs.

These mortality effect estimates are really uncertain, though, so we’ve funded follow-up research (as Dan alluded to in his comment).

Our best guess is still much larger than we’d expect based on the effect of chlorination on diarrhea alone (implying chlorination averts ~3 deaths from non-diarrhea causes for each death averted due to diarrhea) and relies on a lot of judgment calls.
Because of that, we recently made a grant of $1.8 to Michael Kremer and colleagues at the Development Innovation Lab at the University of Chicago to launch an additional RCT in Kenya and scope larger trials in Nigeria and India.
More on why we made the grant is on our grant page.

Thanks again for boosting this question — it’s something we’ve been thinking about a lot, and I’m glad it’s getting some more attention. I think we’d be open to hearing more thoughts about how we could learn more about the extent to which chlorination affects mortality and why, since we’re continuing to explore more grants to chlorination.

NickLaingNov 12 20233

Thanks so much for engaging Alex.

I didn't explicitly say it, but I think GiveWell did a fantastic job of re-analysing and adjusting here, your range and final estimate for estimating the mortality of the intervention are pretty similar to what my intuition would have moved to, and I agree with your methods of getting there - especially doing your own meta-analysis only using the RCTs.

A couple of other comments

1. I agree with you on excluding the Haushofer/Kremer study, but not for the reasons you state. I dont' really understand why exclude a study just because "we believe the effect size it reports is implausibly large, and it has a substantial impact on the pooled estimate" . This seems unnecessarily subjective to me I'm not sure why 30% might be plausible but 60%ish is not? I know 60% does seem subjectively implausible but I'm not sure that's enough of a reason to exclude a study. If a study is methodologically sound, then why not include it.

BUT I think its reasonable to exclude the study because it was done retrospectively, so was not an RCT at all - You say on your website that is is an RCT, but it is not. They retrospectively gather data on mortality post-hoc well after the original RCT was done. The study itself doesn't claim to be an RCT. As you only want to consider RCTs, this would exclude it automatically as it is not one. Its a small thing but maybe you could consider correcting this on the website?

2. (This is very minor) I like your plausibility "cap", but feel like it might be a little low given that 5 meta-analysed RCTs did show a 25-30% mortality reduction. Would it not perhaps be more logical to use the research figure as the cap? Its hard to "reason" our way to plausible percentages here, precisely because we have little idea why the mortality reduction is happening.

The question I'm very interested in of course is why, and I really hope some of the grants you are making will go towards , rather than purely investigating the magnitude of mortality question - which is obviously still the primary purpose of the studies. The most important way to make progress on the answer might be to ascertain the causes of death in those who die in the treatment vs. control groups, so I hope they are planning to collect that data as best they can at the very least. Your study page doesn't mention that any of the RCT work is geared towards the why - but it's not too late ;).

Again amazing job on analysing this, and funding more research on the topic. I don't think it would have happened without you!

Toby Tremlett🔹Dec 22 202320

I'm curating this post.

I don't usually curate posts from further back than about a week ago, but I wanted to make an exception here because this post stuck in my head. I really enjoy the way that it is written- entertaining while staying informative. NickLaing frames the central mystery of the piece very clearly.

It's also been heartening to see the comments filled with people who are excited to contribute, either through sharing information, asking questions, or getting involved. Great post!

LinchNov 9 202314

Thanks for the post! I found it quite interesting and very readable.

In the abstract, it seems kind of wild to me that so many of GiveWell's $-weighted recommendations (deworming, clean water) essentially comes down to the research led by one guy, given how big the fields of empirical development economics and global health are. My immediate reactions:

Man, this Kremer guy seems really impressive. I mean, I know he has a Nobel Prize, but still. How can we learn from his productivity and ability to repeatedly orient towards research on the most important problems?
Still, personal factors aside, why is his research such a high percentage of overall GW recommendations? What are all the other developmental economists and global health academics doing with their time?
I sure hope his research replicates. Given the base rate of failed replications in social science, I intuitively would place an uncomfortably high credence that maybe results that sound too good to be true are just that.

Just very hot takes, please don't take them too seriously.

NickLaingNov 9 20237

Thanks Linch for the encouragement and great comments. I hope its readable given that I sacrificed a bit of content/rigor for readability reasons. I might continue to do that in future given that response.

Yeah Kremer is a bit mindblowing. Especially considering @HStencil's comment below.

"What are all the other developmental economists and global health academics doing with their time? haha love it, I'm mainly a medical man myself so I don't have a lot to add here on the academics front. Us Global Health people are just casually doing amazing things like making new Malaria Vaccines, TB vaccines, Gene drives, etc. so don't cast that shade on us ;).

I actually didn't know he was quite so influential with GiveWell specifically.

Interestingly GiveWell shared some of your skepticism and didn't entirely rely on him in this case. They did their own mini meta-analysis only of the 5 RCTs Kremer included, leaving out 7 other non-RCT studies. This is where they get their high bound of 17% mortality reduction.

For their mid-range figure I think they settled on 12% for their chlorinating water intervention in Rwanda (too lazy to look it up). Even this would be a massive mortality reduction and I think their discounting makes sense, given that the Kremer study is just one study and it was done retrospectively which always adds a big "dodgy" factor for reasons already discussed by commenters above.

I do think though that the finding being similar 100 years ago when we cleaned up water in the west, adds a little "replication" factor here.

HStencilNov 9 202333

HStencilNov 9 20237

And this isn't even to mention Kremer's substantial contributions to growth theory (which are still his second- and third-most-cited papers), much less the major theoretical contribution he made to the modeling of HIV transmission in his free time back in the late 1990s...

Ian TurnerNov 4 202314

I certainly don’t mean to question others’ on the ground experience, but when I asked people in Uganda and Kenya what programs to fund, water projects were the most common response.

I had assumed that the reason for this was something like, drinking poor quality water is unpleasant or stigmatized beyond the simple health effects, or maybe assumptions about the appropriate role for international funders. But I can’t discount the possibility they know something we don’t.

NickLaingNov 13 20231

Thanks so much Ian. I love this take and I kind of hope your are right.

John SalterNov 6 202313

The size of the effect makes me wonder if there's something multiplicative going on. If two independent bad things co-occuring will kill you, then reducing the likelihood of both those things by 30% would reduce the resulting deaths by 51%.

= 0.51

It was three things that needed to co-occur, then 30% reductions would yield a 66% reduction in resulting deaths.

$1^{3} - {0.7}^{3} = 0.66$

I think this would generalise somewhat to a more complicated causal landscape, so long as a sizable proportion of the deaths are caused by these multiplicative interactions. This could perhaps explain why the reduction in deaths amount to more than the sum of their parts?

Examples of relevant bad things:

- Malnutrition
- Parental illness
- Disease
- Hard times with money
- Doctor burnout / Limited hospital capacity

Mo PuteraNov 7 20233

This is a really interesting take. I wish I knew what terms to look for to find papers exploring these multiplicative interactions, if any.

freedomandutilityNov 5 20239

My best guess - this is a noisy estimate from a meta-analysis which doesn't clearly report risk-of-bias assessments of included studies, so the mortality reduction is probably a lot smaller than 30%.

The time in the past where water filtration caused a mortality reduction greater than expected from eliminating typhoid - I imagine we can attribute large amounts to other infectious diseases like cholera, rotavirus etc?

NickLaingNov 5 20236

Yes it's noisy, but the funnel plot looks ok and in this case I don't think major publication bias is likely. I would be very surprised if more than say 1 major study on this topic was not published.

Yeah the typhoid thing is super interesting. I didn't look into it closely, but I suspect when they say "typhoid" they will be including many other diarrhoeal diseases under that banner too. Our microbial knowledge was pretty poor in the early 1900s.

Hayven FrienbyDec 24 20237

Thank you for posting this! As I see it, whether or not we understand the specific pathway of action is irrelevant. The data is clear: clean water initiatives help kids in a measurable way, and that is effective enough to warrant full support imho.

NickLaingDec 28 20233

Thanks Hayven - I can understand that perspective, and completely agree we should crack on full steam with cleaning up the water.

There's a couple reasons I think understanding the "why" could potentially help us focus our efforts and save even more lives. These are just a couple of the possibilities as well.

"Understanding the science here might reveal other hidden benefits. It may help us choose better between different methods of cleaning the water. Or if there is immunology involved, we might even be pointed towards new vaccine techniques or other solutions for infectious diseases."

Dan SteinNov 7 20237

Thanks, super-interesting!

On "We still need a RCT powered for Mortality", someone from GW could confirm, but I heard through the grapevine that they are going to fund Kremer to do a large multi-country RCT powered for mortality to get at these very questions.

NickLaingNov 7 20232

Thanks yeah a couple of people mentioned this too, but the interwebs were not forthcoming lol. I just hope that they organise the study to answer the "why", as well as get a more definitive answer to the mortality question.

It's interesting that they would ask Kremer to do the study though. I wouldn't have thought he was a medical researcher who was experienced with this kind of RCT, but maybe he would be more of a high level organiser.

Otis ReidNov 8 20234

I don't think of this as really a medical intervention FWIW -- it's a community level water infrastructure intervention, which is very much in his wheelhouse (would also be in the wheelhouse of other folks tbc).

NickLaingNov 9 20232

Thanks Otis I think you have a good point here, nice one.

Joel Tan🔸Nov 4 20236

Hi Nick,

Thanks for sharing! I haven't looked much into this, but it does seem a puzzle. Gun to my head, I would hazard a guess that (a) probably a lot of studies are underpowered to detect the smaller non-diarrheal effect sizes, and (b) a plausible mechanisms that may be worth looking more at is the parental channel (e.g. if WASH interventions make parents less ill -> better care + more income and less malnutrition/access to healthcare for children).

david_reinsteinNov 4 20235

I think you meant Michael not Jason Kremer

NickLaingNov 4 20233

Nice one have changed thanks! At least I didn't write John Kramer ;).

RamiroNov 10 20234

Thanks for this.

I have seen many informative comments here, especially on the stats of the paper...

Let's suppose your postulated "30% all mortality reduction" effect is real; on your figure on GBD cause of deaths above "neonatal disorders" occupy almost the same area. So, if such an effect is real, it likely has to affect neonatal disorders - i.e., preterm birth complications, neonatal encephalopathy due to birth asphyxia and trauma, neonatal sepsis and other neonatal infections, haemolytic disease and other neonatal jaundice, and other neonatal disorders.
What would cause an effect on neonatal disorders? Weel, I wonder if many of such "neonatal infections" are not caused by enteric pathogens - or other agents that will be affected by water treatment in general (see here and this). Besides that, diarrohea might affect the mother's health and birth weight, etc.

BTW look at this interesting Lancet case-control study (Levine et al., 2020) funded by Gates & Gates Foundation that follows children with mild and moderate-to-severe diarrhoea that seeked health care, concluding that even mild enteric infections (which might not show up as a factor in GBD) increased the risk of death overall.

Still, I wonder why clean water is not a priority for WHO and World bank (at least according to Kremer's paper). Maybe other interventions have similarly surprising spillover effects - i.e., perhaps Malaria prevention and de-worming affect neonatal deaths, too (I'd have to check their own spreadsheets to see this).

JamesNNov 4 20234

I was intrigued by this when you told me about it at EAG Nick, so it’s great to see you’ve written it in a post here as it absolutely merits further investigation.

As you say, finding out why this (may) be true is highly important. Given the potential scale of impact here and the complexity of interactions which may be creating that effect, this feels a great candidate for some sort of causal/Bayesian model to be created as part of a research project. If done correctly this then could be used to inform several RCTs that can try and find the true effect of potential predictors (of which we seem to only have very data poor priors for currently).

NickLaingNov 4 20233

Thanks James. Yeah its a really good point how poor our priors are here. They are almost straight theory with little empirical evidence to back them up.

Benevolent_RainNov 4 20234

Do you think it could be impactful for someone to be on the ground for some time, turning every rock to try to uncover the mechanisms behind the reduction? I am thinking something a bit like what was done with Ebola where anthropologists on the ground identified burial practices as a mechanism for the spread of the disease. I love your post by the way - fascinating stuff!

NickLaingNov 4 20234

Thanks Ulrik appreciate the encouragement!

I like the idea of doubling down on research, although in this case I'm not sure there's enough to work with right now to do that Ebola style research you speak of. If there was research we could do right now it would likely be physiological or immunological rather than field research

What might be really useful to focus the resarch is to figure out what is causing the excess deaths in those without clean water. That's where a new RCT might come in handy, or this could even potentially be picked up observationally - just look at one community with clean water and another without and see what the kids are dying of in both places and hone in on discrepencies. In saying that accurately identifying cause of death in kids is no really difficult without autopsies in poor settings.

Then once you know what is causing the excess deaths, you can more easily hone in on the underlying cause.

I suspect there has been some of research along these lines already - its a fairly obvious lingering question. Perhaps there has not been much progress, and little has been published or perhaps I'm missing it.

Jonas BeckerNov 10 20234

Thanks, interesting and easy to read! Hot take from an outsider: clean water may actually have more "holistic" effects than just reducing disease prevalence. I'd be curious about the effects on gut microbiome development which could affect general strength of the immune system. Diarrhea is on the worst extreme of the spectrum but there may be a lot more on their that's harder to differentiate but has lots of potential.

André KirschnerApr 18 20243

Totally agree! For all that are not familiar with microbiom:

Imagine a forest that is watered every few days with acid or poisened water. The ecosystem will change, will adapt, will get less strong against pests and parasites or other invasive plant species and herbivore. The plants will be weaker, but also the animals living inside, the worms, insects, ...

This is our gut that has to deal daily with unclean water.

We have more bacteria in our gut than cells in our body. My assumption is that (as in the last 10 years) we will learn alot about our intestine ecosystem in the following years. And I assume that these learnings will answer at least in parts your "WHY" Nick.

Well written and inspiring. Thanks Nick.

NickLaingNov 10 20232

Yeah that's a good call in nicer one. The immunological argument is definitely a strong possibility here. I wonder if people here drive gut microhlbiome studies in comparable places which have clean water available vs. None available?

SjlverNov 9 20233

This is very well written. Thanks! It's the kind of article that sparks (my) curiosity.

I looked for some information on Helvetas' website. Helvetas is a Swiss charity that has been running safe water interventions for about 50 years; they are funded by private donors, but also receive development aid money from the Swiss government.

Helvetas provides some ideas why water interventions might help, besides diarrhea:

Disproportionally helps women and girls: Women and girls in poor communities often spend several hours a day fetching water => big opportunity cost, probably unhealthy for their heads and back.
Unsafe water is used in critical situations, such as during child delivery. (edited to add: There are hints that this might be significant. For example, WHO and many organizations work to promote breastfeeding, and this is shown to reduce child mortality. Presumably many of the averted deaths would be due to unsafe water)

There are also positive effects of water management in general. These don't apply to chlorination or to filters at existing wells... but I found it helpful to consider more holistic approaches to water:

Improving water resource management is also key to equitable development, climate change adaptation, disaster risk reduction, sustainable agriculture and the prevention of conflicts.

Unfortunately, Helvetas' websites and reports are somewhat light on research. They provide numbers for the number of people reached by their programs, but to my knowledge there isn't any cost-effectiveness analysis. +1 that we need more research!

Vasco Grilo🔸Nov 9 20232

Cool investigation, Nick! Although I am confused about whether saving lives is beneficial/harmful, I liked how you described your own journey from being surprised about a topic to eventually understanding it, everything in a concise nice-to-read way!

RamiroNov 15 20234

Adding a new layer to your confusion: since most of the lives in question are probably newborns (as most of the mortality is captured by neonatal disorders), and since the time lag between one's specific donation and its corresponding effects is likely more than 9 months, this means that one's donation will probably impact the identity of the affected kids. If for some reason (e.g., non-identity problem) one believes that helping existing people is more important than helping future people, then this counts against donating to clean water projects.
(I totally disagree with this, ofc)

Vasco Grilo🔸Nov 15 20232

Hi Ramiro,

Interestingly, that point also applies to the effects on animals! Most of the animals who would suffer due to preventing the death of a newborn do not yet exist at the time the newborn was saved. Like you, I totally disagree with this kind of reasoning. Suffering is bad even if the action which led to it happened before the suffering being came into existence.

NickLaingNov 9 20234

Ha thanks man, appreciate it.

I sure hope your are wrong about the saving lives thing, otherwise my life is likely well on the net negative end ;).

Looking forward to seeing more of your interesting posts!

John SalterNov 6 20232

Very nicely written!

YarrowNov 4 20232

Bravo!

Clean water for all!

Peeter LaasNov 19 20231

Clean water is used for drinking, cooking, cleaning, and personal hygiene. Cascading effects that clean water provides to reduce child mortality are very well understood. If you look at the mortality stats for ages 1-4 in Rwanda (https://www.who.int/data/gho/data/themes/mortality-and-global-health-estimates/ghe-leading-causes-of-death) and assume reduction of both leading causes of death (lower respiratory infections - 38% and diarrhoeal diseases - 25%; 2019), then cutting mortality by 33% (ages below 5) sounds about right.

[comment deleted]Nov 5 20232

Deleted by freedomandutility, 11/05/2023