Summary

In April 2022, GiveWell made major updates to their estimates of how cost-effective water chlorination is at improving well-being, with their most recent estimates indicating that such interventions are 4-9x as cost-effective as GiveDirectly. We examined GiveWell’s water quality intervention cost-effectiveness model for errors or methodological issues. We discovered five potential findings and developed approaches to correct for these findings as best as possible given limited available public data on some of these topics. Our major changes to the cost effectiveness analysis were:

• Redoing GiveWell's meta-analysis on the effect of water chlorination on all-cause mortality to include Haushofer et al, 2021. This increases the estimated mortality effect size by ~40%, but due to structure of GiveWell's analysis, has no independent effect on overall cost effectiveness.
• Using the studies on water chlorination and illness to calculate a full probability distribution for the effect of chlorination on all-cause mortality, not just a biologically plausible maximum. Together with the updated meta-analysis, this reduces the cost effectiveness of the interventions by ~7% for in-line chlorination and ~17% for Dispensers for Safe Water.
• Adjusting the moral weight calculations to account for deaths averted from non-waterborne illnesses. This reduces the cost effectiveness of the interventions by ~2%.
• Revising the cost estimate for in-line chlorination to use intervention-specific estimates rather than reuse cost estimates from Dispensers for Safe Water. This increases the cost effectiveness of in-line chlorination by ~13%.
• Discounting future spending and benefits to account for time delays between funding provision and health impacts. This reduces the cost effectiveness of the interventions by ~12% in the long term and ~5% in the short term.

Using our updated approach, the cost-effectiveness of in-line chlorination falls by 0.21% and the cost-effectiveness estimate of Dispensers for Safe Water falls by 23%. Our results are incorporated into an updated version of GiveWell’s water quality cost-effectiveness model spreadsheet, where cells with updated input data are highlighted in red.

Introduction

Diarrheal diseases are a major cause of death and illness worldwide, causing more than 1.5 million deaths annually. Since diarrheal diseases are commonly spread by contaminated water, interventions that clean water are natural areas of interest within global health. For many years, the evidence base on clean water interventions was small and relatively mixed, leading GiveWell to not choose water quality as an area to make significant grants in. However, in 2022, economists Michael Kremer, Stephen Luby, Ricardo Maertens, Brandon Tan, and Witold Więcek published a new Meta-Analysis on water treatment interventions' impact. The authors found that water quality interventions led to major (~25%) reductions in all-cause child mortality, and that they are likely highly cost-effective methods of improving well-being.^[1] In response, GiveWell’s staff undertook their own updated analysis of water quality interventions, concluding that water quality interventions led to a smaller (~14%) but still highly impactful reduction in child mortality, and that they are somewhere between four and nine times as effective as unconditional cash transfers at improving well-being. Based on these results, GiveWell provided a $64.7 million grant to Evidence Action’s Dispensers for Safe Water (DSW) program in 2022. 

The two water quality interventions that GiveWell examines in its cost-effectiveness analysis (CEA) are DSW and in-line chlorination (ILC). Both of these interventions work by adding a small amount of chlorine to drinking water to kill pathogens. DSW involves placing chlorine dispensers next to water sources and teaching communities about the process and importance of using them, so that individuals can add chlorine to containers of water after filling them. ILC involves adding chlorination systems in-line with the water sources, so that dispensed water is automatically chlorinated. In addition to these interventions, several studies have also tested the provision of storage containers designed to keep water clean after filling them, but GiveWell does not examine this type of intervention in its CEA. 

Water quality interventions do not simply improve well-being by decreasing child mortality; they also improve well-being by decreasing adult mortality, decreasing health spending, creating economic benefits, and decreasing morbidity (time spent sick). GiveWell’s CEA for water quality interventions attempts to account for all these benefits, estimating that approximately 30% of the benefits are provided by each of reducing child and adult mortality, approximately 20% of the benefits are provided by each of development effects and reduced medical costs, and a trivial fraction of the benefits are provided by reduced morbidity.^[2] It is important to note that water quality interventions have been found to reduce death due to both waterborne and non-waterborne diseases, perhaps due to negative health effects of waterborne diseases making people more susceptible to other illnesses. This effect has been observed in a large number of contexts for water quality interventions, and is referred to as the Mills-Reincke phenomenon. The Mills-Reincke effect is quite significant for the interventions in GiveWell’s CEA, with GiveWell estimating that water chlorination eliminates approximately 2.7 child deaths from non-waterborne disease for every death averted from waterborne disease. This effect is accounted for in GiveWell’s benefit estimates on child and adult mortality and medical costs averted. 

To complete the benefit side of the CEA, GiveWell applies moral weights to the effects of the intervention and applies downside adjustments to account for issues such as wastage and displacing other donors’ funding. GiveWell’s moral weights are based on the type of benefit (mortality, morbidity, or monetary), as well as the age and income of the recipient. While the weights are standardized between all of GiveWell’s cost-effectiveness analyses, applying them requires understanding the demographics of an intervention’s beneficiaries. GiveWell’s downside adjustments for water quality interventions are more ad-hoc, with justifications of “subjective guess” or “we would guess” underlying several numeric values.^[3] In particular, GiveWell applies a 10% downside adjustment to account for decreasing household sizes (which could cause fewer children under 5 to be reached than estimated), a 5% upside adjustment to account for possible positive spillover effects, and a 2-10% negative adjustment to account for other donors possibly funding water quality interventions if GiveWell did not.^[4] 

The cost side of GiveWell’s CEA for water quality interventions is based on available data for DSW. The interventions have fixed costs for installation and removal of the chlorination systems, as well as ongoing maintenance costs. As a result, the cost per person per year of a given intervention depends on the duration that the systems remain installed and operational. GiveWell’s cost-effectiveness analysis considers two cases: short-term and long-term. The short-term case is related to GiveWell’s grant to Evidence Action in 2022, which covered 4 years of operation in Uganda and Malawi and 3 years of operation in Kenya, assuming that the program would be shut down at the end of the grant. The long-term cost is based on an assumed 10-year life of the equipment, and generally leads to lower cost per person impacted per year estimates than the short term case. 

For the GiveWell Change Our Minds contest, we chose to examine GiveWell’s water quality intervention CEA for potential improvements. We were especially interested in water quality interventions as one of the newest areas considered by GiveWell, and one not examined by Alex Bates in his deep dive into GiveWell’s CEAs. We kept the basic structure of the CEA the same, looking for any places where we thought GiveWell’s CEA was inconsistent with its standard approach or otherwise did not follow best practices. The next section discusses the 5 potential issues we identified within the water quality CEA, and the subsequent sections discuss the overall impact and implications of these potential issues. Our updated parameters and results are located in a copy of GiveWell’s water quality CEA spreadsheet, and additional technical data is provided in the appendices to this writeup. 

Potential Issues Found

Not Including Haushofer et al. (2021) in GiveWell’s Meta-Analysis

GiveWell’s meta-analysis for the effect of water chlorination on all-cause mortality in children under 5 excludes Haushofer et al. (2021). We think that this decision is most likely incorrect, and the decision to exclude the study biases their conclusions. 

GiveWell gives two reasons why they exclude the Haushofer 2021 study from their meta-analysis:

1. It studies the same intervention (Kenya WASH-B) as Null et al. (2018), so including it would cause the studies in the meta-analysis to not be independent.
2. GiveWell considers the effect size too large to be reasonable.^[5] 

We think that neither of these reasons are a reasonable justification for excluding this study.

Both Null 2018 and Haushofer 2021 are based on chlorine dispensers randomly assigned to villages during the Kenya WASH-B study,^[6] but this alone is not a good reason to exclude the study. Fundamentally, this concern would be equally well addressed by removing Null 2018 and including Haushofer 2021, so it cannot give any grounds for preferring one study over the other. 

We also think that there is less overlap between the two studies than GiveWell suggests. The primary analysis in Haushofer 2021 does not include households who participated in the WASH-B study, and they also report subgroup analyses "excluding all children of mothers with any children born within one year of WASH-B". There should be no overlap in participants between the Null 2018 results and the Haushofer 2021 results for this subgroup, minimizing the correlation between the two studies. We therefore think it is appropriate to treat the Null 2018 results and Haushofer 2021 subgroup results as independent in our updated meta-analysis. 

The second reason, that the reported effect size is implausibly large, contradicts current best practice for meta-analysis. The Cochrane Handbook for Systematic Reviews of Interventions states

In general it is unwise to exclude studies from a meta-analysis on the basis of their results as this may introduce bias.

Excluding studies for having too large of an effect runs the risk of biasing the results downward. We think a better approach is to include all the studies in the meta-analysis, regardless of outcome, and to incorporate additional information about the expected effectiveness of water chlorination using a Bayesian framework. This is the approach we adopt, and we discuss the Bayesian framework in more detail in the next section. 

We therefore updated GiveWell’s meta-analysis to include the results of Haushofer 2021. We took the Haushofer 2021 results that exclude “all children of mothers with any children born within one year of WASH-B" and that are not conditioned on pre-intervention mortality rate (Table 7, column 4). Haushofer 2021 reported their results as the change in mortality rate between control and intervention arms. We translated this result into a log relative risk using the baseline mortality rate provided in the text (2.23%) and assuming an equal number of individuals in the control and intervention arms. This led to a calculated log relative risk for all-cause mortality in children under 5 of -1.39 +/- 0.56. In the updated meta-analysis, the large uncertainty in the result means that Haushofer 2021 contributes only 5% to the total weight. 

Including Haushofer 2021 shifts the mortality effect size from a 14% reduction to a 19% reduction, a 42% increase.

Using a Maximum Plausible Effect Size Instead of a Bayesian Framework

GiveWell's analysis calculates the reduction of all-cause mortality due to water chlorination in two ways:

1. Directly, using the results of randomized controlled trials measuring the effect of water chlorination on all-cause mortality.
2. Indirectly, using estimates of the effect of water chlorination on diarrhea morbidity and assumptions about the relationship between diarrhea morbidity and all-cause mortality to calculate a "maximum plausible mortality reduction"

The overall estimate for the effect of water chlorination on all-cause mortality is the minimum of the direct and indirect estimates.  We think that this approach is flawed for multiple reasons:

1. The indirect estimate is supposed to be a maximum plausible mortality reduction, but this term is never strictly defined. Does this correspond to the upper end of the 90% confidence interval, the 95% confidence interval, or something else?
2. While generous assumptions are made for the Mills-Reincke effect and how changes in diarrhea morbidity could affect diarrhea mortality, only the point estimate for the effect of water chlorination on diarrhea morbidity is used. When calculating a plausible upper limit, we think it would be much more appropriate to use the 95% upper confidence limit for the effect of water chlorination on morbidity.
3. More generally, we think the approach of only using the indirect estimate to set a hard upper cap on the mortality reduction is incorrect. We think that a better approach is to treat both the indirect and direct approaches as independent estimates of the mortality reduction, each with their own probability distribution, and select the estimate that maximizes the combined probability distribution from both approaches. This allows all the information from both lines of evidence to be considered at the same time. GiveWell takes a similar approach in their CEA for deworming interventions, in which they use Bayesian methods to combine data sets on deworming’s short-term and long-term impacts. 

We therefore implemented our own combined analysis of the direct- and indirect- estimates using R and Stan. The code for this model is available on github, and the summarized results are included in our updated cost effectiveness spreadsheet. Our model follows the approach used in the current GiveWell CEA spreadsheet, but replaces the point estimates and upper confidence limits with explicit probability distributions for the parameters of interest. We also added two additional correction factors to the indirect estimate: an adjustment for the different levels of adherence in the Clasen et al., 2015 meta-analysis for morbidity and GiveWell's meta-analysis for mortality, and an algebraic correction for diarrheal diseases and other affected illnesses being more common in areas with lower levels of water treatment. A more in-depth discussion of the parameters in our model is provided in Appendix 1. 

The combined effect of the updated meta-analysis and the Bayesian modeling is a 7-21% decrease in the estimated percent reduction in under-5 all-cause mortality and a 18-24% increase in the estimated percent reduction in under-5 morbidity due to diarrheal disease. 

Inconsistent Treatment of the Mills-Reincke Phenomenon

In GiveWell’s CEA, the Mills-Reincke phenomenon is used to explain the large reduction in all-cause mortality despite the low impact of water chlorination on diarrhea morbidity.  However, the Mills-Reincke phenomenon is applied inconsistently to other areas of the analysis. We identified two additional areas where the Mills-Reincke phenomenon should be accounted for in the analysis. 

First, GiveWell does not account for the Mills-Reincke effect was calculating the moral value of each death averted. Because GiveWell assigns different moral values to differently aged individuals, the moral values depend on the age distribution of deaths averted. GiveWell currently calculates moral values and number of deaths averted separately for people under 5 and over 5, but there are significant differences in moral values within these two categories. Currently, GiveWell calculates the moral value per death averted assuming that all deaths are caused by enteric infections (diarrheal diseases, typhoid, and invasive non-typhoidal Salmonella in the Global Burden of Disease dataset). However, the Mills-Reincke effect means that enteric infections accounts for only a minority of deaths averted. Properly accounting for causes other than enteric infection shifts the age distribution of deaths averted, therefore changing the moral values. Because many of the deaths averted via the Mills-Reincke phenomenon are in neonates, which GiveWell assigns a lower moral weight than children aged 1-4, accounting for causes of death other than enteric infection lowers the average moral weight of each under-5 death averted by approximately 3%.

Second, GiveWell’s cost-effectiveness analysis applies a constant Mills-Reincke adjustment factor to the medical costs averted across all 3 countries. We replaced this constant factor with country- and intervention-specific adjustment factors that account for the relative prevalence of enteric infection morbidity and other infectious disease morbidity in the different countries. Using country-specific morbidity factors leads to a 1-30% reduction in the medical costs averted. 

Lack of Discounting Over Duration of Intervention

GiveWell’s CEA for water quality interventions treats both costs and benefits as if they happen immediately. However, both chlorine dispensers and in-line water chlorination systems are durable goods that deliver their benefits over a longer time period, which GiveWell assumes is approximately 10 years. Costs occur at the start of the intervention for equipment installation and project scaling up, during the intervention for maintenance, and at the end of the intervention for equipment removal and project scaling down. At the same time, the benefits of the intervention occur over the full time period the equipment is installed. 

We think that the long duration of the intervention has several implications for the overall cost-effectiveness analysis. First, since not all the costs accrue in the first year of operation, we think that GiveWell should apply a discount factor to the later costs to account for the lower cost of future spending relative to current spending. Second, since the rate of deaths due to diarrheal disease in sub-Saharan Africa is decreasing, we think GiveWell should decrease the expected benefits of the intervention after the first year to account for this decreasing trend. Third, since some of the benefits of water quality interventions accrue in future years, we think that GiveWell should apply a discount factor to these benefits for consistency with how future benefits are modeled in other GiveWell analyses. 

Figure 1: Annual Diarrhea Deaths Per 100,000 Individuals, Sub-Saharan Africa (GBD)

For other interventions, such as New Incentives, GiveWell both decreases its estimates of future effectiveness of an intervention based on expected future changes in disease burden and applies discount rates to the value of future lives saved and future economic benefits. GiveWell’s standard value for discounting future lives saved is 0.5%/year,^[7] and GiveWell’s standard value for discounting future economic benefits is 4%/year.^[8] The discount factor for changes in disease burden depends on the expected trajectory of diarrheal illness rates absent additional interventions from GiveWell. As shown in Figure 1, the rate of deaths due to diarrheal disease in Sub-Saharan Africa has fallen significantly over the past two decades, with a consistent trend. We estimate the rate of decrease at 5.13%/year, meaning that in 10 years, we would expect that even without additional interventions from GiveWell, the rate of deaths due to diarrheal disease would be approximately 60% of what it is today. 

We are not aware of a consistent methodology used by GiveWell to compare present and future costs. For this analysis, we assumed that GiveWell’s discount rate on the value of its own spending is 4% per year, the same as how it values present vs. future economic benefits for aid recipients. However, this value is an extremely coarse estimate, and we recommend that GiveWell use a value that best reflects their estimated balance between the value of present and future dollars. Since installation costs are paid in the first year of the intervention, their cost should not be discounted. Maintenance costs are paid each year of the intervention, but GiveWell’s current grants to Evidence Action’s Dispensers for Safe Water initiative cover 3-4 years of maintenance costs up-front. As a result, we assumed that 3 years of maintenance costs are always paid up-front. Since exit costs are paid in the last year of the intervention they should be fully discounted. 

Table 1. Discounting Adjustment Factors

We computed separate discounting adjustment factors for health benefits, economic benefits, and health spending reduction for both the short-term (4 year) and long-term (10 year) cases. The discounting adjustment factor is the ratio between the discounted benefits and the discounted costs, and the adjusted benefit of the intervention is computed by multiplying the raw benefit by the adjustment factor. Due to the combination of spending being somewhat front-loaded and diarrheal disease rates falling at a faster rate than the rate at which we assumed costs were discounted, all discounting adjustment factors were computed to be less than one. The adjustment factors used are summarized in Table 1. The net impact of discounting is approximately a 5% reduction in short-term effectiveness (assuming a 4-year intervention life) and a 12% reduction in long-term effectiveness (assuming a 10-year intervention life). 

Reuse of Cost Numbers from DSW for ILC 

GiveWell’s cost-effectiveness analysis for in-line chlorination in Kenya currently directly reuses cost numbers for Dispensers for Safe Water In Kenya, which is likely to have substantially different costs relative to in-line chlorination.^[10] Our best guess is that this re-use is intentional, perhaps due to a lack of high-quality cost data for in-line chlorination in Kenya. However, the lack of cost data specific to in-line chlorination does not appear to be documented in comments on the CEA spreadsheet, and no information is given in on either the “ILC Kenya” or “Results Summary” tabs indicating that the ILC cost-effectiveness estimate is likely a coarser estimate than the DSW cost-effectiveness analysis. Moreover, GiveWell has referenced this cost estimate in writing on its website without providing the context behind it. We think this may lead to confusion around the cost-effectiveness of ILC, especially since there is some evidence that ILC may in fact be cheaper to operate than DSW.  

We conducted a basic literature review to identify information about the costs of ILC systems in previous trials. Evidence we consulted indicates that the average cost of commercially available chlorinators was $140/system. From limited case study information, installing a chlorinator requires 1-6 person-hours, which would likely cost <$10 at prevailing wages in Kenya. Data on operating costs is limited, but research in Bangladesh has suggested that annual operating costs for ILC systems would be in the range of $1.20/household/year for a household size of 4.6 people. Assuming a 10-year lifetime, installation costs $150 above DSW, exit costs $10 above DSW, and a 20% addition above the maintenance cost estimate from the Bangladesh study, this estimate indicates that the cost per person per year of ILC in Kenya is $1.48, compared to $1.68 for DSW. This would lead to a 13% increase in the cost-effectiveness estimate for ILC. However, this is a rough cost estimate based on limited literature, and we do not think it should be taken with high confidence. Rather, we think it indicates the extent to which ILC’s costs are much less certain than DSW’s. It may be that the DSW values are indeed the best current estimate of costs for ILC, but we think that if so, GiveWell should make this reuse clear in its documentation.

Updated Results

Our results are incorporated into an updated version of GiveWell’s water quality cost-effectiveness model spreadsheet. Cells with updated input data are highlighted in red, and a summary of the changes is included in the “Results Summary” tab. Tables 2-4 summarize our results, with cost-effectiveness given in multiples of the cost-effectiveness of cash transfers. All together, our changes lead to a negligible change in the cost-effectiveness of in-line chlorination and a decrease of 17-23% in the overall cost effectiveness of Dispensers for Safe Water. The decreases are larger for the long-term estimates than for the short-term estimates, and larger for Uganda and Malawi than for Kenya. 

Table 2: Updated Cost-Effectiveness for In-Line Chlorination

Table 3: Updated Cost-Effectiveness for Dispensers for Safe Water (Short-Term)

Table 4: Updated Cost-Effectiveness for Dispensers for Safe Water (Long-Term)

 Different mechanisms of benefit are affected in different ways by our revisions. Figure 2 shows how each of the five benefit areas considered by GiveWell change for each of the intervention areas. Benefits via under-5 deaths averted, over-5 deaths averted, and developmental effects decrease in all intervention areas. Benefits via medical costs averted increase in some areas and decrease in others. Morbidity benefits increase in all areas, but the changes are negligible compared to the benefits via other mechanisms. 

Figure 2: Units of Value from Each Benefit Mechanism in the Original and Updated CEA (Short-Term). 

Discussion

Our evaluation of GiveWell’s water quality CEA identified five areas for improvement, which combined to reduce our estimate of the cost effectiveness of DSW by approximately 20% and had no overall effect on the cost effectiveness of ILC. This result is the opposite of what we expected when we started the project, as our initial exploration of the topic area made us think that GiveWell was likely underestimating the cost effectiveness of water chlorination. We were surprised as we investigated more thoroughly that our analysis showed the opposite. 

While we found several areas of disagreement with GiveWell’s analysis, our examination of the CEA made us more convinced that the overall approach is sound and that water chlorination has a noticeable impact on all-cause mortality. Our five areas of improvement are all focused on improving how specific values within the CEA are calculated, rather than reconsidering the fundamental assumptions behind the CEA. 

As two outsiders, we found GiveWell’s water quality CEA to be clearly explained and deeply researched. But at the same time, it is clear that the water quality CEA is newer and therefore less well developed than the CEAs for GiveWell’s top charities. The benefits from medical costs averted and development effects appear to have been investigated in much less detail than the mortality benefits, even though these two mechanisms account for an equal amount of value as the deaths averted.^[11] Similarly, the downside adjustments and cost estimates appear more superficial than in GiveWell’s other CEAs. Many of our edits to the CEA involve replacing an ad-hoc adjustment with a more developed technique from other CEAs. This includes our discounting of future lives saved, which follows the technique from the New Incentives CEA, and our use of Bayesian analysis to combine evidence from multiple lines of reasoning, which is similar to the approach used in the Deworm the World analysis.

A 20% change in cost effectiveness is relatively modest compared to the range of cost effectivenesses considered by GiveWell. For comparison, the country-specific numbers in GiveWell’s other cost effectiveness estimates range from -0.5x as effective as cash to 51x as effective. A 20% change in cost effectiveness in either direction isn’t enough to make water chlorination GiveWell’s most effective charity or its least effective, nor would a 20% increase be enough to push either DSW or ILC over GiveWell’s current estimated 10x cash effectiveness bar for funding. 

At the same time, a 20% decrease in cost effectiveness is large enough that it could potentially affect funding decisions. In early 2022, GiveWell used a 6x cash effectiveness bar for funding, which DSW meets according to GiveWell’s CEA but no longer does in our updated version. Our revised analysis also suggests that ILC is a more promising intervention than DSW. In GiveWell’s original analysis, ILC and DSW were approximately equally cost effective. In our updated analysis, ILC is 33% more cost effective than DSW, although this result is highly sensitive to the estimated cost of ILC systems. Efforts to scale up and widely deploy ILC could therefore be more cost effective than expanding the DSW program. ILC is also a relatively new intervention, so there may be more potential for further development to reduce its cost or improve its adherence rates than there is for DSW. 

Appendix 1: Parameters used in Bayesian Analysis

The R and Stan code for our model is available on GitHub. Below is an explanation of how we chose the point estimates and probability distributions for each of the parameters. 

Direct Approach

1. Mortality effect size: Lognormal probability distribution, with mean -0.214 and sd 0.120, from https://docs.google.com/spreadsheets/d/1BnuHlq4b0NSDjDmcEZ-79gjqrsQdi89zVn4ifSkPQ5k/edit#gid=123265793&range=B12. 
2. Mortality internal validity: Fixed parameter, from https://docs.google.com/spreadsheets/d/1BnuHlq4b0NSDjDmcEZ-79gjqrsQdi89zVn4ifSkPQ5k/edit#gid=1674952052&range=B6
3. Mortality external validity: Fixed parameter, varies by country, from https://docs.google.com/spreadsheets/d/1BnuHlq4b0NSDjDmcEZ-79gjqrsQdi89zVn4ifSkPQ5k/edit#gid=1120551419&range=B6

Indirect approach

1. Morbidity effect size: normal distribution with mean 0.77 and standard deviation 0.13, from the Clasen et al., 2015 meta-analysis
2. Morbidity self-reporting bias: fixed parameter, from https://docs.google.com/spreadsheets/d/1BnuHlq4b0NSDjDmcEZ-79gjqrsQdi89zVn4ifSkPQ5k/edit#gid=1336716860&range=B6
3. Adjustment for the different levels of adherence in the Clasen et al., 2015 meta-analysis for morbidity and GiveWell's meta-analysis (new adjustment): Fixed parameter, from https://docs.google.com/spreadsheets/d/1BnuHlq4b0NSDjDmcEZ-79gjqrsQdi89zVn4ifSkPQ5k/edit#gid=864135715
4. Morbidity internal validity: normal distribution with mean 0.9 and standard deviation 0.05, based on https://docs.google.com/spreadsheets/d/1BnuHlq4b0NSDjDmcEZ-79gjqrsQdi89zVn4ifSkPQ5k/edit#gid=1674952052&range=E3 and accompanying note saying "We have not put much thought into this adjustment".
5. Relationship between diarrhea morbidity and diarrhea mortality: lognormal distribution with mean 0 and standard deviation 0.171, based on the calculations described in this tab.
6. Mills-Reincke effect: Modeled as the fraction of all deaths that the X% reduction in mortality applies to. Normal distribution with mean equal to the sum of deaths due to enteric infections, respiratory infections, and a quarter of other infections and nutritional deficiencies, and standard deviation such that "Assumption 4" in GiveWell's mortality plausibility modeling is 2 standard deviations above the mean.
   • We are interpreting GiveWell's "maximum plausible reduction" to be equivalent to a 95% UCL.
   • Our assumption sets the mean roughly halfway between Assumption 2 and Assumption 3 in GiveWell's mortality plausibility modeling.
7. Correction for deaths due to enteric infection being more common in areas without water treatment (new adjustment): GiveWell's plausibility mortality modeling uses the fraction of deaths from different causes on a nationwide basis. However, intervention areas have lower levels of water treatment than the country as a whole and therefore will have a higher percentage of deaths due to enteric infections and other affected causes. We correct for this algebraically using the data reported here.

Appendix 2: Addendum Regarding Funging

In addition to the five issues discussed in the main report, we identified a possible additional issue with GiveWell's downside adjustment for funging. While we did not have sufficient time to fully investigate this possible issue and to incorporate it into our updated version of the model, we think it is worth flagging for further review. In line 138 of GiveWell's DSW CEA spreadsheet, a value is computed for the units of value created if another philanthropic actor were to fund DSW instead of GiveWell. This value is calculated by multiplying the units of value per $10,000 created by DSW by the percent of the DSW program funded by another donor (100%), then subtracting an estimate of the units of value that the other funder would have created had they not funded DSW. However, the subtracted units of value estimate is computed by multiplying the benefit per dollar of other funders' non-DSW spending, the percent of the program funded by the other donor (100%), the cost per person covered, and the number of dollars spent ($10,000). We think including the cost per person in the  multiplication is most likely incorrect, as it leads to the result being the counterfactual value per 10,000 people covered, not per $10,000 spent. Since the downside adjustment section operates entirely in units of value per $10,000 spent, subtracting the counterfactual units of value per 10,000 people covered does not make sense to us. Because the cost per person covered is relatively close to $1/year, this potential error leads to a modest change in the overall cost effectiveness for DSW, which we estimate is approximately 2%. However, we have not done as deep an investigation into this item as we have for the others, and we may be misunderstanding the intent of this calculation.

1. ^{^}

   The authors shared an early draft of the meta-analysis with GiveWell in 2020

2. ^{^}

   See lines 147-152 of GiveWell’s DSW CEA spreadsheet

3. ^{^}

   See comments in cell A106 of GiveWell’s DSW CEA Spreadsheet and cell E7 of GiveWell’s Funging Spreadsheet

^{^}

See lines 87-141 of GiveWell’s DSW CEA spreadsheet

^{^}

"First, it is a follow-up of the Null 2018 RCT, so it does not meet the independence assumption underlying this meta-analysis method."

"Second we find the effect size implausibly large (including the bottom of the 95% confidence interval)  given the modest intervention effect on chlorination rates"

Source

^{^}

 See the “Methods” section of Null et al. (2018)

^{^}

 See line 124 of GiveWell’s cost-effectiveness analysis for New Incentives

^{^}

 See line 37 of GiveWell’s moral weights and discount rate sheet

^{^}

Because decreases in deaths due to diarrheal disease may be caused in part due to increased health spending, it is not clear that health spending on diarrheal disease would be expected to decrease at the same rate as deaths due to diarrheal disease. As a result, we chose to apply only the 4%/year discount rate associated with economic benefits to benefits from decreased health spending, and did not apply any discounting for decreasing diarrheal deaths. 

^{^}

 Cell B72 in the GiveWell’s ILC Kenya CEA spreadsheet contains the formula “=DSW!B74”

^{^}

 As an extremely crude measure, GiveWell's Water Quality Interventions report spends 4100 words discussing mortality effects and only 680 words discussing the developmental and medical cost effects.

Show all footnotes