Thanks for this thoughtful post, Tom. You’ve definitely raised some thoughts that have been on our mind recently, such as how GiveWell could systematically incorporate more external input into our grant-making process. We’ve taken some steps towards this, such as with a beneficiary survey in 2019, and seeking out more external experts, but we’d like to do more – it was really great to read through your recommendations. 

We did want to clarify, however, that GiveWell’s approach to modeling cost-effectiveness and making grant recommendations is heavily context-specific. You’re right that we start with intervention-level analysis in order to get a rough sense of the cost-effectiveness of any given program. But, our next step is to modify our models with many charity- and context-specific data and only fund grant opportunities that are above our 10x bar. For example, when we’re considering making grants to Malaria Consortium's SMC program, we assess funding opportunities at the country level, taking into consideration the differences in malaria prevalence, demographics (age distribution), mortality rates,  program costs, and the spending we might expect from other actors in each setting. The result is that the same program may clear our cost-effectiveness bar in some locations and not in others. For a recent example, see this page about a grant we recommended in January for Malaria Consortium; we decided to extend funding for its SMC program in Nigeria, Burkina Faso, and Togo (where estimated cost-effectiveness was near or above our bar), but provide only exit funding for Chad (which was below our bar).  

If there’s reason to expect variations within countries, we also build out our model at the subnational level. For example, in 2022 we updated our model of Malaria Consortium's SMC program in Nigeria with state-level malaria prevalence and mortality data. While we don’t capture every variance one could expect, we're trying to adjust for the major variances that could exist in different contexts (and which therefore could affect our bottom-line grantmaking decisions). 

We realize that this nuance isn’t captured in our external-facing marketing, and we’re working on updates to our website to address this. We really appreciate your engagement with our work; we aspire to account for context-specific variables in our grantmaking and appreciate the push to consider this further and to make this aspect of our work clearer.

Thanks for this, I found it interesting!

1. I think HTA offers a concrete path to better emulate moral uncertainty and pluralism, though I worry it could soften and blunt the insight EA contributes through its focus on cost-effectiveness. Do you think this is a real trade-off? Is pluralism better emulated at the field-level, such that it is served by EA continuing its narrow focus?
2. I liked the other recent CGD blog on PEPFAR, a case where cost-effectiveness analysis was  misleading. Seemingly the failures were strategic - not considering how budgetary constraints could be loosened nor how prices could be affected by major efforts. Do you think HTA would have fared better?

Thanks for sharing this, Tom!

Some thoughts for future development -- I recognize, of course, that you were working within the confines of a blog post:

On difference 1: Would it be more accurate to describe this as a continuum? E.g., EA organizations tend to invest significantly fewer resources into translating and adapting cost-effectiveness work in Country A before applying it in Country B than organizations applying HTA. Although giving no thought to emergent properties like differences in disease burden would be really unwise, I am not aware of any evidence that major EA funders are missing the super-low-hanging fruit there. (Of course, I could be underinformed, in which case some concrete examples of serious misses would help me update on their methodology!) 

The idea of the suggested reframing is to paint investing in cost-effectiveness translation as a tradeoff -- one can do more extensive work in this area (HTA) or relatively less (EA), and both approaches have their benefits and downsides. It intuitively seems that best approach would depend on the resources that would be going into the intervention in a new location, the costs of additional translation efforts, and the likelihood that the results of those efforts would either shape the intervention-as-deployed or change funding decisions. Indeed, my hunch is that -- in general -- your suggestions grow stronger as the amount of money EA is putting into global health increases, and the odds of EA funders being able to influence other funders through their actions increases.

On difference 2: I think the crux here is about the effect of other donors and the likelihood of successfully coordinating with them. If one views the decisions of other donors as essentially fixed, then it likely makes sense to allocate one's own funds in a way that seeks to gives cost-effectiveness (or other atttributes one thought undervalued by the broader funding ecosystem) the appropriate amount of emphasis in the global health ecosystem as a whole -- rather than weighing a basket of attributes in the way the donor would if they controlled how all donors spent their monies on global health. 

For longer pieces (not blog posts), it would be worth considering that "someone should do X," "EA as a whole should do X," and "Specific Organization C should do X" are different claims. I think the kind of thinking in your blog post has a lot to offer EA organizations, but at later stages of development would need to be actionable by specific people and organizations in light of their particular constraints to achieve impact. In particular, I see GiveWell as acting more as a pure advisor to donors, so it faces the constraint of keeping those donors (which it has attracted with a goal of finding "charities that save or improve lives the most per dollar") happy. Much more so than Open Phil and Founders Pledge, it is also built around the Unix-like philosophy of (mostly) doing one thing and doing it very well and may experience adverse outcomes from even well-intentioned mission creep by funding more "meta" work like your recommendation 4. 

So I am not sure how actionable recommendations 2, 3, and 4 (maybe even 1) are for an organization in the style of GiveWell. On the other hand, I think they are quite actionable for OpenPhil. Given that GiveWell alone is a significant part of the ecosystem, a suggestion for longer pieces would be to offer more action items that clearly fit within its operating philosophy and constraints.