Yes, exactly. The marginal cost of an A/B test in tech is incredibly low, while for NGOs an RCT represents a significant portion of their budget and operational capacity. 

This difference in costs explains why tech can use A/B tests for iterative learning, trying hundreds of small variations, while NGOs need to be much more selective about what to test. 

And despite A/B testing being nearly free, most decisions at big tech firms aren't driven by experimental evidence.

This is a great point. There's an important distinction though between evaluating new programs led by early-stage NGOs (like those coming from Charity Entrepreneurship) versus established programs directing millions in funding. I think RCTs make sense for the latter group.

As far as I can tell ~0% of nonprofits are interested in rigorously studying their programs in any way, RCTs or otherwise

There's also a difference between the typical NGOs and EA-founded ones. In my experience, EA founders actively want to rigorously evaluate their programs, they don't want to work for ineffective interventions.

Good point. Good to clarify that the 80% power standard comes from academic norms, not an inherent RCT requirement. NGOs should chose their statistical thresholds based on their specific needs, budget, and risk tolerance.

I'm considering writing about "RCTs in NGOs: When (and when not) to implement them"

The post would explore:

• Why many new NGOs feel pressured to conduct RCTs primarily due to funder / EA community requirements.
• The hidden costs and limitations of RCTs: high expenses, 80% statistical power meaning 20% chance of missing real effects, wide confidence intervals
• Why RCTs might not be the best tool for early-stage organizations focused on iterative learning
• How academic incentives in RCT design/implementation don't always align with NGO needs
• Alternative evidence-gathering approaches that might be more appropriate for different organizational stages
• Suggestions for both funders and NGOs on how to think about evidence generation

This comes from my conversations with several NGO founders. I believe the EA community could benefit from a more nuanced discussion about evidence hierarchies and when different types of evaluation make sense.

Thanks for the reading list.

Have you considered including some economic complexity literature (Hidalgo, Hausmann et al.)? Their research shows how countries tend to develop by moving from making simple products (like copper or oil) to more complex ones (like electronics or planes), based on how similar the required skills and resources are.

That matches what I've seen. In this case, though, they were tackling multiple issues to increase their chances of funding, not because they identified a common root cause.

My experience with Latin American NGOs and governments at J-PAL corroborates this: most programs lack cost-effectiveness. 

The trend of projects attempting to address multiple complex issues simultaneously is particularly concerning. When reviewing funding applications targeting child labor, deforestation, and wage improvement, most NGOs claimed they could tackle all three. This approach is fundamentally flawed — making significant progress on even one of these issues is challenging enough. IMO focusing on a single, well-defined problem is more likely to yield measurable impact than diluting efforts across multiple fronts.

There is also the Harvard Growth Lab. They work with governments to foster economic growth. There isn't a way to donate, but an actor to keep in mind. Their work is mostly based on the theory of Economic Complexity. 

Great writing. I do feel it conflates cost-effective solutions and "silver bullets". 

For some problems, there are silver bullets — vaccines is one example. For others, like "poverty among smallholder farmers in Uganda", I don't think we have found one. 

A silver bullet, to me, implies that all of our efforts should be going to that one solution. A quick mental model can be that the solution is orders of magnitude more efficient than the alternatives, so it almost doesn't matter how much you spend on the solution, it will still be cost-effective.

GiveDirectly doesn't clear the "silver bullet bar" as this story suggests (compare to vaccines vs smallpox). You might argue it is cost-effective, but that's a different story.

Very interesting argument. As someone fond of Bayesian modeling, I am almost always in favor of replacing point estimates with distributions.

GiveWell links to this paper in their CEA spreadsheet. Their recommendation is a discount rate of 4% for upper-middle-income countries and 5% for low-income countries. This recommendation seems to be based on three factors: Past growth of GDP, implied social discount rate, and projected GDP growth. All three, are measured with uncertainty and will vary by country. I think it would be very interesting to take that variability into account!