Thanks for the context but putting these caveats in the disclaimers doesn't change how the tool works.
The underlying studies are solid. The problem is the layer on top of them. The scores and weights are AI set guesses on a slider, not field data. Again, that's most true for the civic engagement, equity and justice buckets, which is where most of Scott's money goes. The author says himself those buckets get "wide skeptical priors" because no health pathway has credible evidence. That isn't a real effect adjusted for uncertainty. It's a number filling in for missing data. Since those are the biggest buckets, that number is doing a lot of the work behind the headline frontier multiple. The comparison is weakest exactly where most of her money sits.
That's my real concern. The tool takes her hardest to measure work and makes it look like it buys almost nothing, when that low number is a measurement gap, not a fact about the work. A model is only as strong as its weakest layer. For most of this portfolio, that layer is a guess standing in for how funding actually works on the ground, like whether a local coalition can absorb a sudden influx of cash, or the fact that structural change doesn't run in a straight line to a health outcome.
Flagging that in the fine print is good practice, but it doesn't fix it. The calculator is still running precise maths on top of guesses.
This is a really beautifully designed interactive tool and I appreciate how transparent you’ve been with the code.
But looking at this from the perspective of someone who works on the ground in global health and development, I worry that analysing a $26 billion portfolio built for systemic equity through a narrow, health only calculator could mislead philanthropists. It’s a bit like looking at a massive civil rights or education budget and finding it lacking because it didn’t buy malaria nets.
If you talk to practitioners who actually execute this work, a few big real world missing pieces stand out:
The charities this model uses as a benchmark focus on narrow, medical interventions with quick, easily measured metrics. MacKenzie Scott’s giving is intentionally structural. She's investing in areas such as civic engagement, local leadership and systemic justice. Discounting the value of structural work just because it doesn't yield a clinical trial isn't scientific rigour. It's just a measurement blind spot for things that cannot be neatly randomised.
You cannot scale single disease programs linearly. If you suddenly dump billions into narrow medical tracks, you hit a hard wall on the ground. You trigger local wage inflation for scarce medical staff, overwhelm local supply chains and completely pull local nurses away from general primary care just to service a single subsidised metric.
Since the baseline assumptions (the credibility tiers and realisation numbers) were drafted by AI, the model is essentially running complex math on top of text approximations. It takes what is essentially an AI's best guess and wraps it in a bunch of equations to make it look definitive, rather than building the model on actual data from the field.
This is an incredibly interesting data exercise, but by leaving out field realities, the model relies on flat, linear scaling assumptions that completely overlook the law of diminishing returns. If the goal is to make this genuinely informative for philanthropists trying to drive long-term change, I'd love to see it evolve by bringing in some of those real world constraints.
That’s really interesting, Mo. Appreciate you sharing! The Cholesky approach definitely makes sense conceptually.
From a practitioner perspective, the correlations tend to come from fairly intuitive system dynamics rather than anything formal. So in Northern Nigeria, when outreach improved, you would often see several things move together. Coverage would go up, dropout rates would fall and supply chains would stabilise as demand became more predictable. The opposite would happen when systems were under strain. Staffing gaps, stockouts and lower uptake would start reinforcing each other quite quickly.
The tricky part is that those shifts are often uneven and very context specific. Translating them into a stable covariance structure is not straightforward. But I agree there’s probably a useful bridge here between how these dynamics play out operationally and how they could be reflected in models.
That makes sense, and I think the tool does a great job of making those tradeoffs legible.
One thing I’ve found is that spending time in these settings can change how you think about some of the parameters, especially around counterfactuals and how multiple constraints interact in practice. Certain assumptions that look independent in a model often move together on the ground.
It would be interesting to see how that kind of correlated variation could be explored more systematically over time.
Having worked in northern Nigeria, one thing that stood out to me is how dynamic those counterfactuals can be in practice. For example, in Sokoto and Zamfara, DHS coverage numbers capture the endpoint, but underneath that you have shifting factors like outreach consistency, staffing, supply reliability and community trust. I have seen system performance change meaningfully over relatively short periods in ways that would materially affect those parameters.
It also made me think about places like Kenya and Mozambique that are highlighted as “best” countries in your table. Even within the same country, conditions can vary enormously across regions and over time depending on implementation strength and system capacity. Those differences do not always show up immediately in the underlying data, but they can have real implications for how stable those cost effectiveness estimates are.
Curious how you think about parameter stability over time in settings where the system itself is evolving. The model makes the tradeoffs legible, but the inputs themselves can be moving targets.
Thanks for the context but putting these caveats in the disclaimers doesn't change how the tool works.
The underlying studies are solid. The problem is the layer on top of them. The scores and weights are AI set guesses on a slider, not field data. Again, that's most true for the civic engagement, equity and justice buckets, which is where most of Scott's money goes. The author says himself those buckets get "wide skeptical priors" because no health pathway has credible evidence. That isn't a real effect adjusted for uncertainty. It's a number filling in for missing data. Since those are the biggest buckets, that number is doing a lot of the work behind the headline frontier multiple. The comparison is weakest exactly where most of her money sits.
That's my real concern. The tool takes her hardest to measure work and makes it look like it buys almost nothing, when that low number is a measurement gap, not a fact about the work. A model is only as strong as its weakest layer. For most of this portfolio, that layer is a guess standing in for how funding actually works on the ground, like whether a local coalition can absorb a sudden influx of cash, or the fact that structural change doesn't run in a straight line to a health outcome.
Flagging that in the fine print is good practice, but it doesn't fix it. The calculator is still running precise maths on top of guesses.
This is a really beautifully designed interactive tool and I appreciate how transparent you’ve been with the code.
But looking at this from the perspective of someone who works on the ground in global health and development, I worry that analysing a $26 billion portfolio built for systemic equity through a narrow, health only calculator could mislead philanthropists. It’s a bit like looking at a massive civil rights or education budget and finding it lacking because it didn’t buy malaria nets.
If you talk to practitioners who actually execute this work, a few big real world missing pieces stand out:
This is an incredibly interesting data exercise, but by leaving out field realities, the model relies on flat, linear scaling assumptions that completely overlook the law of diminishing returns. If the goal is to make this genuinely informative for philanthropists trying to drive long-term change, I'd love to see it evolve by bringing in some of those real world constraints.
That’s really interesting, Mo. Appreciate you sharing! The Cholesky approach definitely makes sense conceptually.
From a practitioner perspective, the correlations tend to come from fairly intuitive system dynamics rather than anything formal. So in Northern Nigeria, when outreach improved, you would often see several things move together. Coverage would go up, dropout rates would fall and supply chains would stabilise as demand became more predictable. The opposite would happen when systems were under strain. Staffing gaps, stockouts and lower uptake would start reinforcing each other quite quickly.
The tricky part is that those shifts are often uneven and very context specific. Translating them into a stable covariance structure is not straightforward. But I agree there’s probably a useful bridge here between how these dynamics play out operationally and how they could be reflected in models.
That makes sense, and I think the tool does a great job of making those tradeoffs legible.
One thing I’ve found is that spending time in these settings can change how you think about some of the parameters, especially around counterfactuals and how multiple constraints interact in practice. Certain assumptions that look independent in a model often move together on the ground.
It would be interesting to see how that kind of correlated variation could be explored more systematically over time.
Wow Max, this is super impressive!
Having worked in northern Nigeria, one thing that stood out to me is how dynamic those counterfactuals can be in practice. For example, in Sokoto and Zamfara, DHS coverage numbers capture the endpoint, but underneath that you have shifting factors like outreach consistency, staffing, supply reliability and community trust. I have seen system performance change meaningfully over relatively short periods in ways that would materially affect those parameters.
It also made me think about places like Kenya and Mozambique that are highlighted as “best” countries in your table. Even within the same country, conditions can vary enormously across regions and over time depending on implementation strength and system capacity. Those differences do not always show up immediately in the underlying data, but they can have real implications for how stable those cost effectiveness estimates are.
Curious how you think about parameter stability over time in settings where the system itself is evolving. The model makes the tradeoffs legible, but the inputs themselves can be moving targets.
Really thoughtful contribution.