Author: Miranda Kaplan, GiveWell Communications Associate
For many people, GiveWell is practically synonymous with our short list of top charities. But the amount of money we've sent to other organizations, doing other important work, has been increasing. In 2021, we made or recommended about $190 million in grants to non–top charity programs, like water treatment and malnutrition treatment, and in 2022, we set up the All Grants Fund specifically so donors could contribute to programs in this category.
Source: GiveWell, GiveWell Metrics Report — 2021 Annual Review, p. 9
We want to use this blog to give you more frequent, brief insights into these newer areas of our grantmaking before we publish our formal grant write-ups. Below we'll discuss, in light detail, a program that's well outside of our traditional wheelhouse, but that we think significantly improves children's lives—treatment for clubfoot with an organization called MiracleFeet.
The grant
Clubfoot is a congenital (i.e., present from birth) abnormality that causes one or both feet to twist inward and upward. Children born with clubfoot must walk on the sides or backs of their feet, which leads to pain, severely limited mobility, and, reportedly, social stigma. If not corrected, clubfoot is a lifelong condition. In January 2023, we recommended a $5.2 million grant to MiracleFeet to expand its existing clubfoot treatment program in the Philippines and launch two new programs in Chad and Côte d'Ivoire.
In the countries where it works, MiracleFeet and its local NGO partners help health facilities diagnose and treat clubfoot, using a process called the Ponseti method. This generally requires placing the affected foot in a series of casts, performing a minor surgical procedure to improve the foot's flexibility, and bracing the foot during sleep for up to five years. MiracleFeet and its partners provide supplies for casting and bracing, train government health care workers in the above procedures, build awareness of clubfoot, and help health systems collect data on treatment. This makes it comparable to a "technical assistance" program: MiracleFeet doesn't perform clubfoot treatment itself; instead, along with its partners, it helps set health facilities up to successfully find and treat clubfoot cases themselves.
The brace and custom shoes supplied by MiracleFeet for clubfoot treatment. Photograph courtesy of MiracleFeet.
We were excited to recommend this grant because we think it will probably result in a lot more kids being treated for a serious, lifelong condition that nevertheless appears neglected. Clubfoot is debilitating but not life-threatening, and affects only about 1 in 800 babies born. In resource-strapped countries, a relatively new and involved treatment like the Ponseti method may not be prioritized unless an NGO like MiracleFeet is there to advocate for and assist with it. We estimate that MiracleFeet will support treatment of about 10,000 children with this grant, and that only about 10% of those children would get treated absent MiracleFeet, though we don't feel very certain about this (more below, under "What we're still learning").
All in all, after adjustments, we think that this grant will lead to about 3,700 cases of clubfoot successfully treated that otherwise wouldn't have been, and that will result in lifelong mobility gains and pain relief for the children treated.
Why this grant is different
MiracleFeet's program is different from our top charities for a few reasons:
-
The program is expensive compared with our top charities. The Ponseti method requires specialized equipment, training for medical staff, and a multi-step execution with years of subsequent follow-up. In the countries funded by this grant, we estimate that MiracleFeet's program costs a little over $500 per treatment, compared with (for example) about $1 per child given vitamin A supplements through Helen Keller International.
-
But it's also much more targeted. Our current top charities increase coverage of blanket preventive measures—ones given to all children in the target age group, whether or not they would have eventually become sick. By contrast, all the children in MiracleFeet's program have clubfoot and need an intervention to gain full mobility. The higher cost is offset by closer targeting of the kids who will benefit.
-
The main benefit is reduced disability, not lives saved or income gained. While our top charities' work primarily saves lives, MiracleFeet's program focuses on reducing or avoiding disability—thus reducing pain and suffering, and indirectly increasing income (since improved mobility means more opportunity to work). In our cost-effectiveness analysis for this grant, we used our moral weights to compare the value of saving lives to that of avoiding disability and raising incomes.
To do that, we also needed to answer the question of how bad clubfoot is. We have a generic value for a year of avoiding disability, but we needed to multiply it by another value that would represent the burden of clubfoot specifically.
The source we typically rely on for measures of disease burden doesn't provide a disability weight for clubfoot, so we had to find a proxy among the conditions it does list. We considered a couple of options that represented higher and lower degrees of severity, and spoke with a disability expert and reviewed related literature to get a better sense of clubfoot's effects. In the end, we settled on a value that seemed to match clubfoot's combination of pain, impairment, and stigma. This was ultimately a judgment call based on what we had learned about clubfoot over time.
What we're still learning
Our preliminary cost-effectiveness estimate put MiracleFeet's program close to meeting our bar for funding. Still, we had a lot of uncertainty about the ultimate impact of this grant, including the counterfactual impact of the program—that is, how many more kids would be treated for clubfoot because of MiracleFeet's work in these countries? Because MiracleFeet provides assistance for treatment, but not the treatment itself, it's tougher to draw a straight line from its work to impact. We were also unsure about existing access to treatment in the three countries where MiracleFeet would be working; we had been told that treatment is scarce in the Philippines, and essentially nonexistent in Chad and Côte d'Ivoire, but we didn't have independent confirmation of that.
We saw a grant to MiracleFeet as a learning opportunity, in addition to the direct benefit it would have on children's lives. MiracleFeet and its partners already collect some data on the program, but if we could fund additional information-gathering on the number of children treated before program launch (or expansion, in the case of the Philippines), then measure the same numbers a few years later, we'd have a clearer idea of what a program like this accomplishes with philanthropic support. We could use that information to make better decisions not only about whether to fund MiracleFeet again in the future, but also about whether to fund other clubfoot treatment programs.
When we factored in what we'd get out of a separate grant for monitoring and evaluation—what we call "value of information"—the whole opportunity looked cost-effective. In May 2023, we recommended a separate grant of about $600,000 for an external evaluator in each of the three countries to conduct those before-and-after surveys of clubfoot treatment levels, which we hope will give us the extra data we're looking for.
It's possible, of course, that these monitoring activities will tell us that the program doesn't have much of an effect—meaning we might look back several years from now and decide that we wouldn't have made the initial grant, given what we learned. It's a risk inherent in all the work GiveWell does, and, we think, a greater risk for newer-to-us programs like this one. But the upside of a positive finding is considerable! Even learning that the program isn't very effective would be valuable, since it would allow us to more confidently steer our funding toward more cost-effective uses.
There's a lot we don't know about clubfoot treatment, and a lot we hope to learn in the coming years. We're grateful for the more flexible funding we get from our donors so we can continue to invest in our learning and expand our funding's reach beyond top charities.
Notes
Nice one, this instinctively makes good sense to me and I'm super excited to see how this progresses
I'm especially interested by the follow up evaluation. I now do have doubts about the claims of "technical assistance" programs - theoretically they can have a great path to impact, but unfortnuately the few I have seen first hand (n=very small and they were less specific and focusd than Miraclefoot) aren't nearly as effective as they claim to be.
I LOVE this paragraph "Because MiracleFeet provides assistance for treatment, but not the treatment itself, it's tougher to draw a straight line from its work to impact. We were also unsure about existing access to treatment in the three countries where MiracleFeet would be working; we had been told that treatment is scarce in the Philippines, and essentially nonexistent in Chad and Côte d'Ivoire, but we didn't have independent confirmation of that."
Great thing is that you will find out with our evaluation! Its great hat you are doing the before and after studies, and I agree it will make financial sense given the amount that you will learn. I suppose in the Phillipines if only 10% of the kids would have been treated without Miraclefeet, you will assess whether you see around a 900% increase in treatments in that region where miraclefeet works? Chad and Côte d'Ivoire will be especially good case studies due to the apparent complete lack of treatment there, as you wlil be able to attribute almost every successful treatment to Miraclefeet. I'm a little surprised you couldn't confirm treatment was completely non-existent. I would have thought Just calling a few of the biggest hospitals there through a few of your global health contacts would do the trick. I suppose if you are doing a big before and after though, you will answer that in due corse anyway....
I'm not sure how you plan to do external monitoring, but I hope you are doing it yourself rather than hiring external M&E orgs. Here in Uganda anyway those orgs are heavily incentivised to give positive reports, and there is often collusion/corruption between the org itself and the evaluators. The incentives are all wrong, as both the M&E provider and the org they evaluate both benefit hugely from positive rather than negative evaluations - the org through future funding and the M&E provider throughfuture work.
Because of this I've seen a couple of hopeless programs here get bizzarely good evaluation reports. Even something that might seem super simple like following up on clubfoot treatment success percent or counting successful treatments could easily get forged. Here In Uganda I'd almost go as far to say you'd struggle to get an objective and accurate report through a local M&E provider - although many will disagree with me here.
I think there might be a number of surgeries and procedures I think that could potentially be extremely cost-effective to fund - perhaps not at this kind of massive scale though. You seem to suggest that 1 in 800 isn't super common, but in terms of major congenital or genetic disabilities that are curable, that's about as common as it gets. I would compare it with Down Syndrome - that's a similar prevalence to clubfoot and imagine the impact if we could cure that!
This is super exciting - I really hope it works and am super stoked you are funding it!
[Disclaimer: I'm the Chief Economist of IDinsight, an M+E provider who has worked with GiveWell and many others. I have a LOT of experience with evaluators being pressured to sugarcoat results, or lack thereof. ]
Strong disagree on this conclusion that M+E providers are inherently biased.
Yes, there are situations where M+E have incentives that can lead to bias. For instance, if an NGO hires an M+E provider to do an external evaluation of themselves, the NGO is therefore the 'client' of the researchers. This can be problematic, since the NGO will need to approve deliverables before payments are made. I've been involved in these situations and it is tricky.
But in general, arrangements can be made to align incentives with the truth. For instance if a funder (like GiveWell) hires an M+E provider to do an evaluation of one of its grantees, the incentives of the M+E provider are aligned with the funder, who hopefully would like to know the unvarnished truth. We've done numerous evaluations for GiveWell (most notably the New Incentives RCT) and have never felt any incentive to skew results one way or another.
From an organizational perspective, a well-run evaluation organization has much stronger long-term incentives to have a reputation for being honest, transparent, and truth-seeking, rather than getting repeat business from any particular client.
Thanks so much Dan - honoured to get a reply from someone with so much experience on the topic and doing such important work. There's also a decent chance that ID insight has higher standards than many other orgs.
I agree with a decent amount of this - I agree that an NGO hiring its own M&E provider directly is "problematic", and "tricky" only that I would use stronger language ;). Personally I think its a waste of resources for an NGO to hire an external M&E provider as the incentives are so skewed, I don't think there's a lot of added value compared to just internal M&E. Yes incentives are of course all wrong there too, but at least the knowledge and understanding of the operation will be better than the external provider, and the uptake for reform by the org might be better if it is driven from the inside as well.
I also agree if a funder commissions the M&E provider that is far better . At the management level incentives of the funder and the M&E organisation are likely to be aligned. I'm sure you don't feel any incentive at your level to skew a result, but despite that both from evidence what I have seen, the positive skew is very hard to remove due to unfortunately skewed incentives at the local level.
"We've done numerous evaluations for GiveWell (most notably the New Incentives RCT) and have never felt any incentive to skew results one way or another. " - I'm sure you don't at that top level, but at the local "on the ground", assessor level it is very difficult to avoid positive skew.
Both from my personal experience and theory (see incentives below) I think it is likely there will be some degree of positive skew even among the best M&E orgs - this might not mean the orgs shouldn't exist, but we should be doing much more to mitigate the "on-the-ground" positive bias.
Unfortunately there are strong almost unidirectional incentives towards M&E providers at the local level to skew an M&E assessment positively. I can't think of clear incentives towards a negative assessment - maybe someone else can?
Incentives for external M&E Positive bias (A lot of overlap between these)
Positive assessment = more funding for NGO locally = more M&E work in future for me
Negative assessment = less funding for NGO locally = less M&E work in future for me
The equation which you hope your employees will abide by might be(correct me if I'm wrong)
Honest assessment = Correct funding for NGO = increased trust in assessment org = more work for assessment org = more work for all our staff including me
But I think that chain of reasoning is VERY difficult to compute for local people working in doing the assessing. There's also an aspect of prisoners dilemma here, where you rely on everyone assessing in your org to be on board with this long term view in order for it to work for you personally
Incentives for external M&E negative bias- I genuinely fine it hard to think of good ones
I won't get into ways of mitigating these bias now (my comment is too long already haha), but I think the natural lean (quite heavily) towards positive skew in M&E is quite high.
Would love to hear specific rebuttals to this if you have time, but all good if you don't!
Thanks again
NIck.
Interesting discussion. I agree incentives can be tricky and I have seen my fair share of bad evaluations and evaluation organisations with questionable practices. Some thoughts from me as an evaluator who has worked in a few different country contexts:
Wow thanks so much again for the great insights. So good to have experienced development practitioners here!
To give some background I came to EA partly because I saw how useless most NGOs are here where I live, and the EA framework answers many of the questions as to why, and some of the questions as to how to fix the problem. If I was the one doing M&E and had a magic wand, I would probably decide to shut down over 80% of NGOs and programs that I assessed.
Also we have had a bunch of USAID and other funded M&E pass through many of our health centers, and they have almost never either found our biggest problems nor suggested good solutions - with one exception of a specifically focused financial management assessment which was actually really helpful.
I won't respond to everything but just make a few comments :)..
Your M&E might just be better
First, the level of M&E you do might be so much better than I have seen, that some of the issues I talk about might not apply so much.
"For example if you find an education project is cost-effectively increasing test scores, but only for female students and not struggling male students what do you say?"
That you have even done the kind of analysis that allows you to ask this kind of great question would put you above nearly any M&E that I have ever seen here in Northern Uganda. Even the concept of calculated "cost effectiveness" as we know it rarely (if ever) considered here. I can't think of anyone who has assessed either the bigger health centers we operate or OneDay Health who has included this in an assessment.
I'm not sure how you would answer that question, but the fact that you have even reached that point means that in my eyes you are already winning to some degree. Also this analysis is so fantastic thanks for sharing I haven't seen that before! My only comment is that I don't think the analysis generated "mixed' results -they seem very clear to me :D!
External assessors for data collection, local assessors for analysis and change?
For an assessment like this one of Miraclefeet, I favour external assessors to gather the basic data then perhaps local assessors could take over on the analysis? Data collection needs to be squeaky clean otherwise everything else falls down. This particular assessment should be fairly straightforward to assess by first gathering these data
1. Have the clubfoot procedures actually been done as stated? This needs a random selection of a sample of all patients allegedly worked on (say 100 randomly selected from a list of 5000 patients provided by Miricalefeet) then each one of those should be physically followed up in their home and checked. This isn't difficult, and anything else is open to bias.
2. What has the "average intervention" achieved? Then those same 100 patients should be assessed for impact - what is their objective level of functionality and subjective improvement in wellbeing/quality of life after the procedure.
Once these 2 pieces of data are gathered, the organisational analysis and discussion you speak of can start and that might be more productive on a local-to-local level, providing the local expertise is available.
Integrity there but comes second?
I know integrity is an important driver like you say, and I love your comment that it is easy to compute and hard to action. In my experience integrity is usually there, but often falls behind the other "positive skew" motivating factors. Also I agree that M&E shouldn't always be done by external firms partly because of the reasons you state. An added reason is that external firms often hire lots of local people to do much of the work anyway, so the same issues I outlined remain.
A small disagreement?
"I have seen orgs who have been unhappy with an external evaluation because they feel the evaluators made judgements when they didn’t fully understand the operating context (and how can they with often only a fly by visit to project locations) or did not properly take into account the values of the organisation or the community but rather only listened to the funder."
In my experience this response might be a red flag. A sign that the org might dodging and weaving after failing to perform. I believe almost all organisation should do do pre specified actions A,B and C which provides impact X, Y, and Z. If these actions aren't happening and impact isn't produced then that needs to be fixed or maybe the work needs to stop. External evaluators' job isn't to understand the context (how could they possibly do that? Its not realistic. I've been in Uganda for 10 years and in many ways I still don't understand the local context) - that is our job, the practitioners. Their job is to see what the org is doing and whether the impact is happening.
As a side note I'm a little disappointed that we don't have more engagement on this discussion. - the "M&E question" is so important, but perhaps its not sexy and probably isn't accessible to many.
Hi Nick, thanks for the thoughtful response. I think you make a lot of good points and I agree that there are numerous incentives can can lead an M+E provider to bias results positively. That's why there is a ton of bad M+E out there.
One main reaction: for an employee who works in an M+E org, there is arguably no worse situation than being pressured to skew your results positively, or even worse, taking on projects where you know a certain results is expected by your clients. It makes you feel you work is meaningless, and really sucks. And when you are put in this situations, you sure as hell don't want to work for the same client again.
Yes, i hear you that for bean-counters in an organization (or those who get dividends in a for-profit org), there are strong incentives to make clients happy and get more contracts. But I think that the job-satisfaction incentive for rank-and-file employees skews the other way. And in the course of my experience, I think it is this latter incentive toward truth-telling that has dominated in most cases.
Perhaps, like the rules for auditors established after accounting scandals, funders should adopt a policy requiring changes in the M&E provider at certain intervals, maybe with some random selection of interval? Knowing that next year's assessment may be done by a different firm may create a disincentive for gaming the system (and a pathway for easier detection of any gaming). That may only work for projects with longer-term M&E efforts though.
Hi, Nick,
Thanks for your comments, your insight into this grant, and your support!
We do expect to get input from local hospital staff on existing treatment coverage through the baseline surveys. The monitoring grant will fund the creation of a sampling frame that includes both public and private health facilities, which we think will yield more complete data than contacting hospitals through our partners.
We agree that potential bias from external evaluators is a risk for the reasons you've mentioned. While we won't be involved in the selection of evaluators, we plan to do the following to mitigate that risk:
We don't think this will completely eliminate uncertainty about the quality of monitoring results, but we expect it will help. We also think there is some value to be gained from working with evaluators who have a strong familiarity with the local context.
I hope that's helpful!
Best,
Miranda
Thanks for the reply, and most of this makes sense to me.
I'm not sure I understand how you won't be involved in the selection of evaluators, who will do that exactly? Or maybe you mean you won't select the on-the-ground evaluators as in that will be done by the company, which makes sense.
"The monitoring grant will fund the creation of a sampling frame that includes both public and private health facilities, which we think will yield more complete data than contacting hospitals through our partners." This could work (high risk), but seems like a roundabout and inefficient way to do things. Following up on that data from multiple hospitals in West Africa for example could be a nightmare.
I would have thought with this kind of massive funding and the relatively small number of people who get procedures (in the thousands), MiracleFeet could maintain a database the contact details of every kid who gets help - this wouldn't be hard and would make M&E so much easier for everyone. Hospitals might well collect substandard information which makes proper follow up impossible, spoiling your M&E efforts.
If I was going to give one piece of advice on M&E, it would be that your evaluators should follow up personally a completely random sample of individuals who had been treated - both to check that both the interventions actually happened, and that the claimed improvement is real. There should be a list of names, home locations and phone numbers of every single patient who received treatment - I think if that's not there and individuals can't be followed for this kind of intervention, than meaningful M&E becomes close to impossible.
Hi, Nick,
Yes, to clarify, MiracleFeet is selecting the on-the-ground evaluator that will conduct the monitoring in each location, although GiveWell will lightly vet MiracleFeet's choices and meet with each evaluator.
The primary purpose of the monitoring grant is to understand how many children are treated for clubfoot both with and without MiracleFeet's support. So, although MiracleFeet has records of children treated through facilities it's supported, we also want an assessment of baseline treatment coverage before MiracleFeet launches its program (or expands it, in the case of the Philippines). We do plan to incorporate some form of data audit as part of endline activities; we'll work out the details of that at a later date.
Thanks again for your interest in this and for taking the time to ask questions!
Miranda