I want to acknowledge my potential biases for any new comment thread readers (I used to be the senior researcher running the fund at FP, most or all of the errors highlighted in the report are mine, and I now work at GiveWell.) these are personal views.
I think getting further scrutiny and engagement into key grantmaking cruxes is really valuable. I also think this discussion this has prompted is cool. A few points from my perspective-
As Matt’s comment points out- there is a historical track record for many of these grants. Some have gone on the be GiveWell supported, or (imo) have otherwise demonstrated success in a way that suggests they were a ‘hit’. In fact, with the caveat that there are a good number of recent ones where it’s too early to tell, there hasn’t yet been one that I consider a ‘miss’. Is it correct to update primarily from 3 spot checks of early stage BOTECs (my read of this report) versus updating from what actually happened after the grant was made? Is this risking goodharting?
Is this really comparing like for like? In my view, small grants shouldn’t require as strong an evidence base as like, a multimillion grant, mainly due to the time expenditure reasons that Matt points out. I am concerned about whether this report is getting us further to a point where (due to the level of rigour and therefore time expenditure required) the incentives for grantmaking orgs are to only make really large grants. I think this systematically disadvantages smaller orgs, and I think this is a negative thing (I guess your view here partially depends on your view on point ‘3’ below.)
In my view, a really crucial crux here is really about the value of supporting early stage stuff, alongside other potentially riskier items, such as advocacy and giving multipliers. I am genuinely uncertain, and think that smart and reasonable people can disagree here. But I agree with Matt’s point- that there’s significant upside through potentially generating large future room for funding at high cost effectiveness. This kind of long term optionality benefit isn’t typically included in an early stage BOTEC (because doing a full VOI is time consuming) and I think it’s somewhat underweighted in this report.
I no longer have access to the BOTECs to check (since I’m no longer at FP) and again I think the focus on BOTECs is a bit misplaced. I do want to briefly acknowledge though that I’m not sure that all of these are actually errors (but I still think it’s true that there are likely some BOTEC errors, and I think this would be true for many/ most orgs making small grants).
Hey Vasco, these are my personal thoughts and not FP’s (I have now left FP, and anything FP says should take precedence). I have pretty limited capacity to respond, but a few quick notes—
First, I think it’s totally true that there are some BOTEC errors, many/ most of them mine (thank you GWWC for spotting them— it’s so crucial to a well-functioning ecosystem, and more selfishly, to improving my skills as a grantmaker. I really value this!)
At the same time—these are hugely rough BOTECs, that were never meant to be rigorous CEA’s: they were being used as decision-making tools to enable quick decision-making under limited capacity (i do not take the exact numbers seriously: i expect they're wrong in both directions), with many factors beyond the BOTEC going into grantmaking decisions.
I don’t want to make judgments about whether or not the fund (while I was there) was surpassing GiveWell or not— super happy to leave this to others. I was focused on funders who would not counterfactually give to GW, meaning that this was less decision-relevant for me.
I think it's helpful to look at the grant history from FP GHDF. Here’s all the grants that I think have been made by FP GHDF since Jan 2023, apologies if i’ve missed any:
* New Incentives, Sightsavers, Pure Earth, Evidence Action
* r.i.c.e, FEM, Suvita (roughly- currently scaling orgs, that were considerably smaller/ more early stage when we originally granted)
* 1 was an advocacy grant to 1DaySooner (malaria vax roll out stuff), 1 was an advocacy grant to LEEP
* 5 are recent young orgs, that we think are promising but ofc supporting young orgs is hit and miss (Ansh, Taimaka, Essential, IMPALA, HealthLearn)
* 1 was deworming research grant
* 1 was an anti-corruption journalism grant which we think is promising due to economic growth impacts (OCCRP)
I think it's plausible that i spent too little time on these grant evals, and this probably contributed to BOTEC errors. But I feel pretty good about the actual decision-making, although I am very biased:
* At least 2 or 3 of these struck me as being really time-sensitive (all grants are time-sensitive in a way, but I'm talking about ‘might have to shut down some or all operations’ or ‘there is a time-sensitive scaling opportunity’). * I think there is a bit of a gap for early-ish funding, and benefits to opening up more room for funding by scaling these orgs (i.e. funding orgs beyond seed funding, but before orgs can absorb or have the track record for multi million grants). Its still early days, but feel pretty good about the trajectories of the young orgs that FP GHDF supported. * Having a few high EV, ‘big if true’ grants feels reasonable to me (advocacy, economic growth, R and D).
I hope this context is useful, and note that I can’t speak to FP’s current/ future plans with the FP GHDF. I value the engagement- thanks.
Thank you for posting this, really interesting! I am pretty excited for malaria vaccine roll-out, but think that posts which go against current thinking are valuable, and I appreciate this being posted.
Three things that I'd be super curious about (and i think might have the effect of being more optimistic);
(1) IIRC the WHO found a 13% reduction in all-cause mortality (!) with a vaccination coverage rate of ~65%. This was in Ghana/ Kenya and Malawi, I think. I assume these areas already had some availability of SMC/ bednets, and this makes me think that there's likely quite a substantial impact beyond nets/ SMC?
(2) Beyond immediate impacts of the vaccine- will the malaria vaccine likely bring eradication forward beyond the impact of nets/ SMC? (my current understanding is yes, and I think that bringing eradication forward is super important.)
(3) As you already point out, where would money have otherwise been spent. My impression here is that the money which funds malaria vax roll-out would likely come from a mix of sources, rather than nescessarily pulling from SMC/ bednets (but I still have some concern/ uncertainty here).
correct: to be absolutely clear, CE and the orgs themselves definitely incubated and developed the first CEAs for the CE charities (not us!). Thanks for this, will edit my comment. What I meant was is that evaluating and supporting orgs in the 'no longer a seed stage charity but also not yet at scale' stage is a key role that I see for the FP GHD fund, and I think we've had previous success here.
Yep, 1DS' pandemic preparedness has been supported by OP. And thanks, I will mention to our comms team. FYI that our impact report is upcoming onto the website, which will list all recent grantmaking.
(also cheers for this post, and making the subtleties & differences between different GHD funds etc clearer!)
RE: most of the funding going to groups which are already in the EA ecosystem- yep, I think this is correct. I still think this is compatible with the statement of 'being especially interested in stuff falling outside the main GHD ecosystem' though, and that the grantmaking focus has changed somewhat in the last year or two.
* r.i.c.e we granted to an early stage (before they were GiveWell supported) when they were having a severe funding crunch. * FEM, Suvita and LEEP we evaluated and granted to at an early stage. I think there's value here in 'identifying promising charities and getting up to scale asap'. * Our most recent grants aren't up on the website yet: I don't think (? could be wrong) 1daysooner's malaria stuff has gotten EA funding yet, Ubongo isn't EA supported afaik, or Essential.
It's true though that a lot of FP GHD grantmaking is stuff that's already supported by the EA community. I could have been clearer here; our guiding principle is just 'whatever we think is most cost-effective' and many of the existing EA-supported orgs still have funding gaps that are highly cost-effective. Aka I wouldn't want to prioritise something that sounds cool on a 'this is new basis' over something that is already known in EA, and is more cost-effective. Most of the decisions that I really agonise over are 'does this really beat bednets? Most things don't beat bednets, and I'm aware that most things seem less cost-effective as more researcher hours go into probing them'. I do think it is possible though to to find funding opportunities in the 'falling outside the current GHD ecosystem and >10X in expectation' space though, and these have been some of my favourite grants.
RE: the scope of Founders Pledge's global health and development fund being different in the past. This was before my time (I joined FP maybe 1.5 years ago) but yes, I would say that our scope has changed a little to be somewhat more risk tolerant (while still focusing on evidence base) and figuring out where our comparative advantages are (speed being one of them).
RE: being more open with evaluations; yeah I totally agree with you, and think you're right that explaining when we depart from GiveWell is esp important. (Some places where we've just started to do this: Vadim's education work, my mass media stuff. But I haven't published actual FO evals as often). IMO our evals are rigorous, without being as in-depth as GiveWell; there's a process of understanding the area (some of our cause area reports are on the EA forum), then a ~25 page ish eval & CEA, and red-teaming. Our rapid grants are shorter, but still have an eval, CEA/ BOTEC + red-teaming.
I am obviously biased (I co-manage the fund), so please bear that in mind. But I'd like to throw in Founders Pledge GHD fund as another GHD option- it's not the case that you have to be an FP member to donate to this fund.
We tend to fund GiveWell-style recs (e.g. AMF) alongside things that we think are higher risk but potentially more impactful/ younger orgs that we want to get up to scale quickly. Recent grants have included: 1DaySooner to try and speed up malaria vaccine roll-out, Essential for research into producing low-cost proteins in East Africa, Ubongo, Suvita, LEEP, r.i.c.e for kangeroo mothercare, Family Empowerment Media, Taimaka. Some stuff that we're currently checking out include Pure Earth, other lead research-y things, and some anti-corruption stuff via my colleague Vadim.
I'm especially interested in things which could be very good, but might fall outside the bounds of the current GHD funding ecosystem (e.g. things that are not super amenable to RCTs, orgs that are too young to get mainstream funders but too old for seed funding, urgent things that require a very quick turnaround).
Full disclosure that we haven't been the best at publicly publishing evaluations- due to capacity, but still. I hope to improve this/ generally provide more resources about how we do grantmaking from the fund over the next year or so.
Hullo, this report is fascinating! Upvoted. Just a quick note that Founders Pledge estimate of DMI's child survival program is out of date (which totally makes sense, as we haven't published our full eval of DMI's child survival program).
We currently rate this program at 4x GD, not 12x GD- this is after applying a really strong adjustment (16%; internal valdity adjustment of 30%, external of 54%), to try and deal with the uncertainty from the Sarrassat et al. RCT. thanks!
Hello, I've been researching modern slavery at Founders Pledge recently. (Also oops, pressed send before I meant to- hence the edit!)
I really respect the Freedom Fund for pushing forward with evaluative work, esp in an area where there aren't many published evaluations (you can see one of their evaluations here). At the same time, I think the $657 figure is very optimistic. My understanding is that the figure GWWC are pulling from (the $657 amount) is based on weaker, observational evidence—and doesn't really account for the marginal funding amount. By this, I mean that I expect that most high-quality programs would be funded within the next few years (so this is really the cost to speed this process up).
My general sense is that this is an area where the evidence basis for direct work is suprisingly weak; there are hardly any RCTs, or strong quasi-experiments. It's true that modern slavery interventions are difficult to test, but nonetheless this is pretty striking given that modern slavery is an area that's of increasing interest to many policy makers (e.g. movement towards an EU ban on products made with forced labor, SDG goal 8.7).
I suspect the lack of RCT-style evidence, and the sense that it's not super neglected, might be why there hasn't been much EA focus here. Nonetheless, I think focusing on interventions around advocacy, tech and research may be promising here (and I'd probably argue for prioritising this over direct work, given current funding patterns).
Advocacy in this area can push for things like strengthening existing Modern Slavery Acts (e.g. by adding sanctions, existing legislature is very weak) and bringing in 'floor wages' for workers (e.g. see this video from the Asia Floor Wage Alliance). Tech developments to actually be able to locate modern slavery seem critical to me, and like a key bottleneck in this area: to NGOs (who need to be able to prioritise their efforts, measure the impact of their work and so on), to multi-national companies (who need to be able to locate modern slavery within their supply chains, in order to choose contractors who do not use modern slavery), to consumers (to know which products are at risk of modern slavery and apply consumer pressure), and to strengthening legislation (it seems difficult to impose sanctions on multi-national companies, if companies can genuinely argue that they don't have the ability to locate modern slavery in their supply chains). I'm also interested in tech work that helps workers stay safe as they migrate (e.g. apps that provide accurate job info, tell people their work rights, etc etc). So I'd probably lean to charities with a strong focus on this; afraid I don't have a one-stop recommendation yet as work is still in progress, but you might want to check out the Asia Floor Wage Alliance, Global Fund to End modern Slavery, Anti-Slavery International and International Justice Mission as well as the Freedom Fund (for groups that also have an advocacy/ tech focus).
Thanks Stan! This is really helpful- agreed with you that they should be combined as multipliers rather than added together (I've now edited accordingly). I'm still mulling a bit over whether using the word 'discount' or 'adjustment' or something else might help improve clarity.
Hey Aidan,
I want to acknowledge my potential biases for any new comment thread readers (I used to be the senior researcher running the fund at FP, most or all of the errors highlighted in the report are mine, and I now work at GiveWell.) these are personal views.
I think getting further scrutiny and engagement into key grantmaking cruxes is really valuable. I also think this discussion this has prompted is cool. A few points from my perspective-
As Matt’s comment points out- there is a historical track record for many of these grants. Some have gone on the be GiveWell supported, or (imo) have otherwise demonstrated success in a way that suggests they were a ‘hit’. In fact, with the caveat that there are a good number of recent ones where it’s too early to tell, there hasn’t yet been one that I consider a ‘miss’. Is it correct to update primarily from 3 spot checks of early stage BOTECs (my read of this report) versus updating from what actually happened after the grant was made? Is this risking goodharting?
Is this really comparing like for like? In my view, small grants shouldn’t require as strong an evidence base as like, a multimillion grant, mainly due to the time expenditure reasons that Matt points out. I am concerned about whether this report is getting us further to a point where (due to the level of rigour and therefore time expenditure required) the incentives for grantmaking orgs are to only make really large grants. I think this systematically disadvantages smaller orgs, and I think this is a negative thing (I guess your view here partially depends on your view on point ‘3’ below.)
In my view, a really crucial crux here is really about the value of supporting early stage stuff, alongside other potentially riskier items, such as advocacy and giving multipliers. I am genuinely uncertain, and think that smart and reasonable people can disagree here. But I agree with Matt’s point- that there’s significant upside through potentially generating large future room for funding at high cost effectiveness. This kind of long term optionality benefit isn’t typically included in an early stage BOTEC (because doing a full VOI is time consuming) and I think it’s somewhat underweighted in this report.
I no longer have access to the BOTECs to check (since I’m no longer at FP) and again I think the focus on BOTECs is a bit misplaced. I do want to briefly acknowledge though that I’m not sure that all of these are actually errors (but I still think it’s true that there are likely some BOTEC errors, and I think this would be true for many/ most orgs making small grants).
Hey Vasco, these are my personal thoughts and not FP’s (I have now left FP, and anything FP says should take precedence). I have pretty limited capacity to respond, but a few quick notes—
First, I think it’s totally true that there are some BOTEC errors, many/ most of them mine (thank you GWWC for spotting them— it’s so crucial to a well-functioning ecosystem, and more selfishly, to improving my skills as a grantmaker. I really value this!)
At the same time—these are hugely rough BOTECs, that were never meant to be rigorous CEA’s: they were being used as decision-making tools to enable quick decision-making under limited capacity (i do not take the exact numbers seriously: i expect they're wrong in both directions), with many factors beyond the BOTEC going into grantmaking decisions.
I don’t want to make judgments about whether or not the fund (while I was there) was surpassing GiveWell or not— super happy to leave this to others. I was focused on funders who would not counterfactually give to GW, meaning that this was less decision-relevant for me.
I think it's helpful to look at the grant history from FP GHDF. Here’s all the grants that I think have been made by FP GHDF since Jan 2023, apologies if i’ve missed any:
* New Incentives, Sightsavers, Pure Earth, Evidence Action
* r.i.c.e, FEM, Suvita (roughly- currently scaling orgs, that were considerably smaller/ more early stage when we originally granted)
* 4 are ecosystem multiplier-y grants (Giving Multiplier, TLYCS, Effective Altruism Australia, Effektiv Spenden)
* 1 was an advocacy grant to 1DaySooner (malaria vax roll out stuff), 1 was an advocacy grant to LEEP
* 5 are recent young orgs, that we think are promising but ofc supporting young orgs is hit and miss (Ansh, Taimaka, Essential, IMPALA, HealthLearn)
* 1 was deworming research grant
* 1 was an anti-corruption journalism grant which we think is promising due to economic growth impacts (OCCRP)
I think it's plausible that i spent too little time on these grant evals, and this probably contributed to BOTEC errors. But I feel pretty good about the actual decision-making, although I am very biased:
* At least 2 or 3 of these struck me as being really time-sensitive (all grants are time-sensitive in a way, but I'm talking about ‘might have to shut down some or all operations’ or ‘there is a time-sensitive scaling opportunity’).
* I think there is a bit of a gap for early-ish funding, and benefits to opening up more room for funding by scaling these orgs (i.e. funding orgs beyond seed funding, but before orgs can absorb or have the track record for multi million grants). Its still early days, but feel pretty good about the trajectories of the young orgs that FP GHDF supported.
* Having a few high EV, ‘big if true’ grants feels reasonable to me (advocacy, economic growth, R and D).
I hope this context is useful, and note that I can’t speak to FP’s current/ future plans with the FP GHDF. I value the engagement- thanks.
Thank you for posting this, really interesting! I am pretty excited for malaria vaccine roll-out, but think that posts which go against current thinking are valuable, and I appreciate this being posted.
Three things that I'd be super curious about (and i think might have the effect of being more optimistic);
(1) IIRC the WHO found a 13% reduction in all-cause mortality (!) with a vaccination coverage rate of ~65%. This was in Ghana/ Kenya and Malawi, I think. I assume these areas already had some availability of SMC/ bednets, and this makes me think that there's likely quite a substantial impact beyond nets/ SMC?
(2) Beyond immediate impacts of the vaccine- will the malaria vaccine likely bring eradication forward beyond the impact of nets/ SMC? (my current understanding is yes, and I think that bringing eradication forward is super important.)
(3) As you already point out, where would money have otherwise been spent. My impression here is that the money which funds malaria vax roll-out would likely come from a mix of sources, rather than nescessarily pulling from SMC/ bednets (but I still have some concern/ uncertainty here).
correct: to be absolutely clear, CE and the orgs themselves definitely incubated and developed the first CEAs for the CE charities (not us!). Thanks for this, will edit my comment. What I meant was is that evaluating and supporting orgs in the 'no longer a seed stage charity but also not yet at scale' stage is a key role that I see for the FP GHD fund, and I think we've had previous success here.
Yep, 1DS' pandemic preparedness has been supported by OP. And thanks, I will mention to our comms team. FYI that our impact report is upcoming onto the website, which will list all recent grantmaking.
(also cheers for this post, and making the subtleties & differences between different GHD funds etc clearer!)
Hey Vasco,
RE: most of the funding going to groups which are already in the EA ecosystem- yep, I think this is correct. I still think this is compatible with the statement of 'being especially interested in stuff falling outside the main GHD ecosystem' though, and that the grantmaking focus has changed somewhat in the last year or two.
* r.i.c.e we granted to an early stage (before they were GiveWell supported) when they were having a severe funding crunch.
* FEM, Suvita and LEEP we evaluated and granted to at an early stage. I think there's value here in 'identifying promising charities and getting up to scale asap'.
* Our most recent grants aren't up on the website yet: I don't think (? could be wrong) 1daysooner's malaria stuff has gotten EA funding yet, Ubongo isn't EA supported afaik, or Essential.
It's true though that a lot of FP GHD grantmaking is stuff that's already supported by the EA community. I could have been clearer here; our guiding principle is just 'whatever we think is most cost-effective' and many of the existing EA-supported orgs still have funding gaps that are highly cost-effective. Aka I wouldn't want to prioritise something that sounds cool on a 'this is new basis' over something that is already known in EA, and is more cost-effective. Most of the decisions that I really agonise over are 'does this really beat bednets? Most things don't beat bednets, and I'm aware that most things seem less cost-effective as more researcher hours go into probing them'. I do think it is possible though to to find funding opportunities in the 'falling outside the current GHD ecosystem and >10X in expectation' space though, and these have been some of my favourite grants.
RE: the scope of Founders Pledge's global health and development fund being different in the past. This was before my time (I joined FP maybe 1.5 years ago) but yes, I would say that our scope has changed a little to be somewhat more risk tolerant (while still focusing on evidence base) and figuring out where our comparative advantages are (speed being one of them).
RE: being more open with evaluations; yeah I totally agree with you, and think you're right that explaining when we depart from GiveWell is esp important. (Some places where we've just started to do this: Vadim's education work, my mass media stuff. But I haven't published actual FO evals as often). IMO our evals are rigorous, without being as in-depth as GiveWell; there's a process of understanding the area (some of our cause area reports are on the EA forum), then a ~25 page ish eval & CEA, and red-teaming. Our rapid grants are shorter, but still have an eval, CEA/ BOTEC + red-teaming.
I am obviously biased (I co-manage the fund), so please bear that in mind. But I'd like to throw in Founders Pledge GHD fund as another GHD option- it's not the case that you have to be an FP member to donate to this fund.
We tend to fund GiveWell-style recs (e.g. AMF) alongside things that we think are higher risk but potentially more impactful/ younger orgs that we want to get up to scale quickly. Recent grants have included: 1DaySooner to try and speed up malaria vaccine roll-out, Essential for research into producing low-cost proteins in East Africa, Ubongo, Suvita, LEEP, r.i.c.e for kangeroo mothercare, Family Empowerment Media, Taimaka. Some stuff that we're currently checking out include Pure Earth, other lead research-y things, and some anti-corruption stuff via my colleague Vadim.
I'm especially interested in things which could be very good, but might fall outside the bounds of the current GHD funding ecosystem (e.g. things that are not super amenable to RCTs, orgs that are too young to get mainstream funders but too old for seed funding, urgent things that require a very quick turnaround).
Full disclosure that we haven't been the best at publicly publishing evaluations- due to capacity, but still. I hope to improve this/ generally provide more resources about how we do grantmaking from the fund over the next year or so.
Hullo, this report is fascinating! Upvoted. Just a quick note that Founders Pledge estimate of DMI's child survival program is out of date (which totally makes sense, as we haven't published our full eval of DMI's child survival program).
We currently rate this program at 4x GD, not 12x GD- this is after applying a really strong adjustment (16%; internal valdity adjustment of 30%, external of 54%), to try and deal with the uncertainty from the Sarrassat et al. RCT. thanks!
Hello, I've been researching modern slavery at Founders Pledge recently. (Also oops, pressed send before I meant to- hence the edit!)
I really respect the Freedom Fund for pushing forward with evaluative work, esp in an area where there aren't many published evaluations (you can see one of their evaluations here). At the same time, I think the $657 figure is very optimistic. My understanding is that the figure GWWC are pulling from (the $657 amount) is based on weaker, observational evidence—and doesn't really account for the marginal funding amount. By this, I mean that I expect that most high-quality programs would be funded within the next few years (so this is really the cost to speed this process up).
My general sense is that this is an area where the evidence basis for direct work is suprisingly weak; there are hardly any RCTs, or strong quasi-experiments. It's true that modern slavery interventions are difficult to test, but nonetheless this is pretty striking given that modern slavery is an area that's of increasing interest to many policy makers (e.g. movement towards an EU ban on products made with forced labor, SDG goal 8.7).
I suspect the lack of RCT-style evidence, and the sense that it's not super neglected, might be why there hasn't been much EA focus here. Nonetheless, I think focusing on interventions around advocacy, tech and research may be promising here (and I'd probably argue for prioritising this over direct work, given current funding patterns).
Advocacy in this area can push for things like strengthening existing Modern Slavery Acts (e.g. by adding sanctions, existing legislature is very weak) and bringing in 'floor wages' for workers (e.g. see this video from the Asia Floor Wage Alliance). Tech developments to actually be able to locate modern slavery seem critical to me, and like a key bottleneck in this area: to NGOs (who need to be able to prioritise their efforts, measure the impact of their work and so on), to multi-national companies (who need to be able to locate modern slavery within their supply chains, in order to choose contractors who do not use modern slavery), to consumers (to know which products are at risk of modern slavery and apply consumer pressure), and to strengthening legislation (it seems difficult to impose sanctions on multi-national companies, if companies can genuinely argue that they don't have the ability to locate modern slavery in their supply chains). I'm also interested in tech work that helps workers stay safe as they migrate (e.g. apps that provide accurate job info, tell people their work rights, etc etc). So I'd probably lean to charities with a strong focus on this; afraid I don't have a one-stop recommendation yet as work is still in progress, but you might want to check out the Asia Floor Wage Alliance, Global Fund to End modern Slavery, Anti-Slavery International and International Justice Mission as well as the Freedom Fund (for groups that also have an advocacy/ tech focus).
Thanks Stan! This is really helpful- agreed with you that they should be combined as multipliers rather than added together (I've now edited accordingly). I'm still mulling a bit over whether using the word 'discount' or 'adjustment' or something else might help improve clarity.
Thank you Joel! Hope that you had a nice flight.