EA vs. FIRE – reconciling these two movements

Stewed_Walrus

Comment Permalink

jhJul 18 202310

I wouldn't put the key point here down to 'units'. I would say the aggregate units of GiveWell tends to use ('units of value' and lives saved) and of GIF (person-year of income-equivalent, "PYI") are very similar. I think any differences in terms of these units is going to be more about subjective differences in 'moral weights'. Other than moral weight differences, I'd expect the same analysis using GiveWell vs GIF units to deliver essentially the same results.

The point you're bringing up, and that Ken discusses as 'Apples vs oranges', is that the analysis is different. GiveWell's Top Charity models are about the outputs your dollar can buy now and the impacts those outputs will generate (e.g., over a few years for nets, and a few more for cash). As part of its models, GIF projects outputs (and impacts) out to 10 years. Indeed this is necessary when looking at early-stage ventures as most of their impact will be in the future and these organizations do not have to be cost-effective in their first year to be worth funding. If you were considering funding LLINs in 2004, or even earlier, you most likely would want to be doing a projection and not just considering the short-term impacts.

Of course, as has been repeatedly discuss in these comments, when projected impact is part of the model how much contribution to assign to the original donors becomes a big issue. But I believe that it is possible to agree on a consistent method for doing this. And once you have that, this really becomes more of an 'Apples vs apples' comparison.

For example, you might decide that a dollar to a malaria net charity now buys an amount of nets right now, but has limited impact on the future growth of that charity. So the current GiveWell estimates don't need to be modified even if you're including impact projections.

My current understanding of the state of play around projected / forecast impact is:

GiveWell doesn't have public methodologies or models for this.
GIF has a public methodology but no public worked examples. Even an artificial example would help make our discussion more precise.
Other actors have methodologies and examples (not necessarily public). All with their own pros and cons, especially around how they handle contribution. One of the most mature may be PRIME Coalition / Project FRAME for climate impact.

See in context

Global Innovation Fund projects its impact to be 3x GiveWell Top Charities

by jh

Jun 1 20231 min read 12

71

ForecastingGlobal health & developmentImpact investingCost-effectiveness analysisGiveDirectlyGiveWellGrantmaking

Frontpage

The Global Innovation Fund (GIF) is a non-profit, impact-first investment fund headquartered in London that primarily works with mission-aligned development agencies (USAID, SIDA, Global Affairs Canada, UKAID). Through grants, loans and equity investments, they back innovations with the potential for social impact at a large scale, whether these are new technologies, business models, policy practices or behavioural insights. I've been told that so far their investments have been roughly 60% grants and 40% 'risk capital' (i.e., loans and equity).

Recently, they made a bold but little publicized projection in their 2022 Impact Report (page 18): "We project every dollar that GIF has invested to date will be three times as impactful as if that dollar had been spent on long-lasting, insecticide-treated bednets... This is three times higher than the impact per dollar of Givewell’s top-rated charities, including distribution of anti-malarial insecticide-treated bednets. By Givewell’s estimation, their top charities are 10 times as cost-effective as cash transfers." The report says they have invested $112m since 2015.

This is a short post to highlight GIF's projection to the EA community and to invite comments and reactions.

Here are a few initial points:

It's exciting to see an organization with relatively traditional funders comparing its impact to GiveWell's top charities (as well as cash transfers).
I would want to see more information on how they did their calculations before taking a view on their projection.
In any case, based on my conversations with GIF, and what I've understood about their methodology, I think their projection should be taken seriously. I can see many ways it could be either an overestimate or an underestimate.

71 Reactions

Comments12

Sorted by

New & upvoted

Click to highlight new comments since: Today at 5:24 AM

Ken ChomitzJul 17 202323

Let me introduce myself – I’m the Chief Economist of the Global Innovation Fund. Thanks Jonathan for the original post and others for your comments. I’d like to offer a few clarificatory responses on some of the issues raised.

Apples vs oranges

Yes, you could say this is a deliberate apple vs oranges comparison as an aid to constructing an efficient fruit salad :-) Apples: The philanthropist in search of impact can invest in a proven intervention, where the marginal impact of each additional dollar is well understood. Buying more bednets, for instance, will save lives and prevent illness. Oranges: the philanthropist could invest in a risky innovation which might fail – but if successful, might catalyze far-reaching impact.

I don’t know the philosophy underlying Givewell’s portfolio strategy of 75%“Top Charities” vs. 25% innovation, but it doesn’t necessarily mean that Top Charities have a higher expected return. In general terms, allocating a philanthropic portfolio between proven interventions vs. innovations involves considerations of risk. Innovations are inherently risky, so a philanthropist might well seek a balanced impact portfolio with ‘bond-like’ Top Charities (sure impact return) and ‘equity-like’ innovations (high but uncertain impact returns). GIF has a very risk-tolerant strategy, but that may not suit everyone.

Projecting long-term impact

Impact projection for innovations is undoubtedly difficult. Certainly these projections involve informed judgment on things like the probability of failure and speed at which the innovation diffuses. But our projections of impact are no more audacious than the valuation projections that underlie venture capital finance for start-ups and indeed often draw on the same set of assumptions.

Typically our impact projections are based on a rate of growth of the innovation’s take-up or influence (number of customers or beneficiaries), multiplied by our best estimate of impact per person reached. To get the impact per beneficiary, we review the literature and draw on RCTs where possible and appropriate. We adjust for the risk that the innovation fails, both during our investment and during the hoped-for subsequent scale-up. We regularly update these projections based on new evidence on impact, rate of uptake, risks surmounted or not. Many of our grants have built-in RCTs or other evidence generation that lets us update the impact per beneficiary parameter.

Additionality and contribution

As readers of this thread know, these are difficult things to estimate. GIF invests in early-stage innovations that arguably might fail or falter absent the funding round in which we participate. So we think the counterfactual of no impact is a defensible assumption. Regarding contribution, we allocate future impact in proportion to our participation in that funding round. We are eager to keep up with methodological developments in these issues.

Transparency

We will update our website with more information on our methodology. Complete transparency on the underlying calculations presents some issues, since some of the information is sensitive or confidential, especially for risk capital. I’m not aware of any investor that publishes projections of a for-profit investee’s impact – I’d be grateful for any references.

jhJul 18 20238

Thank you for engaging with this discussion, Ken!

It's great to have these clarifications in your own words. As you highlight there are many important and tricky issues to grapple with here. I think we're all excited about the innovative work you're doing and excited to learn more as you're able to publish more information.

JasonJul 18 20234

Welcome to the Forum, Ken!

I think there's a difference between metrics that are appropriate for self-evaluation (as long as the underlying assumptions and limitations are shared with potential donors / investors) and those that are useful for direct comparison to other charitable opportunities.

One statement we were discussing from the report is:

This is three times higher than the impact per dollar of Givewell’s top-rated charities, including distribution of anti-malarial insecticide-treated bednets.

It sounds like there are currently enough differences between "GIF impact units" and "GiveWell impact units" to make such comparisons unwise at this time. The allocation of 100% of impact credit to the funding round GIF participates in, without sharing with funders in other rounds, may be a "defensible" approach, but it seems a significantly more favorable calculation model than GiveWell uses. Although it may be possible at some point to fairly convert GIF impact units into an estimated-equivalent number of GiveWell impact units, it doesn't sound like that is currently viable.^[1]

(This isn't meant to come across as too critical -- it's a very minor statement in your report. But I felt it was worth making, because I hope GIF continues to grow. And in that case, making valid comparisons will become increasingly important.)

^{^}
Among other things, I think we would need to know about future funding rounds, particularly the amount funded by social-impact-minded grantors or investors, in order to appropriately allocate credit between GIF and the other funders. Allowing each grantor/investor to assume 100% of the impact flowed from the round in which they participated could significantly inflate the cost-effectiveness of the GIF approach.

jhJul 18 202310

My current understanding of the state of play around projected / forecast impact is:

GiveWell doesn't have public methodologies or models for this.
GIF has a public methodology but no public worked examples. Even an artificial example would help make our discussion more precise.
Other actors have methodologies and examples (not necessarily public). All with their own pros and cons, especially around how they handle contribution. One of the most mature may be PRIME Coalition / Project FRAME for climate impact.

JasonJul 18 20234

I think we mostly have a semantic difference here. At present, I think the method of analysis is so different that it's better not to speak of the units as being of the same type. That's in part based on clarity concerns -- speaking of GIF units and GiveWell units as the same risks people trying to compare them without applying an appropriate method for allocating impact in a comparative context. I think it's possible to agree on a range, but I think that is going to require a lot of data from GIF that it probably isn't in a position to disclose (and which may require several more years of operation to collect).

If I'm understanding Ken correctly, I do not think GIF's current calculation method is sufficient to allow for comparisons between GIF and GiveWell:

GIF invests in early-stage innovations that arguably might fail or falter absent the funding round in which we participate. So we think the counterfactual of no impact is a defensible assumption. Regarding contribution, we allocate future impact in proportion to our participation in that funding round.

Let's say GIF gave a $1MM grant to an organization in the first funding round, which is 50% of the total round. Another grantor gave a $2MM grant in the second round (50% of that round), and a third grantor gave $4MM as 50% of a final funding round. (I'm using grants to simplify the toy model.)

The organization produces 14 million raw impact units, as projected. If I'm reading the above statement correctly, GIF allocates all 14 million raw units to the first funding round, and assigns itself half of them for a final impact of 7 raw impact units per dollar. For this to be comparable to a GiveWell unit (which represents unduplicated impact), you'd have to assign the other two funders zero impact, which isn't plausible. Stated differently, you'd have to assume the other grantors' counterfactual use of the money in GIF's absence would have been to light it on fire.

A generous-to-GiveWell option would be to assume that the other two grantors would have counterfactually given their $6MM to GiveWell. Under this assumption, GIF's impact is 7 million raw impact units for the $1MM minus how many ever raw impact units GiveWell would have generated with an additional $6MM. Under the assumption that GiveWell converts money into raw impact units 1/3 as efficiently as GIF, that would actually make GIF severely net negative in the toy example because the lost $6MM in GiveWell funding was more valuable than 50% of the organization's impact.

I'm definitely not suggesting that is the correct approach, and it is certainly GiveWell-friendly. However, I'm not currently convinced it's more GiveWell-friendly than allocating all impact to the funding round in which GIF participated is GIF-friendly. If the defensible range of comparisons is anywhere near as broad as the toy example, then no meaningful comparisons can be made on currently available information.

jhJul 19 202312

Yeah, it seems we do have a semantic difference here. But, how you're using 'raw impact units' makes sense to me.

Nice, clear examples! I feel inspired by them to sketch out what I think the "correct" approach would look like. With plenty of room for anyone to choose their own parameters.

Let's simplify things a bit. Say the first round is as described above and its purpose is to fund the organization to test its intervention. Then let's lump all future rounds together and say they total $14m and fund the implementation of the intervention if the tests are successful. That is, $14m of funding in the second round, assuming the tests are a success, produces 14m units of impact.

The 14m is what I would call the org's potential 'Gross Impact', with no adjustment for the counterfactual. We need to adjust for what would otherwise happen without the org to get its potential 'Enterprise Impact' (relative to the counterfactual).

For one, yes, the funders would have invested their money elsewhere. So, the org will only have a positive Enterprise Impact if it is more cost-effective than the funder's alternative. I think the 'generous-to-GiveWell option' is more extreme than it might appear at first glance. It's not only assuming that the funders would otherwise donate in line with GiveWell (GW). It's also assuming that they are somehow suckered into donating to this less effective org, despite being GW donors.

A more reasonable assumption, in my view, is that the org only gets funding if its cost-effectiveness is above the bar of its funders. It also seems likely to me that the org, if successful, will be able to attract funders that are not GW donors. There are plenty of funders with preferences that are not that aligned with cost-effectiveness. As long as these other reasons line up with this hypothetical org, then it could get non-GW funding. Indeed, in GW's model for Malaria Consortium it looks like they are assuming the Global Fund is 2-3x less effective than GW spending and that Domestic Governments are 6x-10x less effective. Furthermore, if the org is able to adopt a for-profit operating model, it could get commercial funding with relatively little impact in the counterfactual.

As an example, let's say GW top charities produce 1 unit of impact per dollar and the org's second round funders typically make grants that are 10x less effective than GW. The counterfactual impact of the funder's alternative grants would be 1.4 million units of impact. So, based on this consideration the potential Enterprise Impact = 14 million - 1.4 million = 12.6 million units of impact.

Another consideration is that if the org didn't exist, or even if it does, then another org may have developed that solves the same problem for the same beneficiaries. Let's say the probability of an alternative org replicating the Enterprise Impact is 21% (just an illustrative number). Then adjusting for this consideration makes the potential Enterprise Impact actually (1-21%) * 12.6 million = 10 million units of impact.

Next, we need to go from potential Enterprise Impact to expected Enterprise Impact. That is, we need to account for the probability the org is successful after the first round tests. Let's say 10% - a fairly standard early-stage success rate. That makes the expected Enterprise Impact equal to 1 million units.

Now we can look at the impact of GIF's funding. That is, how did their decision to fund the $1m in the first round change the expected Enterprise Impact?

This will depend on a combination of how much potential interest there was from other funders and how much the organization is able to scale with more funding (e.g., improving the statistical power of their test intervention, testing in more locations,...). At one extreme, all other funders may be non-cost effective donors and only be willing to participate if GIF led the round, in which case I'd say GIF's $1m enabled the entire $2m round. At the other extreme, it could be the org only really needed $1m and there were plenty of other funders willing to step in, in which case GIF's investor impact would be close to zero.

For this example, let's say there was some potential of other funding but it was far from guaranteed, that the typical cost-effectiveness of the other funders would be small, and there were some diminishing returns to scale in the round. Altogether, suppose this means that GIF's 50% of the round only has a contribution versus the counterfactual equal to 30% of the expected Enterprise Impact. That is, an Investor Impact of 300k units of impact.

To summarize the whole stream of calculations is (14m - 1.4m) * (1 - 0.21) * 10% * 30% = 300k. That is, 0.3 units of impact per dollar. Or, 0.3x GW (according to my illustrative assumptions).

Based on GIF's published methodology and Ken's comments here, I believe GIF's reported numbers for this example would be something like 14m * 10% * 50% = 700k. Or, 0.7x GW. Given they actually reported 3x GW, to calibrate this example to their reporting, I'd increase the value of the (14m * 10%) part by 4.3 = 3/0.7. This can be interpreted as the actual scale or duration of the orgs GIF funds being greater than 14m, or that their average probability of success is higher than 10%.

With the calibration change, my illustrative estimate of GIF's effectiveness would be 4.3 times my original value of 0.3x. That is, 1.3x GW.

The only "real" number here is the calibration to being 3x GW according to my version of GIF's calculations. The point of the 1.3x result is just to illustrate how I would adjust for the relevant counterfactuals. Relative to my version of GIF's calculation, my calculation includes a 71% = (14m - 1.4m)/ 14m * (1 - 0.21) multiplier, that translates Gross Impact into Enterprise Impact, and a 60% = 30% /50% multiplier, that gets us to the final Investor Impact. With these values, that I consider highly uncertain but plausible, the effectiveness of GIF would be above GW top charities.

For emphasis, I'm not claiming GIF is or is not achieving this effectiveness. I'm just seeking to illustrate that it is plausible. And, if someone were to do an independent analysis, I'd expect the results to shape up along the lines of the approach I've outlined here.

JasonJul 19 20232

For one, yes, the funders would have invested their money elsewhere. So, the org will only have a positive Enterprise Impact if it is more cost-effective than the funder's alternative. I think the 'generous-to-GiveWell option' is more extreme than it might appear at first glance. It's not only assuming that the funders would otherwise donate in line with GiveWell (GW). It's also assuming that they are somehow suckered into donating to this less effective org, despite being GW donors.

Yes, I think the "generous-to-Givewell" model should be seen as the right bookend on defensible models on available data, just like I see GIF's current model as the left bookend on defensible models. I think it's plausible that $1 to GIF has either higher or lower impact than $1 to GiveWell.

As for the counterfactual impact that other funders would have, I would expect funders savvy enough and impact-motivated enough to give to GIF-supported projects to be a cut above the norm in effectiveness (although full-GiveWell effectiveness is a stretch as you note). Also, the later-round funders could plausibly make decisions while disregarding prior funding as sunk costs, if they concluded that the relevant project was going to go under otherwise. This could be because they are thinking in a one-off fashion or because they don't think their fund/no fund decision will affect the future decisions of early-stage funders like GIF.

Although I like your model at a quick glance, I think it's going to be challenging to come up with input numbers we can have a lot of confidence in. If there's relatively low overlap between the GiveWell-style donor base and the GIF-style donor base, it may not be worthwhile to invest heavily enough in that analysis to provide a confidence interval that doesn't include equality.

Also, GiveWell's diminishing returns curve is fairly smooth, fairly stable over time, and fairly easy to calculate -- most of its portfolio is in a few interventions, and marginal funding mostly extends one of those interventions to a new region/country. GIF's impact model seems much more hits-based, so I'd expect diminishing returns to kick in more forcefully. Indeed, my very-low-confidence guess is that GIF is more effective at lower funding levels, but that the advantage switches to GiveWell at some inflection point. All that is to say that we'd probably need to invest resources into continuously updating the relevant inputs for the counterfactual impact forumula.

Wayne Chang 🔸Jun 3 202314

Thanks for posting this, Jonathan! I was going to share it on the EA Forum too but just haven't gotten around to it.

I think GIF's impact methodology is not comparable to GiveWell's. My (limited) understanding is that their Practical Impact approach is quite similar to USAID's Development Innovation Ventures' impact methodology. DIV's approach was co-authored by Michael Kremer so it has solid academic credentials. But importantly, the method takes credit for the funded NGO's impact over the next 10 years, without sharing that impact with subsequent funders. The idea is that the innovation would fail without their support so they can claim all future impact if the NGO survives (the total sum of counterfactual impact need not add to 100%). This is not what GiveWell does. GiveWell takes credit for the long-term impact of the beneficiaries it helps but not for the NGOs themselves. So this is comparing apples to oranges. It's true that GiveWell Top Charities are much more likely to survive without GiveWell's help but this leads to my next point.

GiveWell also provides innovation grants through their All Grants Fund (formerly called Incubation Grants). They've been funding a range of interventions that aren't Top Charities and in many cases, are very early, with GiveWell support being critical to the NGO's survival. According to GiveWell's All Grants Fund page, "As of July 2022, we expect to direct about three-quarters of our grants to top charity programs and one-quarter to other programs, so there's a high likelihood that donations to the All Grants Fund will support a top charity grant." This suggests that in GiveWell's own calculus, innovation grants as a whole cannot be overwhelmingly better than Top Charities. Otherwise, Top Charities wouldn't account for the majority of the fund.

When thinking about counterfactual impact, the credit one gets for funding innovation should depend on the type of future donors the NGO ends up attracting. If these future donors would have given with low cost-effectiveness otherwise (or not at all), then you deserve much credit. But if they would have given to equally (or even more) cost-effective projects, then you deserve zero (or even negative) credit. So if GIF is funding NGOs that draw money from outside EA (whereas GiveWell isn't), it's plausible their innovations have more impact and thus are more 'cost-effective'. But we are talking about leverage now, so again, I don't think the methodologies are directly comparable.

Finally, I do think GIF should be more transparent about their impact calculations when making such a claim. It would very much benefit other donors and the broader ecosystem if they can make public their 3x calculation (just share the spreadsheet please!). Without such transparency, we should be skeptical and not take their claim too seriously. Extraordinary claims require extraordinary evidence.

MichaelStJulesJun 1 20236

(I haven't read the report.)

This is interesting, but only 3x marginal cost-effectiveness of GiveWell's top charities is not that impressive given that GiveWell's are supported by RCTs and careful adjustments, whereas there aren't going to be very relevant RCTs on affecting policy or supporting innovation, so the evidence for direct impact is weaker. When you loosen evidentiary standards and are willing to make more bets with weaker links in your causal chain, I suspect you can do even better than 3x, at least if you're moving much less money than GiveWell and its top charities.

I wouldn't be surprised if fundraising for GiveWell's top charities (via GWWC, TLYCS, OFTW, Founders Pledge) beat 3x GiveWell's top charities, although presumably you could fundraise for GIF instead and do even better. And then there are different cause areas entirely, instead of global health and development.

On the other hand, Open Phil seems to have found it hard to beat GiveWell's top charities for helping humans alive today even while loosening these standards, so maybe it is very hard, and either GIF is in fact doing really well, or we should be more skeptical that they've succeeded in beating GiveWell's top charities 3x: https://www.openphilanthropy.org/research/givewells-top-charities-are-increasingly-hard-to-beat/

JasonJun 1 202313

Skimming the report, but especially pp. 30-31, I'd probably adjust their estimates significantly downward as relying on informed speculation about future events, and probably on unclear assumptions about counterfactual funding and grantee/investee growth. I find such projections optimistic until proven otherwise.

That being said, I think it's a good idea to have a variety of effective options for prospective donors to capture as large a slice of the philantrophic pie as feasible. If someone doesn't find saving the lives of children under age 5 -- the means by which GiveWell's top charities produce the bulk of their assessed value -- compelling enough to attract the bulk of their donations, I still would like their money for another effective intervention even assuming that the alternative doesn't score quite as well as GiveWell's best.

SanjayJun 2 20232

I think it's interesting that an impact investing fund is making the comparison to Givewell. This is far from widespread in the philanthropic world, and is even rarer in investing.

I predict that I probably wouldn't agree with the 3x claim if scrutinised properly.

I sympathise with the point made by Michael St Jules about quality of evidence, but I'm more worried about counterfactuals. I.e. if GIF had not made those investments, how likely is it that someone else would have?

jhJun 2 20237

Actually, they are more of a grant fund than an impact investment fund. I've updated the post to clarify this. Thanks for bringing it up.

One might call them an 'investing for impact' fund - making whatever investments they think will generate the biggest long-term impact.

The reported projections aren't adjusted for counterfactuals (or additionality, contribution, funging, etc.). I wonder if the fact we're mostly talking about GIF grants vs GiveWell grants changes your worry at all?

For my part, I'd be excited to see more grant analyses (in addition to impact investment analyses) explicitly account for counterfactuals. I believe GiveWell does make some adjustments for funging, though I'm uncertain if they are comprehensive enough.