Comments
Thanks for posting this, Jonathan! I was going to share it on the EA Forum too but just haven't gotten around to it.
I think GIF's impact methodology is not comparable to GiveWell's. My (limited) understanding is that their Practical Impact approach is quite similar to USAID's Development Innovation Ventures' impact methodology. DIV's approach was co-authored by Michael Kremer so it has solid academic credentials. But importantly, the method takes credit for the funded NGO's impact over the next 10 years, without sharing that impact with subsequent funders. The idea is that the innovation would fail without their support so they can claim all future impact if the NGO survives (the total sum of counterfactual impact need not add to 100%). This is not what GiveWell does. GiveWell takes credit for the long-term impact of the beneficiaries it helps but not for the NGOs themselves. So this is comparing apples to oranges. It's true that GiveWell Top Charities are much more likely to survive without GiveWell's help but this leads to my next point.
GiveWell also provides innovation grants through their All Grants Fund (formerly called Incubation Grants). They've been funding a range of interventions that aren't Top Charities and in many cases, are very early, with GiveWell support being critical to the NGO's survival. According to GiveWell's All Grants Fund page, "As of July 2022, we expect to direct about three-quarters of our grants to top charity programs and one-quarter to other programs, so there's a high likelihood that donations to the All Grants Fund will support a top charity grant." This suggests that in GiveWell's own calculus, innovation grants as a whole cannot be overwhelmingly better than Top Charities. Otherwise, Top Charities wouldn't account for the majority of the fund.
When thinking about counterfactual impact, the credit one gets for funding innovation should depend on the type of future donors the NGO ends up attracting. If these future donors would have given with low cost-effectiveness otherwise (or not at all), then you deserve much credit. But if they would have given to equally (or even more) cost-effective projects, then you deserve zero (or even negative) credit. So if GIF is funding NGOs that draw money from outside EA (whereas GiveWell isn't), it's plausible their innovations have more impact and thus are more 'cost-effective'. But we are talking about leverage now, so again, I don't think the methodologies are directly comparable.
Finally, I do think GIF should be more transparent about their impact calculations when making such a claim. It would very much benefit other donors and the broader ecosystem if they can make public their 3x calculation (just share the spreadsheet please!). Without such transparency, we should be skeptical and not take their claim too seriously. Extraordinary claims require extraordinary evidence.
Let me introduce myself – I’m the Chief Economist of the Global Innovation Fund. Thanks Jonathan for the original post and others for your comments. I’d like to offer a few clarificatory responses on some of the issues raised.
Apples vs oranges
Yes, you could say this is a deliberate apple vs oranges comparison as an aid to constructing an efficient fruit salad :-) Apples: The philanthropist in search of impact can invest in a proven intervention, where the marginal impact of each additional dollar is well understood. Buying more bednets, for instance, will save lives and prevent illness. Oranges: the philanthropist could invest in a risky innovation which might fail – but if successful, might catalyze far-reaching impact.
I don’t know the philosophy underlying Givewell’s portfolio strategy of 75%“Top Charities” vs. 25% innovation, but it doesn’t necessarily mean that Top Charities have a higher expected return. In general terms, allocating a philanthropic portfolio between proven interventions vs. innovations involves considerations of risk. Innovations are inherently risky, so a philanthropist might well seek a balanced impact portfolio with ‘bond-like’ Top Charities (sure impact return) and ‘equity-like’ innovations (high but uncertain impact returns). GIF has a very risk-tolerant strategy, but that may not suit everyone.
Projecting long-term impact
Impact projection for innovations is undoubtedly difficult. Certainly these projections involve informed judgment on things like the probability of failure and speed at which the innovation diffuses. But our projections of impact are no more audacious than the valuation projections that underlie venture capital finance for start-ups and indeed often draw on the same set of assumptions.
Typically our impact projections are based on a rate of growth of the innovation’s take-up or influence (number of customers or beneficiaries), multiplied by our best estimate of impact per person reached. To get the impact per beneficiary, we review the literature and draw on RCTs where possible and appropriate. We adjust for the risk that the innovation fails, both during our investment and during the hoped-for subsequent scale-up. We regularly update these projections based on new evidence on impact, rate of uptake, risks surmounted or not. Many of our grants have built-in RCTs or other evidence generation that lets us update the impact per beneficiary parameter.
Additionality and contribution
As readers of this thread know, these are difficult things to estimate. GIF invests in early-stage innovations that arguably might fail or falter absent the funding round in which we participate. So we think the counterfactual of no impact is a defensible assumption. Regarding contribution, we allocate future impact in proportion to our participation in that funding round. We are eager to keep up with methodological developments in these issues.
Transparency
We will update our website with more information on our methodology. Complete transparency on the underlying calculations presents some issues, since some of the information is sensitive or confidential, especially for risk capital. I’m not aware of any investor that publishes projections of a for-profit investee’s impact – I’d be grateful for any references.
Thank you for engaging with this discussion, Ken!
It's great to have these clarifications in your own words. As you highlight there are many important and tricky issues to grapple with here. I think we're all excited about the innovative work you're doing and excited to learn more as you're able to publish more information.
Welcome to the Forum, Ken!
I think there's a difference between metrics that are appropriate for self-evaluation (as long as the underlying assumptions and limitations are shared with potential donors / investors) and those that are useful for direct comparison to other charitable opportunities.
One statement we were discussing from the report is:
It sounds like there are currently enough differences between "GIF impact units" and "GiveWell impact units" to make such comparisons unwise at this time. The allocation of 100% of impact credit to the funding round GIF participates in, without sharing with funders in other rounds, may be a "defensible" approach, but it seems a significantly more favorable calculation model than GiveWell uses. Although it may be possible at some point to fairly convert GIF impact units into an estimated-equivalent number of GiveWell impact units, it doesn't sound like that is currently viable.[1]
(This isn't meant to come across as too critical -- it's a very minor statement in your report. But I felt it was worth making, because I hope GIF continues to grow. And in that case, making valid comparisons will become increasingly important.)
Among other things, I think we would need to know about future funding rounds, particularly the amount funded by social-impact-minded grantors or investors, in order to appropriately allocate credit between GIF and the other funders. Allowing each grantor/investor to assume 100% of the impact flowed from the round in which they participated could significantly inflate the cost-effectiveness of the GIF approach.
I wouldn't put the key point here down to 'units'. I would say the aggregate units of GiveWell tends to use ('units of value' and lives saved) and of GIF (person-year of income-equivalent, "PYI") are very similar. I think any differences in terms of these units is going to be more about subjective differences in 'moral weights'. Other than moral weight differences, I'd expect the same analysis using GiveWell vs GIF units to deliver essentially the same results.
The point you're bringing up, and that Ken discusses as 'Apples vs oranges', is that the analysis is different. GiveWell's Top Charity models are about the outputs your dollar can buy now and the impacts those outputs will generate (e.g., over a few years for nets, and a few more for cash). As part of its models, GIF projects outputs (and impacts) out to 10 years. Indeed this is necessary when looking at early-stage ventures as most of their impact will be in the future and these organizations do not have to be cost-effective in their first year to be worth funding. If you were considering funding LLINs in 2004, or even earlier, you most likely would want to be doing a projection and not just considering the short-term impacts.
Of course, as has been repeatedly discuss in these comments, when projected impact is part of the model how much contribution to assign to the original donors becomes a big issue. But I believe that it is possible to agree on a consistent method for doing this. And once you have that, this really becomes more of an 'Apples vs apples' comparison.
For example, you might decide that a dollar to a malaria net charity now buys an amount of nets right now, but has limited impact on the future growth of that charity. So the current GiveWell estimates don't need to be modified even if you're including impact projections.
My current understanding of the state of play around projected / forecast impact is:
I think we mostly have a semantic difference here. At present, I think the method of analysis is so different that it's better not to speak of the units as being of the same type. That's in part based on clarity concerns -- speaking of GIF units and GiveWell units as the same risks people trying to compare them without applying an appropriate method for allocating impact in a comparative context. I think it's possible to agree on a range, but I think that is going to require a lot of data from GIF that it probably isn't in a position to disclose (and which may require several more years of operation to collect).
If I'm understanding Ken correctly, I do not think GIF's current calculation method is sufficient to allow for comparisons between GIF and GiveWell:
Let's say GIF gave a $1MM grant to an organization in the first funding round, which is 50% of the total round. Another grantor gave a $2MM grant in the second round (50% of that round), and a third grantor gave $4MM as 50% of a final funding round. (I'm using grants to simplify the toy model.)
The organization produces 14 million raw impact units, as projected. If I'm reading the above statement correctly, GIF allocates all 14 million raw units to the first funding round, and assigns itself half of them for a final impact of 7 raw impact units per dollar. For this to be comparable to a GiveWell unit (which represents unduplicated impact), you'd have to assign the other two funders zero impact, which isn't plausible. Stated differently, you'd have to assume the other grantors' counterfactual use of the money in GIF's absence would have been to light it on fire.
A generous-to-GiveWell option would be to assume that the other two grantors would have counterfactually given their $6MM to GiveWell. Under this assumption, GIF's impact is 7 million raw impact units for the $1MM minus how many ever raw impact units GiveWell would have generated with an additional $6MM. Under the assumption that GiveWell converts money into raw impact units 1/3 as efficiently as GIF, that would actually make GIF severely net negative in the toy example because the lost $6MM in GiveWell funding was more valuable than 50% of the organization's impact.
I'm definitely not suggesting that is the correct approach, and it is certainly GiveWell-friendly. However, I'm not currently convinced it's more GiveWell-friendly than allocating all impact to the funding round in which GIF participated is GIF-friendly. If the defensible range of comparisons is anywhere near as broad as the toy example, then no meaningful comparisons can be made on currently available information.
Yeah, it seems we do have a semantic difference here. But, how you're using 'raw impact units' makes sense to me.
Nice, clear examples! I feel inspired by them to sketch out what I think the "correct" approach would look like. With plenty of room for anyone to choose their own parameters.
Let's simplify things a bit. Say the first round is as described above and its purpose is to fund the organization to test its intervention. Then let's lump all future rounds together and say they total $14m and fund the implementation of the intervention if the tests are successful. That is, $14m of funding in the second round, assuming the tests are a success, produces 14m units of impact.
The 14m is what I would call the org's potential 'Gross Impact', with no adjustment for the counterfactual. We need to adjust for what would otherwise happen without the org to get its potential 'Enterprise Impact' (relative to the counterfactual).
For one, yes, the funders would have invested their money elsewhere. So, the org will only have a positive Enterprise Impact if it is more cost-effective than the funder's alternative. I think the 'generous-to-GiveWell option' is more extreme than it might appear at first glance. It's not only assuming that the funders would otherwise donate in line with GiveWell (GW). It's also assuming that they are somehow suckered into donating to this less effective org, despite being GW donors.
A more reasonable assumption, in my view, is that the org only gets funding if its cost-effectiveness is above the bar of its funders. It also seems likely to me that the org, if successful, will be able to attract funders that are not GW donors. There are plenty of funders with preferences that are not that aligned with cost-effectiveness. As long as these other reasons line up with this hypothetical org, then it could get non-GW funding. Indeed, in GW's model for Malaria Consortium it looks like they are assuming the Global Fund is 2-3x less effective than GW spending and that Domestic Governments are 6x-10x less effective. Furthermore, if the org is able to adopt a for-profit operating model, it could get commercial funding with relatively little impact in the counterfactual.
As an example, let's say GW top charities produce 1 unit of impact per dollar and the org's second round funders typically make grants that are 10x less effective than GW. The counterfactual impact of the funder's alternative grants would be 1.4 million units of impact. So, based on this consideration the potential Enterprise Impact = 14 million - 1.4 million = 12.6 million units of impact.
Another consideration is that if the org didn't exist, or even if it does, then another org may have developed that solves the same problem for the same beneficiaries. Let's say the probability of an alternative org replicating the Enterprise Impact is 21% (just an illustrative number). Then adjusting for this consideration makes the potential Enterprise Impact actually (1-21%) * 12.6 million = 10 million units of impact.
Next, we need to go from potential Enterprise Impact to expected Enterprise Impact. That is, we need to account for the probability the org is successful after the first round tests. Let's say 10% - a fairly standard early-stage success rate. That makes the expected Enterprise Impact equal to 1 million units.
Now we can look at the impact of GIF's funding. That is, how did their decision to fund the $1m in the first round change the expected Enterprise Impact?
This will depend on a combination of how much potential interest there was from other funders and how much the organization is able to scale with more funding (e.g., improving the statistical power of their test intervention, testing in more locations,...). At one extreme, all other funders may be non-cost effective donors and only be willing to participate if GIF led the round, in which case I'd say GIF's $1m enabled the entire $2m round. At the other extreme, it could be the org only really needed $1m and there were plenty of other funders willing to step in, in which case GIF's investor impact would be close to zero.
For this example, let's say there was some potential of other funding but it was far from guaranteed, that the typical cost-effectiveness of the other funders would be small, and there were some diminishing returns to scale in the round. Altogether, suppose this means that GIF's 50% of the round only has a contribution versus the counterfactual equal to 30% of the expected Enterprise Impact. That is, an Investor Impact of 300k units of impact.
To summarize the whole stream of calculations is (14m - 1.4m) * (1 - 0.21) * 10% * 30% = 300k. That is, 0.3 units of impact per dollar. Or, 0.3x GW (according to my illustrative assumptions).
Based on GIF's published methodology and Ken's comments here, I believe GIF's reported numbers for this example would be something like 14m * 10% * 50% = 700k. Or, 0.7x GW. Given they actually reported 3x GW, to calibrate this example to their reporting, I'd increase the value of the (14m * 10%) part by 4.3 = 3/0.7. This can be interpreted as the actual scale or duration of the orgs GIF funds being greater than 14m, or that their average probability of success is higher than 10%.
With the calibration change, my illustrative estimate of GIF's effectiveness would be 4.3 times my original value of 0.3x. That is, 1.3x GW.
The only "real" number here is the calibration to being 3x GW according to my version of GIF's calculations. The point of the 1.3x result is just to illustrate how I would adjust for the relevant counterfactuals. Relative to my version of GIF's calculation, my calculation includes a 71% = (14m - 1.4m)/ 14m * (1 - 0.21) multiplier, that translates Gross Impact into Enterprise Impact, and a 60% = 30% /50% multiplier, that gets us to the final Investor Impact. With these values, that I consider highly uncertain but plausible, the effectiveness of GIF would be above GW top charities.
For emphasis, I'm not claiming GIF is or is not achieving this effectiveness. I'm just seeking to illustrate that it is plausible. And, if someone were to do an independent analysis, I'd expect the results to shape up along the lines of the approach I've outlined here.
Yes, I think the "generous-to-Givewell" model should be seen as the right bookend on defensible models on available data, just like I see GIF's current model as the left bookend on defensible models. I think it's plausible that $1 to GIF has either higher or lower impact than $1 to GiveWell.
As for the counterfactual impact that other funders would have, I would expect funders savvy enough and impact-motivated enough to give to GIF-supported projects to be a cut above the norm in effectiveness (although full-GiveWell effectiveness is a stretch as you note). Also, the later-round funders could plausibly make decisions while disregarding prior funding as sunk costs, if they concluded that the relevant project was going to go under otherwise. This could be because they are thinking in a one-off fashion or because they don't think their fund/no fund decision will affect the future decisions of early-stage funders like GIF.
Although I like your model at a quick glance, I think it's going to be challenging to come up with input numbers we can have a lot of confidence in. If there's relatively low overlap between the GiveWell-style donor base and the GIF-style donor base, it may not be worthwhile to invest heavily enough in that analysis to provide a confidence interval that doesn't include equality.
Also, GiveWell's diminishing returns curve is fairly smooth, fairly stable over time, and fairly easy to calculate -- most of its portfolio is in a few interventions, and marginal funding mostly extends one of those interventions to a new region/country. GIF's impact model seems much more hits-based, so I'd expect diminishing returns to kick in more forcefully. Indeed, my very-low-confidence guess is that GIF is more effective at lower funding levels, but that the advantage switches to GiveWell at some inflection point. All that is to say that we'd probably need to invest resources into continuously updating the relevant inputs for the counterfactual impact forumula.