I actually think it could be quite reasonable for an org to trust or place more weight on an internal evaluation more than an external one, but apart from that fully agree with all you say!
On 2) I think this shouldn't be a barrier, since the first step in impact evaluation is establishing a theory of change, and any individual organisation of a given maturity should have a clear picture of this, and consensus on their goals. (I'm not advocating for a singular ToC across the whole EA movement, just for individual organisations)
On 3) I agree this is a factor for evaluations for funders. But I think organisations should carry out internal impact evaluation according to their theory of change, so I think should still be able to have clear and established goals. They can say to funders "Our Theory of Change is X, and along those lines our impact has been Y". Funders might also carry out their own evaluations in terms of the goals they most care about.
On 4), I agree there's a trade-off between spending on direct activities and impact evaluation. Of course the value of impact evaluation is that it may improve the effectiveness of the direct activities. So I don't think the existence of the trade-off is in itself a reason that impact evaluation shouldn't be prevalent.
I agree independence has advantages. OTOH I think there are also important advantages to internal impact evaluation: the results are more likely to be bought into internally, and important context or nuance is less likely to be missed. For making a theory of change specifically, I think it's quite important this is done internally, usually. Overall I think the ideal setup would quite often be for organisations to have their own internal impact evaluation function.
And that's interesting on funder interest. In a few cases, organisations I've spoken to have been able to get specific grants for impact evaluation. But also org's might choose to reallocate their existing budget, without needing additional funding, if they consider impact evaluation an essential function. E.g. for a fixed budget, they might decide that they should be allocating at least e.g. 5 / 10% to impact evaluation. (But I guess this might be harder if it required pulling back on existing activities)
I kind've see this as more at the org-level than funder-level tbh, similarly to any other spending decision facing an organisation. Perhaps because I'm thinking most about the benefits to org's themselves. But I definitely still agree that funder interest is a big driver.
Yeah I agree, I just mean that $1bn in funds lost to customers across the world is plausibly comparable in welfare terms to other wins on that list. E.g. dividing by 10 to account for differences in income of those affected, it would be around the amount attributed to GiveDirectly on the EA impact page.
(without wanting to make a very direct crude comparison, or getting into the details of that)
The role of the EA movement in the case of FTX seems surely to meet the level of influence for some of the impact win's that EA has had so far here.
Perhaps most prominently, the movement:
Gave the idea of 'Earning to Give' to Sam
Provided a primary motivation to Sam and other FTX leadership to build the exchange
For example, when comparing to the case of Sendwave, the influence seems at least comparable and if not larger e.g. played a motivational role in founding a company, for the purpose of improving the world. (I'm not familiar with Wave's founders motivations, so could be wrong here)
In welfare terms alone, the impact of FTX's collapse on it's customers seems plausibly comparable to some of the impact win's of the movement to date. I.e. of the order of $1bn in lost funds. Given this, I think that an honest impact evaluation of the EA movement would include the harm caused to customers through FTX's collapse.
This is relevant not for blame assignment, but because it's very decision-relevant to EA's mission of improving the world. For example, when in the future deciding how much to emphasise harm avoidance when encouraging the (good and novel) idea of Earning to Give.
On EA Grants:
Will you allow individuals to fund EA Grants in the future? This could either be letting individuals add to CEA's pot of funding for grants, publishing the rejected grants so that individuals can fund them independently or putting the applications on EA funds.
On EA Funds:
"Potential expansion of EA Funds on offer and investigation of different models for running and >using funds"
What types of funds and models might this investigation include?
I actually think it could be quite reasonable for an org to trust or place more weight on an internal evaluation more than an external one, but apart from that fully agree with all you say!
Thanks a lot Robert!
On 1) great point, I agree.
On 2) I think this shouldn't be a barrier, since the first step in impact evaluation is establishing a theory of change, and any individual organisation of a given maturity should have a clear picture of this, and consensus on their goals. (I'm not advocating for a singular ToC across the whole EA movement, just for individual organisations)
On 3) I agree this is a factor for evaluations for funders. But I think organisations should carry out internal impact evaluation according to their theory of change, so I think should still be able to have clear and established goals. They can say to funders "Our Theory of Change is X, and along those lines our impact has been Y". Funders might also carry out their own evaluations in terms of the goals they most care about.
On 4), I agree there's a trade-off between spending on direct activities and impact evaluation. Of course the value of impact evaluation is that it may improve the effectiveness of the direct activities. So I don't think the existence of the trade-off is in itself a reason that impact evaluation shouldn't be prevalent.
Thanks David!
I agree independence has advantages. OTOH I think there are also important advantages to internal impact evaluation: the results are more likely to be bought into internally, and important context or nuance is less likely to be missed. For making a theory of change specifically, I think it's quite important this is done internally, usually. Overall I think the ideal setup would quite often be for organisations to have their own internal impact evaluation function.
And that's interesting on funder interest. In a few cases, organisations I've spoken to have been able to get specific grants for impact evaluation. But also org's might choose to reallocate their existing budget, without needing additional funding, if they consider impact evaluation an essential function. E.g. for a fixed budget, they might decide that they should be allocating at least e.g. 5 / 10% to impact evaluation. (But I guess this might be harder if it required pulling back on existing activities)
I kind've see this as more at the org-level than funder-level tbh, similarly to any other spending decision facing an organisation. Perhaps because I'm thinking most about the benefits to org's themselves. But I definitely still agree that funder interest is a big driver.
Yeah I agree, I just mean that $1bn in funds lost to customers across the world is plausibly comparable in welfare terms to other wins on that list. E.g. dividing by 10 to account for differences in income of those affected, it would be around the amount attributed to GiveDirectly on the EA impact page.
(without wanting to make a very direct crude comparison, or getting into the details of that)
The role of the EA movement in the case of FTX seems surely to meet the level of influence for some of the impact win's that EA has had so far here.
Perhaps most prominently, the movement:
For example, when comparing to the case of Sendwave, the influence seems at least comparable and if not larger e.g. played a motivational role in founding a company, for the purpose of improving the world. (I'm not familiar with Wave's founders motivations, so could be wrong here)
In welfare terms alone, the impact of FTX's collapse on it's customers seems plausibly comparable to some of the impact win's of the movement to date. I.e. of the order of $1bn in lost funds. Given this, I think that an honest impact evaluation of the EA movement would include the harm caused to customers through FTX's collapse.
This is relevant not for blame assignment, but because it's very decision-relevant to EA's mission of improving the world. For example, when in the future deciding how much to emphasise harm avoidance when encouraging the (good and novel) idea of Earning to Give.
Thanks for writing this.
On EA Grants: Will you allow individuals to fund EA Grants in the future? This could either be letting individuals add to CEA's pot of funding for grants, publishing the rejected grants so that individuals can fund them independently or putting the applications on EA funds.
On EA Funds:
What types of funds and models might this investigation include?