Comments
First, we want to sincerely thank Giving What We Can for running this “Evaluating the Evaluators” exercise. We recognize that ACE has set ourselves a difficult task, compounded by the fact that we’re the only organization doing what we do. Therefore, receiving this kind of feedback is both very rare and very welcome. There’s a great deal in GWWC’s report that will help us improve our processes for 2024, which ultimately means more animals will be helped and spared. While we were disappointed that GWWC has decided not to defer to our recommendations this year or recommend our Movement Grants program as a top-rated fund, we were heartened by the positive points in their report and their optimism about ACE’s future, and look forward to receiving further helpful feedback in a future evaluation from them. We were also delighted that GWWC recommended the EA Animal Welfare Fund as an effective giving opportunity.
Second, as an organization that values transparency and seeks to be open about our own limitations, we appreciated GWWC’s same openness about the limitations to, and uncertainties around, their evaluation. As they noted, this included limitations to their animal welfare expertise, the early stage of the charity evaluation space in animal welfare, and the time constraints forcing them to take a minimum viable product approach to this evaluation. The bulk of this year’s process also coincided with the culmination of our charity recommendation decisions—which, combined with GWWC’s demanding deadlines, made some aspects of the process challenging for us. GWWC fully recognized this, and we are confident that any future evaluation exercise will be even more helpful than this year’s.
Third, we were reassured that much of GWWC’s constructive feedback aligns with ACE’s own self-identified areas for improvement. For example, we agree that we need to continuously assess whether we want to give out fewer, larger Movement Grants than we do currently. We also agree that we should be more strategic in using the valuable feedback we get from our Movement Grants grantees to inform our own views on priority tactics and translate this into useful information for the broader animal advocacy movement.
We are also continuously working toward improvements to our charity evaluation methods, such as how to more sensitively capture differences in scope. As in previous years, we will be conducting a thorough review of the top-priority improvements to make to next year’s Movement Grants and Charity Evaluations programs. We will certainly draw on GWWC’s feedback for this while also acknowledging that capacity constraints will inevitably make it impossible to make all of the improvements we would like.
Fourth, there are some elements of GWWC’s report that we did not fully agree with. For example, while we agree that there’s plenty of opportunity for improvements to our Cost Effectiveness model to ensure that it reflects the cost effectiveness of charities’ achievements as accurately as possible, we would like to highlight that this year’s model is the result of considerable research, external guidance, and exploration of alternatives. We built this year’s model systematically to try to capture the most important aspects of what makes achievements impactful, based on empirical evidence wherever possible, and the quantitative metric closest to impact on animals that we could access for all charities’ achievements (e.g., the number of people reached per dollar for an educational campaign). We consulted with several external experts on how to best combine these scores into a single score and went through several iterations to ensure that the scores held up in confidence checks. We also think it’s likely that GWWC is overestimating how easy it is to deliver on their recommendation of reliably estimating the “marginal cost effectiveness of a dollar spent on the charity, based on the charity’s specific context.” In some past rounds of our Charity Evaluations program, ACE carried out back-of-the-envelope-calculation (BOTEC)-type cost-effectiveness modeling using Guesstimate, which aligns with GWWC’s recommendation. However, ACE then decided to change this approach for the reasons outlined here. We continuously review this decision and are open to reintroducing elements of our past approach if we determine it would be valuable for more effectively advancing our theory of change.
As another example of disagreement, GWWC noted that they would prefer our Movement Grants program to focus exclusively on national or international projects rather than the types of regional projects we have sometimes funded. However, given that one of the aims of the Movement Grants program is to build up the movement in priority regions with a relatively small animal advocacy movement, and that for some regions, we receive significantly more applications for regional projects than for high-quality, tractable, national-level applications, we expect that we will continue funding some regional projects that we consider particularly promising. Relatedly, GWWC disagrees with our view that funding projects in countries with very little animal advocacy representation should be a key part of ensuring the animal advocacy movement’s long-term success, noting that others may reasonably be more sympathetic to this view.
Fifth, we share GWWC’s commitment to prioritizing marginal funding to the projects where it will be the most cost effective. However, elements of ACE’s work—none of which are unique to us—make this particularly complex to achieve in practice. With our charity evaluations, for example, we only re-evaluate charities every two years, and we do not directly disburse most of the funding that we influence to our Recommended Charities. As such, ACE does not regularly calculate cost effectiveness over a range of possible allocations and distributes funding only to those above a particular effectiveness bar; instead, we recommend a set of charities that we are confident will put additional funding toward effective use to help animals over a longer time horizon without our oversight. We make this decision based on all of our evaluation criteria, with a strong focus on Cost Effectiveness (which examines the effectiveness of past work) and Room For More Funding (which assesses whether a charity’s planned uses for funding over the next two years will be roughly similar to their past work).
It is also worth noting that because we have shifted to one recommendation level for our charities this year (as opposed to the previous Top Charities/Standout Charities distinction), we plan to develop a new decision-making process that better accounts for the marginal cost effectiveness of funding disbursed from ACE’s Recommended Charity Fund. This will better enable us to leverage our grantmaking role in addition to our recommendation role.
For our Movement Grants—especially smaller projects, projects benefiting species for whom few interventions have been tried, and projects in regions where the movement is particularly small, so there’s particularly little evidence—it is not currently possible to sufficiently investigate each project application to make meaningful cost-effectiveness estimates. We, therefore, rely on proxies such as the coherence of an applicant’s theory of change, the priority of their focus animal groups, and the neglectedness of the region in which they operate.
Additionally, because we believe that supporting a range of different approaches is essential for an effective animal advocacy ecosystem (and because our recommendations influence donor and public opinion), we want to feature a plurality of approaches with a strong potential for impact. Because of the high uncertainty about the most effective ways to help animals, and because the different interventions reinforce and facilitate each other, we think supporting a range of approaches is both necessary and beneficial for the animal advocacy ecosystem as a whole. We view this as preferable to diverting funding near-exclusively to charities and programs with the most convincing shorter-term theories of change, which runs the risk of dismissing potentially pivotal interventions due to measurability bias.
Sixth, we are glad that GWWC chose to recommend The Humane League (THL) based in part on our evaluation. As we note in our 2023 review, we view giving to THL as an excellent opportunity to support initiatives that create the most positive change for animals. At the same time, we are disappointed that GWWC chose to only recommend one of our Recommended Charities and to restrict grants for their corporate campaign work. After months of evaluation, we are confident that all of our Recommended Charities represent highly promising giving opportunities. We are also convinced of the need for a pluralistic and resilient movement incorporating a range of effective tactics toward different outcomes to achieve wellbeing for as many animals as soon as possible globally.
We also want to make clear that while GWWC’s decision might imply that THL represents a superior giving opportunity compared to our other recommended charities, this is not a view that ACE shares. We view all of our recommended charities, including THL, as highly impactful giving opportunities. Following GWWC’s initial conclusion that they weren’t going to defer to our overall charity recommendations this year, we would have welcomed the opportunity to provide supporting materials for more than three of our Recommended Charities and to have had more time to do so.
Lastly, and most importantly, we congratulate, once more, the latest additions to our list of Recommended Charities. We will diligently continue working to ensure that our approach to evaluation and our methods capture the full extent of charities’ work as accurately as possible, and we expect these improvements to be an ongoing endeavor that will continue for as long as ACE exists. At the same time, following months of research, preparation, and evaluation, we are confident that Kafessiz Türkiye, Dansk Vegetarisk Forening, Faunalytics, Fish Welfare Initiative, Good Food Institute, Legal Impact for Chickens, New Roots Institute, Shrimp Welfare Project, Sinergia Animal, The Humane League, and Wild Animal Initiative all do incredible work and represent extremely promising giving opportunities. We are pleased that GWWC’s report recognizes this, both through their direct recommendation of our Recommended Charity The Humane League, and through their strong recommendation to impact-maximizing donors to give to ACE’s recommended charities over the average animal welfare charity.
-ACE Team
Removed.
First of all, thank you for the extensive comments!
I can give more context during our AMA next week if helpful (I won't have much time to engage in the coming few days unfortunately), but wanted to just quickly react to avoid a misunderstanding about our views here. I've copy-pasted from the relevant section from the report below:
So in short, we share your impression that THL's work is (much) less cost-effective than it was a few years ago. We are aware of Open Phil's views on this, and their referral of THL's work to us took these diminished expected returns into account. The FP and RP reports weigh (much) less heavily in our recommendation of THL's current work than ACE's and OP's recommendations, but we think those reports still provide a useful (and publicly accessible) reference on corporate campaigns as an intervention more generally.
Agree with lots of the above.
It also just seems very bizarre that the GWWC's animal fund pays out half to EA AWF and half to THL. Surely if you thought that EA AWF was a good evaluator or donation opportunity for donors, you would just let them manage the entirety of the fund? As then EA AWF would be able to distribute to THL if they actually thought THL was the most effective use of funds on the margin. And if not, even better, as they can give to more effective opportunities.
Also responding to the below points in your ACE evaluation report:
I'm also curious why you felt the need to recommend at least one competitive alternative to the AWF, when the AWF itself is a fairly diversified fund? Arguably, you marked ACE down for similar reasoning in your evaluation of their Movement Grants (that they were spreading their grants across many groups rather than focusing mostly on the most effective groups)
Statements like this make me worry that this evaluation focused too much on the certainty of some positive impact, rather than maximising expected impact (i.e. measurability bias). As mentioned in the comment above, you would struggle to find many experienced animal advocates who would confidently recommend THL as the single best marginal giving opportunity. In reality, they would likely either advocate for a spread of groups using different approaches or just simply give to a fund (e.g. EA AWF or ACE).
Thanks for your comments and questions, James.
The short answer is "no": we don't think we can currently justify the claim that giving to the AWF is better than giving to THL's corporate campaigns, or vice versa. We did indeed conclude from our evaluation that the AWF can likely use marginal funds cost-effectively, but that isn't the same as deferring to them on all fronts (including because we also found significant room for improvement, as explained in the AWF report), nor does it imply the AWF is better at allocating extra capital than THL is.
Our goal is to provide recommendations to help donors maximise their impact from the perspective of a variety of worldviews, and it's in that light that we decided to also recommend THL's corporate campaigns: consider that someone else could have made an (I think justifiable) comment that is entirely the opposite of what you are saying, i.e. that we should only recommend THL because there we actually have some independent evidence of their intervention working and being highly cost-effective, which is lacking for many if not most of the projects the AWF funds (given the early stage of the AW charity evaluation space).
We criticized ACE MG not for making grants to multiple groups but for doing so at the seeming expense of expected impact. As mentioned above, we don't think THL's corporate campaigns are a worse donation opportunity than the AWF, and we think there may be donors who think it's more cost-effective in expectation, for instance because they put less weight on the individual judgement of grantmakers (or on our judgement in evaluating AWF to be a good donation opportunity!) and think the publicly available evidence for THL is stronger.
I think you're right to worry about this - I do as well! - as I would say there is some implicit measurability bias in our recommendations. Most notably, we ended up recommending THL's corporate campaigns over other ACE recommendations not because we have strong evidence that they are a better donation opportunity than any other individual ACE recommendation, but because they are the only one where we think we have sufficient evidence to justify recommending them.
However, this is importantly different from us prioritising certainty of some positive impact over maximum expected impact: THL's corporate campaigns is our best guess donation opportunity to maximise expected impact (alongside the AWF). If we thought we could have easily justified any one of ACE's other recommendations was better - or even just as good - from that perspective, we would have recommended them, but we currently can't. And please note that "justifying" here isn't about finding "certainty of positive impact": we are looking for the expected value case (as we do for the AWF and our other recommendations as well).
This is a much stronger claim than we are making (THL's corporate campaigns being the "single best marginal giving opportunity"): we think it's one of the two best donation opportunities we can, from the information we have available, recommend to a broad set of donors to maximise their expected impact. We are not claiming that nobody could do better (certainly by their individual values/worldview!), and encourage donors to do their own (further) research if they have the time and expertise available. This is also why we host a broader selection of promising programs donors can look into and support on our donation platform.
Based on your paragraph below from the ACE Report, I'm inferring that you only looked at three (out of 11) ACE recommendations, which only included charities evaluated in 2023, rather than 2022? So by default, GFI, Sinergia, Fish Welfare Initiative, Kafessiz and DVF were all excluded from potentially being identified (which seems illogical, as there is no obvious reason to think that charities evaluated in 2022 would be less cost-effective).[1]
Given you only looked at three of the ACE 2023 recommendations (and you didn't say which ones), I'm wondering how you can make such a strong claim for all of ACE's recommended charities?
On a slightly unrelated point: For the referral from OP, I would be curious to hear if you asked them "What is the most cost-effective marginal giving opportunity for farmed animal welfare" (to which they replied THL's corporate campaigns) or something closer to "Do you think THL is a cost-effective giving opportunity on the margin?"
Fair enough! I should have said "One of the top 2 marginal giving opportunities" but I still think I stand by my point that many experienced animal advocates would disagree with this claim, and it's not clear that your charity recommendation work has sufficient depth to challenge that (e.g. you didn't evaluate groups yourself), in which case it's not clear why folks should defer to you over subject-matter experts (e.g. AWF, OP or ACE).
You might say there is weaker evidence of their cost-effectiveness as it's been a year since they were evaluated but since you said you focused on the expected value case rather than certainty of positive impact, I assume this wasn't your issue.
Yes they were, as were any other charities than the three charities we asked ACE to send us more information on (based on where they thought they could make the strongest case by our lights). Among those, we think ACE provided the strongest case for THL's corporate campaigns, and with the additional referral from Open Phil + the existing public reports by FP and RP on corporate campaigns, we think this is enough to justify a recommendation. This is what I meant by there indeed being a measurability bias in our recommendation (which we think is a bullet worth biting here!): we ended up recommending THL in large part because there was sufficient evidence of cost-effectiveness readily and publicly available. We don't have the same evidence for any of these other charities, so they could in principle be as or even more cost-effective than THL (but also much less!), and without the evidence to support their case we don't (yet) feel justified recommending them. We don't have capacity to directly evaluate individual charities ourselves (including THL!), but continue to host many promising charities on our donation platform, so donors who have time to look into them further can choose to support them.
To put this differently, the choice for us wasn't between "evaluating all of ACE's recommendations" and "evaluating only THL / three charities" (as we didn't have capacity to do any individual charity evaluations). The choice for us was between "only recommending the AWF" and "recommending both the AWF and THL's corporate campaigns" because there happened to already be sufficiently strong evidence/evaluations available for THL's corporate campaigns. For reasons explained earlier, we stand by our decision to prefer the latter over the former, even though that means that many other promising charities don't have a chance to be recommended at this point (but note that this is the case in charity evaluation across cause areas!).
Could you clarify which "strong claim for all of ACE's recommended charities" you are referring to? From the executive summary of our report on ACE:
We also expect the gain in impact from giving to any ACE-recommended charity over giving to a random animal welfare charity is much larger than any potential further gain from giving to the AWF or THL’s corporate campaigns over any (other) ACE-recommended charity, and note that we haven’t evaluated ACE’s recommended charities individually, but only ACE’s evaluation process.
The latter, because a referral by OP on its own wouldn't have been sufficient for us to make a recommendation (as we haven't evaluated OP): for recommending THL's corporate campaigns, we really relied on these four separate pieces of evidence being available.
We're not even claiming it is one of the top 2 marginal giving opportunities, just that it is the best recommendation we can make to donors based on the information available to us from evaluators. If you could point us to any alternative well-justified recommendations/evaluators for us to evaluate, we'd be all ears.
And we don't claim people should defer to us directly on charity evaluations (again, we don't currently do these ourselves!). Ultimately, our recommendations (including THL!) are based on the recommendations of the subject-matter experts you reference. The purpose of our evaluations and reports is to help donors make better decisions based on the recommendations and information these experts provide.