ACE's Response to John Halstead

ToniA

ACE's Response to John Halstead

ToniA

14 min readSep 7, 2018

Comments 14

Sorted by

New & upvoted

zdgroff

As I noted on the original post, I am grateful this dialogue is happening so respectfully this time around.

adom_hartell

Hi Toni, thanks for posting. My apologies if the questions below have been answered elsewhere; I have not engaged very much with your research over the past year or so.

I'm wondering if you could provide a bit of clarification regarding the role of CEEs in the formation of your overall view of cost effectiveness. You describe CEEs and (e.g.) leadership quality as being independent features in determining the marginal effectiveness of donations. I understand "independent" here as meaning that each feature can vary as the other(s) are held constant (roughly, of course there will be some correlation between all of the criteria).

This makes sense, but I think it isn't hitting the core of Halstead's argument. The criteria seem to comprise a multiplicative model, where setting any of the variables to a sufficiently low value is enough to bring the estimated marginal impact close to zero. If donations don't cash out in the implementation of cost-effective interventions, then the rest of the features don't matter; likewise if the leadership is so poor that the organization disbands. In this view, something like CEEs are critical even if they are not "core" in a sense of being much more important than the other criteria, such that a compelling argument against the evidence of cost effectiveness does feel per se like a compelling argument against the evidence for marginal cost effectiveness.

While the criteria may vary independently, it doesn't seem that they independently contribute to animal welfare (i.e. contribute additively rather than multiplicatively).

Also relevant are one's prior on the cost effectiveness of animal welfare charities, and one's confidence in arguments in the style of CEEs. If I think that most animal welfare interventions have little impact, then I need compelling evidence that a particular charity is doing better in order to form a positive overall view of the org. If my prior is more optimistic, then other considerations pointing to organizational quality will be sufficient to think the org is a worthwhile target for donations (especially if I think my ability to form accurate CEEs is weak).

Reviewing your paraphrase of Halstead's argument:

ACE's corporate outreach intervention report is flawed.

Corporate outreach accounts for 90% of THL's and Animal Equality's CEEs.

Each charity's CEE is the primary piece of evidence that the charity improves animal welfare. Therefore,

ACE does not provide much reason to believe that their recommended charities improve animal welfare.

It seems to me that a compellingly positive CEE, primary evidence or no, is nonetheless a necessary component in the belief that an organization will improve animal welfare, particularly if one has a pessimistic prior. As such, effectively attacking the CEE is basically decisive. I'll note that my argument somewhat rests on a conceptual confusion: the CEE as you use it isn't actually your estimate of cost-effectiveness, just the subset that is easily quantifiable. The argument still seems to carry given a pessimistic prior and a lack of justification for the subset of impact that is hard to quantify.

I expect you have a more optimistic prior on the benefits of animal welfare charities, and I take your point (as I understand it) that there are benefits that are hard to capture in a CEE, such as movement capacity should we find some great intervention later on, and the maintenance of enthusiastic grassroots support for use as leverage in corporate campaigns. Does ACE have material explicating and justifying its understanding of this systemic/hard-to-quantify value? Is there something else I'm missing about your argument for independence, such that arguments for the value of something like strong leadership don't rely on/flow through arguments for the cost-effectiveness of programs?

ToniA

Hi Adom,

Thanks for your post, and no worries about asking questions we’ve answered elsewhere; we have a lot of research on our website, so we don’t expect anyone to know about all of it!

When I said that we consider each criterion to be an indication of a charity's marginal cost-effectiveness “independently” of the charity's average cost-effectiveness, I meant that—regardless of whether the charity has a high average cost-effectiveness or not—we still consider our six other criteria to be indications of marginal cost-effectiveness. There’s no one or two (or three, or four…) criteria that we think are perfect indications of marginal cost-effectiveness, though we think that all seven of them together are a very good indication. We discuss this a bit in our page on cost-effectiveness estimates, here: https://animalcharityevaluators.org/research/methodology/our-use-of-cost-effectiveness-estimates/

I won’t write more about this right now because we actually have a forthcoming blog post about how we weigh our criteria against each other to make our recommendation decisions. It’s being edited now and then we’ll likely seek external feedback before publishing, so I’d expect it in a month or so.

“It seems to me that a compellingly positive CEE, primary evidence or no, is nonetheless a necessary component in the belief that an organization will improve welfare, particularly if one has a pessimistic prior.”

We think it’s totally possible to make well-reasoned, evidence-based decisions about how to help animals, even in the absence of quantitative CEEs. After all, we don’t even publish quantitative CEEs for some charities that we review (especially if they are working towards long-term or difficult-to-measure outcomes). Take The Good Food Institute, for example. They are one of our Top Charities, but we have not published a quantitative CEE for them. It would be very difficult for us to quantitatively estimate the good they have done so far, since they are working to change the food system in a way that could take years or even decades. Still, we think they have excellent leadership, strong strategy, a healthy culture, we think their programs are likely to have a high long-term impact, and so on. We explain why in their review, and we think we’ve provided a compelling case for donating to them based on their marginal cost-effectiveness.

Regarding your question about “material explicating and justifying [ACE’s] understanding of this systemic/hard-to-quantify value,” we explain some of our thinking about long-term outcomes on the page about our cost-effectiveness estimates, linked above. If you’re asking for explanations of our assessment of the long-term value of particular charities or interventions, that would be in each charity review (mostly discussed in the “high-impact” section with the theories of change) and in our specific intervention reports. For instance, our protest report discusses the importance of movement building.

Hope that helps to answer some of your questions, and watch our blog for the post on our weighing of each criterion!

Best, Toni

[anonymous]

Hi Toni,

Thanks for this detailed response. I don't have much to add over what I said in my post. I have three comments.

Clarificatory question - I calculated the percentage of impact of the different interventions by multiplying the spending on the intervention by the c-e of the intervention. Could you clarify how I should have calculated the impact? Also, I'm not sure I understand how the percentages you give can be right. You say "By our calculations, corporate outreach accounts for about 63% of THL's 2017 CEE and about 36% of Animal Equality's 2017 CEE". But you model the effect of only 3 interventions carried out by Animal Equality - grassroots outreach, investigations, and corporate outreach. But as far as I can see from the CEE, your mean estimate of the effect of grassroots outreach and investigations is negative i.e. they do harm. So, I don't see how they could comprise 64% of the impact of Animal Equality. I would assume that similar things apply to the CEE of THL. Though maybe I have misunderstood
Clarifying my own position - I don't argue that the evidence on the impact of THL and Animal Equality comes from the CEE. My argument is that the arguments and evidence provided for the CEEs, and the views in the intervention reports and the charity reviews is not of the standard we should expect.
Corporate campaigns - it's fine to rely on the open phil report on cage-free welfare, but ACE's research doesn't tell the reader this or provide any acceptable justification for the view that corporate campaigns are beneficial.

ToniA

Hi John!

1- Sure, happy to discuss this further. In the example we gave in footnote 3, we only used the proportional expenditure (PE) to calculate the weighting of each program’s “animal years averted” (AYA) estimate (i.e., weighting for AYA_1 = PE_1/Sum(PE_modelled)). So this gives a weighting that we apply to each AYA estimate, and is independent from the AYA estimate itself. Stopping here is not ideal, however it is not as straightforward to use a similar method for the AYA estimates, due to their distributions.

Including the mean values of the AYA estimates without the rest of the distributions introduces some inconsistencies that make this approach of questionable use. If you consider example 1 in this model, we have two calculations for total AYA. They would be identical if it weren’t for for the distribution of the third AYA. The impacts of the third AYAs would have the same result using your method of calculation, however they clearly impact the model differently (with 3a having a much larger impact on the overall result). In example 2, we have the issue of the mean being very small for one AYA. While the two distributions are of even size and have the same expenditure weighting, an estimate using the mean would attribute 99% of the impact to program 2.

A different way of considering the impact of each part of the model is not to consider the proportional magnitude of each program but to use a sensitivity analysis (Guesstimate has one built in). This tests which parts of the model would have the biggest impact on the final result, should they be adjusted. Running this for both models indicates that the THL model is most sensitive to corporate outreach, while the Animal Equality model fluctuates between corporate and grassroots outreach, depending on how guesstimate populates the model.

2- That’s fair! I agree that we did not sufficiently explain all of the evidence we used in our CEEs, and I agree that our old intervention reports were not of our current standard. You did not state explicitly that the evidence for supporting THL and Animal Equality comes only from their CEEs. However, you seemed to conclude that our reviews provide only weak evidence for supporting each charity simply because our CEEs are weak evidence. My point is just that we provide a lot of other evidence, as well.

3- Agreed—we should have mentioned this! We are trying to do better this year, and we appreciate your insights as our Criterion 3 consultant : )

Best, Toni

Avi Norowitz

Thanks for the informative post, Toni!

With regard to:

We do search online for evidence in the news of each charity's achievements. The problem is: there usually is no such evidence, particularly in the field of corporate outreach. Of course, the absence of evidence of a charity's involvement in a corporate campaign is not evidence that the charity was not involved. We've also looked up corporations' press releases announcing their commitments, but these generally do not mention animal charities. (As far as I can remember, I've never seen one that does.) We have little reason to believe that a corporation could or would share detailed information about their decisions with us if we asked them. I don't know who Mr. Halstead has in mind when he mentions checking with "experts," though we've certainly spoken with many experts in corporate campaigning, if that is what he means.

I've found that the causal role of animal charities in corporate commitments is often supported by publicly available evidence. This evidence generally takes one of two forms:

Some corporations do name animal charities in their press releases. This often occurs when the charity secured the commitment through a cooperative approach, though it also sometimes occurs after a public campaign.
In cases of public campaigns, the timeline of events often provides some evidence of causality. I've found that the following pattern is typical: An animal charity launches a public campaign, leaving historical evidence in the form of a petition, tweets, media coverage, etc. Weeks or months later, the corporation publishes a press release agreeing to the commitment. (For reference, I've found Twitter to be a helpful resource for establishing these timelines.)

For example, here's is a list of corporate commitments that CIWF USA was allegedly involved in for the period from January 2016 to March 2017. (I had originally compiled this back in March 2017.) In 15 of the 22 cases, I found that their causal role was supported by publicly available evidence.

ToniA

Hi Avi,

Thanks for your comment!

I think you’re right that some corporations do name organizations in their press releases, and it seems more likely that groups will be named if they are using a more collaborative approach. For what it's worth, in the paragraph you quoted, I now think that I anchored too heavily on my impression that groups such as THL, Mercy For Animals, and Animal Equality are quite rarely (if ever) named in the news or press releases associated with the welfare policy statements, or in the welfare policy statements themselves. As the majority of the organizations we evaluate usually use a less collaborative approach, I think the paragraph you quote will usually hold for the groups that we evaluate.

Still, even in those cases, I think that you’re also right and there should often still be some indirect evidence available from the timeline. That is, evidence of an organization campaigning at t1 and then, usually a short time later, at t2 evidence of a corporation making a commitment to the related welfare standards. For particularly important commitments we do look at this evidence but for the majority of commitments we don’t.

I think that your comment helps provide some important nuance to this discussion and I have left a link to this comment in the piece itself. Thank you again for the comment!

Best, Toni

Jamie_Harris

Thanks very much for posting this reply. And thanks a lot for all the work ACE does in general. Some clarifications were useful to have, e.g. "The Relationship Between our Intervention Research and our Charity Reviews" - I had felt confused about this when I first looked through the reviews in depth.

Here are some specific comments:

Reviews of existing literature

I agree that the new intervention reports are much better on this front. I'm especially keen on the clear tables summarising existing literature in the protest report. I suspect that there's still room for more depth here, especially since the articles summarized are probably just the most relevant parts of much wider debates within the social movement studies literature. For example, I notice a couple of items by S.A. Soule; although I haven't read the book and analysis you (or whoever wrote the protest report) cite, I have read another article of her's which was partially directed at considering the importance of the "political mediation" and "political opportunity structure" theories for assessing the impact of social movement organizations, and suspect that some of the works you cite might consider similar issues. I think the protest report goes into an appropriate amount of depth, given limited time and resources etc, but I've recently gained the impression that a literature review of social movement impact theory in a broad sense, or more systematic reviews of some of the more specific sub-areas, is a high priority in EAA research. I'd be keen to hear views about how useful this would be, and I'm happy to share more specific thoughts if that would help.

Unclear sources of figures

With some older intervention reports I agree with John Halstead that there are some confusing, unexplained numbers, although I think he exaggerates the extent of this (perhaps unintentionally), since some of the figures are explained. I don't think this needs further comment since, as noted, the new intervention report style is much clearer. My impression was that the Guesstimate models from more recent charity evaluations also had some slightly unexplained figures on there. E.g. THL guesstimate model – “Rough estimate of number of farmed animals spared per dollar THL spent on campaigns” is -52 to 340. Tracking this back through the model takes you to a box which notes "THL did not provide estimates for the number of animals affected by cage-free campaigns they were involved with. We have roughly based this estimate on estimates from other groups active in promoting cage-free policies and have attempted to take into account the greater amount of resources THL dedicates towards this program area." I feel like some explanation of this (perhaps a link to an external Google sheet) might have been helpful? I don't think this is a big issue though. There's also a chance I've just missed something / don't fully understand Guesstimate yet.

General comment on use of CEEs

ACE does make very clear that it only sees CEEs as one part of a charity evaluation. I'd just suggest that, in spite of these warnings, individuals looking at the reports will naturally gravitate towards the CEEs as one of the more tangible/concrete/easily quotable areas of the report. E.g. when I've organised events and created resources for Effective Animal Altruism London, I've quoted some of the CEEs for charities (and pretty much nothing else from the report) to make broad points about the rough ballpark for cost effectiveness of different groups. Given this, it still makes sense to treat the CEEs as more important than some other parts of the report, and to try and be especially rigorous in these sections. So doing things like using a single disputed paper by De Mol et al (2016) (although this example is from the old corporate campaigns intervention report) as a key part of a cost effectiveness analysis seems inadvisable, if it is avoidable.

ToniA

Hi Jamie,

Thanks for those thoughts. I agree that there’s room for more depth in the literature review portion of our intervention reports. We’ve prioritized breadth over depth in our intervention research so far. That’s because there’s usually no existing survey of the literature on a given intervention, and beginning with a survey helps us identify the areas that we’d like to explore more in depth. (We usually identify “questions for further research” at the end of our reports.) I agree that a review of the literature on social movement impact theory would likely be very useful for the movement. I’m not sure whether ACE is the best-positioned group to do that kind of research, but we can certainly consider it!

Regarding the sources of the figures in our CEEs, I agree that this is an area where we can improve. I do think Guesstimate can be a little hard to read, and that might be part of it, but there are also some places where our 2017 CEEs did not include enough information. We are being more careful about this in 2018, and are publishing a separate “CEE metric library” that will explain the figures that crop up in every CEE.

Yes, we’ve definitely noticed that people naturally gravitate towards our CEEs : ) That corporate outreach report will be archived, and we are focusing on improving our research every year.

Best, Toni

Jamie_Harris

Thanks for the reply. Just wanted to note that I agree with ACE's breadth over depth strategy, and that ACE might not be best-placed for a fuller review of social movement impact literature. It's something I'm considering prioritizing doing personally in my work for Sentience Institute.

Elizabeth

How does ACE intend the CEEs to be used, if they're not a major determinant of recommendations?

ToniA

Hi Elizabeth,

Good question! We discuss how we make and use our CEEs on this page: https://animalcharityevaluators.org/research/methodology/our-use-of-cost-effectiveness-estimates/#2

Jc_Mourrat

People at GiveWell state that they base their recommendations on four criteria: evidence of effectiveness, cost-effectiveness, room for more funding, and transparency. For ACE, as you reminded us here, there are seven criteria, including:

"6. Does the charity have strong leadership and a well-developed strategic vision? A charity that meets this criterion has leaders who seem competent and well-respected. The charity’s overall mission puts a strong emphasis on effectively reducing suffering, and the charity responds to new evidence with that goal in mind, revisiting their strategic plan regularly to ensure they stay aligned with that mission.

"7. Does the charity have a healthy culture and a sustainable structure? A charity that meets this criterion is stable and sustainable under ordinary conditions, and seems likely to survive the transition should some of the current leadership move on to other projects. The charity acts responsibly to stakeholders including staff, volunteers, donors, and others in the community. In particular, staff and volunteers develop and grow as advocates due to their relationship with the charity."

I have some questions about this.

1) Would you agree that your evaluation criteria are different from those at GiveWell? If so, do you think that one organization should update its criteria? Or is it that the criteria should be different depending on whether we consider human or non-human animals?

2) W.r. to point 6: if a charity does outstanding work, but happens to not emphasize effectively reducing suffering in its mission statement (e.g. they emphasize a proximal goal which turns out to be useful for the reduction of animal suffering), would that be a reason to downgrade its evaluation?

3) W.r. to point 7: if a charity does outstanding work, but staff and volunteers do not become advocates due to their relationship with the charity, would that be a reason to downgrade its evaluation?

ToniA

Hi Jc,

1- Yes, our criteria are different from GiveWell’s. As John alluded to in his original post, our work is quite different from GiveWell’s in a number of ways. For one thing, there is generally much less evidence available about the cost-effectiveness of animal advocacy interventions than about the cost-effectiveness of direct health interventions. As a result, our models of average cost-effectiveness are much less certain than GiveWell’s, which is one reason why we rely more heavily on other indicators of marginal cost-effectiveness. It’s possible that GiveWell could also benefit from considering some of the other criteria we consider, but I’m not enough of an expert on their work to be comfortable drawing that conclusion.

2- We look for charities that emphasize effectively reducing suffering in their mission statement so that we can be confident that their future activities will still align with that goal. Suppose a charity does outstanding work influencing diet change/meat reduction, but they do it with the goal of improving human health. We would be concerned that such a charity could dramatically shift their activities if something caused their mission to be less aligned with ours (for instance, if new research suggested that meat is good for human health). This concern wouldn’t necessarily prevent us from recommending the charity, but it would factor into our decision.

3- As above, this is a concern that would factor into our decision but it wouldn’t necessarily prevent us from recommending a charity.

Best, Toni

Comments

Curated and popular this week

Cultivating hope: calibrating the expectations for cultivated meat to end factory farming

PabloAMC 🔸·1w ago·Curated 2d ago·22m read

GWWC's 2025 impact evaluation (executive summary)

Aidan Whitfield🔸, Giving What We Can🔸·4d ago·2m read

This post presents the executive summary from Giving What We Can’s impact evaluation for 2025. At the end of this post we share links to more information, including the full report and...

Maybe do the thing you wish CEA would do

alejoacelas 🔸·2d ago·2m read

I used AI to fix transcription errors, rerrarange the ideas, and suggest tweaks to the title and some sentences. Three of the most exciting projects to come out of EA in recent years are, in a vague sense, CEA spinouts: * Kairos is directly a spinout of CEA and now handles most support for university AI safety groups. Basically everyone I've found who knows them is really excited about what they do * NEST is an opinionated ideas-fi...

ACE's Response to John Halstead

ACE's Response to John Halstead

Introduction

Our Position on our Older Intervention Research

The Role of Cost-Effectiveness Estimates in our Work

The Relationship Between the Cost-Effectiveness Estimates and the Overall Views Expressed in our Intervention Reports

The Relationship Between the Cost-Effectiveness Estimates and Overall Views Expressed in our Charity Reviews

The Relationship Between our Intervention Research and our Charity Reviews

Further Thoughts on the Role of Cost-Effectiveness Estimates in our Charity Reviews

Our Plans for ACE's Research Department

Comments on Some Miscellaneous Claims