Summary
I outline some concerns about ACE’s research. I show that some of ACE’s older research is of low quality, and should be removed from the website - ACE’s new Research Director agrees with this. More importantly, ACE’s research on the impact of corporate campaigns is flawed, and consequently ACE’s research does not provide much reason to believe that their recommended charities actually improve animal welfare. This is not a criticism of ACE’s recommended charities. I conclude with some thoughts about how ACE could improve and note a cause for optimism, as a new Research Director arrives.
***
ACE’s research has been criticised in the past, most notably in a December 2016 blogpost by Harrison Nathan. ACE’s research has improved since then with some of the most serious problems being resolved. For example, their November 2017 leafleting report was of a good standard, and was a major improvement on their previous leafleting report, which was of poor quality. However, I have examined ACE’s research in 2018 and 2017 and believe that it still contains some serious flaws.
Evaluating the impact of animal charities is generally more difficult than evaluating the impact of charities carrying out direct health interventions because evidence is sparse and much hinges on difficult questions about animal sentience. Consequently, ACE’s research team faces a harder problem than GiveWell’s. However, this point notwithstanding, ACE’s research still falls short of what we should expect. The problems concern the failure to remove low quality older research and, more importantly, the reasoning for judgements about the effectiveness of corporate campaigns.
1. Old research
a. Many of ACE’s older intervention reports are of low quality, and should have been removed from ACE’s website.
b. ACE now acknowledges in private that these reports are of low quality and for the most part does not rely on them in its charity reviews, but in many cases ACE fails to inform reader of this fact.
2. Corporate campaigns
a. ACE does not have up to date research of sufficient quality on the welfare effects of corporate campaigns.
b. ACE also does not check whether their recommended charities are genuinely causally responsible for the corporate policy successes that they claim.
The problems I discuss here pertain only to the reviews of The Humane League (THL) and Animal Equality, as I have not had time to look into their research on the Good Food Institute. This piece is in no way a critique of ACE’s recommended charities.
I publish this critique firstly in the hope that it will encourage an improvement in standards at ACE; secondly, that it will encourage external scrutiny of ACE research going forward; and finally that it will provide information relevant to individual donors. In August 2018, ACE appointed a new Research Director, Toni Adleberg, who should not be held responsible for the mistakes discussed here. My interactions with Ms Adleberg and other members of ACE’s current research staff have been very positive, and I am optimistic that there will be improvements in ACE’s research in the future.
Disclosure: I interned at ACE for a few months in 2015. I am currently an unpaid external research consultant for ACE, and have thus far carried out about 2 hours of work for them in this role. ACE reviewed this piece prior to publication. The views expressed here are my own, not those of Founders Pledge.
1. ACE’s view on the impact of their recommended charities
Before we begin, it is useful to distinguish three platforms in which ACE presents its views:
1. Intervention reports
2. Cost-effectiveness analyses
3. Current all-things-considered view expressed in charity reviews
The view expressed in each these three things are often different. For example, the view expressed in the intervention report on investigations is different to the view expressed in the cost-effectiveness analyses of investigations. ACE is also at pains to point out that their cost-effectiveness analyses are only supposed to give a very rough picture of the cost-effectiveness of their charities, and they say that cost-effectiveness analyses play “only a limited role in our overall opinions of which charities and interventions are most effective”.[1]
With this clarified, we can now outline ACE’s current view on THL and Animal Equality have impact. The interventions pursued by these two organisations can be divided into grassroots outreach and corporate outreach. As I am defining the term, grassroots outreach includes things like leafleting, online ads, social media outreach, humane education, and undercover investigations.
Corporate outreach accounts for the majority of the modelled impact of both charities in the cost-effectiveness analyses of THL and Animal Equality: for THL, ~90% of the modelled impact is from corporate outreach, and ~10% from online outreach; and for Animal Equality, >90% of the modelled impact is from corporate outreach. This seems consistent with ACE’s all-things-considered view of which interventions are likely to be impactful. ACE says that “[corporate campaigns] can be highly impactful when implemented thoughtfully”.[2] ACE seems to be pessimistic about grassroots outreach, saying:
“THL works to effect change through several different kinds of outreach, including leafleting, online ads, and education. While there is little evidence available about the effectiveness of these interventions, we do not currently recommend the use of leafleting or online ads as we suspect that they are not as effective as some other means of public outreach.”[3]
Thus, ACE’s view as of August 2018 is that grassroots advocacy has close to no effect, though ACE does estimate that THL’s online outreach is beneficial. I discuss this in section 2b.
2. Grassroots outreach
In this section, I discuss ACE’s research on the impact of grassroots outreach. Due to lack of data on the impact of most forms of grassroots outreach, in their cost-effectiveness analyses, ACE estimates the impact of most forms of grassroots outreach – humane education, investigations and Animal Equality’s i360 video outreach – by assuming that they were X times as effective as leafleting. For example, we would, the argument goes, intuitively expect humane education to be ~5 times as cost-effective as leafleting, and leafleting spares X animals from factory farming, so humane education spares ~5x animals from factory farming.
Prior to November 2017, ACE believed that leafleting produced fairly substantial benefits. However, in November 2017, ACE published a new leafleting report, stating that their view on the impact of leafleting had changed. They now believe that leafleting has tiny impact, and is in fact very slightly harmful. However, ACE continues to infer the impact of other forms of grassroots outreach from the impact of leafleting, using a scaling factor. Thus, because they judge humane education to be ~5 times as effective as leafleting and they estimate leafleting to be harmful, they now estimate that the mean effect of humane education is 5x as harmful as leafleting. It is not clear that continuing to use a scaling factor makes sense when the sign of the effect has changed, but that is perhaps open to debate. They continue to use this approach for modelling the impact of humane education, investigations, and Animal Equality’s video outreach.
Setting the specifics of the cost-effectiveness analysis to one side, as mentioned, ACE’s apparent current all-things-considered view is that all forms of grassroots outreach (with the possible exception of online ads) have close to zero effect. I agree that grassroots outreach is likely to have very small effects because securing dietary change through advocacy appears to be very difficult, with the best data suggesting that the percentage of veg*ns has barely increased since the 1990s, as discussed in this excellent recent ACE blog.
2a. Investigations
Of THL and Animal Equality, only Animal Equality carries out investigations. The Feb 2016 intervention report on investigations on ACE’s website evaluates the impact of investigations in a different way to the cost-effectiveness analyses of Animal Equality, and produces a different estimate of their impact. The quality of the intervention report is low. One major concern is that the impact calculations are done in a table, rather than in a spreadsheet or Guesstimate model, making it difficult to understand important inputs, calculations and outputs.
Direct suffering avoided per intervention unit
An ‘intervention unit’ is the measure of the length of an intervention. For investigations, 1 intervention unit is 1 investigation. A unit of suffering is a year of farmed captivity, or equivalent, averted.
ACE provides pessimistic and realistic (but not optimistic) estimates of ‘direct suffering per intervention unit’. No explanation is given for these figures and they are actually the same as the ‘indirect suffering avoided per intervention unit’. Assuming that direct and indirect suffering averted are intended to be different, this suggests a mistake that should have been noticed in a review of the page. Here is a screenshot (I have added the column labels from a separate screenshot); these numbers appear without any explanatory calculations.
(Screenshot: 15th August 2018, about 95% of the way down the page)
Indirect suffering avoided per intervention unit
This is a screenshot of the relevant part of the table on ‘indirect suffering avoided per intervention unit’ (again, I have added the column labels from a different screenshot further up the page):
(Screenshot: 15th August 2018, about 95% of the way down the page)
There are three problems with this. Firstly, ACE estimates the reach of a campaign stating:
‘A person does not have to be directly contacted by a staff member in order to be “reached”. They must merely encounter the campaign in some capacity, including living under a legal jurisdiction being targeted by a legislative campaign.’
The condition specified in the second sentence is clearly inappropriate: obviously, not everyone who lives in a legal jurisdiction targeted by a campaign should count as being reached by the campaign.
Secondly, the optimistic estimate cannot be correct and is therefore not a reasonable rendering of ‘optimistic’. It implies that each undercover investigation makes 2.8% of those reached vegetarian or vegan. If, as estimated in the Feb 2016 intervention report, each campaign will reach ~9m people, this would imply that each campaign will create ~250,000 new vegetarians or vegans. Since Animal Equality runs 20 investigations a year,[4] this would imply that they alone will create ~5m new vegetarians or vegans per year exclusively through their undercover investigations. It is somewhat difficult to estimate how many vegetarians there are in the US because accurately tracking dietary habits is challenging and people tend to lie about their animal product consumption. While up to 6% of people in the US self-identify as vegetarian or vegan, but studies that ask people whether they ate meat or fish on two consecutive 24 hour periods show that the vegetarianism rate remains below 1.5%. See this ACE blog for more. Even on the higher estimate based on self-identification, the estimate of the impact of investigations cannot be correct.
Thirdly, no explanation is provided for the pessimistic or realistic estimates; as you can see in the above screenshot, headline figures appear without any calculations. This is concerning. When I asked ACE about this in 2017, they said that the basis for the figure was another page on the impacts of media coverage on meat demand, which is not referenced at that point on the cost-effectiveness analysis of undercover investigations.[5]
I pointed all of this out to ACE in September 2017, and they acknowledged that this was a mistake, but the page has still not been altered as of August 2018. I think this page should be immediately removed. In conversation with ACE’s new research team, they have told me that they were already considering removing these old intervention reports before I sent them this report, which is an encouraging sign.[6]
2b. Online ads
As I mentioned above, in ACE’s cost-effectiveness analysis, online ads are responsible for around 10% of THL’s impact. There are two main problems with ACE’s treatment of online ads.
Firstly, it models the effect of online ads in a different way to how it models other similar forms of grassroots outreach. As discussed above, ACE uses the ‘leafleting scaling factor’ approach for all other forms of grassroots outreach. It does not use this approach for online ads, despite the fact that it seems that whatever rationale justifies using the ‘leafleting scaling factor’ approach for other forms of grassroots outreach must also apply to online ads. For example, Animal Equality’s i360 video outreach seems functionally similar to online ads, and yet a different approach is taken for each.
Secondly, in its cost-effectiveness analysis of THL’s online ads, ACE relies on an August 2016 intervention report on online ads. This report is highly opaque and it is very unclear how ACE arrived at certain crucial figures, such as number of vegetarians created per click on an online ad. Here is the crucial section in the cost-effectiveness estimate of online ads:
Source: intervention report, online ads (as of 10th August 2018)
The reasoning presented for these numbers is limited, and because the numbers are presented in a table rather than a spreadsheet, it is not clear how they are combined together.
When I asked about this in October 2017, ACE said the researcher who made the calculation has left and that the factors bearing on it were not weighted in a formal way that could be publicly explained.[7] Indeed, in the comprehensive review of Animal Equality, ACE says “There is little evidence, thus far, about the effectiveness of online outreach...”[8]
In summary, ACE has used figures which it cannot publicly explain and which it now rejects, to estimate the impact of one of its recommended charities
3. Corporate campaigns
In their cost-effectiveness analyses, ACE estimates that corporate campaigns account for the vast majority of the impact of their recommended charities. It also appears to be ACE’s all-things-considered view in its charity reviews that corporate campaigns are one of the highest impact interventions. For example, in its review of THL, ACE says of interventions aiming to influence industry “We find that these interventions can be highly impactful when implemented thoughtfully”.[9]
However, ACE’s research on corporate campaigns is flawed in four ways. These flaws are the most important ones I have found, given how important corporate campaigns are in ACE’s current research.
- In the November 2014 intervention report on corporate campaigns, ACE did not carry out a comprehensive literature review of the evidence on the welfare effects of cage-free systems. They relied only on a paper by De Mol et al (2006), which seems quite badly flawed, as discussed in a report by the Open Philanthropy Project published in September 2017 (specific section of the report on De Mol et al (2006)).[10] The evidence on the welfare benefits of cage-free systems is much more unclear than suggested in ACE’s assessment of corporate campaigns.
- ACE no longer endorses the conclusions in its intervention report on corporate campaigns, but does not inform the reader of this on the intervention report page or anywhere in the reviews of its recommended charities. The intervention report on corporate campaigns apparently plays no role in the cost-effectiveness estimate of corporate campaigns or in the charity reviews, but this is not explained to the reader. This leaves a major justificatory gap in ACE’s research. In its cost-effectiveness analyses, ACE estimates that their mean estimate of the “proportional improvement in welfare due to cage-free policies” is ~0.05,[11] but provides only a one sentence explanation for this estimate. This crucial figure should be defended with multiple page report and review of the science on the welfare effects of cages.
- ACE have been aware of the problems with the De Mol et al (2006) paper at least since September 2017, and told me in October 2017 that revising the corporate campaigns intervention report was a high priority for the coming year.[12] As of August 2018, this report has still not been updated and ACE have told me that it will very likely be updated in early 2019.[13] The report has now not been updated for almost four years. This progress seems much too slow given how important corporate campaigns are in ACE’s evaluations, and according to conventional wisdom in effective animal advocacy.
- In its review of THL’s and Animal Equality’s corporate outreach, ACE relies only on the charities’ self-reported corporate policy successes, which it then discounts by an arbitrary uncertainty factor: ~0.4 for both Animal Equality and THL. ACE does not check with third party news sources, experts or with the companies themselves on whether the claims of the charities are accurate.[14] This seems to me like basic due diligence that one should carry out when assessing corporate advocacy.
Conclusion
ACE’s research has improved somewhat over the last year or so, but serious problems remain. I believe that at present, donors interested in animal welfare would learn little from ACE’s research. Some of the problems (those regarding grassroots outreach) chiefly concern the failure to remove older poor quality research and to adequately communicate their views to consumers of their research. Other problems (those regarding corporate outreach) are highly relevant to crucial judgements about whether ACE’s recommended charities have any impact at all, or are even harmful (which is possible on some apparently reasonable views on the effects of cage-free systems).
1. Grassroots outreach
a. ACE believes that grassroots outreach has small to zero impact, which seems right.
b. However, ACE’s intervention reports on many forms of grassroots outreach are of poor quality and should have been removed from ACE’s website.
c. ACE does not communicate their current view of the quality of some of these reports, nor does it explain that the conclusions therein diverge from their current view.
2. Corporate outreach
a. Corporate outreach accounts for most of the recommended charities’ impact in ACE cost-effectiveness analyses and in their all-things-considered view of the impact of their charities.
b. In spite of that, ACE still does not have a high quality up to date review of corporate outreach or of the welfare effects cage-free systems.
c. ACE does not independently check the claims of charities about their corporate policy successes.
To be clear, I do not think that these problems can be explained by the fact that cost-effectiveness analyses for animal interventions are extremely uncertain and should not be taken overly literally. The main problems pertain to research questions absolutely crucial to figuring out whether animal charities are beneficial or harmful, which have not been adequately answered by ACE.
As I mentioned at the start of this brief, ACE research has improved over the last two years, and I am optimistic that with the arrival of their new Research Director, Toni Adleberg, ACE’s research will improve at a faster pace in the future. Going forward, I think the main improvements would come from:
1. Removing the older lower quality intervention reports from the ACE website. As I have mentioned, it is a positive sign that ACE were already considering doing this before I sent them this report.
2. Producing a report on the welfare effects of corporate campaigns and independently checking charities’ claimed corporate policy successes.
3. Regularly updating intervention reports and charity reviews to reflect ACE’s current views, rather than updating them wholesale every >1 year.
I should also add that I do not think it would be wise for ACE to rely less on quantification in cost-effectiveness analyses in the future. Quantifying the impact of animal charities is very difficult, but without quantifying impact, comparisons between charities become much more difficult.
[1] See criterion 3 of the review of THL here - https://animalcharityevaluators.org/charity-review/the-humane-league/#c3
[2] See the ‘influencing industry’ section under Criterion 2 here - https://animalcharityevaluators.org/charity-review/the-humane-league/#c2
[3] See the ‘influencing public opinion’ section under Criterion 2 here - https://animalcharityevaluators.org/charity-review/the-humane-league/#c2
[4] See ACE’s Animal Equality cost-effectiveness analysis.
[5] Conversation with Allison Smith, former Research Director at ACE, 12th October 2017.
[6] Conversation with Toni Adleberg and Jamie Spurgeon, 28th August 2018.
[7] Conversation with Allison Smith of ACE, 12th October 2017.
[9] See the ‘influencing industry’ section under Criterion 2 here - https://animalcharityevaluators.org/charity-review/the-humane-league/#c2
[10] Rudi M. De Mol et al., “A Computer Model for Welfare Assessment of Poultry Production Systems for Laying Hens,” NJAS Wageningen Journal of Life Sciences 54, no. 2 (October 25, 2006): 157–68.
[11] See the cost-effectiveness analysis of THL at https://www.getguesstimate.com/models/9604?token=CkrbzlMlrvv6XJOuZ3wmLJwtvnkJoD2aIY79V6DuAe3Wk5t0WP-q4O5Eu9eDuL-jzqz5w1I_YARjtTakfkWesA
[12] Conversation with Allison Smith of ACE, 12th October 2017.
[13] Email correspondence with Jamie Spurgeon of ACE, July 17th 2018.
[14] Email correspondence with Jamie Spurgeon of ACE, July 17th 2018.
Thanks, John! I've posted a response on behalf of ACE here: http://effective-altruism.com/ea/1sq/response_to_john_halstead/.
I'm grateful to see this dialogue being had so respectfully and am grateful to both sides for this dialogue.
I had arrived at similar conclusions. Lately we were busy preparing material to put up on our freshly minted website at EA France. When it comes to recommendations, we naturally turn to charities such as GiveWell or ACE and refer to their work. But before putting these recommendations onto our website, I wanted to double-check that the evaluations are sound, because I want that we be ready to back them up publicly with confidence. So I tried to locate contradicting voices. For GiveWell, I found out that they disagree with other organizations such as the Campbell institute or Cochrane on the effectiveness of deworming programs. So I spent quite a bit of time reading the arguments of each party, and after that I came out extremely impressed by the depth and seriousness of the analysis of GiveWell. Of course I did not check all of what they do, but this experience gave me very high confidence that they are doing an outstanding work.
Then I moved to ACE. I took Nathan's article as a starting point for the disagreeing voice. Of course I was appalled by some of the points raised there, in particular in relation with leafletting. Also, a friend at AEF dig up a Facebook thread that happened around the time of publication of this article, and my recollection is that half of the people discussing this where just busy explaining that Nathan was really a very very mean person that we could not possibly imagine talking to.
I understand that this is old news, but I want to pause for a moment and reflect on the bigger picture. On human poverty, GiveWell is one among several very serious actors. It engages very thoroughly in discussions and explanations when diverging views emerge. We can argue about whether Nathan was diplomatic enough, etc, certainly he did not take all the precautions that Halstead has taken when writing this piece. But we have to realize that when it comes to animal suffering, as far as I know ACE is the only game in town. In my opinion, this is a precarious state of affairs, and we should do our best to protect criticism of ACE, even when it does not come with the highest level of politeness. Of course, I do not mean that people at ACE have bad intentions, but checks and balances are important, we are all human beings, and right now there seems to be precious little of these.
And as it turns out, as pointed out here by Halstead, at least some of the criticism of Nathan was actually correct, and is now acknowledged on the website of ACE (e.g. on leafletting).
And then I must stay that I fell off of my chair when I looked up the evaluation of corporate outreach and found out that the single argument in support of cage-free reform was this paper by De Mol et al (2016). To give an element of context, I have no previous exposure to animal wellfare, and yet it jumped at me that this was very bogus. How can this possibly happen?? I know that now ACE changed their position about this, but how they could come up with that in the first place, and how on Earth can this argument be still available online completely eludes me. All this while, as far as I can tell, corporate outreach is one of the flagship interventions advocated by ACE.
But again, I want to pause and think about the bigger picture for a while. The fact is that at the time of writing this argument, the organisation Direct Action Everywhere (DxE) had put up a rather comprehensive report explaining that they had come up with the opposite conclusion! (That cage-free reform is actually detrimental to animal wellfare.) I will refrain from discussing it at length here because this comment is already long, but this report of DxE was, in my opinion, dismissed with precious little good argument.
So again I see a small dissenting voice in the otherwise rather monopolistic position of ACE which is being dismissed without due consideration. And of course I get more worried. (To be perfectly clear, my point has nothing to do with whether DxE's conclusions were right or not; only with the fact that they were dismissed without proper consideration.)
I know that ACE no longer considers the article of De Mol et al (2016) as relevant, and things are clearly moving in a positive direction. Yet my confidence in ACE's work is at an all-time low. For full disclosure, I was planning to propose to my colleagues at AEF that we spend some time doing a "case study" in relation with the recommendations of ACE (similarly to studying the controversy about deworming for GiveWell). The "case study" I have in mind is the comparative evaluations of the Good Food Institute vs. New Harvest (As a side remark, I am interested in any previous discussion about this point.)
To sum up, I want to stress that I write this with the best intentions, and I appreciate that ACE has been improving a lot on all the points I have raised so far. Also, I understand that we cannot "go meta" and ask for evaluators of evaluators, evaluators of evaluators of evaluators, etc. Yet it is my impression that when it comes to human poverty, the situation is much, much healthier. In this area, there are several influential actors that have their own decision processes, can then compare their conclusions, and engage in serious debate about them. For animal suffering, I think it would do us a lot of good to make sure that dissenting voices are heard and protected, and that ACE engages with their arguments with much greater consideration than has been the case so far.
It sounds like AEF is doing a fantastic job of ensuring rigour in its messaging!
I think in cases where there is little primary research, it's all the more important to ensure that discourse remain not merely polite, but friendly and kind. Research isn't easy at the best of times, and the animal space has a number of factors making it harder than others like global poverty (eg historic neglect and the difficulty of understanding experiences unlike our own). In cases like this where people are pushing ahead despite difficulty, it is all the more important to make sure that the work is actively appreciated, and at baseline that people do not end up feeling attacked simply for doing it. Criticisms that are framed badly can easily be worse than nothing, in leading those working in this area to think that their work isn't useful and they should leave the area, and by dissuading others from joining the area in the first place.
This makes me all the more grateful to John for being so thoughtful in his feedback - suggesting improvements directly to ACE in the first instance, running a public piece by them before publishing, and for highlighting reasons for being optimistic as well as potential problems.
The problem with HN's article wasn't just that it was "impolite" but that it mixed in a number of unfounded ad hominem attacks along with the more serious criticisms, arguments that implied people at ACE and other organisations were deliberately acting in bad faith. It seems to me that the proper way to respond to a mix of good arguments and bad arguments is to take the good arguments into account while dismissing the bad ones, and that seems to be what happened.
And just as a broader point, if someone regularly mixes ad hominem attacks with more serious points, it's not a good idea to signal boost the mixed arguments. If I say, "so-and-so is a liar and a thief who only cares about self-promotion, and also so-and-so incorrectly reported a particular study," the best response is to take the valid point about the study into consideration without promoting the argument as a whole.
Moreover, the methodological arguments put forward weren't entirely new; HL Lab director Harish Sethu's presentation at the EAA conference in Princeton, for example, had extremely detailed methodological criticisms that far surpassed the other criticisms I had seen to that point.
In other words, there's not a binary decision between ignoring the points altogether and praising the deeply flawed HN article. The good methodological points are being taken into consideration, as seen in some of the changes made and in this essay.
The diverging views in the case of the GiveWell example you gave are from respected research organizations Campbell and Cochrane, with all parties arguing in good faith. This was very different from the case of Nathan's criticisms of ACE.
But ACE did reply to Nathan Harrison's criticisms:
https://animalcharityevaluators.org/blog/responses-to-common-critiques/ https://animalcharityevaluators.org/blog/response-to-a-recent-critique-of-our-research/
My understanding is that ACE did in fact take DxE's arguments into consideration, and that their relatively pessimistic estimate that cage-free represents a ~5% improvement is informed by many different views, including DxE's arguments that cage-free is harmful. (This is from conversations I've had and I'm not sure if ACE has published this. I agree it would be helpful if ACE published their reasons for this estimate.)
I'm not sure how you define "the only game in town." There are currently a number of other organizations who do research on effective animal advocacy, including Open Philanthropy, Sentience Institute, Rethink Charity, Faunalytics, Humane Society of the United States, Humane League Labs, Animal Welfare Action Lab, Wild Animal Suffering Research, etc.
The fact of the matter is that ACE exists since 2013, that corporate outreach is central to its strategy, and that Nathan, DxE, and now Halstead have serious reservations about how the impact of this intervention is assessed. Moreover, it seems to be agreed upon that past mistakes have been substantial and yet have stayed online way too long. So I'm not sure if I found the best way to express it, and I surely wasted a lot of time on material that is regarded as outdated, but I think the point remains that ACE's research has not been sufficiently "tested". I am very happy to see all the progress that has been made and is to be coming at ACE. But I also hope with Halstead that these observations "will encourage external scrutiny of ACE research going forward".
Wouldn't referring to other groups likely confirm that it is the only game in town? If they were working on similar issues then there would be cross referencing and a greater degree of accountability. But it seems that hasn't happened at least in some cases and it may or may not be the case there are further issues to be examined elsewhere. In my view there are around moral theory (particularly managing more polarising issues), whilst i would disagree with Jc that meta evaluation isn't useful. Likely it would provide some useful information to consider in one sweep if other organisations aren't doing that work or people too time constrained or just willing to trust in the process. I think it would at least be worthwhile seeing whether it has value in this context and it could also give people more confidence in the process.
I started out with a negative impression of ACE when I found it years ago. Since then, I've seen substantial improvements in their research quality, substantial willingness to update to new evidence, and substantial willingness to publicly state unpopular conclusions (e.g. leafletting has a slightly negative expected value). I was also impressed with the conference they ran in 2016. My overall impression is now positive, and I appreciate their contributions. I'd also suggest putting ACE in context: GiveWell, which generally has a positive reputation, also got off to a rough start.
I agree that ACE still has room to improve, and I appreciate that you have provided constructive feedback in good faith.
With regard to:
I agree that ACE should provide more justification for this estimate, but I think there are a few points worth noting:
This is a fairly pessimistic estimate, far more pessimistic than a reasonable reading of De Mol et al 2006. (Of course a negative estimate would be even more pessimistic.)
GiveWell also makes some subjective judgements in their cost-effectiveness estimates that are not supported by comprehensive literature reviews, especially regarding moral weights.
With regard to:
I assume you're referring to the metric "THL's responsibility for changes"? My understanding that this is mostly supposed to reflect the case that it's often the case that multiple charities are involved in securing a given corporate commitment, making it incorrect to assign 100% of the effectiveness to just one charity.
With regard to:
I agree that ACE should do this, but I predict most of the claims would withstand this scrutiny. As I mention here, I found that 15 out of the 22 corporate commitments that CIWF USA was allegedly involved in from January 2016 to March 2017 had some publicly available evidence to support their causal role.
With regard to grassroots outreach, it's worth noting that a large amount of THL's grassroots work in the past few years has been in coordinating in person protests against food corporations. This differs quite a bit from activities like leafletting, so it's understandable that evidence on the effectiveness of leafletting may not be the most relevant consideration to an evaluation of THL's grassroots work. (ACE has published an intervention report on protests earlier this year, which I haven't read yet.)
I think the challenges inherent in evaluating animal charities go beyond even the difficult questions of animal sentience. Unlike in the case of direct health interventions, the impact of almost all of the activities by animal charities depends on extremely complicated facts about human psychology. Asking "how likely is leafletting to change behaviour," is more akin to asking "how likely is leafletting to change someone from a Republican to a Democrat" than it is to asking "how likely are bed nets to prevent malaria?"
Just a few of the complications of predicting long-term behaviour change through specific interventions:
-Unlike certain health risks which occur in specific types of situations, human psychology is extremely complicated and could in principle be influenced by events/conversations/memory recollections taking place at almost any moment throughout one's life.
-There are individual differences in how likely people are to be persuaded by particular tactics. So if only 2.5% of the population are even in principle persuadable by leaflets, and you happen to reach those 2.5% in an initial campaign (and, with perfect measurement, record the change), the prediction that a similar future campaign would cause change in 2.5% of the population would be completely wrong (because the 2.5% of reachable people would have already been reached). -Unlike in the case of health interventions, changing behaviour in relation to animals is taking place in direct opposition to a hugely powerful set of industries that are themselves spending millions of dollars trying to produce the exact opposite behaviour changes. So even if we discovered the holy grail intervention X that caused 100% of people to go vegan, Tyson foods would be busily working on messaging to convince people to eat more meat.
I'm not sure what the upshot of all of this is, but it does seem to me that influencing change in relation to animals is much more like running a political campaign. And people have been studying political tactics for thousands of years without developing any type of infallible playbook for winning. So while I think measuring animal charity effectiveness is extremely important and interesting, it also seems like it's a mistake to think that it could ever come close to reaching the type of reliability seen in other types of interventions.