Hide table of contents

Comment Permalink

I do think there are downsides with sharing draft reviews with organizations ahead of time, but I think they're mostly different from the ones listed here. The biggest risk I see is that the organization could use the time to take an adversarial approach:

Trying to keep the review from being published. This could look like accusations of libel and threats to sue, or other kinds of retaliation ("is publishing this really in the best interest of your career...?").
Preparing people to astroturf the comment section
Preparing a refutation that is seriously flawed but in a way that takes significant effort to investigate. This then risks turning into the opposite of the situation people usually worry about: instead of people seeing a negative review but not the org's follow-up with corrections they might see a negative review and a thorough refutation come out at the same time, and then never see the reviewer's follow-up where they show that the refutation is misleading.

I also think what you list as risk 2, "Unconscious biases from interacting with charity staff", is a real risk. If people at an evaluator have been working with people at a charity, especially if they do this over long periods, they will naturally become more sympathetic. [1]

Of the other listed issues, however, I agree with the other commenters that they're avoidable:

There are many services for archiving web pages, and falsely claiming that archives have been tampered with is a pretty terrible strategy for a charity to take. If you're especially concerned about this, however, you could publish your own archives of your evidence in advance (without checking with the org). The analogy to police is not a good one, because police have the ability get search warrants and learn additional things that are not already public.
If the charity says "VettedCauses' review is about problems we have already addressed" without acknowledging that they fixed the problems in response to your feedback I think that would look quite bad for them. There is risk of dispute over whether they made changes in response to your review or coincidentally, but if you give them a week to review and they claim they just happened to make the changes in that short time between their receiving the draft and you releasing it I think people would be quite skeptical.
On "It is not acceptable for charities to make public and important claims (such as claims intended to convince people to donate), but not provide sufficient and publicly stated evidence that justifies their important claims", I don't think you've weighed how difficult this is. When I read through the funding appeals of even pretty careful and thoughtful charities I basically always notice claims that are not fully backed up by publicly stated evidence. While this does sound bad, organizations have a bunch of competing priorities and justifying their work to this level is rarely worth it.

Overall, I don't think these considerations appreciably change my view that you should run reviews by the orgs they're about.

[1] Charities can also trade access (allowing a more comprehensive evaluation) for more favorable coverage, generally not in an explicit way. I think this is related to why GiveWell and ACE have ended up with a policy that they only release reviews if charities are willing to see them released. This is a lot like access journalism. But this isn't related to whether you share drafts for review.

See in context

Request for Guidance: Reaching Out to Charities Before Publishing Reviews and Our Concerns

by VettedCauses

Feb 215 min read 54

56

Opportunities to take actionCommunityRequests (open)Criticism of effective altruist organizationsCriticism of work in effective altruismEffective altruism cultureTransparency

Frontpage

Request for Guidance: Reaching Out to Charities Before Publishing Reviews and Our Concerns

Introduction

Acknowledging The Potential Benefits of Reaching Out to Charities Before Publishing Reviews on Them

Potential Risks of Reaching Out to Charities Before Publishing Reviews About Them

Risk 1: Charities could alter, conceal, fabricate and/or destroy evidence to cover their tracks.

Risk 2: Unconscious biases from interacting with charity staff.

Other Reasons

Reason 1: Organizations should be held accountable for serious mistakes.

Reason 2: Charities should be incentivized to provide sufficient and publicly stated evidence to justify their important publicly stated claims.

Conclusion

55 comments

Introduction

Vetted Causes writes reviews of charities. Thus far, we have published reviews of two charities. In both cases, we did not reach out to the charities before publishing reviews on them. We did consider doing so for our most recent review, but did not due to concerns which we've described below.

We hope that through this post, members of the community can offer solutions that resolve our concerns. Should this happen, we would be more than happy to send reviews to charities before releasing them going forward.

Acknowledging The Potential Benefits of Reaching Out to Charities Before Publishing Reviews on Them

We would like to start by acknowledging some of the potential benefits of reaching out to charities before publishing reviews on them:

Charities could inform us of public information that we failed to locate.
Charities could inform us of strong counterarguments that we did not consider.
Charities may expect us to do this due to social norms, and failing to do so may lead to less productive discourse.

There are other potential benefits as well, which you can find stated and suggested by community members on our previous posts.

Potential Risks of Reaching Out to Charities Before Publishing Reviews About Them

Note: none of the risks we mention are directed at any particular charity. When we begin evaluating any charity, we are concerned about all of these risks.

The two risks we are primarily concerned about relate to biased evidence, and unconscious biases.

Risk 1: Charities could alter, conceal, fabricate and/or destroy evidence to cover their tracks.

Much of the evidence we cite is from charities’ own webpages. Charities have the ability to change their webpages to potentially alter, conceal, and/or destroy evidence that we have cited. In some cases, there is also an incentive for charities to do this.

If we reach out to charities before publishing negative reviews about them, there is a risk that they will take these actions. While it is possible charities will still do this even when we don’t reach out before publishing negative reviews about them, it is riskier for charities to do this since the general public has already seen the review, and people may notice that evidence was changed after the review was released. This serves as deterrence against charities taking these actions.

Currently, there are no known statistics on how often charities alter or remove evidence in response to critical reviews. However, there are many documented cases of evidence tampering in various fields. Additionally, police investigators often do not want an organization under investigation to know they are under investigation for fear they will tamper with evidence to avoid consequences. We believe similar concerns exist when we are investigating a charity and plan to release a negative review.

Prior to releasing our reviews, we have always taken screen recordings of important evidence. This was done in case one of the above scenarios were to ever happen. In the future, we also plan to create internet archives for the important evidence we use. However, we remain concerned that in the case of a dispute, we would be accused of creating fake screen recordings/archives.

If there is a third-party service that is trusted by the community that could verify the accuracy of our screen recordings/archives prior to us showing reviews to charities, we’d be much more open to the idea of showing reviews to charities before releasing them. Please let us know if you’re aware of one.

Risk 2: Unconscious biases from interacting with charity staff.

When we evaluate a charity, we want to evaluate them based on their work, not based on how much we like their employees. Accordingly, we do not want to acquire unconscious biases.

If anyone has solutions to this problem, please let us know below, as it would make us more open to showing reviews to charities before releasing them. We would also like to acknowledge that we may be misunderstanding what people are suggesting when they say they'd like us to show our reviews to the charities before publishing them. If this simply entails sending them an email and nothing more, we are more open to that than having meetings with charity employees to discuss their review.

Other Reasons

Below are additional reasons we have been inclined to not reach out to charities before publishing reviews about them. This section largely reflects our own personal opinions.

Reason 1: Organizations should be held accountable for serious mistakes.

If a company makes a serious mistake that causes significant harm, even though it was a mistake, the company is generally still liable for the harm they caused. Simply correcting the mistake does not erase the harm that was caused. Similarly, if a charity makes misleading claims, it should not be allowed to quietly revise them without being held accountable for both the misinformation itself and the harm it caused.

If we reached out to a charity we have written a negative review about, we would hope they address the mistakes we identified. However, even if they addressed the mistakes, we believe they should still be held accountable for making the mistakes in the first place, and for whatever damage the mistakes caused.

If we gave a charity a negative review before publishing it and they immediately fixed the problems we noted, we think it is likely that once we published our review, the charity would say something like “VettedCauses' review is about problems we have already addressed.” We believe this unreasonably reduces the level of accountability the charity is held to.

If anyone has a solution to this concern, please let us know below, as it would make us more open to showing reviews to charities before releasing them.

Reason 2: Charities should be incentivized to provide sufficient and publicly stated evidence to justify their important publicly stated claims.

It is not acceptable for charities to make public and important claims (such as claims intended to convince people to donate), but not provide sufficient and publicly stated evidence that justifies their important claims.

If a charity has done this, they should not be given the benefit of the doubt, because it is their own fault that there is not sufficient publicly stated evidence to justify their important claims; they had the opportunity to state this evidence but did not. Additionally, giving a charity the benefit of the doubt in this situation incentivizes not publicly stating evidence in situations where sufficient evidence does not exist, since the charity will simply be given the benefit of the doubt.

Note: If a charity is unable to disclose all relevant evidence due to legal or strategic reasons, they should at least acknowledge this fact up front. A simple statement such as: "Unfortunately, we cannot disclose everything here due to legal reasons. Thus, the evidence we have presented is incomplete. We hope you trust that we have properly assessed our full evidence and reached an accurate conclusion" would be far more responsible than presenting the disclosed information as complete.

Further, members of the public should not be expected to reach out to charities to figure out if they have hidden evidence that justifies their claims, since this also incentivizes not providing sufficient publicly stated evidence (charities could make claims based on poor evidence that is hidden, but many people would not know the evidence is poor since it is hidden). The expectation should be for charities to provide sufficient and publicly stated evidence to justify their publicly stated important claims. We are concerned that the norm of reaching out to charities before publishing critiques about them worsens these issues.

We are interested to hear your thoughts about this concern.

Conclusion

We apologize for violating the social norms of the community. It was not our intention to disrespect anyone. We are genuinely concerned about the issues above, and look forward to hearing the community’s thoughts and solutions.

56 Reactions

More posts like this

Comments55

Sorted by

New & upvoted

Click to highlight new comments since: Today at 1:06 AM

Neel NandaFeb 2176

I pretty strongly disagree with the case that you make care and think that you should obviously give charities a heads up and it is bad for everyone's interests if you don't. The crucial reason is that it is very easy to misunderstand things or be missing key context and you want to give them a chance to clarify the situation

Regarding your concerns, just use archive.org or archive.is to make archives or relevant web pages. This is a standard and widely used third-party service, and if the charity changes things secretly but the archived evidence remains, they look bad.

Regarding unconscious biases. I don't actually think this is a big deal, but if you are concerned it's a big deal then you can just email the charity and be like: Here is the document, we will publish it, but you have a chance to comment in this Google doc to point out factual inaccuracies and provide us evidence which we will attempt to take into account. And we will give you the opportunity to prepare a written response that can either be a comments immediately after this goes live or that we will include at the end of our piece. Readers can make up their own mind.

Regarding charities being held accountable for making mistakes, I don't think there's a big difference between the charity being publicly called out for a mistake and then they quickly fix it or the charity fixed it just before it was publicly called out. They still made the mistake and it's still obvious it was fixed because of you

Re your point about charities should be incentivised to provide sufficient public evidence. I think this is an extremely unreasonably high standard. Charities Should try to provide as much evidence as they can, but each person will have different objections and confusions and edge cases and it is just completely impractical to provide enough evidence to address all possible cases an adversarial reviewer might ask about. You can criticise charities for providing significantly insufficient rather than just mildly insufficient evidence. But ultimately there is always going to be the potential for misunderstanding and details that feel important to you that the charity did not predict would be important

VettedCausesFeb 210

Re your point about charities should be incentivised to provide sufficient public evidence. I think this is an extremely unreasonably high standard.

Hi Neel, thanks for the reply. Could you clarify what standard you're referring to?

Neel NandaFeb 228

Quoting your post:

Note that I interpret this standard as "provide sufficient evidence to support their claims in the eyes of any skeptical outside observer", as that's the role you're placing yourself in here

VettedCausesFeb 22-2

Thanks for clarifying, Neel, and for giving us the opportunity to clarify our statements. Quoting our original post:

It is not acceptable for charities to make public and important claims (such as claims intended to convince people to donate), but not provide sufficient and publicly stated evidence that justifies their important claims.

By “sufficient evidence" we did not mean proof eliminating all possible doubt. We meant evidence that would satisfy a reasonable person. For example, a reasonable person thinks there is sufficient evidence that Neil Armstrong landed on the moon, but some people do not.

Additionally, we do not expect charities to provide sufficient evidence for all of their publicly stated claims, just their important ones. For instance, if a charity claims to have provided 10,000 meals to homeless individuals, they should provide sufficient evidence that this actually happened. However, if they announce that their end-of-year party was at Disneyland, no additional evidence is necessary.

JasonFeb 2221

This rationale only works if you very carefully calibrate your claims. "Charity X has not provided sufficient, publicly available evidence to substantiate its claim to have provided 10K meals to homeless individuals" is a much different claim than "Charity X didn't feed 10K meals to homeless individuals." The former is about transparency; the latter is about whether the claimed results are true or not. An assertion that lack of transparency is not acceptable does not warrant an inference linking a lack of transparency to the falsity of the claim -- at least not without giving the charity a chance to correct its alleged transparency error ahead of time.

VettedCausesFeb 22-2

Thanks for the insights Jason, we completely agree.

We are curious if you would you also agree that it is (1) reasonable to assert “Charity X didn't feed 10K meals to homeless individuals” and (2) unreasonable to give Charity X the benefit of the doubt, if:

"Charity X has not provided sufficient, publicly available evidence to substantiate its claim to have provided 10K meals to homeless individuals" and
There is publicly available and sufficient evidence indicating Charity X did not provide 10K meals to homeless individuals.

JasonFeb 2211

I think your second strand of evidence is doing almost all the work here. There's a big difference between evidence sufficient to establish a prima facie case, such that a reasonable mind would accept the conclusion if no additional evidence were offered and evidence that so conclusively establishes the truth that it's a waste of time to ask the charity for a response.

As with other considerations, this is a fact-bound consideration. If you're relying on it to justify not giving a specific organization notice and an opportunity to prepare a response for concurrent publication, you can always say that -- and thereby accept the loss in your own credibility if readers later judge that the charity had a reasonable response that it could have provided if asked.

I would include situations in which the charity could establish the substantial truth of the assertion. E.g., if the charity could show that it made a math error and actually fed 9,823 individuals, that is information a reasonable reader would find highly relevant to evaluating the assertion of falsity. Not giving the charity a chance to concurrently explain that does a disservice to both the charity and the reader.

Neel NandaFeb 2217

Yeah thi. In particular anytime you criticise an organisation and they are only able to respond a few weeks later, many readers will see your criticism but will not see the organization's response. This inherently will give a misleading impression, so you must be incredibly confident that there is no mitigating context that this organisation would give lest you do their reputation undue damage empirically. I think it is obviously the case empirically that when an organisation gets critiqued including your two previous examples that there is valuable additional context they're able to provide when given notice

VettedCausesFeb 220

Thanks for the insights, Jason.

If you're relying on it to justify not giving a specific organization notice and an opportunity to prepare a response for concurrent publication, you can always say that

Could you clarify what you are saying we are relying on?

JasonFeb 236

It's a hypothetical about feeding homeless people, so it doesn't track any of your prior decisions not to give advance notice to the target organization.

In the hypothetical, you could assert that advance notice wasn't warranted because your external evidence was overwhelming enough that providing advance notice and opportunity for comment would have been a waste of time. This would be consistent with my interpretation of the community norm, which is that critics should provide advance notice unless they can (and do) articulate a good cause on a situation-specific basis to dispense with doing so. But if you did provide this justification, and the organization was later able to provide a response that materially changed the conclusions to be drawn, then your organization would suffer a loss in credibility as a result.

Neel NandaFeb 226

Seems pretty clear that he meant your second statement

There is publicly available and sufficient evidence indicating Charity X did not provide 10K meals to homeless individuals.

Caroline MillsFeb 2361

I'm writing this comment as both a fundraiser and someone who has been involved in leading an organization's participation in an evaluation process. I don't have the knowledge base to engage on whether ACE or Sinergia's claims or evaluations are accurate, but reading your posts makes me worry that you lack some essential understanding of how charities work, which would be greatly improved if you discussed your reviews with charities prior to publishing.

I think you're underestimating how difficult cost-effectiveness estimates are in animal advocacy. The work is highly complex and interdependent, and reliable impact data is not always available. My organization does not currently publish our own cost-effectiveness estimate because we have found it too complex and time consuming to meet the standard of accuracy we anticipate some supporters or potential supporters might hold—essentially out of fear of reviews like yours—and rely on external evaluators like ACE and Rethink Priorities, though their estimates are not really suitable for a general donor audience as they are not widely accessible.

As we have been developing our measurement & evaluation capacity, we are working on creating a cost-effectiveness estimate for our work. Seeing your reviews makes me very hesitant to go through with that process, even though I think overall publishing such an estimate would increase our funding and enable us to spare more animals from suffering. We know that our estimate would be imperfect (different reviewers use different methodologies—there is simply disagreement about what matters and the best methodology) and we would plan to publish our assumptions and work, in line with your expectation that "charities to provide sufficient and publicly stated evidence to justify their publicly stated important claims."

In fact, I think reviewers reaching out to us makes us more likely to publish evidence for our claims, contrary to your Other Reason #1, as we understand the expectation for this and what kind of evidence we might need to collect and publish. On the other hand, anonymous, scathing reviews make us less likely to publish anything at all. In addition, I disagree with the level of seriousness you assign these types of mistakes and the idea that there needs to be some sort of severe public accountability. There is enormous pressure to produce cost-effectiveness estimates because they are such effective fundraising tools, and too make them broadly legible, contrary to what you or an EA reviewer would expect of them. I would conservatively estimate that I've been asked by a potential supporter, meta-charity, or other interested party for a cost-effectiveness estimate at least every month since I started fundraising for an animal advocacy non-profit over 8 years ago. In that time, I've seen plenty of cost-effectiveness estimates be revised, or fall in and out of favor. I'm not defending deliberate misinformation, but without talking to charities I think you have no way of knowing whether those mistakes are deliberate, simply mistakes, or valid philosophical disagreements.

JamesÖz 🔸Feb 2142

IMO the risks you state are much less severe relative to missing key information about a specific charity (as likely happened with your Sinergia work) and therefore misleading people. This also makes people less likely to take your claims seriously in all future reviews.

Risk 2: Unconscious biases from interacting with charity staff.
When we evaluate a charity, we want to evaluate them based on their work, not based on how much we like their employees. Accordingly, we do not want to acquire unconscious biases.
If anyone has solutions to this problem, please let us know below, as it would make us more open to showing reviews to charities before releasing them. We would also like to acknowledge that we may be misunderstanding what people are suggesting when they say they'd like us to show our reviews to the charities before publishing them. If this simply entails sending them an email and nothing more, we are more open to that than having meetings with charity employees to discuss their review.

To clarify, yes I think most people think you should just share a Google Doc of your review and give the organisation time to leave comments about factual inaccuracies or other relevant context. I don't think anyone is suggesting you meet with organisations and discuss things via a call. If anything, it's best to share your review relatively early so you don't spend 100s of hours, as you claim you did with Sinergia, down some rabbit hole which may just be a lack of understanding or context-specific issues.

Lorenzo Buonanno🔸Feb 21*6

I strongly agree that the benefits of sharing the evaluation greatly outweigh the risks, but I'm not sure if sharing the it relatively early is best

There is a risk of starting a draining back-and-forth which could block or massively delay publication. See e.g. Research Deprioritizing External Communication which was delayed by one year
It would cost more time for the org to review a very early draft and point out mistakes that would be fixed anyway
It could cause the org to take the evaluation less seriously, and be less likely to take action based on the feedback

I think the minimal version proposed by @Jason of just sending an advance copy a week or two in advance is an extremely low-cost policy that mitigates most of the risks and provides most of the benefits (but some limited back-and-forth would be ideal)

Aidan AlexanderFeb 2218

From first hand experience, I think critics should make more of an effort to ensure that the charities have actually received your communications and had a chance to review it. When my organization was the subject of a critical post, the email from the critic had landed in my spam, so I only learned of the critique when reading it on the forum.

What's more, as an organization of 2 people, one of which was on leave at the time of the notice, we couldn't have reasonably responded to it without stopping critical work to keep the lights on. The smaller the organization being critiqued is, the less flex capacity they have to respond quickly to these sorts of things, and so they should be given a longer grace period.

JamesÖz 🔸Feb 224

Yep fair enough! That was one bit I wasn’t sure about and can definitely see the downsides of sharing too early. I guess the trade-off I was considering was Vetted Causes’ time spent on the evaluation but definitely think an advance finished version with two weeks notice would be something most groups would be happy with.

John SalterFeb 2135

Updating your website feels extremely low urgency as a charity founder. Often you're facing existential threats, deadlines for funding applications, operational issues that can literally be the difference between life and death, and a million other pressing issues you know you'll never get round to because of the aforementioned.

I'm firmly on the side of sharing a draft.

MichaelStJulesFeb 2129

However, we remain concerned that in the case of a dispute, we would be accused of creating fake screen recordings/archives.
If there is a third-party service that is trusted by the community that could verify the accuracy of our screen recordings/archives prior to us showing reviews to charities, we’d be much more open to the idea of showing reviews to charities before releasing them. Please let us know if you’re aware of one.

https://web.archive.org/ seems good enough to me in most cases?

I think it doesn't do so well for Google Spreadsheets or videos, though:

For example, this archive of ACE's CEA of ALI only includes the first tab of the sheet. I tried seeing if I could archive the other tabs via their separate links, but it just redirects me here, where only the one tab is archived. Maybe there's a way around this, or a better acrhiving service.
AFAIK, videos are not archived at all, either.

I've also used https://archive.ph/. There's a browser plugin Archive Page for it, but I often get nginx errors when I try to archive pages with it. You can archive specific tabs by the tab links, e.g. here, but then it doesn't let you scroll through the sheet, which means you won't be able to access sheet cells you'd have to scroll to see. It also doesn't show how cells are calculated.

You could just save the whole sheet and upload it somewhere with a timestamp, and save archives anyway. Maybe there are better options than web.archive.org and archive.ph.

VettedCausesFeb 213

Hi Michael, thanks for the reply.

https://web.archive.org/ seems good enough to me in most cases?

That archive service is great, we use them all the time. From our understand though, it is actually possible to manipulate web archives.

MichaelStJulesFeb 216

Didn't they already address this specific vulnerability with the measures described on that page?

VettedCausesFeb 213

The aforementioned page states that they took action "to mitigate these attacks," so from our understanding it is still possible to do.

Also, the organization who completed the study still cautions users who rely on Wayback Machine (the archive platform that was manipulated).^[1]

^{^}
https://rewritinghistory.cs.washington.edu/index.html See section "I rely on Wayback Machine -- what should I do?"

Lorenzo Buonanno🔸Feb 2112

The original information is still archived, my understanding is that those attacks just inject other data that changes what is shown to the user, but as they mention it's easily detectable and the original information can still be recovered.

A bigger risk would be that the organization asks the archive to delete their data, but that would look very suspicious, and you could use multiple archives (e.g. https://archive.is/ )

VettedCausesFeb 21-1

Thank you for your reply and technical insights, Lorenzo.

To clarify, we are actually not that concerned about archived documents being manipulated. From what we understand, this is extremely rare.

What we are quite concerned about is that we will be falsely accused of manipulating archives, and the charity accusing us will be given the benefit of the doubt. They could cite articles like the one we cited earlier, and most people do not have the technical expertise to evaluate disputes over archive integrity.

Lorenzo Buonanno🔸Feb 21*8

I think that is extremely unlikely, they have a lot to lose as soon as it's confirmed that the archived data is not manipulated.

Also, from the page you cite:

we emphasize that these attacks can in most cases be launched only by the owners of particular domains.

So they would need to claim that you took control of a relevant domain as well.

But even if something like that happened, you could show that the archive has not been tampered (e.g. by linking the exact resource containing the information, or mentioning the "about this capture" tool that was added by the web archive to mitigate this)

LinchFeb 256

I think that is extremely unlikely, they have a lot to lose as soon as it's confirmed that the archived data is not manipulated.

Not just that, I expect charities to have a lot to lose just from the fight alone, for better or worse. Getting into fights about your integrity generally has negative effects on your reputation and fundraising capacity.

VettedCausesFeb 211

Thanks for the tool! It seems very useful.

they have a lot to lose as soon as it's confirmed that the archived data is not manipulated.

We think our team still has some disagreements with you over how effective disinformation campaigns can be (especially when the disinformation is technical and the audience is mostly non-technical). That being said, we really appreciate your insights—you’ve made some great points.

Neel NandaFeb 2217

I think the typical member of the EA community has more than enough technical skill to understand evidence that a web page has been edited to be different from an archived page, if pointed to both copies from a reliable source

MichaelStJulesFeb 210

My first impression is that these techniques are pretty obscure and technical, and charities would not think to use them or know how to by default. In fact, sharing them here might make it more likely that charities use them (an infohazard).

EDIT: But maybe if motivated and strategic enough, they would find them through online search.

Jeff Kaufman 🔸Feb 23*25

Trying to keep the review from being published. This could look like accusations of libel and threats to sue, or other kinds of retaliation ("is publishing this really in the best interest of your career...?").
Preparing people to astroturf the comment section
Preparing a refutation that is seriously flawed but in a way that takes significant effort to investigate. This then risks turning into the opposite of the situation people usually worry about: instead of people seeing a negative review but not the org's follow-up with corrections they might see a negative review and a thorough refutation come out at the same time, and then never see the reviewer's follow-up where they show that the refutation is misleading.

Of the other listed issues, however, I agree with the other commenters that they're avoidable:

There are many services for archiving web pages, and falsely claiming that archives have been tampered with is a pretty terrible strategy for a charity to take. If you're especially concerned about this, however, you could publish your own archives of your evidence in advance (without checking with the org). The analogy to police is not a good one, because police have the ability get search warrants and learn additional things that are not already public.
If the charity says "VettedCauses' review is about problems we have already addressed" without acknowledging that they fixed the problems in response to your feedback I think that would look quite bad for them. There is risk of dispute over whether they made changes in response to your review or coincidentally, but if you give them a week to review and they claim they just happened to make the changes in that short time between their receiving the draft and you releasing it I think people would be quite skeptical.
On "It is not acceptable for charities to make public and important claims (such as claims intended to convince people to donate), but not provide sufficient and publicly stated evidence that justifies their important claims", I don't think you've weighed how difficult this is. When I read through the funding appeals of even pretty careful and thoughtful charities I basically always notice claims that are not fully backed up by publicly stated evidence. While this does sound bad, organizations have a bunch of competing priorities and justifying their work to this level is rarely worth it.

Overall, I don't think these considerations appreciably change my view that you should run reviews by the orgs they're about.

JasonFeb 2224

Risk 1: Charities could alter, conceal, fabricate and/or destroy evidence to cover their tracks.

While this is possible in the abstract, I don't think it is generally applicable enough to credit without a case-specific application to the facts at hand.

For example: Your most recent post asserts that Singeria has made "false claims" and that this is an "important issue because Sinergia receives millions of dollars in grants from EA organizations." You note specifically that it has been a ACE top charity since 2018 and recently received a $3.3MM grant from Open Phil. I'll allude to that at points in the discussion as an example of what types of reports you might be producing, but emphasize that the discussion is entirely hypothetical.

One problem is that your argument for importance also underscores how difficult and risky for an organization to "alter, fabricate, and/or destroy evidence" in the way you describe. If the charity challenges your work, presumably you'd produce the screenshots. At that point, the charity would have to double down and accuse you of fabricating the screenshots; there's no plausible way to claim "misunderstanding" at that point in the game.

That would be a serious charge, not least because I would expect (and generally hope) that grantmakers would impose an organizational death penalty on an organization that fraudulently claimed that an evaluator's screenshots were forged. Stuff like misattributed credit and poor calculations can be written off as non-malicious error, but deceit about something as concrete as what the organization's website said would be very hard to spin as anything other than career/organization-ending lack of trustworthiness.

In addition to general Internet archives, there are a number of places not under the organization's control where corroborating evidence could potentially be found. And I think it would be very hard for an organization to reduce the risk of information leakage to a level it deemed an acceptable risk:

Presumably, the alleged false claims would appear in fundraising and fundraising-adjacent materials in the possession of third parties, such as Open Phil, ACE, other grantmakers, recipients of fundraising e-mails, etc.
It's likely that references to the false claims would also exist in some form in individual memories and in written discussions under the control of disinterested third parties (e.g., discussions on the Forum, which I believe are regularly scraped by various people). Although this might not be conclusive, it would leave the organization under a cloud of suspicion probably worse than admitting the truth of the original allegations!
There's also the angle that falsely accusing someone of forging screenshots would constitute libel -- and even in the US, the organization isn't likely to have any good defenses other than non-falsity of the accusation. And litigation would bring even more tools to help uncover the truth:
- If a preservation demand were filed quickly enough, third-party service providers (e.g., a webhost, an Internet archive that received takedown requests) may have evidence establishing the attempted cover-up. Some of these might even voluntarily release information without a lawsuit even being filed.
- The organization might have to turn over its own records in discovery that would establish the coverup. Of course, the organization could try to hide documents, but that comes with risks!
- Of course, there might not be litigation . . . but I wouldn't underestimate the risk to the organization. If there were enough suspicion, I think people would fund it if you were able to conditionally commit some funds to it upfront and/or if there were some third-party recollections of the disputed statements. The litigation would be fairly simple and thus cost-effective as a way to determine whether the organization should receive funds.
  Given the difficulties and risk of accusing an evaluator of committing forgery, I'd generally be inclined to give this issue meaningful weight where there was a reasonable basis to believe that:
  * Distribution of the information to third parties was limited enough that the organization could predict that evidence tampering had a high degree of success.
  * The report was so damning that the organization had little to lose from claiming forgery (cf. the Hail Mary pass situation in American football).
  * Organization leadership was irrational enough to claim forgery even when doing so was strongly against the organization's interests and their personal interests.

LinchFeb 2515

Yeah the causal model here is extremely implausible to me. I'm not saying fraud or falsified evidence is rare in charities, on balance I expect it to be slightly higher than numbers I've seen of the economy as a whole (3-5%). But the specific causal model of (summarizing)

charity was reviewed-> charity gets sent evidence from the review -> charity hides the evidence that was previously on the public internet -> they were successful in doing so

just seems extremely implausible to me.

VettedCausesFeb 22-1

Thanks for your reply Jason—great to hear your perspective.

One problem is that your argument for importance also underscores how difficult and risky for an organization to "alter, fabricate, and/or destroy evidence" in the way you describe.

We understand it is risky, but could you clarify why it would be difficult for an organization to "alter, fabricate, and/or destroy evidence?" In many cases, wouldn't it be as simple as changing values in a spreadsheet?

JasonFeb 2211

It's not difficult to try. I believe it would be difficult to succeed (or even not get easily caught) because the evidence exists in too many third-party hands outside the organization's custody and control. Thus, absent special circumstances, it is very unlikely an organization would even try; the difficulty of success would deter them.

(I meant "risk" to refer more to the consequences of being caught -- that it would pose a grave risk to the organization's continued existence, and to the careers of those who attempted the tampering.)

Carolina Galvani - Sinergia AnimalFeb 2618

Hi everyone,

I’m Carolina, International Executive Director of Sinergia Animal.

I want to acknowledge that members of this community have shared this post with us, and we truly appreciate your engagement and interest in our work. A deep commitment to create real change, transparency and honesty have always been central to our approach, and we will address all concerns accordingly.

To clarify in advance, we have never taken credit for pre-existing or non-existent policies, and we will explain this in our response. We always strive to estimate our impact in good faith and will carefully review our methodology based on this feedback to address any concerns, if valid.

This discussion comes at a particularly busy time for us, as we have been attending EA Global while continuing our critical work across eight countries. We appreciate your patience as we prepare a thorough response.

As a best practice, we believe organizations mentioned by others in posts should have the chance to respond before content is published. We take the principle of the right to reply so seriously that we even extend it to companies targeted in our campaigns or enforcement programs. In that spirit, we will share our response with Vetted Causes via the email provided on their website 24 hours (or as much time as Vetted Causes prefers) before publishing it on the Forum.

The EA community has been a vital supporter of our work, and we hope this serves as an constructive opportunity to provide further insight into our efforts and approach.

Best,
Carolina

Vasco Grilo🔸Feb 2115

Thanks for asking for feedback, and for all your reviews! I very much encorage you to reach out to organisations before you publish your reviews. However, I also think reviews of organisations which have not been reviewed by them are much better than no reviews.

Risk 1: Charities could alter, conceal, fabricate and/or destroy evidence to cover their tracks.

I do not recall this having happened with organisations aligned with effective altruism.

Prior to releasing our reviews, we have always taken screen recordings of important evidence. This was done in case one of the above scenarios were to ever happen. In the future, we also plan to create internet archives for the important evidence we use. However, we remain concerned that in the case of a dispute, we would be accused of creating fake screen recordings/archives.

I would be surprised if organisations aligned with effective altruism claimed the above type of evidence had been fabricated by you.

Risk 2: Unconscious biases from interacting with charity staff.

You can try correcting for this by not updating as much towards being positive about the organisation as naively warranted by the feedback you received from people working there. In addition, you can avoid meeting with people from the organisation, restricting yourself to sharing a doc with your review, and discussing it via messages.

Reason 1: Organizations should be held accountable for serious mistakes.

Organisations would be held accountable if risk 1 was mitigated.

Reason 2: Charities should be incentivized to provide sufficient and publicly stated evidence to justify their important publicly stated claims.

I agree with the points you make in this section, although I think sharing the reviews is still worth it overall.

Habryka [Deactivated]Feb 21*19

> Risk 1: Charities could alter, conceal, fabricate and/or destroy evidence to cover their tracks.

I do not recall this having happened with organisations aligned with effective altruism.

(FWIW, it happened with Leverage Research at multiple points in time, with active effort to remove various pieces of evidence from all available web archives. My best guess is it also happened with early CEA while I worked there, because many Leverage members worked at CEA at the time and they considered this relatively common practice. My best guess is you can find many other instances.)

Sarah LevinFeb 2229

At one point CEA released a doctored EAG photo with a "Leverage Research" sign edited to be bizarrely blank. (Archive page with doctored photo, original photo.) I assume this was an effort to bury their Leverage association after the fact.

Jeff Kaufman 🔸Feb 224

To the extent that you update against an org, of currently existing orgs this would be 80k, not CEA. At the time that this happened current CEA and current 80k were both independently managed efforts under the umbrella organization then known as CEA and now known as EV (more).

Separately, I agree this editing was bad, but doing it in the context of a review would be much worse.

Sarah LevinFeb 2213

The current version of CEA employs Julia Wise, your wife. Previously Alexey Guzey sent Wise a draft of a post critical of her superior Will MacAskill and a request for confidentiality. Wise accidentally (or "accidentally") leaked the draft to MacAskill, who then used it to prepare an adversarial public response to the upcoming post rather than to give Guzey feedback ahead of publication as he'd requested. Neither Wise nor MacAskill disclosed this until after the leak was caught because MacAskill publicly responded to parts of the draft which were removed before publication. Wise remains in her role as CEA's community liaison, where she is the point person for confidential information from people who worry that leaks would provoke adversarial action from powerful community insiders.

Jeff Kaufman 🔸Feb 234

I'm confused why you're posting this?

Are you trying to say I should have included some sort of disclosure in my comment? Or trying to give this as an example of the kind of thing VettedCauses is worried about with sharing reviews before publication? Something else?

Sarah LevinFeb 244

You pointed out the lack of staff continuity between the present CEA and the subset of then-CEA-now-EV which posted the doctored image, to argue that their behavior does not reflect on the present CEA, so that we have no particular reason to expect sketchy or adversarial comms from the present CEA.

Your argument about lack of staff continuity is valid as a local counterpoint which carries some weight (IMO not an extreme amount of weight, given the social and institutional links between the different orgs siloed under then-CEA-now-EV, but others might reasonably disagree). Nevertheless I object to your conclusion about present CEA, largely because of a separate incident involving present CEA staff. So, I brought up this other incident to explain why.

It's true that this is also an example of the kind of thing VettedCauses is worried about, but that's not what made me think of it here.

Jeff Kaufman 🔸Feb 248

I object to your conclusion about present CEA

I don't think I gave any conclusion about CEA? I was pointing out that 80k's past actions are primarily evidence about what we should expect from 80k in the future.

I think your comment is still pretty misleading: "CEA released ..." would be much clearer as "80k released ..." or perhaps "80k, at the time a sibling project of CEA, released ...".

separate incident involving present CEA staff

FYI I'm not getting into the separate incident because, as you point out, it involves my partner.

Vasco Grilo🔸Feb 22*4

Thanks, Sarah! I have checked the links, and I agree that is a clear example of faking evidence. I assume this is an exception, and encourage CEA to disclose any similar instance where they have clearly faked evidence.

Habryka [Deactivated]Feb 245

(I think this level of brazenness is an exception, the broader thing has I think occurred many dozens of times. My best guess, though I know of no specific example, is that probably as a result of the FTX stuff, many EA organizations changed websites and made requests to delete references from archives, in order to lower their association with FTX)

pseudonymFeb 245

Which EA organizations do you know have made requests to delete references from archives?

MichaelStJulesFeb 214

Were they successful in getting evidence removed from web archives?

Habryka [Deactivated]Feb 217

Yes, many of my links over the years broke, and I haven't been able to get any working copy.

Jeff Kaufman 🔸Feb 226

That sort of "it's hard to archive things reliably long-term" seems less relevant in the context of a review, where there's a pretty short time between sharing the doc with the charity and making the review public.

Habryka [Deactivated]Feb 244

To be clear, many of my links were to archive.is and archive.org and stuff, and they still broke. I do agree I could have taken full offline copies, and the basic problem here seems overcomable (if requiring at least a small amount of web-development expertise and understanding).

Vasco Grilo🔸Feb 222

Thanks for sharing, Habryka! If VettedCauses reviewed a random organisation recommended by Animal Charity Evaluators, and shared their review before publication, I guess there would only be a 20 % chance they would regret having shared the review specifically due to risk 1. What would be your guess?

[comment deleted]Feb 214

Deleted by abrahamrowe, 02/21/2025

Reason: I decided not to share this.

VettedCausesFeb 213

Thanks for the feedback, Vasco!

Risk 2: Unconscious biases from interacting with charity staff.
You can try correcting for this by not updating as much towards being positive about the organisation as naively warranted by the feedback you received from people working there. In addition, you can avoid meeting with people from the organisation, restricting yourself to sharing a doc with your review, and discussing it via messages.

These are fair points. We agree the risk of bias is likely minimal if this is how you handle it.

Vasco Grilo🔸Feb 224

Thanks!

We agree the risk of bias is likely minimal if this is how you handle it.

I think there is still some significant risk. My reviews of organisations very often become more positive towards the organisations after I integrate their feedback, although I try to update the reviews based on what makes sense to me instead of what appeals to the organisations. I try to counter the selection bias naturally present in the feedback I receive from organisations by actively searching for evidence against them after receiving their feedback, especially if I updated significantly as a result of their feedback. I search for contrary evidence not only in the updated parts of the review, but also elsewhere. In addition, you can ask feedback from people who you think are pessimistic about the organisations or their interventions.

David TFeb 248

Charities removing false claims from their website is usually a good thing that should happen as soon as possible.

The exception to this would be if they are removing them to deny the claim was ever made and attack your credibility, but a mixture of screenshots, archive links and sharing reviews in advance with other trusted third parties who don't have any stake in those companies should be enough to make that approach very unlikely to work.

Frankly it's much lower risk for charities to respond with "we have corrected this. these are our excuses. but thanks anyway" even if its a really bad excuse than try to claim they never said anything

Matt_SharpFeb 215

"Much of the evidence we cite is from charities’ own webpages. Charities have the ability to change their webpages to potentially alter, conceal, and/or destroy evidence that we have cited"

https://web.archive.org/ (aka Wayback Machine) regularly saves old versions of webpages. Maybe select a bunch of charities at random to see how thorough it is? In theory the charities could ask for old versions of their websites to be deleted from the archive, but there's no guarantees the archive would comply with their request and if they did it would look very suspicious if basically every other charity is on there.