Introduction
Vetted Causes writes reviews of charities. Thus far, we have published reviews of two charities. In both cases, we did not reach out to the charities before publishing reviews on them. We did consider doing so for our most recent review, but did not due to concerns which we've described below.
We hope that through this post, members of the community can offer solutions that resolve our concerns. Should this happen, we would be more than happy to send reviews to charities before releasing them going forward.
Acknowledging The Potential Benefits of Reaching Out to Charities Before Publishing Reviews on Them
We would like to start by acknowledging some of the potential benefits of reaching out to charities before publishing reviews on them:
- Charities could inform us of public information that we failed to locate.
- Charities could inform us of strong counterarguments that we did not consider.
- Charities may expect us to do this due to social norms, and failing to do so may lead to less productive discourse.
There are other potential benefits as well, which you can find stated and suggested by community members on our previous posts.
Potential Risks of Reaching Out to Charities Before Publishing Reviews About Them
Note: none of the risks we mention are directed at any particular charity. When we begin evaluating any charity, we are concerned about all of these risks.
The two risks we are primarily concerned about relate to biased evidence, and unconscious biases.
Risk 1: Charities could alter, conceal, fabricate and/or destroy evidence to cover their tracks.
Much of the evidence we cite is from charities’ own webpages. Charities have the ability to change their webpages to potentially alter, conceal, and/or destroy evidence that we have cited. In some cases, there is also an incentive for charities to do this.
If we reach out to charities before publishing negative reviews about them, there is a risk that they will take these actions. While it is possible charities will still do this even when we don’t reach out before publishing negative reviews about them, it is riskier for charities to do this since the general public has already seen the review, and people may notice that evidence was changed after the review was released. This serves as deterrence against charities taking these actions.
Currently, there are no known statistics on how often charities alter or remove evidence in response to critical reviews. However, there are many documented cases of evidence tampering in various fields. Additionally, police investigators often do not want an organization under investigation to know they are under investigation for fear they will tamper with evidence to avoid consequences. We believe similar concerns exist when we are investigating a charity and plan to release a negative review.
Prior to releasing our reviews, we have always taken screen recordings of important evidence. This was done in case one of the above scenarios were to ever happen. In the future, we also plan to create internet archives for the important evidence we use. However, we remain concerned that in the case of a dispute, we would be accused of creating fake screen recordings/archives.
If there is a third-party service that is trusted by the community that could verify the accuracy of our screen recordings/archives prior to us showing reviews to charities, we’d be much more open to the idea of showing reviews to charities before releasing them. Please let us know if you’re aware of one.
Risk 2: Unconscious biases from interacting with charity staff.
When we evaluate a charity, we want to evaluate them based on their work, not based on how much we like their employees. Accordingly, we do not want to acquire unconscious biases.
If anyone has solutions to this problem, please let us know below, as it would make us more open to showing reviews to charities before releasing them. We would also like to acknowledge that we may be misunderstanding what people are suggesting when they say they'd like us to show our reviews to the charities before publishing them. If this simply entails sending them an email and nothing more, we are more open to that than having meetings with charity employees to discuss their review.
Other Reasons
Below are additional reasons we have been inclined to not reach out to charities before publishing reviews about them. This section largely reflects our own personal opinions.
Reason 1: Organizations should be held accountable for serious mistakes.
If a company makes a serious mistake that causes significant harm, even though it was a mistake, the company is generally still liable for the harm they caused. Simply correcting the mistake does not erase the harm that was caused. Similarly, if a charity makes misleading claims, it should not be allowed to quietly revise them without being held accountable for both the misinformation itself and the harm it caused.
If we reached out to a charity we have written a negative review about, we would hope they address the mistakes we identified. However, even if they addressed the mistakes, we believe they should still be held accountable for making the mistakes in the first place, and for whatever damage the mistakes caused.
If we gave a charity a negative review before publishing it and they immediately fixed the problems we noted, we think it is likely that once we published our review, the charity would say something like “VettedCauses' review is about problems we have already addressed.” We believe this unreasonably reduces the level of accountability the charity is held to.
If anyone has a solution to this concern, please let us know below, as it would make us more open to showing reviews to charities before releasing them.
Reason 2: Charities should be incentivized to provide sufficient and publicly stated evidence to justify their important publicly stated claims.
It is not acceptable for charities to make public and important claims (such as claims intended to convince people to donate), but not provide sufficient and publicly stated evidence that justifies their important claims.
If a charity has done this, they should not be given the benefit of the doubt, because it is their own fault that there is not sufficient publicly stated evidence to justify their important claims; they had the opportunity to state this evidence but did not. Additionally, giving a charity the benefit of the doubt in this situation incentivizes not publicly stating evidence in situations where sufficient evidence does not exist, since the charity will simply be given the benefit of the doubt.
Note: If a charity is unable to disclose all relevant evidence due to legal or strategic reasons, they should at least acknowledge this fact up front. A simple statement such as: "Unfortunately, we cannot disclose everything here due to legal reasons. Thus, the evidence we have presented is incomplete. We hope you trust that we have properly assessed our full evidence and reached an accurate conclusion" would be far more responsible than presenting the disclosed information as complete.
Further, members of the public should not be expected to reach out to charities to figure out if they have hidden evidence that justifies their claims, since this also incentivizes not providing sufficient publicly stated evidence (charities could make claims based on poor evidence that is hidden, but many people would not know the evidence is poor since it is hidden). The expectation should be for charities to provide sufficient and publicly stated evidence to justify their publicly stated important claims. We are concerned that the norm of reaching out to charities before publishing critiques about them worsens these issues.
We are interested to hear your thoughts about this concern.
Conclusion
We apologize for violating the social norms of the community. It was not our intention to disrespect anyone. We are genuinely concerned about the issues above, and look forward to hearing the community’s thoughts and solutions.
I pretty strongly disagree with the case that you make care and think that you should obviously give charities a heads up and it is bad for everyone's interests if you don't. The crucial reason is that it is very easy to misunderstand things or be missing key context and you want to give them a chance to clarify the situation
Regarding your concerns, just use archive.org or archive.is to make archives or relevant web pages. This is a standard and widely used third-party service, and if the charity changes things secretly but the archived evidence remains, they look bad.
Regarding unconscious biases. I don't actually think this is a big deal, but if you are concerned it's a big deal then you can just email the charity and be like: Here is the document, we will publish it, but you have a chance to comment in this Google doc to point out factual inaccuracies and provide us evidence which we will attempt to take into account. And we will give you the opportunity to prepare a written response that can either be a comments immediately after this goes live or that we will include at the end of our piece. Readers can make up their own mind.
Regarding charities being held accountable for making mistakes, I don't think there's a big difference between the charity being publicly called out for a mistake and then they quickly fix it or the charity fixed it just before it was publicly called out. They still made the mistake and it's still obvious it was fixed because of you
Re your point about charities should be incentivised to provide sufficient public evidence. I think this is an extremely unreasonably high standard. Charities Should try to provide as much evidence as they can, but each person will have different objections and confusions and edge cases and it is just completely impractical to provide enough evidence to address all possible cases an adversarial reviewer might ask about. You can criticise charities for providing significantly insufficient rather than just mildly insufficient evidence. But ultimately there is always going to be the potential for misunderstanding and details that feel important to you that the charity did not predict would be important
Hi Neel, thanks for the reply. Could you clarify what standard you're referring to?
Quoting your post:
It is not acceptable for charities to make public and important claims (such as claims intended to convince people to donate), but not provide sufficient and publicly stated evidence that justifies their important claims.
If a charity has done this, they should not be given the benefit of the doubt, because it is their own fault that there is not sufficient publicly stated evidence to justify their important claims; they had the opportunity to state this evidence but did not. Additionally, giving a charity the benefit of the doubt in this situation incentivizes not publicly stating evidence in situations where sufficient evidence does not exist, since the charity will simply be given the benefit of the doubt.
Note that I interpret this standard as "provide sufficient evidence to support their claims in the eyes of any skeptical outside observer", as that's the role you're placing yourself in here
Thanks for clarifying, Neel, and for giving us the opportunity to clarify our statements. Quoting our original post:
By “sufficient evidence" we did not mean proof eliminating all possible doubt. We meant evidence that would satisfy a reasonable person. For example, a reasonable person thinks there is sufficient evidence that Neil Armstrong landed on the moon, but some people do not.
Additionally, we do not expect charities to provide sufficient evidence for all of their publicly stated claims, just their important ones. For instance, if a charity claims to have provided 10,000 meals to homeless individuals, they should provide sufficient evidence that this actually happened. However, if they announce that their end-of-year party was at Disneyland, no additional evidence is necessary.
This rationale only works if you very carefully calibrate your claims. "Charity X has not provided sufficient, publicly available evidence to substantiate its claim to have provided 10K meals to homeless individuals" is a much different claim than "Charity X didn't feed 10K meals to homeless individuals." The former is about transparency; the latter is about whether the claimed results are true or not. An assertion that lack of transparency is not acceptable does not warrant an inference linking a lack of transparency to the falsity of the claim -- at least not without giving the charity a chance to correct its alleged transparency error ahead of time.
Thanks for the insights Jason, we completely agree.
We are curious if you would you also agree that it is (1) reasonable to assert “Charity X didn't feed 10K meals to homeless individuals” and (2) unreasonable to give Charity X the benefit of the doubt, if:
I think your second strand of evidence is doing almost all the work here. There's a big difference between evidence sufficient to establish a prima facie case, such that a reasonable mind would accept the conclusion if no additional evidence were offered and evidence that so conclusively establishes the truth that it's a waste of time to ask the charity for a response.
As with other considerations, this is a fact-bound consideration. If you're relying on it to justify not giving a specific organization notice and an opportunity to prepare a response for concurrent publication, you can always say that -- and thereby accept the loss in your own credibility if readers later judge that the charity had a reasonable response that it could have provided if asked.
I would include situations in which the charity could establish the substantial truth of the assertion. E.g., if the charity could show that it made a math error and actually fed 9,823 individuals, that is information a reasonable reader would find highly relevant to evaluating the assertion of falsity. Not giving the charity a chance to concurrently explain that does a disservice to both the charity and the reader.
Yeah thi. In particular anytime you criticise an organisation and they are only able to respond a few weeks later, many readers will see your criticism but will not see the organization's response. This inherently will give a misleading impression, so you must be incredibly confident that there is no mitigating context that this organisation would give lest you do their reputation undue damage empirically. I think it is obviously the case empirically that when an organisation gets critiqued including your two previous examples that there is valuable additional context they're able to provide when given notice
Thanks for the insights, Jason.
Could you clarify what you are saying we are relying on?
It's a hypothetical about feeding homeless people, so it doesn't track any of your prior decisions not to give advance notice to the target organization.
In the hypothetical, you could assert that advance notice wasn't warranted because your external evidence was overwhelming enough that providing advance notice and opportunity for comment would have been a waste of time. This would be consistent with my interpretation of the community norm, which is that critics should provide advance notice unless they can (and do) articulate a good cause on a situation-specific basis to dispense with doing so. But if you did provide this justification, and the organization was later able to provide a response that materially changed the conclusions to be drawn, then your organization would suffer a loss in credibility as a result.
Seems pretty clear that he meant your second statement