Hide table of contents

In relevance to ongoing outreach work in EA Israel, an upcoming Charity Effectiveness Prize, and our local charity-evaluation work, I've become interested in how does the evaluation process affects charities who are under evaluation. 

I can (sort of) quantify and understand the direct impact of the recommendation on the top charities. I can also (sort of) imagine the kind of impact the recommendation process has on popularizing cost-effectiveness (although I'd love to read a detailed report on the topic).

What I'd like to understand better at the moment are the generally-framed questions around how does more evidence lead to higher performance: 

  1. How do self-reflecting charities adapt to new evidence? GiveDirectly, for example, performs a lot of RCTs on direct cash transfers. I'd be very interested in examples of GiveDirectly and other charities making strategic or practical changes due to new evidence, and examples of randomized trials or other analyses being performed to make high-level decisions. Are there examples of nonprofits that completely started over when their interventions proved ineffective? I'd also be interested to learn more about what charities recommended by evaluation orgs learn from the evaluation process itself and whether this helps them improve (or cause harm).
  2. How are charities that aren't empirically grounded impacted from doing charity evaluation? Most charities, unfortunately, are not performing RCTs or otherwise invest in gathering evidence about the impact of their actions. Generally speaking, how do such charities respond to evidence or claims of (in)effectiveness? How reasonable is it to expect charities without a strong self-evaluation history to improve as a result of performing an analysis later on? Are there examples where organizations like GiveWell and ACE successfully helped improve the performance of investigated charities?

26

0
0

Reactions

0
0
New Answer
New Comment


1 Answers sorted by

I can only address one of your points from question 1. Evidence Action has abandoned and completely started at least one of their interventions which proved to be ineffective in light of new evidence. If I remember correctly, it was in Bangladesh.

Thank you!

I've searched and found this post describing it. The summary:

Evidence Action is terminating the No Lean Season program, which was designed to increase household food consumption and income by providing travel subsidies for seasonal migration by poor rural laborers in Bangladesh, and was based on multiple rounds of rigorous research showing positive effects of the intervention. This is an important decision for Evidence Action, and we want to share the rationale behind it.  

Two factors led to this, including the disappointing 2017 evidence on

... (read more)

Thank you. Yes, this is exactly what I referring to.

Curated and popular this week
 ·  · 5m read
 · 
[Cross-posted from my Substack here] If you spend time with people trying to change the world, you’ll come to an interesting conundrum: Various advocacy groups reference previous successful social movements as to why their chosen strategy is the most important one. Yet, these groups often follow wildly different strategies from each other to achieve social change. So, which one of them is right? The answer is all of them and none of them. This is because many people use research and historical movements to justify their pre-existing beliefs about how social change happens. Simply, you can find a case study to fit most plausible theories of how social change happens. For example, the groups might say: * Repeated nonviolent disruption is the key to social change, citing the Freedom Riders from the civil rights Movement or Act Up! from the gay rights movement. * Technological progress is what drives improvements in the human condition if you consider the development of the contraceptive pill funded by Katharine McCormick. * Organising and base-building is how change happens, as inspired by Ella Baker, the NAACP or Cesar Chavez from the United Workers Movement. * Insider advocacy is the real secret of social movements – look no further than how influential the Leadership Conference on Civil Rights was in passing the Civil Rights Acts of 1960 & 1964. * Democratic participation is the backbone of social change – just look at how Ireland lifted a ban on abortion via a Citizen’s Assembly. * And so on… To paint this picture, we can see this in action below: Source: Just Stop Oil which focuses on…civil resistance and disruption Source: The Civic Power Fund which focuses on… local organising What do we take away from all this? In my mind, a few key things: 1. Many different approaches have worked in changing the world so we should be humble and not assume we are doing The Most Important Thing 2. The case studies we focus on are likely confirmation bias, where
 ·  · 1m read
 · 
Although some of the jokes are inevitably tasteless, and Zorrilla is used to set up punchlines, I enjoyed it and it will surely increase concerns and donations for shrimp. I'm not sure what impression the audience will have of EA in general.  Last week The Daily Show interviewed Rutger Bregman about his new book Moral Ambition (which includes a profile of Zorrilla and SWP). 
 ·  · 2m read
 · 
Americans, we need your help to stop a dangerous AI bill from passing the Senate. What’s going on? * The House Energy & Commerce Committee included a provision in its reconciliation bill that would ban AI regulation by state and local governments for the next 10 years. * Several states have led the way in AI regulation while Congress has dragged its heels. * Stopping state governments from regulating AI might be okay, if we could trust Congress to meaningfully regulate it instead. But we can’t. This provision would destroy state leadership on AI and pass the responsibility to a Congress that has shown little interest in seriously preventing AI danger. * If this provision passes the Senate, we could see a DECADE of inaction on AI. * This provision also violates the Byrd Rule, a Senate rule which is meant to prevent non-budget items from being included in the reconciliation bill.   What can I do? Here are 3 things you can do TODAY, in order of priority: 1. (5 minutes) Call and email both of your Senators. Tell them you oppose AI preemption, and ask them to raise a point of order that preempting state AI regulation violates the Byrd Rule. * Find your Senators here. * Here’s an example of a call:  “Hello, my name is {YOUR NAME} and I’m a resident of {YOUR STATE}. The newest budget reconciliation bill includes a 10-year ban pre-empting state AI legislation without establishing any federal guardrails. This is extremely concerning to me – leading experts warn us that AI could cause mass harm within the next few years, but this provision would prevent states from protecting their citizens from AI crises for the next decade. It also violates the Byrd Rule, since preempting state AI regulation doesn’t impact federal taxes or spending. I’d like the Senator to speak out against this provision and raise a point of order that this provision should not be included under the Byrd Rule.” See here for sample call + email temp