Local charity evaluation: Lessons from the "Maximum Impact" Program in Israel

TL;DR

Maximum Impact program, led by Effective Altruism Israel, aims to significantly increase the impact of Israeli philanthropy by conducting nonprofit evaluations, identifying high-impact local nonprofits (even if their recipients are abroad), and promoting effective giving. Additionally, we aim to strengthen the community internally by developing a replicable model that could achieve these goals in other local communities and by building the Israeli EA community through action-oriented projects.

During the last year, we joined forces with promising non-profits and conducted and published 21 cost-effectiveness analyses. Together with an acclaimed team of expert judges, we identified 3 nonprofits as relatively cost-effective, including Smoke-Free Israel, a tobacco taxation nonprofit, which might be competitive with top charities. We learnt a lot from the pilot and have many ways to improve the program, but overall the pilot was successful: we identified high-impact local nonprofits and have engaged Israeli philanthropies interested in cost-effectiveness and in the results of the program. Finally, having a large-scale program on nonprofit effectiveness was a huge boost for EA Israel’s community building, and many people heard about EA and became more involved because of the program. We’re looking forward to continuing to maximize the impact of Israelis and to share the knowledge and experience we gained with the EA community.

 Motivation

• We identified that global effective giving platforms currently don’t “meet donors where they are” and only offer recommendations on global nonprofits without being able to engage donors. Donors who are initially interested in local donations but can later be introduced and nudged to give to global high-impact nonprofits are currently not targeted by global effective giving initiatives.
• Israeli and Jewish donors abroad (~17.2bn per annum) currently donate almost entirely without evidence and cost-effectiveness estimations, as the Israeli non-profit ecosystem doesn’t yet emphasize these as prerequisites for donations.  
• With almost no research currently available on local effectiveness and a strong bias of donors towards donating locally, there are not enough impact-based donations in the local nonprofit sector.
• Based on discussions with EAs worldwide, we assume this problem is not unique to Israel and would like to find a cost-effective way to solve it. 

What have we done so far?

Our research program

In our 10-month program, we chose 25 promising nonprofits from more than 160 applications from Israeli-based NGOs, 21 of which successfully published their first-ever cost-effectiveness reports. In the pilot program, we provided guidance and assistance to support non-profits to execute a cost-effectiveness analysis. After the research period, we sent the reports to a team of expert judges for review, and chose the most promising non-profits based on the expert review. We offered monetary incentives to encourage nonprofits to participate: 15 nonprofits received preliminary research grants of $6k, the 3 top outstanding nonprofits received $24k prizes, and $26k was allocated to a donation matching campaign to all nonprofits who publish a report. A private donor and an Infrastructure Fund grant funded the pilot.

Guiding nonprofits on evaluation and improvement

We implemented various tools and resources to help incentivize and enable non-profits to conduct robust effectiveness analyses, including:

• An extensive set of templates and how-tos.
• Research grants for 10 young academic researchers (MA and PHDs) from Israel's top universities to conduct research with the organizations and join the EA community during the process. 
• Personal guidance from skilled community members to plan and conduct the reports, aligned with the EA view and principles. The community members gained hands-on evaluation experience while improving the research quality. 
• Monthly 1-1 meetings with our Head of Research were highly valuable, helping researchers and organizations resolve specific research issues and ensure deadlines were met. 

Judges and advisory board

We are grateful for the guidance and expertise that went into developing our model. We received support from esteemed local economics experts, as well as known researchers from the EA community, in developing our program and assessing the results. Our panel of judges included Claire Walsh from J-PAL, Dan Stein from IDinsight, and Karolina Sarek from Charity Entrepreneurship. Additionally, we consulted with members of SoGive, GiveWell, and GWWC about research questions and program design. Within Israel, we received counsel from leaders such as Assaf Kovo, Chief Economist for the Israel Innovation Authority, Analia Schlosser, an acclaimed empirical economics professor at Tel Aviv University, and Ahinoam Zeigel of Social Finance Israel.  We truly appreciate the time and insights this distinguished group contributed to help us develop and refine a robust and successful work model.

Success

This was the program's pilot year, and there are positive outcomes and things that can be improved. 

Top nonprofits found

In terms of output, we published 21 cost-effectiveness analyses on Israeli nonprofits, becoming the first public database on nonprofit effectiveness in Israel. Although there is variance in research quality, all analyses were conducted based on a relatively high standard. We've shown we can conduct thorough analyses at a relatively low cost while still using high-quality methods. Our analyses cost an average of just $6,300 each yet over half employed quasi-experimental designs. With adequate resources, we can scale this process and maximize the potential we’ve identified during the pilot.

While further research is required to verify the results, we identified a few promising nonprofits. The most promising organization among those we evaluated (and probably one of the most effective nonprofits in Israel) is Smoke-Free Israel. Our evaluations suggest they may be more effective than GiveWell's top charities, with an estimated 7.18 QALYs per dollar (This is in line with the CE report, WHO, and multiple resources, suggesting that tobacco taxation may be highly cost-effective). Although they currently have limited room for funding, identifying and quantifying their success is important in helping them continue their work (they’ve already presented the research done in the program to the Israeli Knesset in a debate on tobacco legislation) and to show that highly cost-effective global health nonprofits could potentially be found outside of GiveWell’s top charities and in local contexts in developing countries. We rewarded two other organizations for their relatively high levels of effectiveness. NALA, a WASH intervention in Ethiopia, and  Kav LaOved that help migrant workers from developing countries in recovering unpaid wages, an important yet neglected cause. We were also impressed by the Better Plate accelerator operated by MAF, a local accelerator focused on alternative protein startups. 

To improve our identification of top nonprofits, we collaborated with the EA group at Tel Aviv University for a hackathon. Over 80 participants worked on developing a technological tool and a robust methodology for identifying promising non-profits using publicly available data. We are currently collaborating with the winning team to build an MVP that we hope will be operational soon. We aim to implement this tool in the upcoming year to enhance our outreach efforts and proactively recruit high-potential non-profits based on the outcomes.

Collaboration with the philanthropic sector in Israel

Our most significant achievement has been cultivating partnerships with leading philanthropic organizations and individuals across Israel. We aim to bridge research and philanthropy advising establish ourselves as a major influencer in the sector, and establish our reputation as providing high-quality effectiveness research.

We’ve had some success since the end of the research program. We launched a website and donation platform published this year's research and which will soon feature global high-impact nonprofits (GiveWell, GWWC recommended charities) alongside Israeli nonprofits. We had a successful event where we presented the program and discussed the importance of evaluation in nonprofit work. Two major local philanthropies sponsored the event, a few others attended, and we have meetings scheduled with a few more in the coming weeks to explore potential collaborations and talk about the importance of effectiveness based giving. We were featured on Israel's most popular economics podcast, reaching over 300k monthly listeners, and we’ve been told that a senior official in Israel’s Ministry of Finance heard about EA and became interested through the podcast. 

We have leveraged the success of pilot by actively building new connections and reinforcing existing ones. We have established valuable partnerships with numerous Israeli philanthropic entities, including the largest funders network operated by Jewish philanthropists internationally. Additionally, we have forged strong ties with the largest local Donor-Advised Fund (DAF). Furthermore, we have initiated new collaborations with multiple Impact-tech companies, expanding our network and fostering innovative partnerships.

Onboarding young professionals to high-impact career paths

Part of the impact comes from onboarding promising young professionals into impactful research since we see ourselves as coming from and working with the EA community. By connecting young professional researchers, and local nonprofits, our process provides valuable skills to organizations while offering hands-on research experience aligned with EA principles and cost-effectiveness analysis methods to the researchers. 

Along the way, we've helped nurture emerging careers. One participant from our hackathon is joining the Rethink Priorities “AI Governance & Strategy” team next month. Meanwhile, four researchers continue to collaborate with their non-profits locally and abroad (at the moment, one of them is running RCTs in Zambia), three others have asked to further build expertise by joining our program managing team, one applied to a research role at GiveWell and another one joined the evaluation team at one of the philanthropies we’ve collaborated with, hands-on research experience seems to inspire new career paths. We believe diving into fieldwork with guidance from experts can be a useful complementary product to EA traditional career advising.

Build a replicable model

Part of the Maximum Impact program’s goal is to roll out our model to other countries so that they can run their own version of the program and in order to replicate our success. EA Israel has already done this in the community building space, and our templates are referred to often and serve as the basis for an SOP (Standard operating procedure) for local and national groups being built by CEA. 

During the pilot, we created an extensive toolkit that can support the implementation of our program in additional countries and communities. It may also prove useful as easy-to-use resources for monitoring and evaluation planning or teaching purposes. We have received expressions of interest in carrying out our work in multiple locations such as India, Brazil, Switzerland and Turkey.

What was less successful?

• Our research design still can be better - To be as rigorous as we can, we’ve most of our research is empirical and quasi-experimental designed (and RCTs when possible) based on the outputs of each nonprofits. However, the final result can be improved to achieve higher internal and external validity, and we have many ideas how we can improve the quality of research in coming rounds,
• Our selection process should be improved - some of the non-profits we’ve decided to collaborate with turned out to not be mature enough to carry out an in-depth evaluation. This led to a lower-quality database and limited staff power we could use to support our research. For the next round, we will revamp our screening process and do active outreach to the most promising nonprofits.
• Insufficient funds shifted to effective nonprofits -  With the research completed and a donation infrastructure in place, we hope to shift funds to high-impact nonprofits. However, we haven’t yet seen the results we’d like, and we have shifted only $10k in the month to the outstanding nonprofits on the websites. It is promising that over 85% have supported the most effective nonprofits in the program, even though it’s possible to donate (and receive matching) to all participating nonprofits. We are now dedicating significant effort to establishing stronger connections with local donors and philanthropies and developing our marketing efforts and brand.  We believe that some hard work on this aspect should deliver much more significant outcomes in the near future.

What’s next?

• We’re evaluating the program (Keen thanks to the Rethink Priorities survey team, which helps us a lot in the questioning process!), conducting exit interviews with the participating nonprofits, and planning the next round. We have some open questions about the format, but we want to base the next program on the success we’ve gained and the proof of working closely with nonprofits and philanthropy.
• The format we’re leaning towards is one where we’d partner with Israeli philanthropies and provide Evaluation-as-a-service to them while independently researching a smaller group of especially promising nonprofits. This would enable us to strengthen our connections with philanthropy and strengthen our sustainability, while creating more public knowledge on the best giving opportunities in Israel.
• We aim to add global effective giving nonprofits to our website and fundraise for them, while “nudging” Israeli donors who are interested in effectiveness to donate globally.
• Funding permitting, we want to finish our template toolkit and implement the program in another country, ideally one that has a bias to local giving similar to Israel and potential for high-impact nonprofits (India is a promising option).

How you can get involved

• If you know funders interested in spreading nonprofit evaluation worldwide or in an innovative approach to action-based community building, please contact us!
• If you know Israeli nonprofit funders interested in maximizing their portfolios' impact, let them know about the program.
• If you want to implement a nonprofit evaluation program in your country, please reach out!
• If you’re interested in reviewing the cost-effectiveness analysis of the top nonprofit we identified and how it compares to global top nonprofits - we’d love to see that done!