This is a Draft Amnesty Week draft and is also my first EA post. This was a June 2024 submission toward my Nonprofit Management Certificate aimed at informing my peers about the mechanics of general charity navigator ratings, and to introduce them to EA-aligned charity evaluators. Few of my peers were aware of EA’s existence and purpose, and none reported being aware that rigorous evaluation of effectiveness is undertaken by EA-aligned charities themselves, much less by external charity evaluators focusing on effectiveness. The Certificate Program provided excellent education on how to quantitate output and outcome (charity-wide and program-specific: TOC, output vs. outcome, % program activities expense, etc.), but it lacked discussion about standard charity evaluation methods and comparison to established effectiveness benchmarks. I am hopeful that future nonprofit management curricula adopt these principles when educating future founders, and that the differences between evaluator methodologies become better understood by the general public’s donors, or at least by donor advisors, DAF sponsors, grant-makers, etc.

Benefits of charity evaluators 

When people go to a marketplace to shop for products, they expect to be able to determine in advance how many units of product they will receive for a given purchase price. When individuals donate to nonprofits to support their mission, they need a means of determining how many nonprofit units each dollar can purchase. To fill this void, charity evaluators were born. This article is designed to help nonprofit executives understand the evaluation methods of modern charity evaluators, how they report their results, and how to ensure a high charity evaluator score or top recommendation for their nonprofit.

While foundations and major corporate donors use professional grant-makers or employ their own research teams to perform charity evaluation and prioritization, few of the general public perform their own charity organization research prior to contributing to a nonprofit. Many trust the output of various free online charity evaluators, although their grading schemes vary significantly, and their rating values cannot be directly compared. Examples of the most popular charity evaluators’ ranking systems are provided in table 1 along with links to their methodology and a sample report for the same nonprofit to allow the reader to compare evaluator outputs.

Traditional ranking systems

Some ranking systems focus on the organization’s transparency alone, but do not analyze the data provided, while some do include basic calculations to measure financial efficiency. For example, GuideStar is focused on transparency, and it awards higher Seal of Transparency levels based on how much information the organization voluntarily provides. The details listed from lowest weight to highest includes: the organization’s mission, leadership information, key program details, populations/geographic locations served, audited financial statement, leadership DEI information, goals and strategy and mission-related metrics. No analysis of the self-reported metrics is involved in determining whether the organization achieves the Bronze, Silver, Gold or Platinum Seal; they need only provide the requested information. 

CharityWatch pulls data from the 990 alone and calculates only two values (the percent of budget spent on Programs and the cost to raise $100), and reports “the efficiency with which a charity raises and spends public dollars” as a single grade, A+ through F. The grade, however, does not report the individual calculations separately (one for program % and one for cost to raise $100). Charity Navigator provides a 0 - 4 Star rating of based on a weighted sum of the organization’s individual Beacon scores which include Accountability & Finance, Impact & Measurement, Leadership & Adaptability and Culture & Community. The report includes a single analytic which is the ratio between costs and “outcomes” and is used “to determine how much good a nonprofit can accomplish per dollar” compared to a predetermined benchmark for cost-effectiveness. While reporting a “pass” or “no pass” for 20 individual Standards for Charity Accountability, Better Business Bureau’s Wise Giving Alliance reports on organizations’ accountability with the requirement for policy and reporting across several broad categories including governance, measuring effectiveness, finances and solicitations and informational materials which includes fundraising. Wise Giving, however, “does not evaluate the effectiveness or worthiness of the charitable program itself” although it does require that the organization spend at least 65% of its total expenses on program activities to receive a Pass under the Program Expenses Standard. This concept of “excessive overhead” being a problem in nonprofits, however, should be questioned.¹

It is evident from just these few examples that your organization’s transparency is extremely important in earning high charity evaluator rankings. While simply providing basic financial data can earn you a high score with some, others will require an independent financial audit, and having well-established policies and DEIA in the workplace will rank you higher with more rigorous evaluators; however, the most rigorous will require that you have a policy in place to routinely measure your organization’s effectiveness (more below) and that you share those findings. But do these high rankings translate into higher donor trust? A meta-analysis including 42 studies sampling more than 81,000 people in 31 countries confirmed a positive association between trust and giving across diverse measures; specifically, organizational trust was more strongly associated with giving than sectoral trust but both were correlated.² The trust of some donors can be earned simply by receiving your organization’s text, email or snail mail or viewing your ad online explaining what you do and asking for a donation. No further data is needed. But for others, the assurance of a high charity evaluator score may nudge them to give to your organization rather than a lower-scoring one (and steer them away from organizations with terrible scores).

Defining impact

There is an increasing number of highly-motivated donors who belong to the Effective Altruism movement who will need to dig even deeper into your organization’s details before giving. It will not be enough for them that your organization has self-reported its policies, provided reports in all required fields, or earned an A+, 4 Stars, 100%, and a Platinum Seal. They will need to review not only the content of your policies and the figures within your provided financial reports, but your organization will need to show its actual outcome per dollar. The charity evaluators are not organized to report or compare outcome, only what they call “impact” or “efficiency”, which are financial figures. For the purpose of nonprofit work, output measures how many units of product or service are delivered through the organization’s programs per dollar, while outcome refers to how that unit has affected the life of a person (or animal, or environmental unit). This is much more difficult to assign a quantitative value to and to physically measure, but units such as the Quality-Adjusted Life Year (QALY, combines length of life and level of health into a single metric)³ and the Disability-Adjusted Life Year (DALY, combines burden of mortality and morbidity into a single metric)⁴ have been created to allow comparisons of interventions affecting human lives. GuideStar’s “impact” reporting requirement could be fulfilled by reporting “number of donors or volunteers”. This figure does not necessarily translate into a higher output or outcome; it only counts individuals working toward your mission. If your organization does not use its dollars or volunteers efficiently or does not produce a product or service that actually improves the lives of those it serves, there will be little good accomplished. In fact, a fraudulent organization could have a huge number of donors and volunteers and massage the figures to look highly impactful, but ultimately be spending program funds for personal gain. It is likely that an overtly fraudulent nonprofit organization would eventually be found out by at least one of the charity evaluators. 

Fraudulent organizations aside, we must consider those organizations who do aim to produce good outcomes (even those who score well on traditional charity evaluators’ “impact” metric) but which only deliver a high output, not a high outcome. For example, the number of classroom textbooks delivered to children in a village elementary school per dollar (output) is very high for a top-rated nonprofit, but it is later discovered through responsible organizational program follow-up and analysis that the provision of textbooks (its output) did not result in the students performing better in school or having other measurable benefits (outcome). In contrast, Against Malaria Foundation, a Top Charity as rated by the “outcome-focused” charity evaluator, GiveWell (Table 2), can save a life with $4500 by distributing malaria nets in sub-Saharan Africa.⁵ Saving a life in the Global South and other under-developed regions is generally much more cost-effective than saving a life in the United States because basic needs such as fresh water, minimum required nutrition, anti-parasitics, vaccinations, simple healthcare, education and other fundamental survival resources are not being met and can be provided at very little cost. Interventions can be measured and compared using QALY and DALY values across various nonprofit organizations to determine their relative effectiveness. While I am not suggesting that US-based nonprofits transition their programs to focus on these geographical regions in order to become more effective with each dollar, many do consider it a nonprofit’s obligation to determine the most cost-effective means of accomplishing your mission and quantitating your effect. See table 2 for additional information on example outcome and impact-focused charity evaluators.

Internal evaluation and adaptation

Nobody (including the traditional charity evaluators) will know whether your nonprofit truly produces good outcomes effectively except for your organization itself (and if you engage with them, the outcome and impact-focused charity evaluators), and only if your organization adopts and integrates outcome and effectiveness goals in its mission statement, values, bylaws, and policies. It is not enough to measure your output, outcome and effectiveness, but you must then act on the results. If they suggest that you are not delivering good outcomes to your target population effectively (high output per dollar raised, as compared to benchmarks), then you must still report this to your stakeholders and take steps to change your mission, programs, organizational structure, etc. This is what true transparency is. Reporting the truth about what your organization does, whether good or bad, and how well or poorly. 

At the very core of philanthropy is the idea that we are aiming to doing good. Nonprofits aim to create a positive change in the world, and by operating their programs in exchange for donors’ dollars, they are either selling good outcomes (hopefully in the most financially-efficient and socially-responsible way possible) for their recipients, or they are selling a "warm glow” - the emotional utility donors derive from the act of giving - to their donors.⁶ For those seeking the warm glow effect, your organization may need to do nothing more than ask for a donation! To recruit the savvier donor, your organization may need to score highly with a traditional evaluator - establish and uphold policies which represent best nonprofit practice, pursue an independent annual financial audit, submit your annual 990 transparently, measure and report your program output, and comply with the reporting requirements on the evaluators’ websites. But for those organizations looking to make a greater change in the world, they will need to measure their outcome and effectiveness using the principles discussed here. They might be surprised by what they find.

Table 1. Traditional Charity Evaluators

Table 2. Outcome and effectiveness-focused Charity Evaluators

References: 

1. Misconceptions and Concerns About Effective Altruism and Charity Evaluation. Giving What We Can. URL: https://www.givingwhatwecan.org/misconceptions-and-concerns-about-effective-altruism-and-charity-evaluation Accessed 6/8/24.

2. Chapman, C. M., Hornsey, M. J., & Gillespie, N. To What Extent Is Trust a Prerequisite for Charitable Giving? A Systematic Review and Meta-Analysis. 2021. Nonprofit and Voluntary Sector Quarterly, 50(6), 1274-1303. URL: https://journals.sagepub.com/doi/pdf/10.1177/08997640211003250 Accessed 6/8/24.

3. Alex Zhavoronkov, PhD. How To Live This Life Better: Effective Altruism In The Context Of Longevity. 2/20/19. Forbes Cognitive World Contributor Group. URL: https://www.forbes.com/sites/cognitiveworld/2019/02/20/how-to-live-this-life-better-effective-altruism-in-the-context-of-longevity/

4. Interpreting the Disability-Adjusted Life-Year (DALY) Metric. GiveWell. URL: https://www.givewell.org/research/DALY Accessed 6/8/24.

5. How Much Does It Cost to Save a Life. GiveWell. URL: https://www.givewell.org/how-much-does-it-cost-to-save-a-life Accessed 6/8/24.

6. Andreoni, J. Impure Altruism and Donations to Public Goods: A Theory of Warm-Glow Giving. 1990. The Economic Journal, 100(401), 464–477. URL: https://doi.org/10.2307/2234133. https://www.jstor.org/stable/2234133 Accessed 6/8/24.