Are there good introductory materials that explain how charity evaluation works?

Most of the organisations publish this but I'm not aware of a single resource that comprehensively compares them. 

The closest I've seen is the "choosing a charity" article on Giving What We Can.

Also, many of the specific details of the different approaches change over time. And some organisations use different methods depending on the cause.

Hello, 
I am a research analyst at the Happier Lives Institute where we do cost-effective analyses (CEAs) of charities and interventions such as StrongMinds and GiveDirectly.

I am not aware of a document that collects all of the groups who do CEAs and compares and contrasts their methods. I would be quite interested in seeing one. The creation of such a document could look like a table with evaluating groups (GiveWell, OpenPhil, SoGive, Founders Pledge, Charity Entrepreneurship, Happier Lives Institute, etc.)  as rows and important methodological questions as columns. This could be useful as it allows evaluators to learn from each other's methodology and allows people to see the different assumptions under which different evaluators are operating.

This raises the question, what are the important elements/questions/assumptions to consider? I will give my list based on thinking and methods at the Happier Lives Institute.

In a CEA, first you need to think about the effect. In EA, we are concerned with doing (the most) good. This leads to the first big question, what is ‘good’? A lot of people might agree that income or health are maybe not intrinsic goods that we want to increase, but instead that these are good for people, they are instrumental. Instead, an intrinsic good (which is good in of itself) would be wellbeing. There are different theories of wellbeing, and I am not going to get into them here, but you can read some summaries here and here.

Say that, like the Happier Lives Institute, we think wellbeing is the good we want to maximise and that we take a subjective wellbeing interpretation of this: what is good are the mental states of people, their evaluations of their lives and the positive and (lack of) negative feelings they experience. Great, we have defined the good/the effect. Note that you don’t have to 100% agree with an evaluator for their evaluation to be useful, you might value elements outside of human wellbeing and still find someone’s evaluation in terms of wellbeing very useful for your prioritisation about charities.

Next step, how do we measure this good/effect? This isn’t trivial as we can agree on the good we want and measure it in very different ways. For example, we - as evaluators - could make a list of things we believe improve people’s subjective wellbeing (e.g., income and health) and make inferences about how much they improve wellbeing (e.g., for every 1 DALY prevented, X units of wellbeing are gained). Alternatively, and this is the method we use at the Happier Lives Institute, we could use the most direct measures of subjective wellbeing: Answers to questions about how happy people are, how satisfied with life people are, how many positive or negative feelings people have been experiencing.

Once we know what we are measuring and how, there are a bunch of methodological questions that can be asked, at different levels of precision. For example, what sort of data do people value? Only RCTs, or any sort of empirical data? If we combine multiple studies, what kind of technique do you use (what kind of meta-analysis)? Do you use predictions from experts? Do you combine the priors (subjective beliefs) of evaluators - which might or might not be informed by empirical data - with the data? Should evaluators give subjective estimates for variables there isn’t any data for? Should we make subjective discounts concerning the quality of the data? [At the Happier Lives Institute we are into meta-analyses of multiple studies, if possible RCTs, and we insert some reasoning about causal mechanisms but we try to avoid subjective estimates as best we can.]

I’d like to mention some bonus - but quite important - considerations about how we go about the effects at the Happier Lives Institute. We are not just interested in the effect of people when they receive the intervention (e.g., when they receive a cash transfer), but also the longterm effect: How long does the effect last? Does it decay? Does it grow? How quickly does it change? Then we also want to think about the effects on people beyond the recipients of the intervention; spillovers. How much does helping one individual affect their household or their community?

Finally, there’s the cost. I think this can be simple for charities at point of delivery: How much money do they receive divided by how many interventions they deliver gives you the cost to deliver an intervention to one person. But this can likely be more complicated in other situations. For example, what is the cost of influencing policy?

A meta consideration when one is comparing evaluators, how much time and how many people are involved in doing the analysis? Is this the work of 2 weeks or 2 months?

These are all really interesting questions and getting an idea of how different evaluators answer them is IMHO useful for the public, current evaluators, and future evaluators trying to learn how to do CEAs.

To onlookers: I’m not exactly answering the OP’s question and there’s high mansplaining potential, but my guess is that this comment has a chance of being useful.
 

Below is a narrative, not something I fully believe, because I think it’s incomplete (and I didn’t actually read the sources I quoted). But I think there is a chance this narrative is useful for the perspective it tries to share.
 

Narrative:

Basically, the root underlying value of evaluators comes from deep competence and powerful “models of reality”, this value does not come from a method. 

Evaluators always share something like a “public method”, like spreadsheets and long essays, but these are not at all a complete presentation of how they reached their views, so reliance on them is problematic.

To be clear, all evaluators use numbers and tools like spreadsheets, and develop systems and processes. But in the end, these are surface methods, sort of a window dressing, explanation or heuristic to explain decisions whose information are stretched over a much deeper latent pool of skill and understanding. This skill and understanding isn’t legible to most people.

This is directly relevant to some motivations of the OP’s question, if they are trying to find and compare each org’s "Way to Measure Charities". This is because, in the narrative you are reading, it’s hard to learn where the actual content comes from, and copying the surface presentation is an anti-pattern.

Much of SoGive's methodology is outlined on this blog, which I think is pretty accessible for beginners (but I think some parts are out of date)