Cost-effectiveness assessment for Sahara Nepal and Swallows of Finland. It regards their project’s Outcome 1: “Improved food  security and livelihoods of target communities through improved farming practices, value addition and better access to markets.” 

Link to the spreadsheet containing the calculations: Cost-Effectiveness Assessment Swallows of Finland and Sahara Nepal

Background

I had the idea of a business that makes cost-effectiveness assessments (later CEA) for charities. It would help them to be more effective by pointing out the most and least effective interventions they are doing. Allocating resources better might lead to improvements in the total impact of the charity. I wanted to know if this could work. So I proposed to the Swallows of Finland that I make them a pilot assessment for free, and they agreed. 

Swallows of Finland and their partner organization Sahara Nepal had a three-year development project in Masta Rural Municipality in Nepal. The project plan aimed for four different outcomes: Improved Food Security, Sustainable Livelihoods and Empowerment of Women plus increased capabilities of the partner organization.

Methods 

The evaluation that meets the standards of OECD/DAC and EU was already taken care of by another actor. It is mandatory for this project, like it is for all the other development projects that get funding from the Finnish Ministry of Foreign Affairs. It takes into account many important aspects, but it doesn’t really measure and quantify the absolute impact.

To find out which interventions were most effective, I decided to use GiveWell’s cost-effectiveness assessments as a template, which I modified and simplified. I also found out that I must narrow down the focus of the CEA to keep the workload manageable. Some intended outcomes, like empowerment or nature were also hard to measure or compare against each other. I evaluated the impact of enhanced farming practices on income, concentrating solely on agricultural livelihood improvement, excluding the project goal of food security. How efficiently were euros spent for the good achieved?

Metric: Doubling of income

One euro is much more valuable when it’s your only one, compared to the situation where you have thousands of them. That’s why I used both the absolute increase of income, and the doubling as the metric of impact. Many other assessments use the doubling of consumption instead of doubling of income, but it would have made the calculations more complicated. The sum of the income changes caused by the project was turned to the equivalent sum of income doublings. Doubling is compared to the baseline.

Literature research

I did the literature review to get an “outside view” - to know what to expect from a project like this and avoid the optimistic bias that may occur when someone is evaluating their own work. I looked for data about similar projects and their results. I used some factors to adjust the values to better reflect the contemporary situation. At some points I didn’t find the information, so I had to guess. 

In the project, there were a lot of different actions made to improve the agricultural livelihoods. For all of them, the literature didn’t tell clear numbers that would tell the effect size. The project reports had information about the outputs achieved, but I utilized them only partly, to keep the calculations somewhat simple. Instead, I looked at the whole list of actions in the project and found out they were quite similar in the studies. 

Theory of Change

The project has been measuring the implemented interventions/actions, so there’s data about that. The thing we ultimately care about, and what I’m trying to estimate here, is the impact. The higher you go in the results chain, the more uncertain it becomes and the more there are external factors affecting the outcomes. There’s not much data on the assumptions or outcomes. 
 

This visual Theory of Change is from the project document prepared by Sahara Nepal and Swallows of Finland 2020. 

The spreadsheet

The calculation spreadsheet is color-coded: Orange cells are guesses, the least certain values. Green cells are from a certain source. Blue cells are calculations based on those values. The reasoning and sources are mentioned in the cell notes. You can see the notes or formulas by hovering the cursor on the cell. 

Feel free to make your own copy of the spreadsheet and improve it, or use it as you wish.

I use several sources and calculations and take an average value of them to estimate the baseline and the impact. 
 

 Supposed improvement is written as a fixed amount for a fixed time, to equal the average effect time and income increase. In real life both factors have more variation. 

Results

In the Swallows project, 300 people initially earned around 850 € /person per year. As a result of the project, all of them will see their incomes rise by an average of 67€ over 5 years. 300 * 67 € * 5 = 100 000 €. It is also assumed that people outside the target group (eg. family and neighbors) will also adopt the improved farming practices, with a total benefit of 78 000 €. Total 178 000 €. It equals 321 yearly individual incomes doubled. 

Dividing this figure by the costs incurred for this part of the project gives the cost-effectiveness. 

The estimated cost of the project in terms of agricultural development was € 149 000, of which € 48 000 was local funding. The cost-effectiveness was therefore 178 000€ / 149 000€ = 1.194, i.e. a total return on investment of 19.4% over time. Yearly individual incomes doubled per 1000 € spent was ( 321 / 149 000 € ) * 1000 = 2.16

Discussion

The project resulted in 2.16 yearly individual incomes doubled per 1000€ used.

If the money used to this project would be given to the beneficiaries, it would have caused 1.17 yearly individual incomes doubled per 1000€ used, supposing no investments made. 

Give Directly, which serves direct transfers to the poorest, is reported to cause 3.97 yearly individual consumptions doubled per 1000€ used. Their beneficiaries are even poorer than in this project and they have included the investment returns in their calculations. (The poor don’t buy  shares from the stock market, but tin roofs that prolong the life cycle of their houses for example.)

Disclaimer

This figure should not be taken literally, as some of the assumptions in the calculation are rather vague, (especially the sustainability of the results or size of the spillover effect.) It’s also important to keep in mind that one number doesn’t tell much about the qualitative situation like income differences or healthiness of the diet, when reading any cost-effectiveness assessment. 

The causal chain used in this cost-effectiveness assessment is weak. There are many different inputs causing many different outputs. Calculating all the effects of each of them would have become too complicated, so I’ve jumped rather straight to impact from the inputs, skipping many of the middle steps.

Conclusion

The plan and the measurement plan defines what things you can know about the project. Being able to verify the assumptions, or even measure the ultimate outcomes would make impact calculation more reliable. 

Although the calculation is rather uncertain, it does show the order of magnitude and allows us to see which factors are most important for the final result, in this case the increase in income. For example, the choice of crops can have a significant impact on income. 

The importance of remittances from those working elsewhere as part of income was surprising in its magnitude, being as much as farming, or other income combined. 

In Masta Rural Municipality, agricultural income does not appear to be far from the Nepalese average, but perhaps exceeding the average may be possible and certainly valuable.

To be a household with big agricultural incomes you also need more land. Improved practices help only so much. In other words, the land area is a limiting factor. An average household in the project area has 7,3 people and less than 0,35 ha farming area. Average Nepali uses 0,1 ha to feed himself, which is 20 times more. Considering this, achieving food self-sufficiency seems very hard. On the other hand, productivity of agricultural work is only a fifth of the world average, so there is a lot to improve. 

Reflection on my work

I focused heavily on external literature partly because I felt like just taking the numbers from the project reports and putting them to a spreadsheet would be too easy and not provide much additional value. In a complicated case like this, when there’s no exactly similar records, I would now start with the reports and use additional information only to fill the gaps. Though for a complicated case like this, doing a precise CEA may require more work than what the project benefits from it. This took some 210 hours for a first-timer like me.

How useful are the findings? If it can lead to decisions that improve the results by 5% or more, it’s worth it, but I guess it won’t.

After three months of work, three weeks ago, I read the baseline study that was made for the development project. The data about the baselines told a somewhat different story than my guesses based on the literature and project plan. I should have acquired that sooner.

Technical

I use several sources and calculations to estimate the baseline and the impact. Instead of just taking an average of them, a weighted average might give more accurate results.

Making a more simple calculation would have reduced the risk of calculation errors and made it easier to read.

The success factor is probably the most uncertain value in the spreadsheet.

Acknowledgments

Timofey Zabelin, Anssi Lehtonen, @Vesa Hautala, Elsa Kivinen, (@Elskivi) and @Stan Pinsent have been helping me. Thanks also for Raimo Lilja, Heli Janhunen and other people from the Swallows of Finland.    

Sources 

I’ve linked all the sources to the spreadsheet cells that they refer to.