Livelihood interventions: overview, evaluation, and cost-effectiveness

Rethink Priorities; Greer Gosnell; JamesHu🔸; Melanie Basnak🔸; Ruby Dickson

Livelihood interventions: overview, evaluation, and cost-effectiveness

Rethink Priorities,

Comments 8

Sorted by

New & upvoted

Karthik Tadepalli

This is a really impressive endeavor. I want to flag an important argument that I think is quite important for EAs to take away.

We are familiar with the idea of some interventions having more leverage than others and thus being better. For example, Open Philanthropy does global aid advocacy as a cause area because every dollar they spend on advocacy can increase aid spending by much more than $1. But livelihood interventions point to a different source of leverage: the private market. Market failures are everywhere in developing countries - frictions that prevent mutually beneficial market transactions. Thus, correcting those market failures can unlock transactions that increase incomes by much more than the actual money spent. The success of each of these interventions reflects a corresponding market failure that they are solving: for example, Building Markets is impactful because SMEs in poor countries and buyers in rich countries can't find each other to make mutually beneficial transactions.

For a long time, EA's favored approach to development interventions was to focus singlemindedly on health because health impacts are large and direct. I think it's a good thing that we are now recognizing that there are so many indirect interventions that have much higher leverage. Solving frictions that prevent mutually beneficial market transactions - whether in labor markets or export markets - deserves a lot more attention as a class of interventions.

Michael_Wiebe

I'm surprised the Nigerian business plan competition was not included. (Chris Blattman writeup from 2015 here: "Is this the most effective development program in history?".)

Henry Howard🔸

IPA and J-PAL are underrated. They've had a hand in producing the evidence for many of GiveWell's recommendations. They seem to be significantly better at cause discovery than the Effective Altruism community.

Arturo Macias

I find this work fascinating. Is there any project about agricultural mechanization based on open source standards. I only know about of this:

https://www.opensourceecology.org/

But I think that Research and Developement for low tech innovation adapted to severely underdeveloped coutries looks a possible kind of potentially ultra leveraged aid.

One-time pad

-2

"We estimate with medium-high confidence that the Spark intervention generates about $28 in village-level income per dollar spent over the first 10 years of the program when villages purchase livestock."

This screams bullshit. If you believe this to be true, why don't you invest in this for profit? You could subsequently donate 10x more in just 10 years even if you only capture 1/3 of the value added.

Karthik Tadepalli

How would you make a profit from this? If you charged villagers for the grant, it would just be a loan, and takeup would be dramatically smaller, so you wouldn't get effects like this.

One-time pad

If I offered you a free solar panel upgrade, in return for 1/3 of the additional revenue it generates, would you accept?

Karthik Tadepalli

The revenue generated by a solar panel is very attributable to solar panels because of electric metering. This estimate in contrast is at the village level, includes lots of things that wouldn't be attributable to the cow (e.g. income multipliers from people spending in village shops).
It's village level income. Who individually would pay you?
Contracting frictions make it virtually impossible to enforce payment after the fact. If people refuse to pay you the revenue fraction, or claim that they got much less revenue from it then they actually did, then you are shit out of luck.
It's a well established fact that development interventions have a massively reduced take-up when the price goes from zero to even a small amount. This was a big debate with malaria nets, and the evidence showed that people would dramatically reduce takeup of nets if they had to pay even a negligible amount for them.

Comments

^{^}

In the process of collecting information to identify promising interventions, we began by reviewing a wide range of agricultural and vocational training interventions in Ethiopia, the Democratic Republic of the Congo, and Tanzania due to their large populations and relatively low income levels. As we narrowed the project scope, we quickly pivoted to consider only the J-PAL/IPA interventions that specifically targeted income and earnings as an outcome of interest and included all studies in sub-Saharan African countries, and gathered information only on those studies that were completed and had reported results.

^{^}

Wealthier rural households are more likely to be engaged in non-farm activities and non-agricultural wage labor.

^{^}

A “nudge” is an intervention intended to change behavior without changing actors’ choice sets or significantly changing their economic incentives to do so.

^{^}

Note that these are the effects reported in the J-PAL (2013) summary, though the study design over two seasons — with Treatment 1 only provided in the first year, and the rest provided in the second year with a smaller comparison group — leads the authors to present variations on these estimates for T1. The comparison group is much smaller in the second year, which may contribute to the lack of significance for T3.

^{^}

Somewhat surprisingly, it appears that the authors did not measure actual fertilizer usage (as opposed to adoption), so it is unclear whether the timing of fertilizer receipt would have implications for actual usage (e.g., due to the ability to plan, the salience of the delivery, or seasonal variations in available labor or weather).

^{^}

ISFM practices include compost, blended fertilizer, improved seeds, line seeding, and lime application to soil, and “is viewed as a strategy to sustainably intensify agricultural productivity and combat land degradation caused by excessive deforestation and inappropriate land-use practices by agriculture” (J-PAL, 2020).

^{^}

A microwatershed is an agglomeration of rural communities that share a water outlet.

^{^}

Basis risk is the risk of imperfect hedging that arises due to differences in the cash price and futures price of a commodity (Hecht, 2021).

^{^}

Spark provides $4,200 in year one and $3,800 in year two to encourage learning.

^{^}

To calculate their social return on investment, Spark uses a “convenience sample of 19 communities from Rwanda,” though we do not understand — and have not asked — why it restricts its sample to just these communities.

^{^}

Other intangible benefits suggested by quotes on their website are the development of a savings culture, reduction in domestic violence, relationship development between communities and local governments, and increases in civic action.

^{^}

Given that pigs’ natural lifespan is about 15-20 years, i.e. longer than the 10-year period we consider in assessing cost-effectiveness, we do not separately model pork consumption but instead assume that households consume pork relatively infrequently.

^{^}

Given some idea about the diets of villagers from Spark’s 2021 end-of-year survey that it shared with us, we adjusted the village’s crop diversification in the “Crop inputs” sheet of our CEA to closely match their consumption, which ultimately eliminates revenues from crop sales and costs from food purchases. Therefore, all of the revenue comes from the sale of piglets, and households do not supplement their food consumption with off-farm crop purchases, in line with anecdotal evidence from Spark.

^{^}

We use a discount rate of 4% on future returns; using a discount rate of 0% increases the cost-effectiveness to $34 in income generation per program dollar.

^{^}

As such, the results we discuss below are intent-to-treat effects (i.e., the effect on women who were offered the opportunity to participate in the program), and we would expect treatment-on-treated effects (i.e., the effects on women who actually participated in the program) to be higher.

^{^}

We use the conversion of 1,700 Ugandan shillings (UGX) per dollar based on the January 2008 exchange rate reported in the paper. Per Table A3 of the online appendix, baseline earnings and the increase in earnings at endline translate to $16 and $50, respectively, such that earnings are $76 at endline.

^{^}

The researchers do not include effects past the four-year mark. It is possible that impacts persist or drop off after this period; we don’t have a sense of which assumption is more realistic.

^{^}

This number results from a simple back-of-the-envelope calculation, i.e., (12 months/year × 10 years × $10/month) ÷ $18.20 for the 10-year time horizon. One could also imagine income gains becoming even higher, as any proportional wage increases would be made relative to the higher baseline income induced by the treatment effect. Thus, it seems plausible that the effect on income would be even greater in the long run, while the cost of the intervention remains constant, leading to an even higher long-run cost-effectiveness.

^{^}

Kamau stated the cost was around $5,000, which varies based on the economic situation in the country. However, operational data suggests that the total cost, including recruitment, administration, and tuition, may be closer to $6,500.

^{^}

In the US market, software engineers typically make 5%-10% more than UX designers (Coren, 2016). Our very limited data from AkiraChix indicates this gap may be somewhat smaller in the Kenyan tech scene.

^{^}

In our model, we use a 5% reduction in raise year-on-year, that is, each year’s raise is 95% the size of the previous year, in real terms. Because the initial raises, from intern to junior engineer, and from first- to second-year junior engineer, are so large (122.55% and 52.31% respectively), this still translates to substantial raises after 10 years. We think this is a reasonable assumption given the high initial benefits, but the model’s output does depend on it.

^{^}

We assume that graduates get large initial annual raises which taper off at 5% per year. If, for example, we change this assumption to allow for increasing raises, or a less smooth salary trajectory, the results can change substantially.

^{^}

We aren’t sure how this squares with the assertion that around 40% of employment in LMICs comes from the formal sector. A large proportion of SMEs in Dalberg Global Development Advisors (2011) are informal or “non-employer” firms (p. 7, Figure 2).

^{^}

Blattman et al. (2016) also measure a consumption treatment effect in their paper, which is three times the earnings treatment effect ($15.50 vs. $5.19 per month). They therefore suggest that the consumption effect act as an upper bound and the effect on cash earnings effect act as a lower bound, leading to a $13-$39 treatment effect in purchasing power parity terms (pp. 49-50).

^{^}

The group dynamics intervention appears to have additionally insulated against the increased neighbor disputes and verbal abuse identified in Treatment 1 villages, with qualitative research suggesting these were acts of jealousy or resentment from a handful of households (Blattman et al., 2016, p. 55).

^{^}

To obtain this estimate, we used the NPV formula the authors specify in Table 8, with a time horizon t of 10 years and a discount rate of 5% (in line with the authors’).

^{^}

The authors report the intent-to-treat results as their headline findings, but also report the somewhat higher treatment-on-treatment result of 28%-42%.

^{^}

The treatment-on-treatment results show exporting firms score 1.35 points, or 46% higher, on rug quality than control.

^{^}

Priority sectors included agriculture, small scale industries, and “the weaker sections of society” (e.g., self-employed and microcredit; p. 3). The Government of India mandate required that 40% of net bank credit be allocated to the priority sector.

^{^}

Note that the authors chose to use the growth rate instead of growth, which strikes us as unconventional. We did not have time to investigate why they did so, and if it makes more sense to analyze this variable in this specific study.

^{^}

To obtain the total increased profit over three years, we assumed that the increase in monthly profits between years one and three was linear, implying an intermediate profit value in year two.

^{^}

This cost might include overhead, rental facilities, and physical inputs such as lumber and screws.

^{^}

GiveWell values an additional year of healthy life approximately 2x-3x as much as a doubling of income, which it does not view as particularly robust given the limitations of the value-of-statistical-life and disability-adjusted-life-year literature it examined (GiveWell, 2017).

^{^}

We received data for both of these organizations the day before our deadline for finishing the report, so we could only undertake minimal analysis. More sophisticated analysis and the opportunity to ask questions and seek additional data could help to refine our CEA inputs.

Livelihood interventions: overview, evaluation, and cost-effectiveness

Livelihood interventions: overview, evaluation, and cost-effectiveness

Editorial note

Key takeaways

Introduction

Agricultural interventions

The importance of agriculture for income in sub-Saharan Africa

Promising interventions

Subsidized watchmen improved income per acre by 15% and reduced neighbor disputes, but costs to farmers exceeded the intervention’s benefits

Fertilizer adoption nudges and incentives can improve income by around 11%-18%, though the cost-effectiveness of the interventions is unclear

In-person and video-based extension services improved uptake of helpful agricultural technologies among Ethiopian farmers, though further research is needed to understand the impact on incomes as well as the interventions’ cost-effectiveness

Offering crop insurance to informal insurance groups improved demand for rainfall insurance in Ethiopia, though impact on income and cost-effectiveness are not measured

Organization spotlight on Spark Microgrants

Background on Spark Microgrants

Our modeling suggests that village-level investments in pig farming lead to an expected cost-effectiveness of $28 in income generation per dollar spent on the program over 10 years

Training and labor market interventions

Promising interventions

Subsidized vocational training reduced unemployment by 23% and increased wages by 55% among low-skilled Ugandan workers, but the cost-effectiveness is less than $1 in income generation per program dollar

Combined vocational and life skills training for young women improved annual earnings among young Ugandan women by about $50, with an estimated cost-effectiveness of $5.88 in income generation per dollar spent on the intervention over 10 years

Job-search support workshops improved income significantly and cost-effectively among Ethiopian job seekers, generating $6.6 ($66) in income per dollar spent on the intervention over a one-year (10-year) time horizon, which could potentially improve with better targeting

Computer skills training paired with a job referral increased income by 37% among Kenyan youths, though the intervention’s cost-effectiveness is unknown

Organization spotlight on AkiraChix

Entrepreneurship and small business support

Promising interventions

The Women’s Income-Generating Support (WINGS) program improved cash earnings by 67%-151% among Ugandan women, with an estimated cost-effectiveness of $4.70 per dollar spent on the intervention (assuming persistent effects)

Access to exporting opportunities improved Egyptian rug manufacturers’ profits by 16%-26% with only a 5% increase in labor hours, though the authors do not measure cost-effectiveness

Subsidized credit access increased small-scale industrial firms’ total bank borrowing by 20% in India, increasing export growth rates by 24%.

In-kind grants to female-owned enterprises that were already above subsistence-level profits improved profits by about $830 ($2,828) over the first year (three years), with a cost-effectiveness of $6.90 ($23.60) per dollar spent

Conclusion

How we would spend more time

Acknowledgments

References

Notes