Summary
Evidence Action is terminating the No Lean Season program, which was designed to increase household food consumption and income by providing travel subsidies for seasonal migration by poor rural laborers in Bangladesh, and was based on multiple rounds of rigorous research showing positive effects of the intervention. This is an important decision for Evidence Action, and we want to share the rationale behind it. Two factors led to this, including the disappointing 2017 evidence on program performance coupled with operational challenges given a recent termination of the relationship with our local partner due to allegations of financial improprieties. Ultimately, we determined that the opportunity cost for Evidence Action of rebuilding the program is too high relative to other opportunities we have to meet our vision of measurably improving the lives of hundreds of millions of people. Importantly, we are not saying that seasonal migration subsidies do not work or that they lack impact; rather, No Lean Season is unlikely to be among the best strategic opportunities for Evidence Action to achieve our vision.
Background
No Lean Season is a program in Evidence Action’s Beta Incubator portfolio, which is designed to rigorously test promising programs along the criteria of evidence, cost-effectiveness, potential for scale, and strategic fit.
In 2018, the evidence from a 2017 at-scale randomized controlled trial (RCT) showed that the program did not have the desired impact on inducing migration, and consequently did not increase household income or consumption. Our hypothesis was that the poor performance in 2017 was due to implementation challenges, and in 2018 we set out to adjust the program design to address these issues and conduct another at-scale RCT. We ceased further fundraising for No Lean Season pending the results of the 2018 RCT, anticipating these results would determine the path forward for the program – including the possibility that further implementation may not be warranted.
Scaling programs is inherently difficult, and we expect some programs in the Beta pipeline to ‘fail’ in initial years due to implementation issues, issues relating to the underlying causal mechanism and theory of change, or some combination thereof that result in not meeting our criteria. Our focus in 2019 was on seeking to untangle this, as our research partners completed the RCT. The research team is still in the midst of collecting that data.
Allegations of financial irregularities
In 2019, a new challenge emerged that we want to share. In February 2019, we learned of allegations that the local Bangladeshi organization that Evidence Action contracted to implement the No Lean Season program had sought to bribe the Bangladeshi agency responsible for program approval. Specifically, Evidence Action staff were allegedly told by the local organization that the program approval had been forged, and that in trying to rectify the matter, a government agency official may have asked for an additional bribe that the local organization sought to pay. Such actions would have violated the local organization’s agreement with Evidence Action, and we therefore terminated our relationship with that organization.
We promptly engaged experienced, global legal and forensic audit firms to conduct an independent investigation into the allegations. After an extensive review of thousands of emails, documents, and staff interviews, the independent investigation was unable to corroborate allegations of any bribe. However, we also faced limited partner cooperation and contradictory Evidence Action staff reports during the investigation, which hindered a clear understanding of what transpired.
During the course of the investigation, we did learn of one instance in which Evidence Action policies and procedures were not followed. Specifically, the investigation discovered a payment of approximately $400 by an Evidence Action staff member in 2018 that was allegedly made to a consultant to assist with obtaining program approval. The investigation was unable to conclude that the payment was actually made to a consultant or that any services were performed by such a consultant. The payment of this amount did not follow established Evidence Action processes and procedures for contracting and paying consultants. These procedures are in place to minimize risks of financial improprieties, and we have taken appropriate internal action in connection with this non-compliance. Additionally, we are taking this opportunity to review and strengthen our compliance controls and provide additional staff training to ensure that our work, including that carried out by our partners, is conducted ethically, legally, and effectively.
Decision to end the No Lean Season program
Following this recent development, Evidence Action assessed whether to continue the No Lean Season program. In 2018, we decided that, though the program did not have the desired impact in 2017, it merited further efforts to seek to resolve any implementation challenges. Given our existing program and partner infrastructure, these efforts could be achieved in a resource-effective way. However, with the termination of our relationship with our local partner – who bore primary responsibility for implementing the program – we have made the difficult decision to discontinue the program. Identifying a new partner and rebuilding the program would require significant investments of time and resources, and substantial leadership and attention, possibly over 1-2 years, just to regain the ground we have lost with the termination of the existing partnership.
Like many things we struggle with in our work, this was a complex decision. Beta is deliberately experimental, and we expect many programs will not work. It is also quite clearly assessing potential programs against a range of factors – evidence, cost-effectiveness, potential for scale, and strategic fit, with operational feasibility serving as a critical underpinning. For Beta to work – and for Evidence Action to continue to ensure we are using scarce development resources as effectively as possible – we have to be willing to make tough calls and exit programs. Ultimately, the termination of the partnership and resulting operational challenges, coupled with the 2017 RCT results, led to our decision that the opportunity cost for Evidence Action of rebuilding the program is too high relative to other opportunities we have to meet our vision of measurably improving the lives of hundreds of millions of people.
Accident in Cumilla, Bangladesh
Separately, in January 2019, one of the known risks of the program came to bear in a tragic manner: an overloaded truck fell onto a temporary shelter in which several seasonal migrants who had migrated to work at a brick kiln were sleeping, killing 13 individuals, five of whom were affiliated with households that participated in No Lean Season. Moreover, four of the five were underage males aged 15-17. We were deeply saddened by this incident and the implications for these five No Lean Season households. Although this incident did not drive our decision to end the program, we want to share an update on it because the investigation we launched into the incident has recently concluded.
From the outset of working on seasonal migration subsidies, Evidence Action and our research partners have been deliberate in carefully examining risks and unintended consequences of the No Lean Season program, such as underage migration. We have sought to minimize these, where possible, in our program design. We recognize that migration is a complex phenomenon that can result in important gains. For example, the promising research that underpinned our enthusiasm to bring seasonal migration subsidies into our Beta Incubator showed significant gains in household food consumption and income. However, migration can create both opportunities and risks for migrants, families, and the broader population in both rural and urban areas, as individuals migrate from villages to cities. Recognizing this, Evidence Action and our research partners carefully developed a research agenda to understand the magnitude of potential risks, including studying the impact of seasonal migration subsidies on, for example, gender and social norms, welfare of migrants, long-term migration behavior, wages in origin and destination communities, and underage migration. We also put in place programmatic safeguards to minimize risks where feasible, including those associated with underage migration.
In response to the tragic incident, we undertook a thorough review of program protocols and procedures focused on preventing underage migration, including commissioning an independent investigation. The investigation found that the safeguards we had in place were robust, though ultimately could not fully eliminate the risk of an adult recipient choosing to pass their cash transport subsidy to a teenager in his place, contrary to program rules and protocols. These protocols were multilayered, and included verbally informing subsidy recipients of the condition that migrants must be at least 18 years of age; requiring subsidy recipients to sign or thumbprint an acknowledgement that both recipients and migrants (where different individuals) must be at least 18 years of age; reviewing national identification cards to verify that the subsidy recipient and any person that the recipient says plans to migrate from the household is at least age 18; and utilizing mobile data collection software that is programmed to prohibit field staff from including individuals reporting to be under the age of 18, in order to prevent accidental enrollment.
In parallel, we analyzed both program administrative data and data from past and ongoing RCTs of the program to understand the scope of underage migration in the program and, importantly, the causal impact of the program on underage migration. We recognize that underage migration occurs in Bangladesh, where severely limited options may lead a deeply impoverished family to believe that sending a teenage son to the city to find work is their best available option. As such, we analyzed the RCT data to understand whether the program may have inadvertently led to additional underage migration beyond that which would have occurred in the absence of the program. We have concluded that, across RCTs conducted in 2014, 2017, and 2018, there is no evidence to suggest such a causal relationship.
Next steps
Importantly, we are not saying that seasonal migration subsidies do not work nor that they lack impact. Seasonal migration subsidies may be a useful and cost-effective tool for addressing the critical issue of seasonal poverty. What we are saying, however, is that No Lean Season is unlikely to be among the best strategic opportunities for Evidence Action to achieve our mission, and as such we are exiting it from the Beta portfolio.
We are nonetheless committed to ensuring that the at-scale testing of the program underway with our research partners can inform the evidence base on seasonal migration subsidies, and therefore will complete the RCT that is currently being conducted by an independent organization. Though there is low likelihood of our further work in this area, we will consider the findings from the trial, once available, to inform our organizational view on seasonal migration as an intervention.
As a result of our decision to terminate No Lean Season, our commitment to ensuring good stewardship of donor funding comes to the fore. We stopped fundraising for No Lean Season in November 2018 due to the 2017 RCT results and continued to use funds already received for the program for our existing commitments; we will continue to do so to complete the ongoing RCT and undertake a responsible close-out of the program. Evidence Action expects that some program funding received from foundations under grant agreements will remain following the completion of these activities. We are engaging with these donors and, where possible, will seek to reallocate the unspent funds to other evidence-based, cost-effective programs within our organization. Though all funds donated directly by individual donors to Evidence Action for No Lean Season have been used for existing program commitments, donors may contact [email protected] to request reallocation of a portion of the donation to another Evidence Action program. Any request of this nature should be made by July 31, 2019.
We also want to ensure that the takeaway is not that if a program faces challenges, an NGO should walk away from doing work that measurably improves the lives of tens or hundreds of thousands of people. International development programs are complex – operating in high-risk settings where the risk of corruption and a whole host of other challenges is high. Moreover, the reality of working on the ground – and at scale – is often complicated. Our job must be to cost-effectively and responsibly mitigate these risks, rather than turn away from opportunities that dramatically improve lives.
Evidence Action is and will continue to seek to be transparent in sharing these risks and challenges, consistent with our organizational values and those of many of our supporters and partners. To the extent possible, we share the issues as we face them so that we can contribute to an active exchange of learnings among our partners and the broader development community. By doing so, our goal is to ultimately drive better outcomes for the hundreds of millions of people living in poverty that we all seek to serve.
This has been shared before in a very similar context; do you still want this post up?
I would recommend this post over the GiveWell one as a case study / postmortem on charity entrepreneurship.
While it covers similar ground, the GiveWell post (which is essentially a metacommentary on this one) seemed to be written partly with the intention of reassuring donors to GiveWell that they shouldn't update too negatively. This post felt like a more straightforward summary of Evidence Action's decision-making process about No Lean Season.
I think this post also more clearly emphasizes the various factors that contributed to the decision to shut down the program; not just uncertainty about its impact but also the need to relaunch with a new partner, such that:
This conclusion is also expressed in the GiveWell post, but I found it more clear here.