Epilogue: I am happy to report that the donors have decided today to donate 1 million dollars (CAD) over the next five years to support Teaching at the Right Level, Africa. This initiative is helping to change education systems in African countries so that students in primary school can learn to read, write and count (which is sadly not the case right now, for the most part). Since writing my report, Teaching at the Right Level, Africa has also been reviewed and recommended by Founders Pledge.
I thank all of you for your thoughts and comments and am very excited to have been able to contribute to this :)
@HStencil, TechnoServe does seem like an interesting organization. There is definitely some similarity to Village Enterprise, but Village Enterprise is focused on the graduation approach while TechnoServe runs a broader range of programs including support to small and medium enterprises. I was also very interested in small and medium enterprise support, but my review of the evidence on this suggests that it's inconclusive whether such programs have had success at creating jobs or reducing poverty. Of course, it's entirely possible that TechnoServe's programs do indeed do this. If so, that would be great! ImpactMatter's evaluation does suggest this, but as they point out the evidence is not very strong: it's based on a pre-post analysis conducted by TechnoServe themselves, response rates post-intervention were low and enterprises that did not experience increased revenues were excluded from the analysis (see excerpt from ImpactMatter's evaluation below).
"Impulsa Tu Empresa’s beneft/cost ratio is based on data collected by TechnoServe on businesses’ gross revenues before and after participating in the program. We subtract from these figures the growth we assume businesses would have experienced had they not in fact participated in the program. However, only a small number of firms responded to TechnoServe’s surveys conducted two to three years after they completed the program. TechnoServe also did not count businesses that had lower revenues after the intervention than before the intervention, biasing upward the benefit/cost ratio"
tomwein, I just looked through the RCT. The reduction in violence is indeed significant and promising. In terms of test scores, though, they report "There was no evidence that the intervention had an impact on any educational test scores". Test scores and income were the main metrics we were focusing on for education interventions, so this intervention doesn't seem like a good match to the donors' preferences at this stage. From reading the RCT, it seems that studies about violence prevention in school are quite novel and that the evidence base is still thin, so again this wouldn't match well with the donors' wish to support interventions with strong evidence of effectiveness. Though obviously it's very valuable to be gathering evidence given the lack of previous studies.
samcart, just a quick note that TaRL Africa is (or was) part of J-PAL's Innovation in Government Initiative (or its predecessor, the Government Partnership Initiative). See these links:
@smclare Thanks for giving some background on the Founders Pledge cost-effectiveness scenarios. For TaRL, I'm surprised that you say that the optimistic scenario is the unfathomably best case scenario. Even in that scenario, impacts are assumed to last 20 years, and the impact of test scores improvements on earnings does not use the most optimistic cases mentioned in the Founders Pledge education report. It seems fathomable impacts could last a whole career (say 40 years). As you can see from my cost-effectiveness estimates for TaRL, my unfathomably best case scenario is significantly more optimistic than the one from Founders Pledge (I included in the cost-effectiveness spreadsheet a worksheet using the worksheet from Founders Pledge as a starting point, but with my own scenarios in there). And in both cases, we only include the impact on income. It seems quite plausible that education would have impacts beyond that which aren't taken into account.
@smclare: I'm glad you liked the report :) I definitely hope it can be helpful to others, since alot of work went into it! If it can save others some time, that would be great!
I'll check out the Marginal Revolution post on Millenium Villages and see if I can include a few sentences about that in the report.
As for TaRL Africa, alot of what they are doing is directly implementing TaRL in different African countries with partners and trying to ensure that the scale up there is a success. So I don't think of it as being mainly advocacy. You're right that there's no guarantee that it will work. I think there is a tension between the donors' goal of backing things that have been shown to work and having a long term and transformative impact, since the evidence on long term impacts is generally lacking (beyond say 7 years or so at most, in the case of the graduation approach). I tried to find a middle ground between their different goals. One of the reasons I included various charities in Table 1 was a recognition that different interpretations of their goals might lead to different recommendations.
Regarding the other organizations that implement the graduation approach in Africa, I did indeed look into them (see the report on cash transfers for details). BRAC might have been a good choice, but they run a very wide range of programs, and it was alot harder to figure out what would be the net impact of even a targeted donation to their graduation approach program (see e.g. this GiveWell post on BRAC). I was also influenced by discussions with Founders Pledge who also looked into these different charities and ended up recommending Village Enterprise.
As for J-PAL's Innovation in Government Initiative, I personally find it very interesting. I don't know if it would be a good fit with the donors' preferences. They will be going over the comments here, though, so they'll get a chance to see some of the other options that have been proposed.
Responses to the questions you posted:
1)Yes, I did try to assess organizations' strength and track record, though admittedly that was harder to do than going over RCTs and literature. It was part of the motivation for recommending TaRL Africa and Village Enterprise. I'll look over the text again and see if I can make that clearer. Mind you, all the other organizations that made it to Table 1 also seem to have a strong track record. I partly relied on the fact that many of these had been vetted by GiveWell and Founders Pledge.
2) and 3) I'll do that, thanks for the suggestion :)
I think the donors do indeed intend to commit for 5 years, for the reason tomwein invokes. But of course if new evidence suggests an intervention really isn't having the impact that we expected, or something else that seems much more promising comes along, presumably they could still revisit their commitment on an annual basis.
Regarding TaRL, the intervention has been studied extensively. The main uncertainty is whether and to what extent gains in test scores translate to long-term outcomes like higher income. But since the donors also care about improvements in learning outcomes per se, there is a bit of a hedge here. It just isn't captured in the cost-effectiveness analyses, which only incorporate effects on income.
Thanks for your feedback Carl, and for the links to the Open Phil reports on Center for Global Development and Target Malaria. I will check them out. One thing I have to keep in mind though is the donor's preferences/mandate. While these are somewhat fuzzy/open-ended, they did not seem very interested in policy advocacy or interventions where there isn't yet clear evidence of impact.
Epilogue: I am happy to report that the donors have decided today to donate 1 million dollars (CAD) over the next five years to support Teaching at the Right Level, Africa. This initiative is helping to change education systems in African countries so that students in primary school can learn to read, write and count (which is sadly not the case right now, for the most part). Since writing my report, Teaching at the Right Level, Africa has also been reviewed and recommended by Founders Pledge.
I thank all of you for your thoughts and comments and am very excited to have been able to contribute to this :)
@HStencil, TechnoServe does seem like an interesting organization. There is definitely some similarity to Village Enterprise, but Village Enterprise is focused on the graduation approach while TechnoServe runs a broader range of programs including support to small and medium enterprises. I was also very interested in small and medium enterprise support, but my review of the evidence on this suggests that it's inconclusive whether such programs have had success at creating jobs or reducing poverty. Of course, it's entirely possible that TechnoServe's programs do indeed do this. If so, that would be great! ImpactMatter's evaluation does suggest this, but as they point out the evidence is not very strong: it's based on a pre-post analysis conducted by TechnoServe themselves, response rates post-intervention were low and enterprises that did not experience increased revenues were excluded from the analysis (see excerpt from ImpactMatter's evaluation below).
Excerpt from ImpactMatter's evaluation report:
"Impulsa Tu Empresa’s beneft/cost ratio is based on data collected by TechnoServe on businesses’ gross revenues before and after participating in the program. We subtract from these figures the growth we assume businesses would have experienced had they not in fact participated in the program. However, only a small number of firms responded to TechnoServe’s surveys conducted two to three years after they completed the program. TechnoServe also did not count businesses that had lower revenues after the intervention than before the intervention, biasing upward the benefit/cost ratio"
tomwein, I just looked through the RCT. The reduction in violence is indeed significant and promising. In terms of test scores, though, they report "There was no evidence that the intervention had an impact on any educational test scores". Test scores and income were the main metrics we were focusing on for education interventions, so this intervention doesn't seem like a good match to the donors' preferences at this stage. From reading the RCT, it seems that studies about violence prevention in school are quite novel and that the evidence base is still thin, so again this wouldn't match well with the donors' wish to support interventions with strong evidence of effectiveness. Though obviously it's very valuable to be gathering evidence given the lack of previous studies.
samcart, just a quick note that TaRL Africa is (or was) part of J-PAL's Innovation in Government Initiative (or its predecessor, the Government Partnership Initiative). See these links:
https://www.povertyactionlab.org/blog/1-16-18/increasing-use-data-and-evidence-real-world-policy-stories-j-pals-government
https://www.povertyactionlab.org/blog/8-8-17/zambia-scale-teaching-right-level-program-1800-schools
@smclare Thanks for giving some background on the Founders Pledge cost-effectiveness scenarios. For TaRL, I'm surprised that you say that the optimistic scenario is the unfathomably best case scenario. Even in that scenario, impacts are assumed to last 20 years, and the impact of test scores improvements on earnings does not use the most optimistic cases mentioned in the Founders Pledge education report. It seems fathomable impacts could last a whole career (say 40 years). As you can see from my cost-effectiveness estimates for TaRL, my unfathomably best case scenario is significantly more optimistic than the one from Founders Pledge (I included in the cost-effectiveness spreadsheet a worksheet using the worksheet from Founders Pledge as a starting point, but with my own scenarios in there). And in both cases, we only include the impact on income. It seems quite plausible that education would have impacts beyond that which aren't taken into account.
@smclare: I'm glad you liked the report :) I definitely hope it can be helpful to others, since alot of work went into it! If it can save others some time, that would be great!
I'll check out the Marginal Revolution post on Millenium Villages and see if I can include a few sentences about that in the report.
As for TaRL Africa, alot of what they are doing is directly implementing TaRL in different African countries with partners and trying to ensure that the scale up there is a success. So I don't think of it as being mainly advocacy. You're right that there's no guarantee that it will work. I think there is a tension between the donors' goal of backing things that have been shown to work and having a long term and transformative impact, since the evidence on long term impacts is generally lacking (beyond say 7 years or so at most, in the case of the graduation approach). I tried to find a middle ground between their different goals. One of the reasons I included various charities in Table 1 was a recognition that different interpretations of their goals might lead to different recommendations.
Hi samcart,
I'm glad you liked the report :) Thanks for the detailed and constructive feedback! Note: One reason that it isn't too long is that other reports preceded it (on education, early childhood development, small and medium enterprise support, rural electrification and long-term effects of cash transfers). This final report tries to summarize things.
Regarding the other organizations that implement the graduation approach in Africa, I did indeed look into them (see the report on cash transfers for details). BRAC might have been a good choice, but they run a very wide range of programs, and it was alot harder to figure out what would be the net impact of even a targeted donation to their graduation approach program (see e.g. this GiveWell post on BRAC). I was also influenced by discussions with Founders Pledge who also looked into these different charities and ended up recommending Village Enterprise.
As for J-PAL's Innovation in Government Initiative, I personally find it very interesting. I don't know if it would be a good fit with the donors' preferences. They will be going over the comments here, though, so they'll get a chance to see some of the other options that have been proposed.
Responses to the questions you posted:
1)Yes, I did try to assess organizations' strength and track record, though admittedly that was harder to do than going over RCTs and literature. It was part of the motivation for recommending TaRL Africa and Village Enterprise. I'll look over the text again and see if I can make that clearer. Mind you, all the other organizations that made it to Table 1 also seem to have a strong track record. I partly relied on the fact that many of these had been vetted by GiveWell and Founders Pledge.
2) and 3) I'll do that, thanks for the suggestion :)
I think the donors do indeed intend to commit for 5 years, for the reason tomwein invokes. But of course if new evidence suggests an intervention really isn't having the impact that we expected, or something else that seems much more promising comes along, presumably they could still revisit their commitment on an annual basis.
Regarding TaRL, the intervention has been studied extensively. The main uncertainty is whether and to what extent gains in test scores translate to long-term outcomes like higher income. But since the donors also care about improvements in learning outcomes per se, there is a bit of a hedge here. It just isn't captured in the cost-effectiveness analyses, which only incorporate effects on income.
Thanks for the recommendation, tomwein. Could you provide a link to the RCTs or other experimental evidence?
Thanks for your feedback Carl, and for the links to the Open Phil reports on Center for Global Development and Target Malaria. I will check them out. One thing I have to keep in mind though is the donor's preferences/mandate. While these are somewhat fuzzy/open-ended, they did not seem very interested in policy advocacy or interventions where there isn't yet clear evidence of impact.