Tyler Cowen posted a link to this paper(PDF), outlining how effective programs are when transported to new contexts, or scaled up by governments.
Two key quotes:
The program implementer is the main source of heterogeneity in results,
with government-implemented programs faring worse than and being poorly predicted
by the smaller studies typically implemented by academic/NGO research teams, even
controlling for sample size
The average intervention-outcome combination is comprised 37% of positive, significant studies;
58% of insignificant studies; and 5% of negative, significant studies. If a particular result is positive
and significant, there is a 61% chance the next result will be insignificant and a 7% chance the
next result will be significant and negative, leaving only about a 32% chance the next result will
again be positive and significant.
Thanks for sharing, this is important data. My experience working in non-profits throughout my career (and the contact with the charity world that it gave me) has steadily moved me to the conclusion which that paper speaks towards: namely that having a diligent program implementer, which is sincerely focused on valuable end goals, is comparably important to having a program which seems effective on the face of it. It's interesting that " the smaller studies typically implemented by academic/NGO research teams" typically had better results - it suggests that they might be skewed towards being unrepresentatively positive.
The document and summary makes a good read and a good point, viewing fig1 and fig2 alone is good thought provoking stuff. I personally am not surprised by the findings, they simply show we need more people to adopt an EA approach towards and within such programs and organisations. Less is more for sure. Less is even more when enough of it is done and learnt from! Im with Tom!
This sort of assessment, while different, is not a million miles from the experience of most private let alone public sector spending/investments. The 3rd and 4th sector are no differe... (read more)