Just to say, and speaking from the Happier Lives Institute perspective, I really liked both of the thoughtful precommittment and the care you've taken to plan this RCT. We look forward with great interest to digging into this further!
I really admire your commitment to gathering rigorous evidence and the transparency around the process! Wishing you the best of luck with the study, I’m very excited to see the results.
One suggestion: you mentioned IPV prevalence in Rwanda is ~25%*, but you're only powered to detect a 2pp effect? 2pp reduction with a baseline 25% prevalence is an 8% relative reduction. Anything below that is unlikely to be significant. You're unable to reliably detect a 6 or 7% relative reduction which is above your 5% bar, but would still send you into "pause and explore". I might be better to commit on the confidence interval rather than on significance. Lakens' work on the smallest effect size of interest (SESOI) and equivalence testing could be useful here, since it would let a null result mean you have ruled out effects above some size rather than just failed to find one.
*across a year, while you're measuring across four months. Baseline prevalence is probably smaller in that time period, meaning the minimum detectable effect gets larger in relative terms.
Thanks Laura, you are right that we cannot be sure a 5% reduction would come out significant, since that depends on the baseline prevalence. We do expect prevalence in our study population to be somewhat higher than the national figure, as we are working in mostly rural areas. Thank you for mentioning SESOI! We will look into it and raise it with the study PIs.
I understand you are planning to measure against another show as a control. Which makes a lot of sense, but raises some questions: Have you accounted for a possibility that control treatment itself has effect on gender norms related to IPV? There is at least some evidence that exposure to media ( e.g. Jensen/Oster in India) has effect on gender norms and views on women autonomy. If control has positive effect, you would need to be comparing two potentially beneficial interventions, not your show vs. no program. Have you costed the control show? Also, you intention is to change attitudes to IPV, if you rely on self reports, how will you distinguish actual change in violence from treatment-induced change in reporting rates? I could see how change in attitudes may both increase the rates it is disclosed because the behavior is seen as more objectionable, or decrease them because there is now more stigma attached to the behavior.
On the control condition: yes, in principle the control programme could itself affect gender norms or related outcomes. We tried to minimise that risk by choosing a control programme with no gender, relationship or violence content at all (it's about how to declare taxes mostly).
On reporting: our primary IPV measures reduce this problem by asking about specific acts within a defined window rather than asking whether someone experienced or perpetrated "abuse" or "violence." That is the standard approach in the field, and it makes responses much less dependent on how people label the behaviour.
Even so, treatment-induced changes in reporting remain possible, as they do for any violence-prevention programme measured by self-report. Behaviourally specific items remove some forms of reporting bias but not all of them. So we will read the IPV results alongside attitudes, controlling behaviours, communication and other secondary outcomes.
I have a lot of admiration for your transparency and public commitment. In general, I think this kind of public commitment and pre-analysis planning for key evaluation results is laudable and pushes the sector towards evidence-based decision making.
That said, I do have some concerns about setting a pre-commitment on an effect size from an RCT, particularly at such an early stage of an intervention's maturity. How are you thinking about mitigating the risk of a null result? I understand that in that scenario you'd pause for a few months and conduct exploratory research. But would there not be an opportunity, at earlier stages of your theory of change, to better test the inputs-to-outputs and outputs-to-outcomes links through things like:
simple observational studies
mixed methods
process evaluations, run in parallel with or prior to the RCT?
I'll put my biases on the table. I think RCTs are incredibly useful tools, but they're often not the most helpful tools for understanding the complex theory of change behind an intervention. They give you a single output, usually an effect size, that can be hard to disentangle and act on operationally, especially if that effect size turns out to be null.
Thanks Tony! Completely agree that the RCT are not the most helpful tools to understand ToCs. Both our first and second seasons were created through an iterative process that involved a lot of formative (qualitative research) that helped us get more insights in how listeners actually perceive the drama and its characters. See for example our (small) mix method evaluation of season 2 prior to the RCT (http://www.novah.ngo/qual-eval-twubakane-s2-infographic).
I appreciate the commitment to rigor here! I'd be curious if there are any non-linearity or saturation effects of listenership on outcomes, or if that has already been analyzed in other studies.
Thanks Warner! This has not been studied well yet in the context of edutainment and violence prevention. Some studies suggest higher dosis is higher effect, but the literature is thin. Our RCT is not designed to give answers on this as well, but we will ook at listening dose against outcomes descriptively at endline.
What would make us scale or stop? NOVAH's pre-commitment before seeing the RCT results
Disclaimer: This blog post focuses on a new piece of research from Rethink Priorities. I was not involved with the funding or execution of this research, and therefore write this piece merely as a consumer. However, I am the Executive Director of Giving Green and a board member of Rethink Priorities, and acknowledge that these affiliations may bias my...
I am writing up two charity ideas I’ve wanted to investigate, but haven’t been able to, in the hopes someone will investigate them further. If you end up investigating either, I'd be most interested to see what conclusions you come to!
Vulture preservation as a global health intervention
Since the mid 1990's the vulture population in India has been rapidly declining. The reason is due to a few chemicals which are fed to livestock, which is highly poisonous to vultures.
...
Summary
1. With new funding potentially entering the effective altruism ecosystem, we wanted to estimate how much more the wild animal welfare (WAW) movement could productively absorb. We asked organizations with a public, dedicated WAW workstream to estimate their own room for more funding (RFMF). Although this does not capture the whole movement, we believe it captures a substantive fraction of it.
2. Summed across the five organizations that took part, the estimates point to...
NOVAH (No Violence At Home) was incubated by Charity Entrepreneurship (now Ambitious Impact) in 2024 to test a promising idea: preventing intimate partner violence through edutainment, in our case a serialised radio drama. Over the past two years we have produced and aired two seasons in Rwanda.
We are currently evaluating our second season through a randomized controlled trial with 2,400 couples in Rwanda in partnership with Innovations for Poverty Action (IPA). Preliminary results are expected in October 2026. The end-line analysis is run by IPA and our academic partners, not by NOVAH.
Edutainment to prevent intimate partner violence ispromising, but it is still young as a field. We do not want to spend donor money scaling it unless the evidence shows it works at least as well as the strongest existing approaches. The RCT is what will tell us that. Therefore, before seeing those results, we want to make our decision rule public:
We will scale our programming only if the RCT finds a statistically significant relative reduction of at least 5% in physical or sexual intimate partner violence prevalence. How quickly we proceed will depend on the size and certainty of the effect.
Our current plan is:
RCT result (relative change in violence prevalence)
What we will do
Below 5%
Pause, investigate what happened, and decide whether to adapt the programme or stop. Publish what we learned either way.
5%–20%
Proceed with our planned 2027 programme. Budget ~550k$.
Above 20%
Accelerate our 2027 programme. Budget ~750k$.
We are sharing this now to anchor and commit as an organization to only continue if our programming meets our threshold.
Note: AI was used to help writing and polishing this post, but ideas are ours.
Background on NOVAH’s cause area and intervention
Globally, 1 in 4 women are beaten or sexually assaulted by their partner in their lifetime, and 1 in 4 each year in Rwanda.
NOVAH was founded in 2024 to test whether radio edutainment could prevent domestic violence (“intimate partner violence”, IPV) cost-effectively. At that time, and still today, the intervention appeared particularly promising but faced a relatively thin literature.
Our radio drama, Twubakane (“Let’s build together”) follows couples dealing with recognizable challenges around money, alcohol, communication, gender roles and conflict. Rather than telling listeners what to do, the programme shows characters making mistakes, experiencing consequences and gradually changing their behaviour with the help of other role model characters.
In early 2025, our first six-episode season aired regionally and reached an estimated 30,000 listeners in Rwanda.
The initial findings were encouraging. In qualitative interviews, listeners frequently described better communication, more joint decision-making and healthier relationships. A pilot study with 200 couples found a suggested six-percentage-point decline in women’s reports of physical or sexual violence—equivalent to a 40% relative reduction—as well as improvements in relationship quality and women’s participation in financial decisions. However, the study was small and not rigorously designed, so it could not provide definitive evidence of impact. We treated its findings as a reason to run a better test, not as proof that the intervention works.
We decided to run a rigorous study, with the idea that positive or negative results could be particularly helpful both for us and the ecosystem. Thanks to the help of many partners and supporters, we were able to do a large-scale randomized controlled trial (RCT), in a naturalistic setting.
How we are evaluating our impact
We have now completed a second season of 12 episodes, which we recently aired regionally as the implementation of the RCT. We expect preliminary RCT results in October 2026. We deliberately held back national broadcasts until endline data collection was complete, so that control villages were not exposed to the programme before we measured them. Unless the results indicate that the programme is causing harm, we plan to air nationally shortly after the end-line.
Although it is not the key objective of the post, we make a short detour to explain the RCT’s design, as we know mass media interventions are particularly hard to evaluate - feel free to skip this and move to the next section.
The evaluation includes ~2,500 couples (5,000 participants), living across ~100 villages in Northern Rwanda. We distributed one solar-powered radio for each household and trained 2 youth volunteers per village to support the programme and encourage couples to listen each week, on top of small monthly incentives for couples to listen. Randomization took place at the village level: in half of the villages, couples are incentivised to listen to our program on a regional radio station, and others are encouraged to listen to an unrelated radio program airing on the country’s most popular radio station at the same time on Monday evening.
The evaluation is being conducted with Innovations for Poverty Action (IPA) and academic research partners. It measures a large range of outcomes and impact metrics from our theory of change, in particular physical and sexual IPV, subjective wellbeing, mental health, communication quality, gender attitudes, and alcohol use. The analysis plan is being preregistered publicly.
Our pre-commitment
Our central rule is:
We will proceed with our programming only if the RCT finds a statistically significant relative reduction of 5% or more in physical or sexual IPV.
For example, if IPV prevalence in the comparison group were 20%, a 5% relative reduction would correspond to prevalence of 19% in the intervention group.
Note: The exact metric is any self-reported physical or sexual IPV in any of the two recall periods (last 4 months, last month). This bar is not a prediction of what the RCT will find. We will only consider statistically significant effects (depending on IPV’s baseline, the RCT may not be powered to detect a 5% relative reduction, as it has been designed to detect a 2pp effect).
Even if IPV is reduced by more than 5%, if any safety concern arises (e.g. a negative effect for depression or a reduction in one form of violence alongside an increase in another) and/or we discover serious implementation problems or evidence of harm, we would not proceed with airing the show nationally.
Below, we first discuss our plans depending on results and this pre-commitment more broadly.
What happens under each result?
Effect below 5%, or not statistically significant: pause and explore
We would pause our expansion plans for a few months, and conduct exploratory research. We would then make one of two decisions:
There is a specific and credible explanation for the weak result, and an adaptation is sufficiently promising to justify another test; or
We shut down NOVAH (as this intervention) and openly share what we learned.
Effect of 5%–20%: continue as planned
We would implement our central programme for 2027: producing a new season in Rwanda and in Burundi, and exploring a third country of operations. We would continue developing processes and methodologies for others to replicate our work from 2028 onward. The planned budget for 2027 is approximately 550k$.
Effect above 20%: accelerate
Results would land in the "exceptionally good" category and we would aim to accelerate, by working in an additional country already in 2027 (for an approximately 750k$ budget).
Why did you choose a 5% reduction?
Our currently projected cost-effectiveness is ~$60 per IPV-year prevented, but we have set ourselves a bar of $200 per IPV-year prevented. This bar reflects the estimates from the current most cost-effective interventions in the field (*).
(*) According to TLYCS, the implementation of SASA! by CEDOVIP achieves $150 but doesn’t take into account overhead organization costs. Another community mobilization program finds $194.
A 5% relative reduction in IPV, under our current assumptions, would make NOVAH meet the bar of $200-per-IPV-year averted. Under our current assumptions:
Relative reduction in IPV
Approximate cost per IPV-year
Interpretation
Above 20%
Below $50
Strong enough to consider accelerating
5%–20%
$50–$200
Clears our current bar
Below 5%
Above $200
Does not clear our current bar
Is everything driven by this RCT?
These RCT results are only the first decision gate. A media intervention needs to both work and reach people cheaply.
The RCT tells us whether Twubakane changes outcomes among the population offered and encouraged to listen - which seems the hardest challenge.
It does not tell us how many people will choose to listen during an ordinary national broadcast. Our current model assumes 7% active listenership during a first national broadcast - a hypothesis we will test in the Q4 2026 airing and measure in a survey planned for January 2027. The combination of the two pieces should lead to a robust CEA.
Low listenership would be a different kind of problem from a null result. If the January 2027 survey shows listenership well below 7%, we are reasonably confident we can improve this: the main levers (station and time slot, co-broadcast partnerships, promotion, and adapting the show for a national rather than a regional audience) are ones we control. We would treat weak reach as something to fix rather than a reason to stop. A null or negative effect in the RCT would, for us, suggest that something more fundamental is not working in our intervention.
Why make this public?
Once an organisation has hired staff, developed partnerships, raised money and spent several years building a programme, there are strong reasons to keep going, and we do not think we are immune to that.
We are hopeful about the results (especially when reading listeners’ feedback) but we want to stay true to the commitment we made when starting: we want to design and scale an intervention that is more cost-effective than the current best alternative. Also, we do not want to use valuable donor money to scale interventions that don’t work.
How you can hold us to this
A pre-commitment is only worth something if someone outside the organisation can check it. Three things make ours checkable.
We have pre-defined in advance exactly the metric that will be used.
The estimate is produced by IPA and our academic partners, who are external to NOVAH. We do not compute the number we are judged against.
We will publish the results on this Forum.
Should more non-profits do this?
We are also curious whether more evidence-focused NGOs should make similar public commitments, or whether the disadvantages outweigh the benefits.
There are pros (avoiding bias, being transparent, staying evidence-based) and cons (false precision from a numerical threshold, strategic optimization, etc.) for such communications. Also, pre-commitments are easier for direct work organizations than for research, policy, or ecosystem building ones.
We plan to share more on that after we get the results.
Final words
When the preliminary results become available, we will come back to this post and explain what they mean for NOVAH, including if the conclusion is that we should stop.
If you find this approach useful and would like to back it, we would welcome conditional support. Our conditional support plan for funders sets out what a commitment at each threshold would look like. It lets a funder commit now on the condition that the evidence clears the bar.
If you have any feedback on our approach, please get in touch.
Note: if the effect is 5% or higher, in ~October 2026, we would hire a Head of Research, responsible for the research parts of our Iterative Content Production process in Rwanda and in Burundi - subscribe to our newsletter so you don't miss the job ad!
Why do you use IPV-year instead of DALYs in your cost-effectiveness?
We believe this metric best reflects our work, although this choice has been the result of tradeoffs between different options (see table). We also use WELLBYs and DALYs as complementary indicators.
NOVAH’s cost-effectiveness indicators and the tradeoffs they offer:
Cost-effectiveness metric
Importance for NOVAH
Robustness & Credibility
Alignment with our mission
Comparability (helpful for NOVAH internally and for donors)
Low - we believe the existing data to convert IPV to DALYs is conservative
Direct
Life-saving interventions
Correct
~320$ (***)
(*) Self-reported by women, but this follows all standards in the domain
(**) We work with the Happier Lives Institute to ensure we follow best practices in this domain - we are very grateful to them for their support and time
(***) We have recently updated our CEA, working with external researchers and revising assumptions. This led us to lower our estimate of the DALY burden caused by a year of IPV and we think the current version is more conservative and credible.
Just to say, and speaking from the Happier Lives Institute perspective, I really liked both of the thoughtful precommittment and the care you've taken to plan this RCT. We look forward with great interest to digging into this further!