Epistemic status: I think this is a statistical “fact” but I feel a bit cautious since so few people seem to take advantage of it
Summary
It may not always be optimal for cost or statistical power to have equal-sized treatment/control groups in a study. When your intervention is quite expensive relative to data collection, you can maximise statistical power or save costs by using a larger control group and smaller treatment group. The optimal ratio of treatment sample to control sample is just the square root of the cost per treatment participant divided by the square root of the cost per control participant.
Why larger control groups seem better
Studies generally have equal numbers of treatment and control participants. This makes intuitive sense: a study with 500 treatment and 500 control will be more powerful than a study with 499 treatment and 501 control, for example. This is due to the diminishing power returns to increasing your sample size: the extra person removed from one arm hurts your power more than the extra person added to the other arm increases it.
But what if your intervention is expensive relative to data collection? Perhaps you are studying a $720 cash transfer and it costs $80 to complete each survey, for a total cost of $800 per treatment participant ($720 + $80) and $80 per control. Now, for the same cost as 500 treatment and 500 control, you could have 499 treatment and 510 control, or 450 treatment and 1000 control: up to a point, the loss in precision from the smaller treatment is more than offset by the 10x larger increase in your control group, resulting in a more powerful study overall. In other words: when your treatment is expensive, it is generally more powerful to have a larger control group, because it's just so much cheaper to add control participants.
How much larger? The exact ratio of treatment:control that optimises statistical power is surprisingly simple, it’s just the ratio of the square roots of the costs of adding to each arm i.e. sqrt(control_cost) : sqrt(treatment_cost) (See Appendix for justification). For example, if adding an extra treatment participant costs 16x more than adding a control participant, you should optimally have sqrt(16/1) = 4x as many control as treatment.
Quantifying the benefits
With this approach, you either get free extra power for the same money or save money without losing power. For example, let’s look at the hypothetical cash transfer study above with treatment participants costing $800 and control participants $80. The optimal ratio of control to treatment is then sqrt(800/80) = 3.2 :1, resulting in either:
Saving money without losing power: the study is currently powered to measure an effect of 0.175 SD and, with 500 treatment and control, costs $440,000. With a 3.2 : 1 ratio (*types furiously in Stata*) you could achieve the same power with a sample of 337 treatment and 1079 control, which would cost $356,000: saving you a cool $84k without any loss of statistical power.
Getting extra power for the same budget: alternatively, if you still want to spend the full $440k, you could then afford 416 treatment and 1,331 control, cutting your detectable effect from 0.175 SD to 0.155 SD at no extra cost.
Caveats
Ethics: there may be ethical reasons for not wanting a larger control group, for example in a medical trial where you would be denying potentially life-saving treatments to sick patients. Even outside of medicine, control participants’ time is important and you may wish to avoid “wasting” it on participating in your study (although you could use some of the savings to compensate control participants, if that won’t mess with your study).
Necessarily limited samples: obviously if there is a practical limit to increasing your control group size, such as only being able to operate in a limited geography, this may not be an option.
Natural skepticism? This isn’t a common technique, you might just trust that the market for ideas is efficient and if this really was a thing you would have heard about it from somewhere else by now. It kind of blows my mind that this isn’t done more often, which both makes me want to tell people about it and be skeptical. We used this approach for a pretty large RCT I worked on in Tanzania, and no one complained.
Conclusion
If you treatment is quite expensive relative to data collection costs, consider using a larger control group in the ratio of sqrt(treatment_cost/control_cost) and enjoy that spare money or additional statistical power.
Appendix
I am not claiming to have discovered this myself. I first read this equation in Running Randomized Evaluations and was able to derive the same result myself here.
I believe this holds for cluster RCTs, just remember that the increased control sample here would come in the form of additional control clusters, rather than larger clusters.
If you are doing power calculations in Stata and want to factor in different treatment/control group sizes, you just add ratio(X) to the sampsi command, where “X” is the treatment/control ratio. For a cluster RCT using clustersampsi you... need to do something involving harmonic means, I forget exactly, but poke me on the Forum and I'll happily dig through some old code.
The idea makes a lot of sense, but my guess is that the circumstance where the cost is driven by the intervention itself isn’t that common: In the context of charities, we’re thinking about applying RCTs to test whether an intervention works. Generally the intervention is happening anyway. The cost of RCTs then doesn’t come from applying the intervention to the treatment group - it comes from establishing the experimental conditions where you have a randomised group of participants and the ability to collect data on them.
Hey Aidan-- that's a good point. I think it will probably apply to different extents for different cases, but probably not to all cases. Some scenarios I can imagine:
Overall, I think cases 2/3/4 benefit from the cheaper study. Scenario 1 seems more like what you have in mind and is a good point, I just think there will be enough scenarios where the cheaper trial is useful, and in those cases the charity might consider this treatment/control optimisation.
Thanks Rory - I think your general idea is good, and in some cases could be a good option!
I could be wrong, but from my experience working in the development world these 4 scenarios aren't really how RCTs generally happen. Usually there will be a partnership with a RCT running NGO (like IPA) or a university department (J-PAL at MIT) where the partner organisation pays for and organise everything.
Sometimes scenario 4 could happen as part of a grant application
This doesn't change the existence of a budget constraint, though. The partner organization, especially a grant funder like JPAL/IPA, will grant you a certain amount of their resources to use. I don't see why you wouldn't want to optimize the use of their resources.
100% the original post stands, in any scenario we would want to optimise use of resources. I don't think JPAL/IPA is generally a funder though - they do the research themselves so they are the ones to convince ;).
Ah, that's helpful data. My experience in RCTs mostly comes from One Acre Fund, where we ran lots of RCTs internally on experimental programs, or just A/B tests, but that might not be very typical!
Would be super interested to see the results of some of these RCTs / AB tests. Were any of them published apart from the Lime SMS study? We're looking for great examples of learning orgs that do this and some studies from 1AF would be a great motivator/example.