Why randomized controlled trials matter

Hello! I'm Saloni Dattani and I work at Our World in Data.

I wanted to share an article I wrote recently for Our World in Data, where I explain how randomized controlled trials (RCTs) work and why (and when) they matter: http://ourworldindata.org/randomized-controlled-trials

Since RCTs are considered a high-quality source of evidence for our knowledge of the effects of treatments, policies, and interventions, I think it’s important to understand how they work to improve our ability to read scientific literature and to help us make better decisions.

I make two main arguments:

1. That RCTs are a powerful source of evidence because of the procedures that they are expected to follow. For each, I illustrate how they work and how they might affect the results of studies with examples.

• A control group, which gives us the possibility to see a counterfactual (“what might happen otherwise”).
• Placebos, which can account for placebo effects. For example, people may feel better from taking  a pill because they expect it to make them feel better. More generally, these are changes that occur because of the procedure of the treatment rather than the treatment itself.
• Randomization, which ensures that the two groups (control and treatment) have comparable risks (at the beginning of the study) of developing the outcome, and can enable us to attribute differences between them to whether they received the treatment.
• Concealment and blinding, which prevent researchers and participants from knowing which group they are allocated to.
• Pre-registration, a procedure where researchers declare in advance how they are going to carry out the study. This allows us see whether they’ve deviated from their plans, for example, because they found results that were disappointing. It can store research that is not published in a journal, because of ‘publication bias.’
• Some other key features of RCTs that I mention but don’t detail: experimentation and intention-to-treat analysis.

Importantly, this means that when RCTs do not follow these procedures, this makes them less reliable. Sometimes other types of studies (apart from RCTs) follow some of these procedures, which strengthens them as sources of evidence.

2. That RCTs are particularly useful in some circumstances. I illustrate this with two examples. 

We know that smoking has a large causal effect on the risk of lung cancer, without evidence from RCTs. This is because many lines of evidence converge on this conclusion, and other explanations fall short of accounting for the massive association that we see.^[1]

In contrast, when scientists looked for treatments for HIV/AIDS, many candidate drugs they expected to work actually failed, while some that worked were unexpected and led to insights about how the virus caused disease.^[2]

With this, I argue that RCTs matter when: we don’t have enough data from other lines of evidence, when we don’t know how to rule out other explanations, and when research is affected by biases (of the researcher, of participants, and publication bias). A catchier version is that they matter when we don't know enough, when we're wrong, and when we see what we want to see.

___

As I explain, I think understanding these ideas is very important because we point to evidence from RCTs when we want to evaluate treatments, policies and interventions.

Hopefully, the examples that I give are intuitive and help to apply these concepts more widely. If the points in the summary above are already familiar to you, hopefully there are some cool charts or examples that are still new to you. 

I'm also happy to answer questions or correct errors, if you spot any. You can also contact me at [email protected] or find me at @salonium on Twitter. Thank you!

1. ^{^}

   If you are interested in reading more about the smoking/lung cancer debate, I highly recommend these two papers.

   The first is a fascinating paper from 1960 that summarises the evidence that existed at the time, and why that led epidemiologists to be confident that the rise in lung cancer was a result of smoking cigarettes. 
   (It was highly influential at the time, and also the first example of ‘sensitivity analysis’ which is used in epidemiology to find out whether there might be remaining confounding in a study.)

   Cornfield, J., Haenszel, W., Hammond, E. C., Lilienfeld, A. M., Shimkin, M. B., & Wynder, E. L. (1959). Smoking and Lung Cancer: Recent Evidence and a Discussion of Some Questions. JNCI: Journal of the National Cancer Institute. Available as a PDF here.

   The second is a paper summarising the history of the smoking/lung cancer debate, and the counterclaims made by some famous statisticians (such as Ronald Fisher).

   Hill, G., Millar, W., & Connelly, J. (2003). “The Great Debate”: Smoking, Lung Cancer, and Cancer Epidemiology. Canadian Bulletin of Medical History, 20(2), 367-386. Available as a PDF here.

2. ^{^}

   If you're interested in the history of the discovery of HIV/AIDS treatments and the role of clinical trials, I strongly recommend this book chapter. It includes discussion about how activists pushed for these trials to be accelerated and de-regulated.

   National Research Council (U.S.) (1993). The social impact of AIDS in the United States. 4. Clinical Research and Drug Regulation. National Academy Press. Available in full text here.