Mindfulness meditation seems to be quite popular in the EA community and in adjacent communities. Many EAs I meet seem to be interested in meditation or actively practicing it, and there has been some written discussion of its value by EAs.[1]
I personally find mindfulness useful for reducing rumination. Its main value is revealing that your mind is basically going completely bananas all the time, that patterns of thought emerge which are difficult to control. I also find that loving and kindness meditation improves my mood in the short-term. However, I think the strength of the evidence on the benefits of meditation is often overstated by EAs and quasi-fellow travellers like Sam Harris.
Here, I discuss the overall strength of the evidence on meditation as a treatment for anxiety and depression.
1. What is mindfulness?
With its roots in Buddhism, attention towards mindfulness has grown enormously since the early 2000s.[2] Two commonly studies forms of mindfulness are Mindfulness-Based Stress Reduction and Mindfulness-Based Cognitive Therapy. Mindfulness-Based Stress Reduction is an 8-week group-based program consisting of:
● 20-26 hours of formal meditation practice, including:
○ Weekly 2 to 2.5 hour sessions
○ A whole-day retreat (6 hours)
○ Home practice ~45 mins per day for 6 days a week.[3]
Mindfulness-Based Cognitive Therapy incorporates cognitive therapy into the sessions. Mindfulness-Based Stress Reduction can be led by laypeople, whereas Mindfulness-Based Cognitive Therapy must be led by a licensed health care provider.
In my experience, most people practising mindfulness use an app such as Headspace or the Sam Harris Waking Up app. I personally find the Waking Up app far superior to the Headspace app.
2. How over-optimistic should we expect the evidence to be?
Mindfulness has many features that should make us suspect that the strength of the evidence claimed in the literature is overstated:
● A form of psychology/social psychology research
● Most outcome metrics are subjective
● Many of those researching it seem to be true believers[4]
● Hints of religion, alternative medicine, and woo
Research fields with these features are ripe for replication crisis trauma. We should expect inflated claims of impact which are then brought back to Earth by replications or further analysis of the existing research.
3. Main problems with the evidence
3.1. Reporting bias
Reporting bias includes:
- Study publication bias - the publication of significant results and the failure to publish insignificant results
- Selective outcome reporting bias, in which outcomes published are chosen based on statistical significance with non-significant outcomes not published.
- Selective analysis reporting bias, in which data are analyzed with multiple methods but are reported only for those that produce positive results.
- Other biases, such as relegation of non-significant primary outcomes to secondary status when results are published.
There is good evidence of reporting bias in mindfulness research.
Coronado-Montoya et al (2016) test for reporting bias by estimating the expected number of positive trials that mindfulness-based therapy would have produced if its effect size were the same as individual psychotherapy for depression, d = 0.55.[5] As we will see below, this is very likely an overestimate of the true effect of mindfulness-based therapy, and therefore the method used understates reporting bias in mindfulness studies.
Of the 124 RCTs included in Coronado-Montoya et al’s (2016) study, 108 (87%) were classified as positive and 16 (13%) as negative. If the true effect size of mindfulness-based therapy was d = 0.55, then we would expect 68 of 124 studies to be positive, rather than 108, meaning that the ratio of observed to expected positive studies was 1.6.[6] This is clear evidence of reporting bias.
Moreover, Coronado-Montoya et al (2016) also looked at 21 trials that were pre-registered. Of these, none specified which variable would be used to determine success, and 13 (62%) were still unpublished 30 months after the trial was completed.[7]
A recent Nature paper found that in psychology, due to selective reporting, meta-analyses produce significantly different effect sizes to large-scale pre-registered replications in 12 out of 15 cases. Where there was a difference, on average, the effect size in the meta-analysis was 3 times larger than the replications.[8] This shows that reporting bias is usually not adequately corrected for in meta-analyses.
3.2. Effect size of meditation compared to other interventions
Goyal et al conducted a meta-analysis of the effect of mindfulness-based therapy for well-being.[9] They key facts are:
● Effect size
○ Cohen’s d ranging from 0.22 to 0.38 for anxiety symptoms.[10]
○ 0.23 to 0.30 for depressive symptoms.[11]
○ These were each usually compared to a nonspecific active control.
○ However, neither of these estimates correct for reporting bias.[12] I think it is plausible that this biases the estimate of the effect size upwards by a factor of 2 to 3.
○ Comparison to alternative treatments
■ In the 20 RCTs examining comparative effectiveness, mindfulness and mantra programs did not show significant effects when the comparator was a known treatment or therapy.[13]
■ Sample sizes in the comparative effectiveness trials were small (mean size of 37 per group), and none was adequately powered to assess noninferiority or equivalence.[14]
■ Antidepressants
● According to a recent meta-analysis, antidepressants have an effect size of 0.3 for depression vs placebo.
■ CBT
● According to one meta-analysis, compared to wait-list controls, CBT has a Cohen’s d = 0.88 on depression
● Compared to care as usual or non-specific controls, it has a Cohen’s d of 0.38.[15]
● Goyal et al assessment of strength of evidence
○ Only 10 of the 47 included studies had a study quality rating of ‘good’, with the remainder having a rating of only ‘fair’ or ‘poor’.[16]
○ Goyal et al state that “none of our conclusions yielded a high strength of evidence grade for a positive or null effect.”[17]
This suggests that the strength of the evidence on meditation is weak, but that there is some evidence of small to moderate positive effect on anxiety and depression. However, the evidence seems to be much weaker than the evidence for CBT and antidepressants, and CBT and antidepressants seem to have a greater effect on depression.
We should beware the man of one study, but also beware the man of a meta-analysis that doesn’t correct for reporting bias or other sources of bias. Indeed, as argued in the section on reporting bias, there is good reason to think that a pre-registered high-powered replication would cut the estimated meta-analytic effect size for meditation by a factor of 2 to 3.
3.3. Large variation among mindfulness-based interventions
Most mindfulness-based therapy has been based on the idea of mindfulness-based stress reduction. However, there are large variations among studied mindfulness-based interventions.
● Time commitment[18]
○ The practice hours of the intervention included in the Goyal et al meta-analysis range from 7.5 hours to 78 hours.
○ The homework hours are: often not specified, exceed 30 hours in many studies and even reach up to 1,310 hours in one study.
● Methods for teaching and practicing mindful states.[19]
Van Dam et al (2017) contend that there is far greater heterogeneity among mindfulness interventions than among other intervention types such as CBT.[20] This heterogeneity across intervention types means that we should be cautious about broad claims about the efficacy of mindfulness for depression and anxiety.
It is especially important to consider this heterogeneity given that most people practicing meditation practice for 10-20 minutes per day using an app, making their experience very different to a full mindfulness-based stress reduction course.
3.4. Shaky fMRI evidence
Many studies assess the impact of meditation on brains states using fMRI imaging. These methods are highly suspect. There are numerous potential confounds in fMRI studies, such as head movement, pace of breathing, and heart rate.[21] These factors can confound a posited relationship between meditation and change in activity in the amygdala. Moreover, calculating valid estimates of effect sizes across groups in neuroimaging data is very difficult. Consequently, the practical import of such studies remains unclear.
Nonetheless, according to Van Dam et al (2017), meta-analyses of neuroimaging data suggest modest changes in brain structure as a result of practicing mindfulness. Some concomitant modest changes have also been observed in neural function. However, similar changes have been observed following other forms of mental and physical skill acquisition, such as learning to play musical instruments and learning to reason, suggesting that they
may not be unique to mindfulness.[22]
It would be interesting to compare the effects of meditation on the brain with the effects of other activities such as reading, exercise, sport, or having a conversation with friends. I suspect that the effects on fMRI scans would be quite similar for many mundane activities, though I have not looked into this.
4. Useful resources and reading
● Coronado-Montoya et al, ‘Reporting of Positive Results in Randomized Controlled Trials of Mindfulness-Based Mental Health Interventions’, Plos One (2016)
○ A high-quality analysis of reporting bias in mindfulness research.
● Goyal et al, ‘Meditation Programs for Psychological Stress and Well-being: A Systematic Review and Meta-analysis’ JAMA Internal Medicine 2014.
○ A review commissioned by the U.S. Agency for Healthcare Research and Quality. Provides a good overview of the quality of the evidence and estimates of effect size but crucially does not correct for reporting bias.
● Van Dam et al ‘Mind the Hype: A Critical Evaluation and Prescriptive Agenda for Research on Mindfulness and Meditation’, Perspectives on Psychological Science (2017)
○ A very critical review of the evidence on mindfulness, which raises several problems with the evidence. However, I think the tone is overall too critical given the evidence presented.
● Sam Harris - Waking Up book
○ An in my opinion overly rosy review of the evidence on meditation, especially in chapter 4.
● The Waking Up meditation app.
○ The best meditation app I have tried.
References
[1] See for example this piece by Rob Mather, and this by Louis Dixon
[2] Nicholas T. Van Dam et al., ‘Mind the Hype: A Critical Evaluation and Prescriptive Agenda for Research on Mindfulness and Meditation’, Perspectives on Psychological Science 13, no. 1 (2018): 36–37.
[3] Stephanie Coronado-Montoya et al., ‘Reporting of Positive Results in Randomized Controlled Trials of Mindfulness-Based Mental Health Interventions’, PLOS ONE 11, no. 4 (8 April 2016): 1, https://doi.org/10.1371/journal.pone.0153220.
[4] For example, a review of RCT evidence on mindfulness by Creswell opens with a quote from John Kabat-Zinn, a leading figure in mindfulness studies: “There are few people I know on the planet who couldn’t benefit more from a greater dose of awareness”. [creswell ref]
[5] Coronado-Montoya et al., ‘Reporting of Positive Results in Randomized Controlled Trials of Mindfulness-Based Mental Health Interventions’, 5.
[6] Coronado-Montoya et al., 9.
[7] Coronado-Montoya et al., 10.
[8] Amanda Kvarven, Eirik Strømland, and Magnus Johannesson, ‘Comparing Meta-Analyses and Preregistered Multiple-Laboratory Replication Projects’, Nature Human Behaviour, 23 December 2019, 1–12, https://doi.org/10.1038/s41562-019-0787-z.
[9] Madhav Goyal et al., ‘Meditation Programs for Psychological Stress and Well-Being: A Systematic Review and Meta-Analysis’, JAMA Internal Medicine 174, no. 3 (1 March 2014): 357–68, https://doi.org/10.1001/jamainternmed.2013.13018.
[10] Goyal et al., 364.
[11] Goyal et al., 364.
[12] Goyal et al., 361.
[13] Goyal et al., 364.
[14] Goyal et al., 365.
[15] Ellen Driessen and Steven D. Hollon, ‘Cognitive Behavioral Therapy for Mood Disorders: Efficacy, Moderators and Mediators’, Psychiatric Clinics 33, no. 3 (1 September 2010): 2, https://doi.org/10.1016/j.psc.2010.04.005.
[16] Goyal et al., ‘Meditation Programs for Psychological Stress and Well-Being’, Table 2.
[17] Goyal et al., 365.
[18] Goyal et al., Table 2.
[19] Van Dam et al., ‘Mind the Hype’, 40.
[20] Van Dam et al., 46.
[21] Van Dam et al., 49–51.
[22] Van Dam et al., 50–51.
I briefly and informally looked into this several years ago and, at the time, had a few additional concerns. (Can't promise I'm remembering this perfectly and the research may have progressed since then).
1) Many of the best studies on mindfulness's effect on depression and anxiety were specifically on populations where people had other medical conditions (especially, I think, chronic pain or chronic illness) in addition to mental illness. But, most people I know who are interested in mindfulness aren't specifically interested in this population.
My impression is that Jon Kabat-Zinn initially developed Mindfulness-Based Stress Reduction (MBSR) for people with other conditions and my intuition from my experience with it is that it might be especially helpful for things like chronic pain. So I had some external validity concerns.
2) There were few studies of long-term effects and it seems pretty plausible the effects would fade over time. This is especially true if we care about intention-to-treat effects. The fixed cost of an MBSR course might only be justified if it can be amortized over a fairly long period. But it wouldn't be surprising if there are short-to-medium term benefits that fade over time as people stop practicing.
By contrast, getting a prescription for anti-depressants or anti-anxiety has a much lower fixed cost and it's less costly and easier to take a pill every day (or as needed) than to keep up a meditation practice. (On the other hand, some meds have side effects for many people.)
3) You already mention that "many of those researching it seem to be true believers" but it seems worth reemphasizing this. When I looked over the studies included in a meta-analysis (I think it was the relevant Cochrane Review), I think a significant proportion of them literally had Jon Kabat-Zinn (the founder of MBSR) as a coauthor.
---
All that said, my personal subjective experience is that meditating has had a moderate but positive effect on my anxiety and possibly my depression when I've managed to keep it up.
On the true believers point, I have also hear second-hand stories from people who went to mindfulness conferences to find that they were full of people who really wanted mindfulness to have a big effect