Our paper “The effects of creatine supplementation on cognitive performance - a randomised controlled study” is out now!
→ Paper: https://doi.org/10.1186/s12916-023-03146-5
→ Twitter thread: https://twitter.com/FabienneSand/status/1726196252747165718?t=qPUghyDGMUb0-FZK7CEXhw&s=19
Jan Brauner and I are very thankful to Paul Christiano for suggesting doing this study and for funding it.
My understanding of the results: for the preregistered tasks you measured effects of 1 IQ point (for RAPM) and 2.5 IQ points (for BDS), with a standard error of ~2 IQ points. This gives weak evidence in favor of a small effect, and strong evidence against a large effect.
You weren't able to measure a difference between vegetarians and omnivores. For the exploratory cognitive tasks you found no effect. (I don't know if you'd expect those tests to be sensitive enough to notice such a small effect.)
At this point it seems a bit unlikely to me that there is a clinically significant effect, maybe I'd bet at 4:1 against the effect being >0.05 SD. That said I still think it would be worthwhile for someone to do a larger study that could detect a 0.1 SD effect, since that would be clinically significant and is very weakly suggested by this data (and would make supplementation worthwhile given how cheap it is).
(See also gwern's meta-analysis.)
Unless I'm misunderstanding, does this mean you'd bet that the effects are even smaller than what this study found on its preregistered tasks? If so, do you mind sharing why?
Yes, I'd bet the effects are even smaller than what this study found. This study gives a small amount of evidence of an effect > 0.05 SD. But without a clear mechanism I think an effect of < 0.05 SD is significantly more likely. One of the main reasons we were expecting an effect here was a prior literature that is now looking pretty bad.
That said, this was definitely some evidence for a positive effect, and the prior literature is still some evidence for a positive effect even if it's not looking good. And the upside is pretty large here since creatine supplementation is cheap. So I think this is good enough grounds for me to be willing to fund a larger study.
Great that you did this, really appreciate it.
I'm no expert on the biology, but my intuition would in any case have been that the effect size would be tiny/negligible for 6 weeks of supplementation, and that for non-trivial effects, you would need sustained supplementation over a longer time period.
Is there any reason to doubt my intuition on this?
I believe prior work showed large effects from short periods of supplementation. (Edit: Note that this work seems to debunk prior work, but this should explain the study design)
We also see moderate to large effects from short periods of supplementation for the non-cognitive effects, so we might expect similar saturation for the cognitive effects.
Bravo! This really sets a bar for the quality of inquiry we should strive for in this community.
Thanks for comparing vegetarian and omnivore diets. Interested to see vegan diet results too since vegan diets restrict even more sources of dietary creatine than vegetarian diets
Strongly upvoted. I'd love to see more work like this: well-powered, preregistered, and directly answering an important question.
Would be interesting to see a SEM analysis of this looking at the impact on a latent IQ measure behind these test results that each try to get at it separately. This model should have much better power to find an effect and not suffer from multiple hypothesis testing issues looking at the different tests separately. Model would be of the form
Latent variable definition
IQ =~ RAPM+BDS+Other tests
Regression
IQ ~ Creatine+Other explanatory variables(Age,vegetarian etc.)