Effectiveness of a theory-informed documentary to reduce consumption of meat and animal products: three randomized controlled experiments

with a 95% confidence interval ranging from a 6 ounce reduction to a 5 ounce increase.

Thanks for including the CI bounds, that makes it much more interpretable as an 'actual tight null' rather than an underpowered study.

CF I think that is relative to an average US consumption of about 80 oz per week ... so under +/- 10% with 95% CIs ... and the 80% CI would be obviously even tighter

(Or am I wrong, maybe this isn't that tight)

Disclaimer: Jacob and I both work at Rethink Priorities

Do you think your study is sufficiently well powered to detect very small effect sizes on meat consumption? It seems plausible that effects on meat consumption would be very small in expectation plus many people would not reduce meat no matter what, so you may be needing to detect a small shift within a small subpoulation.

It looks like you have 80% power to detect an effect size of d=0.24 - which is actually substantially larger than the effects we usually find for animal interventions even on more moveable things like attitudes/signing a petition/agreeing that "factory farms aren't great". Their null result on effect on meat consumption was not at all tightly bounded: -0.3oz [-6.12oz to + 5.46oz]

So I think the different results here seem possibly explained just by the fact that you could find effects on the moveable attitudes but were underpowered to detect differences in meat consumption. I'd be curious to estimate what effect size would we be looking at if say 3-5% of people stopped eating meat (an optimistic estimate IMO).

This is perhaps further confounded by a large amount of probable noise - how good are people at estimates oz of meat eaten in different time periods, is oz of meat something that is distributed in a way to corresponds to what a t-test is assessing?

This is really interesting, thanks for this! In particular, it was really helpful comparing it to previous less-rigorously designed surveys, as I'm sure you expected pushback using those results. I had a few quite preliminary questions:

• Do you think the effects of this could be different for different documentaries, and is this something you would consider testing in the future? Whilst in the paper you state that "Good For Us" uses psychological theory to make the documentary as compelling as possible to shift attitudes and behaviour, it feels quite hard to predict the emotional/attitudinal impact of a documentary. Some random thoughts I had was that maybe more sensationalist documentaries (What the Health, Cowspiracy, Dominion, etc.) could be more effective even though they ignore best practice, and it would be interesting to see how this stacks up against Good For Us. As these are touted as being the most effective/popular pro-animal documentaries, it would be interesting to see how these perform under the same controlled conditions.
• Obviously whilst difficult to measure, do you think these documentaries might be important in shaping beliefs that later affect eating behaviour? A common analogy we hear is about "planting a seed" whereby one exposure to pro-animal content might not cause any behaviour change, but it primes them for later exposures which might then have more significant impacts on behaviour change. You talk about repeated exposures briefly in the paper but it would be interesting to hear your thoughts on how plausible you think this mechanism is (see point below)
  • If repeated exposures to pro-animal content might be effective, we still might expect there to be some significant changes in this study as it should be repeated exposure for some people (unless you screened them out) so maybe this point isn't so strong
• Do you think there are other long-term mechanisms that might be at play here e.g. the documentary causes more animal-focused conversations with friends and family, which might cause behaviour change past the 12-day mark? Do you think a follow-up after 2-3 months (for example) would introduce too much noise to have strong causal evidence?
• More broadly, what implications do you think this has for the farmed animal movement in terms of funding documentaries vs other interventions, and where do you think more work is needed?

This might be a stupid question, but do you know if anyone looked at how many new activists can a documentary make? :) I met some people who were animal activists because they watched some inspiring movies about animal suffering. I wonder if despite the fact that the movies don't work as a thing that makes people change their diet, they might make some very specific people into animal activists? :) Probably the cost of producing them would not justify the results of getting like 100 activists out of one movie. But on the other hand, if the activist will later work in an organization that leads to a major policy change in a country, maybe it is worth it? :) Just some fun speculations about flow-through effects :) I wonder also about things like effects on the press coverage of animal-related topics, effects on influencers and celebrities, effects on the perception of animal suffering (e.g. would there more likely vote on a law that e.g. forbade using crates in farming systems, or more likely sign a petition to the government). We all know by now that habits are very hard to change, but opening minds to a different viewpoint - maybe not as hard.