Proposed methodology for leafleting study

Alex_Barry

Proposed methodology for leafleting study

Alex_Barry

14 min readFeb 6, 2017

Comments 17

Sorted by

New & upvoted

jimrandomh

The questions about diet before and after the change seem to be pushing people strongly into claiming to be or to have been some sort of vegetarian; the only option you have there that isn't somehow anti-meat is "Other', which requires typing.

A better version of this question would have a no-dietary-restrictions option first, and a few options that aren't animal-welfare related like "low carb" and "Mediterranean".

Alex_Barry

Yeah that seems right, I think it would not be too bad as they would not be shown the options until having already reported changing their diet, but it would be worth adding more options as you describe.

jimrandomh

Statistics nitpick: I believe you should be using a two-sided test, as it is also possible for leafleting to reduce the rate of people going vegetarian if the leaflets alienate people somehow.

Alex_Barry

Ah yeah, my mistake, thanks! I'll update the post.

Derek

Good to see some thought being put into this topic. A few comments/concerns below.

To pseudo-randomise based only on names, I wouldn't use the first letter. I haven't checked, but I'd guess that some strongly correlate with confounders like ethnicity or class: many starting with an X could be Chinese, with a Z Arabic, etc.

Instead, you could assign based on whether they have and odd or even number of letters in their name. It seems unlikely that this would be correlated with confounders, or that there would be very uneven numbers in each group, but I suppose it's possible, e.g. if the most common few names all happen to be even (or odd) and also be associated with class/ethnicity/sex. Using their full name would mitigate this but would slow down leafleting compared to only using first or last, and perhaps introduce counting errors. Maybe you can get a fairly random sample of student names from somewhere and check whether using just one name is likely to be problematic.

Obviously you need to ensure that the name they use on the survey is exactly the same as the one on the pigeonhole (Elizabeth/Beth/Liz/Lizzy; middle name included or not, etc), unless you specifically ask whether they received a leaflet, which introduces other problems. I would probably suggest adding to the questionnaire something like: "(Please write your name exactly as it is on your pigeonhole.)". Presumably family names are less likely to be modified/dropped, but as noted, only using those may hinder randomisation.

I'd guess that many of the leaflets would be seen by the control group. People will show them to their friends, leave them lying around, etc. This would dilute the observed effect of the intervention. I'm not sure how to avoid it without going for cluster randomisation, which is even more difficult to get right. I suppose it would at least give you some basis for saying your findings are conservative; Type 1 errors (false positives) are generally seen as worse than Type 2 errors (false negatives), and given the state of evidence for animal-focused interventions, it is probably wise to be cautious. There would still be some factors pushing the other way though, so I'm not sure it would actually be conservative.

Opportunity permitting, I would consider some follow-up research to establish mechanisms. For example, you could ask (a random subset of?) changers why they changed, whether they remember seeing the leaflet, whether they showed it to others, what they thought about it, whether they were planning to change anyway, etc. This could increase (or decrease) confidence that the result shows a true effect. You might be able to get one or two of these into the original survey; to avoid influencing responses, use a web form that requires them to submit the 'change' answers before viewing subsequent questions - this is quite a common design in my experience. You could also include a social desirability instrument, as in the ACE study.

The use of incentives could introduce response bias. Presumably the extent to which money motivates response will be associated with SES and perhaps ethnicity and other characteristics. Could still be justified, though, in order to boost response. Not sure. Given the huge impact this has on costs, it might make sense to do an unincentivised one first, then perhaps add the incentives in later studies (obviously at different institutions). This would also function as a cheap 'test run', allowing you to refine the methodology and logistics.

I suspect the referral system would introduce considerable sampling bias. Those who pass it on, and perhaps those who are friends with referrers and who respond to a referred task, are unlikely to be representative of the study population: they'd presumably be more conscientious, have more friends, have more interest in dietary change, etc. It seems odd to go to all that effort to randomise then include a method that would undermine it so much. I'd only do it if it was impossible to get an adequate response otherwise.

Likewise publishing articles about it in advance. Those who read them may not be representative, e.g. more likely to be native English speakers, conscientious/motivated, interested in dietary change (or study design), etc. Some people's decision to go veg could also be influenced by the articles themselves. And obviously you couldn't publish them if you were trying to conceal the purpose of the study as others have suggested.

When calculating sample sizes, I'm not sure it's a good idea to rely heavily on one estimate of base rate change. Presumably it fluctuates considerably in response to time of year (new year's resolutions), news stories, other campaigns, etc., and correlates with location and many other factors. It would be a shame to waste resources on an under-powered study.

I would be a little surprised if the true effect was more than 1/200 for any dietary change (at least a durable one), and not surprised if it was orders of magnitude smaller. If you want to look at subgroups (vegan, reducetarian, etc), and maybe even if you don't, I'd guess you'd need a much larger sample than proposed. But these are just hunches.

As usual with trials (and life), it seems there would be a number of hard trade-offs to be made here. I suppose whether it's worth doing depends on what results you expect and how they would influence action. To play devil's advocate: You think a null result is likely but that it wouldn't justify stopping leafleting, in which case the study would not have been helpful in terms of allocating resources. Depending on the size and quality of the study, it would also be unwise to put much weight on a weak positive or negative effect. A strong negative is unlikely and in such a case we should probably conclude that there was a problem with these particular leaflets or with the study design (or that we just got really unlucky). So only a strong positive (>1/150?) seems like a very useful outcome. Even then, we should still suspect influence from chance, bias and/or confounding, but it would give some justification for continuing, and perhaps increasing, leaflet campaigns; and for investing in further studies to confirm their effectiveness and investigate what strategy works best, for whom, why, and in what circumstances. However, a strong positive seems unlikely (<15%?). Therefore, perhaps the question is whether it is worth doing the study if there is only a small chance of getting a result that would (or should) change our activities.

I suspect the answer to the last question is "yes", and I'm not actually sure the other results would be useless (e.g. a null or negative result might prevent inordinate future investment in leafleting based on poor quality past studies). There are also other benefits to running the study, like boosting the credibility of, and skill development within, the movement. But it's worth thinking carefully about what exactly we expect to get from the study before leaping into it.

Peter Wildeford

Instead, you could assign based on whether they have and odd or even number of letters in their name.

You could SHA-256 hash the names and then randomize based on that. Doing so should remove all chances of confounding effects.

Derek

It's been a long time since I wrote the comment, but I think I was under the impression the allocation had to happen at the point of distribution, using only the names on the pigeonholes. But if you could get a list of students that exactly matched the names on the pigeonholes in advance of distribution, then I agree randomising hashes would be ideal. I doubt you'd get this from administrators due to data protection issues, but presumably you could go round and manually record the names. That would be very time-consuming for a large study, but perhaps worth it to avoid any doubts about the randomisation.

Note that this would not remove all risk of selection bias because the allocation would still not be concealed, i.e. the people putting the leaflets in the pigeonholes would know who was assigned to each group. It is possible they would consequently give some of the leaflets to the wrong people, e.g. if they wanted to increase the effectiveness of the intervention they might be influenced by personal knowledge of individuals (better to give a leaflet to someone who is not already vegan) or presumed characteristics (gender, ethnicity, etc) that are correlated with likelihood of being influenced by the leaflets. This may seem far-fetched, but unconcealed allocation is associated with higher effect sizes in medical trials so we shouldn't be too quick to assume it wouldn't matter here.

https://handbook-5-1.cochrane.org/chapter_8/8_10_1_rationale_for_concern_about_bias.htm

https://handbook-5-1.cochrane.org/chapter_8/8_10_2_assessing_risk_of_bias_in_relation_to_adequate_or.htm

One solution is to give every student a leaflet inside an opaque sealed envelope, with some getting the 'treatment' and some a 'control'. But this introduces additional complexity, e.g. it could cause (further) 'contamination' as students compare what they got in that weird envelope, it reduces external validity (assuming leaflets would not normally be in envelopes), and the control leaflet would have to be very carefully designed so as not to affect the outcomes of interest while being very similar in every other respect (e.g. a leaflet promoting exercise may indirectly influence dietary choices).

jimblowiak

I was wondering if you planned on submitting this proposal to ACE?

http://researchfund.animalcharityevaluators.org/about/

They're currently funding research like this. I'm actually spending my Sunday morning trying to figure out the relative benefit of additional research (in animal welfare it would be tremendous I think, since the existing body of evidence leaves much to be desired).

The next question is how can we fund this sort of research effectively? While ACE is running their funding operation, and I think it's extremely smart of them to do so, it'd be a good hedge to have multiple gatekeepers.

Jeff Kaufman 🔸

"I reused the diet questions in my plan from MFA 2013 study on leafleting"

In my view, this study asked way too much. When you try to ask too much detail people drop out. Additionally, it asks about things like diet change, but to pick up on changes we should be comparing the experimental and control groups, not comparing one group with its (reported) earlier self.

What I'd like to see is just "do you eat meat" along with a few distractor questions:

Are you religious?
Is English your native language?
Do you eat meat?
Do you own a car?

Yes, we'd like to know way more detail than this, and in practice people are weird about how they use "meat", but the main issue here is getting enough responses to be able to see any difference at all between the two groups.

Alex_Barry

"I reused the diet questions in my plan from MFA 2013 study on leafleting"

Ah sorry again I was not quite clear, what I meant by this was the question about diet is one I had copied from the MFA study, not that I would reuse all of them. The questions I list in 2.1 are the only ones I would ask (probably with a bonus distractor question and maybe some extra options as suggested by jimrandomh).

Additionally, it asks about things like diet change, but to pick up on changes we should be comparing the experimental and control groups, not comparing one group with its (reported) earlier self.

Asking about 'change in diet' vs just diet generally is basically required to get sufficient statistical power, as the base rate of people saying yes to "have you become vegetarian in the last two weeks" is much much lower than "are you vegetarian" but the effect size we are looking for in each case is the same. One can then compare the control and experimental groups on this metric.

To illustrate the size of this effect, in the post I calculate that with a sample of 5000, by asking about change in the last 2 weeks you would have a 90% chance to find an effect of 1/124 leaflets creating one vegetarian, but if you just asked "are you vegetarian?" you would only be able to find a 1/24 effect at the same power. (Assuming a 20% base rate of vegetarianism, and using this calculator ).

When you try to ask too much detail people drop out.

I agree about using as few questions as possible, and that the MFA study asked far too much (although I think it was administered by volunteers as opposed to online, which would hopefully counteract the drop out effect in their case).

Jeff Kaufman 🔸

"A question to determine if they were leafleted or not, without directly asking."

People who were leafleted but ignored it and don't remember enough to answer this one accurately is a problem here.

What would you think of: at a college that allows students to mass pigeonhole directly, put experiment leaflets in odd mailboxes and control ones in even boxes. Then later put surveys in the boxes, with different links for odd and even boxes.

Instead of having the links be example.com/a and example.com/b it would be better for them all to look like example.com/gwfr so people don't know what's going on. You could generate two piles of follow-up links and use one for the odd boxes and the other for even. QR codes might be good to add so people have the option not to type.

Alex_Barry

People who were leafleted but ignored it and don't remember enough to answer this one accurately is a problem here.

Ah my wording might not be quite clear in that section, ideally the question would rely on the surveyors knowledge of who was leafleted and who was not, without the students having to remember if they were leafleted. E.g. if the leafleting was split by college, asking what college they were from.

Then later put surveys in the boxes, with different links for odd and even boxes.

A reason I would push for trying to get a survey emailed out is that a previous study on leafleting that attempted to use QR codes or links on leaflets got an extremely low (about 2% if I remember correctly) response rate.

I am not sure if giving out the surveys desperately later would boost the response rate, I think the added inconvenience of having to type in a web address or scan a QR code would significantly damage the response rate.

Still it would be worth bearing in mind as strategy that could currently be implemented at any university that allowed mass-leafleting, without being able to email all students.

LaurenMcG

Do you have updates on the impact of any recent university leafleting activities?

rossaokod

Thanks for writing this up! I think this type of research is very important in the animal advocacy movement.

One thing I would like to see in one of these studies is using food purchase data instead of relying on surveys. This might be possible in colleges/universities where a large proportion of students have some sort of meal plan, and pay for their food using their student ID or some other similar card. In some places the students just pay to enter a food hall, and so you wouldn't know what food they ate, but in other places I think each item is scanned individually. This approach would obviously require the university and/or food service company being supportive of this type of work, and being willing to share the data. They would of course also have to anonymize the data and it might be a challenge getting ethics approval.

I see two main benefits of this type of data. (1) it is likely much more accurate than survey data and (2) you are likely to have far more power in your statistical tests. There are a few reasons I believe (2): there would be less noise in the data (i.e. even if self reports are unbiased, they will likely have measurement error), you would likely have a larger sample size (no problems with response rates), and you could use continuous measures of meat consumption (i.e. 'number of food items bought with meat over a week/month' rather than just 'do you eat meat?')

Bernadette_Young

Ethics approval would probably depend on not collecting identifying data like name, so it would be important to build that into your design. College name would work, but pseudo-randomising by leafleting some colleges would introduce significant confounding, because colleges frequently differ in their make up and culture.

Alex_Barry

Yes I mention the issues associated with college-based randomization in section 3.1.

Good point about not collecting identifying data, it should just be possible to ask for whatever information was used to decide who to leaflet, such as first letter of last name, which should avoid this issue.

Bernadette_Young

Ah sorry, I missed that bit!

Comments

Curated and popular this week

Counting animals: Stable population size is not equivalent to priority level

abrahamrowe, mal_graham🔸·6d ago·Curated 2d ago·16m read

AI Use Note: Main body text entirely human written. Claude (Opus 4.8) helped develop models of animal life histories in the appendix. Cross-posted from Good Structures. Executive Summary * Animal advocates sometimes make claims like “there are X of this animal...

151

Let's taboo the V-word

lincolnq·6d ago·8m read

“How long have you been v*g*n?” This is one of the most common icebreakers at animal protection events. It’s a baseline assumption, and it mostly holds true: if you’re out advocating for animals not to be tortured or abused, realistically these days you are v**n, or close. And it makes for good conversation. It seems fairly safe to assume when you meet strangers. But this assumption is hurting the movement in a way which we don’t always notice: someone new comes into the sp...

105

Spiro: an update 2.5 years on and a fundraising ask for expansion

Habiba Banu·3d ago·6m read

Summary Back in November 2023 I posted here to launch Spiro and raise our first $198k. Two and a half years later this is an update and a fundraiser for the next step. The short version: we've now reached over-5,900 people with TB preventive medicine, including over 3,000 children under five years old. Our early results have held up well an...

Derek

Good to see some thought being put into this topic. A few comments/concerns below.

Proposed methodology for leafleting study

Proposed methodology for leafleting study

1. Background

1.1 A brief overview of previous research

1.2 A summary of my plan

1.3 My aim for this post

2. Detailed plan

2.1 Preparations

2.2 Implementation

2.3 Analysis

3.Explanation of the Details

3.1 How to split the population into two groups

3.2 Survey Questions

3.3 Practical survey considerations

3.4 Incentivisation

3.5 Statistical power estimations

4. Final Thoughts
4.1 Limitations

4.2 Predicted cost

4.3 Conclusion

Proposed methodology for leafleting study

Proposed methodology for leafleting study

1. Background

1.1 A brief overview of previous research

1.2 A summary of my plan

1.3 My aim for this post

2. Detailed plan

2.1 Preparations

2.2 Implementation

2.3 Analysis

3.Explanation of the Details

3.1 How to split the population into two groups

3.2 Survey Questions

3.3 Practical survey considerations

3.4 Incentivisation

3.5 Statistical power estimations

4. Final Thoughts4.1 Limitations

4.2 Predicted cost

4.3 Conclusion

4. Final Thoughts
4.1 Limitations