Vegan Outreach ran its first annual Leafletting Effectiveness Survey (LES) last fall and we had a dismal response rate as well (around 2%). We were offering $5 incentives for people to take a 2-part survey, where Part 1 was filled out immediately and then an email was sent out two months later to complete Part 2 and claim their gift card. We've been running small response rate studies since then to figure out what kind of incentives we need to hit our targets, but we're seeing significant variation based on what city / state we're operating in. This is making it really difficult to find one incentive level to rule them all.
I wonder if you've looked at the geographical distribution of where your 2% came from? And do you have any theories why your actual response rate differed from your pilot response rate?
As I've said elsewhere, I'm skeptical that the approach to take is to do more such RCTs. I worry about us having to spend extremely large sums of money for such things.
It's probably a good idea to consider the global amount of money being spent on an AR intervention when evaluating the cost to investigate it. Like how much money is being spent across the different AR orgs on FB ads? If a proper study costs $200K and there is only $500K a year being spent globally, then it's hard to see the value proposition. If the total being spent annually is $20M, then a full fledged RCT is probably in order.
Does anyone know of estimates of how much the AR movement as a whole is investing in different interventions? This might help prioritize which interventions to study first and how much to pay for those studies.
Lot of great points here and angles to follow up on.
I do want to re-assert that enormous 80% recidivism rate, and how strong these (very valid) counter-narratives would have to be to defeat our original assumptions. Consider:
A huge number of people would have to actively curate their Facebook profiles to give current-vegetarians the majority in these groups.
A curiously large number of current-vegetarians would have to click on an ad that claims to "help you eat vegetarian again" and order a "Vegetarian Starter Guide" to make the majority of the respondents non-recidivists.
Again, there have been a lot of great points raised and I'm tempted now to fund a follow up implementing some of these fixes. At the same time, I think it's important to step back and think about how strong these influences would have to be to overcome that monster 80%.
I think I follow you here. Facebook's ad engine is more likely to target people that have liked several of our selected groups / terms. Liking multiple groups indicates stronger identification with being veg, so these people are more likely to still be veg, as opposed to people who liked one but not the others. Is that right?
Excellent thoughts here. As I mentioned in another comment, a follow up study could probably handle that second issue by including a question asking if the requestors of the VSG are current, former or (aspiring) new vegetarians.
This would probably shed some light on your first point as well. If most of the people requesting the VSG identified as current veg, then that would indicate either the ads aren't working at enticing former vegs to try again, or there just aren't any former vegs in the audience. Either of these would be enough to kill this as a strategy for reengaging recidivist. Although, this would open up the question of why so many current vegetarians are interested in a Vegetarian Starter Guide?
It's definitely possible that some current vegetarians might have requested the the Vegetarian Starter Guide. A follow up study could probably parse out this variable by having a simple required question for obtaining the VSG (along with their email) asking if they are currently veg, former veg and interested in trying again, or never been vegetarian at all.
The only reason I wouldn't put that document out publicly is because it wasn't written for wide release, so maybe Nick would want to clean it up before having it shared around. I know I usually spend more time polishing the look and language of a document that I intend to be passed around publicly. But that is the only reason, we're definitely happy to share any details people are interested in.
Nick wrote up a pre-study plan that I can send your way if you (or anyone else) would like to see it. Really though, it was a pretty simple study. We targeted people who liked one or more of the following terms / pages (below) with ads encouraging them to give eating veg another shot. But definitely let me know if you have any specific questions and Alan or I can get you the details. As an aside, can you confirm for me that the images are showing up now?
Hey guys, sorry I didn't realize this had posted. I was still editing the draft and just now finished cleaning it up only to find it went live last night (damn!). Here are some of the numbers (extracted from an email Alan sent me summarizing the results):
"To get one person in the US to pledge to go vegetarian and order a VSG it usually takes $5-$8 but in this trial we were able to get VSG orders for an average of only $2.65. This means that 2-3x more people ordered a VSG than if we were to target the general US population with the same amount of advertising dollars...Below are the detailed results of the ads that we ran. The two most important stats for us are (1) 756 people ordered a VSG and were subscribed to our 12-part email series about eating veg, and (2) 157,512 people saw one of our pro-veg ads in their Facebook Newsfeed...
157,512 people saw one of our pro-veg ads
1,571 people clicked on the ad and read more about eating vegetarian on our landing page
756 people ordered a Veg Starter Guide and were subscribed to our 12-part email series about eating veg
941 people liked MFA on Facebook and will now see our posts about veg eating
286 people shared one of the ads reaching even more people"
A quick note on what encouraged me to donate to EAO.
I navigate robotic spacecraft to destinations in deep space at JPL. If you're trying to get somewhere like Jupiter or Saturn, the most important course corrections you can make are right after launch. We always have a crack team of analysts closely monitoring a spacecraft just after it leaves Earth, because the energy required to change the spacecraft's heading grows exponentially with time; point in the wrong direction too long and the mission is lost.
EA is moving really, really fast, and small adjustments to its development now are likely to have huge consequences down the road. With EAO, we have a team of talented people focused on nothing but making sure it's heading in the right direction. They are doing a lot of really impressive, concrete work (like book promotion, EAG, VIP outreach etc), but I think the greatest value in keeping them well funded is to have a vigilant eye watching for obstacles and helping navigate them at this very important, early stage of the movement.
Vegan Outreach ran its first annual Leafletting Effectiveness Survey (LES) last fall and we had a dismal response rate as well (around 2%). We were offering $5 incentives for people to take a 2-part survey, where Part 1 was filled out immediately and then an email was sent out two months later to complete Part 2 and claim their gift card. We've been running small response rate studies since then to figure out what kind of incentives we need to hit our targets, but we're seeing significant variation based on what city / state we're operating in. This is making it really difficult to find one incentive level to rule them all.
I wonder if you've looked at the geographical distribution of where your 2% came from? And do you have any theories why your actual response rate differed from your pilot response rate?
It's probably a good idea to consider the global amount of money being spent on an AR intervention when evaluating the cost to investigate it. Like how much money is being spent across the different AR orgs on FB ads? If a proper study costs $200K and there is only $500K a year being spent globally, then it's hard to see the value proposition. If the total being spent annually is $20M, then a full fledged RCT is probably in order.
Does anyone know of estimates of how much the AR movement as a whole is investing in different interventions? This might help prioritize which interventions to study first and how much to pay for those studies.
Lot of great points here and angles to follow up on.
I do want to re-assert that enormous 80% recidivism rate, and how strong these (very valid) counter-narratives would have to be to defeat our original assumptions. Consider:
A huge number of people would have to actively curate their Facebook profiles to give current-vegetarians the majority in these groups.
A curiously large number of current-vegetarians would have to click on an ad that claims to "help you eat vegetarian again" and order a "Vegetarian Starter Guide" to make the majority of the respondents non-recidivists.
Again, there have been a lot of great points raised and I'm tempted now to fund a follow up implementing some of these fixes. At the same time, I think it's important to step back and think about how strong these influences would have to be to overcome that monster 80%.
I think I follow you here. Facebook's ad engine is more likely to target people that have liked several of our selected groups / terms. Liking multiple groups indicates stronger identification with being veg, so these people are more likely to still be veg, as opposed to people who liked one but not the others. Is that right?
Excellent thoughts here. As I mentioned in another comment, a follow up study could probably handle that second issue by including a question asking if the requestors of the VSG are current, former or (aspiring) new vegetarians.
This would probably shed some light on your first point as well. If most of the people requesting the VSG identified as current veg, then that would indicate either the ads aren't working at enticing former vegs to try again, or there just aren't any former vegs in the audience. Either of these would be enough to kill this as a strategy for reengaging recidivist. Although, this would open up the question of why so many current vegetarians are interested in a Vegetarian Starter Guide?
It's definitely possible that some current vegetarians might have requested the the Vegetarian Starter Guide. A follow up study could probably parse out this variable by having a simple required question for obtaining the VSG (along with their email) asking if they are currently veg, former veg and interested in trying again, or never been vegetarian at all.
The only reason I wouldn't put that document out publicly is because it wasn't written for wide release, so maybe Nick would want to clean it up before having it shared around. I know I usually spend more time polishing the look and language of a document that I intend to be passed around publicly. But that is the only reason, we're definitely happy to share any details people are interested in.
Nick wrote up a pre-study plan that I can send your way if you (or anyone else) would like to see it. Really though, it was a pretty simple study. We targeted people who liked one or more of the following terms / pages (below) with ads encouraging them to give eating veg another shot. But definitely let me know if you have any specific questions and Alan or I can get you the details. As an aside, can you confirm for me that the images are showing up now?
Terms used to target study audience:
Hey guys, sorry I didn't realize this had posted. I was still editing the draft and just now finished cleaning it up only to find it went live last night (damn!). Here are some of the numbers (extracted from an email Alan sent me summarizing the results):
"To get one person in the US to pledge to go vegetarian and order a VSG it usually takes $5-$8 but in this trial we were able to get VSG orders for an average of only $2.65. This means that 2-3x more people ordered a VSG than if we were to target the general US population with the same amount of advertising dollars...Below are the detailed results of the ads that we ran. The two most important stats for us are (1) 756 people ordered a VSG and were subscribed to our 12-part email series about eating veg, and (2) 157,512 people saw one of our pro-veg ads in their Facebook Newsfeed...
A quick note on what encouraged me to donate to EAO.
I navigate robotic spacecraft to destinations in deep space at JPL. If you're trying to get somewhere like Jupiter or Saturn, the most important course corrections you can make are right after launch. We always have a crack team of analysts closely monitoring a spacecraft just after it leaves Earth, because the energy required to change the spacecraft's heading grows exponentially with time; point in the wrong direction too long and the mission is lost.
EA is moving really, really fast, and small adjustments to its development now are likely to have huge consequences down the road. With EAO, we have a team of talented people focused on nothing but making sure it's heading in the right direction. They are doing a lot of really impressive, concrete work (like book promotion, EAG, VIP outreach etc), but I think the greatest value in keeping them well funded is to have a vigilant eye watching for obstacles and helping navigate them at this very important, early stage of the movement.