I haven't reread this post (I find it aversive/painful), but for outside view reasons I think you should heavily discount any conclusion or analysis reached by the author of this post, for reasons outlined here and here.
I would guess that the operational details and factual claims are relatively more trustworthy.
You should assume that the author was pretty junior to this type of analysis and not very good at impact assessments or related points.
I started a concerted campaign where we tried to get access to "celebrities" [...] two journalists and an artist have agreed to take the pledge
Anecdotally, one person who basically never heard of EA before came to our local group's lunch about Giving What We Can on Jan.9th, and took the pledge in time for the drive--this might set a record for fastest turnaround for hearing about a thing and then committing to it!
That's strikingly successful! Can you say a little more about these people and why they were so unusually receptive? Did they fit into groups of people typically receptive to EA? What was their attitude to charity before? What sort of contact did you have with the journalists and artist?
In addition, both EA London and EA Madison (my local meetup) organized local meetups to talk to people about the GWWC pledge and encourage people to take the pledge/Try Giving, with very positive results.
Can you expand on these results?
The journalists and the artist were all people who are familiar with EA already, and likely already give 10% or considerably more of their income to global poverty alleviation efforts. So the counterfactual benefit of their donations is low (but nonzero--commitment devices are great!), though we're hoping that the public interviews, once they're done/published, will help get people interested or excited about EA.
In a sense, most of the benefit will ultimately come in the upcoming months if not years, as they hopefully increase interest/investment in the EA movement and publicly give, encouraging more people among their audience to do so.
"Can you expand on these results?" I can't speak too much for the London group, but the Madison meeting just generally felt very well. People talked about why they pledged, and it generally seemed very convincing to the non-pledgers in the audience. And like I said, somebody pledged who didn't hear of EA before, which is strongly counterfactual (since if he did not come to this one, it might be months if not years before he hears about EA or came to another EA Madison meeting).
I've forwarded your comment to somebody who ran the London meeting. Hopefully they can provide more info!
All views here are my own, as a volunteer for Giving What We Can with a fairly significant influence on the pledge drive. This post does not represent the official position of Giving What We Can but is written with their permission. I suspect that this post is primarily of interest to employees and volunteers of Giving What We Can. However, I decided to release this to the general EA audience because we've had a lot of help from so many different people in EA, and also because this may be of interest for people in other EA organizations who face similar problems, or for people interested in volunteering for Giving What We Can.
I learned a lot from the pledge drive, and will likely make at least 2 more posts, one suggesting process improvements for the next pledge drive, and the other analyzing pros and cons of paid Facebook advertising.
Because there were so many different people who contributed, and different people have different privacy settings, I've tried my best to scrub identifying details in this public post whenever possible. Please understand that I have no problem with giving people full credit for their work, and would love to publicly thank you for them. In addition, this summary is a work in progress, and I probably missed a lot of the very important work related to the drive that I'm less familiar with.
Executive Overview
Giving What We Can ran a pledge drive to get as many people as possible to take the pledge or Try Giving in the same time frame, between December 10th 2015 and January 10th 2016. The drive encouraged somewhere between 60 and 90 additional people (who otherwise would not have taken it during the same time period) to take the Giving What We Can pledge, and also a smaller number of Try Givers. This is the second pledge drive: the first one was held in 2014/5 and organised by Ravi Patel from GWWC Cambridge.
Approaches We Tried
We used a multi-pronged approach to get as many people as possible to be excited and willing to take the pledge. Most of these approaches mobilized volunteer effort of the Giving What We Can and EA community.
Facebook was the primary tool for our social media outreach, and the Facebook event was this pledge drive's focus point. Facebook outreach involved the following:
The overall Facebook event was fairly successful, with 13K invited, 311 who marked themselves as "going" (of course, not all of them actually pledged), and 420 "Interested." However, this is worse than our reference class of the 2014/2015 pledge event, with 14K invited, 571 went, and 202 "Maybe."
Other Social Media
Mobilizing the Giving What We Can community
There were many times when we asked/mobilized the general GWWC and EA community to help. We set up an excel sheet for people to record contacting their personal networks (to keep them on track and minimize overlap). Eleanor Stephens also wrote a nice post on how individuals can help with the pledge drive, which was promoted on Facebook to followers of GWWC's page and shared on the GWWC Community Facebook group. We also set up a "Pledge Event Organizers" Facebook group.
Contacting EA Leaders
We had an internal list of 20-30 "EA Leaders" (people exceptionally involved in EA Movement-building) who we asked to help share information about the pledge drive. We have received mostly positive responses and have had a lot of help from leaders of other EA organizations, etc.
Reaching out to EA Bloggers
We reached out to many different EA bloggers, asking them to write about the pledge. Many of them agreed, for example unitofcaring and Jaibot. We also had GWWC blog posts by: Ali Ladak, Eleanor Stephens, Zach Leather. Some bloggers have also very kindly written posts without prompting from anybody in GWWC, for example ThingofThings.
Celebrity Outreach
I started a concerted campaign where we tried to get access to "celebrities"-writers, journalists, artists, bloggers and other people whose influence vastly exceeds the monetary value of their donations, on the fringes of EA. While we didn't manage to talk to many of the people I wanted (which is hardly a surprise!), two journalists and an artist have agreed to take the pledge and potentially do an interview with us. Hopefully we'll be able to get those interviews up soon!
Media
Unfortunately, despite our best efforts, we were unable to obtain significant media attention for this pledge drive. However, aside from the two journalists already mentioned, 2 other journalists (from fairly prominent publications) have expressed an interest in covering us in the future. There are also 3 articles out on Huffington Post, by Jacy Reese from Animal Charity Evaluators, Daniel Selwyn, and myself.
Local Meetups/Chapters
Many of the GWWC chapters and local EA meetups and pages have helped promote the Giving What We Can Pledge Event by sharing our articles, links, and personal messages. GWWC Cambridge was especially impressive, they created a video for the drive (see below) and posted pictures of people who have taken the pledge every day, Humans of New York-style. This seems like an incredibly effective way of showcasing how human GWWC members are, and I will highly encourage other GWWC chapters and EA meetups to try it too.
In addition, both EA London and EA Madison (my local meetup) organized local meetups to talk to people about the GWWC pledge and encourage people to take the pledge/Try Giving, with very positive results.
Video
Giving What We Can: Cambridge created a short, very light-hearted silent movie to encourage people to take the pledge! The video has 1654 views (To be fair, I don't know how many of them were from me) and 62 likes. It's a really cute, fun video.
Progress over Time
Here I include a chart of GWWC's progress during the pledge drive. If you want to learn more about GWWC's progress, you can check out the dashboard yourself.
I included data from July 2015 onwards so people have the appropriate reference class, and used 5-day moving averages to smooth out spikes. There was a dip right after the pledge drive started, followed by a peak about a week in (December 18th/19th), followed by a large dip again for Christmas, and a peak in the last days of December/Jan.1st. There's a dip in the early days of January but the number of pledgers grew very quickly near the end of the pledge drive, cumulating in 17 people pledging on the last day of the drive (Jan.10th), or about half the average number of new pledgers a month in 2014.
Workload/Timeline
Jon Courtney and Marinella Capriati (GWWC staff) have been working on setting up the event, coordinating with chapters, and EA Outreach since October 2015. The core "pledge drive team" also includes Harri Bescelli, Eleanor Stephens and I (volunteers) who worked primarily during this event. I volunteered essentially full-time from December 10th to January 10th. In total, I estimate about 2.5 person-work-months spent on this pledge drive (one from me, 1.5 from the rest of the team).
We also had help from dozens if not hundreds of EAs/GWWC pledgers with technical help, sharing our articles, retweeting, writing personal stories and posts, about why they took the pledge and entreating friends to do the same, etc, etc. I can't imagine how to estimate this time.
Impact
We can't really have a good gauge on our counterfactual impact until people who took the pledge during our pledge drive have been interviewed. That said, there are some ways to estimate it. For example, counting people who marked themselves as "attending" or "interested in" the Facebook event, as of Jan.12th, 66 of them have taken the pledge on or after Dec. 10th 2015, and 13 of them have "Tried Giving" on or after December 10th. 66 may not necessarily be the lower bound, for example some people may have decided to take the pledge in December already and then just signed up for the FB event after hearing about it as an afterthought.
Another way to gauge the impact of our event is to look at before/after. Giving What We Can has about 1400 members (+-5) as of December 10, 2015, and 1555 by the end of Jan.11th, 2016, or a difference of 155. Eyeballing the dashboard, there usually isn't a spike in the number of pledges in December outside of the two drives, suggesting that the base rate of 50-60 people per month in 2015 is the appropriate counterfactual to consider if we didn't have a drive. So just looking at before/after rates will suggest that the impact of the pledge drive is on about 90 people rather than 66. One plausible reason why people may be inspired by the resources from the pledge drive (blog posts, in-person explanations, etc.) but did not take the pledge is that they don't have a Facebook account, another is that they're very private, a third is that our resources didn't explicitly emphasize the value-added of the FB event.
That said, 90 is not necessarily the upper bound for the long-term impact of this pledge drive. For example, we've had 3 HuffPo articles and many different blog posts come out of this pledge drive, which may marginally increase people's likelihoods of pledging in the ensuing months if not years.
Impact Evaluation
Note that even more so than the rest of the post, this is #justmyopinion, and does not represent the official view of Giving What We Can.
So the most naive way to look at our impact is by just adding up the total pledged donations ($17.5 million, plus Try Giving that hasn't been added) from all the people who marked themselves as "attending" the FB event. Not bad for 2.5 work-months!
But clearly this is not counterfactual or time-adjusted. GWWC's estimations is that each new pledger is worth ~60K in donations (counter-factual and time-adjusted). However, since GWWC's own "realistic" estimations is that their impact is ~100x, then the correct internal value of a new pledge relative to other things that GWWC could do, is about 600 dollars each.
By this estimation, the impact of work on this pledge drive, relative to earning to give and donating to GWWC, is somewhere approximately between 66*$600=$39,600 and 90*$600=$54000.
But this is still somewhat naive. If most of the new pledgers would have ended up taking the pledge anyway, the real counterfactual impact is the delta in how much sooner this pledge drive has resulted in people taking the pledge. For example, if you guesstimate that the average is one year, then the internal estimation of how valuable this pledge drive is could be as low as 66*$600/40=$990! However, I think that this is too pessimistically low for three core reasons: 1)Haste consideration means that hearing about this pledge now is worth more than hearing about the pledge later, 2)The "average" of one year is too low: the median might be around there, but the right end of the counterfactual tail could end up being really high, 3) "just hearing about" the GWWC pledge sometimes happens, but a lot of the time it takes staff or concentrated volunteering time. So many of the people who counterfactually would not have heard of the pledge in the context of the pledge drive would probably have taken up staff and volunteering time longer down the line.
In conclusion, the value of the pledge drive relative to other activities GWWC could undertake is inconclusive, but I’m relatively confident (95%+) that it is more worthwhile than 2.5 person-months spent working at a “typical” job and donating to top GWWC charities.
Other Metrics/Outcomes
Hey, awesome!
I've always been very pleasantly surprised that these things are so much of a success. My prior that you can just ask people to donate 10% for life and they will is very, very, very low -- it just doesn't seem like something that can happen and no other charity I know of does it. Everyone emphasizes making very small asks (though monthly donation asks are common) and ramping up steadily over time. So whenever I see the pledge drive work (twice now), I always feel like I'm looking at a miracle and wondering what the catch is.
My current working hypothesis is that the pledge works, but only on people who were already weakly involved in the EA movement and know other people who have pledged. So this would mean that the pledge event does create genuine impact by spurring people to action, but might be weak in (a) trying to build massive success outside EA and (b) in considering the counterfactual of whether pledgers who pledge during the drive would have just pledged later. (Overall, still probably worth doing.)
I think this makes sense of the evidence that you tried posting in various non-EA online groups that seem relevant but had low engagement and that the Tumblr posts went so well (since they were writing to somewhat EA audiences). This hypothesis also makes sense of the data that the 2014 pledge went better than the 2015 one despite the higher effort put into 2015, since in 2014 there were more EAs on the fence that you could convert over.
I'd use this hypothesis to predict that virtually no one will have joined from a Facebook ad and very few people will join from reading the HuffPo post. Instead, most of your people will report being somewhat familiar with EA before pledging and will come from Tumblr or FB posts. I'm super curious whether your interviews will validate this or not.
-
I do think it's worth seriously considering that you're only bringing forward in time the pledge by a year or so.
Also, I think it's worth seriously considering that the people you bring into the pledge through drive events will not be as committed as people who came to the pledge through their own volition (which constitute much of the people from which the $60K estimate was derived).
That being said, even if the 90 people are worth $600 each instead of $60K each, you're still netting ~$22K/mo in counterfactually-adjusted impact, which seems clearly worth doing.
-
This doesn't seem that valuable to me, except in so far as you get another year of donations. I would expect some social effect of getting more members sooner, but I expect this effect would not be large (say larger than the value of the additional year of donations) because my prior is that not that many GWWC members are successful at getting their friends to pledge... and even if they do get a friend to pledge, it will likely just bring them in one year sooner than they otherwise would.
-
This seems incorrect and needlessly unfair to the pledge event, since the pledge event could just be one of GWWC's activities with the 100x ROI. Obviously GWWC has to do something and you should compare the pledge event to the calculated impact of other GWWC activities, but there's no reason to just drop the impact by 100x.
-
I'll admit that the "one year" number is quite made up, but I don't see any reason to think that it is distributed with a wide right tail except that it tells a better story for counterfactuals. At minimum, I think chances are high that they'll have just heard about it in the next pledge drive. ;)
Hi Peter, I really appreciate the detailed comments and nice words!!
"I've always been very pleasantly surprised that these things are so much of a success" Agreed.
"My prior that you can just ask people to donate 10% for life and they will is very, very, very low -- it just doesn't seem like something that can happen and no other charity I know of does it." Well, there are many things other charities don't know and EAs do. :P
"My current working hypothesis is that the pledge works, but only on people who were already weakly involved in the EA movement and know other people who have pledged." Yes, this is my working hypothesis as well. It's actually explicitly addressed in GWWC's "Membership Pathway Model": https://www.givingwhatwecan.org/sites/givingwhatwecan.org/files/attachments/givingwhatwecanpathway.pdf
"This hypothesis also makes sense of the data that the 2014 pledge went better than the 2015 one despite the higher effort put into 2015, since in 2014 there were more EAs on the fence that you could convert over." Hmm...I would be surprised if there are more EAs on the fence in 2014 than 2015. I think there are specific features of the 2014 pledge drive that I will talk about in a follow-up post (not least the sheer degree that the Cambridge group was able to mobilize) that are difficult to reproduce. That said, we have more people pledging in Jan 2016 (104 so far, and still growing as of Jan 25th) than in Jan 2015 (94). (But December numbers are lower, 117 for Dec.2015 vs. 129 for Dec.2014, and since the base rate for pledges is considerably higher in 2015 vs. 2014, on balance I will definitely agree that 2014 was the more successful drive).
"I'd use this hypothesis to predict that virtually no one will have joined from a Facebook ad and very few people will join from reading the HuffPo post" Agreed. That said, when the base rate is 50-60/month in a normal month and ~150 during the pledge drive, any nonzero number is significant! Another point to make is that the counterfactual value of people who pledge from HuffPo articles or FB ads is likely greater than the counterfactual value of pledges who hear from their friends.
"even if they do get a friend to pledge, it will likely just bring them in one year sooner than they otherwise would." Hmm...how will you realistically define the Haste consideration? I generally think of it as "A persuades B to join x months earlier who persuades C to join y months earlier, etc." Put another way, the entirety of the Haste consideration is shifting the curve earlier, I don't think persuading people earlier will significantly change the final number of the people who wound up doing EA things. That said, when the growth is exponential, I think persuading people to join earlier is quite significant.
"This seems incorrect and needlessly unfair to the pledge event, since the pledge event could just be one of GWWC's activities with the 100x ROI." Not really. Consider: Assuming that the average ROI of GWWC's activities are 100x. Then if GWWC doesn't get unusually high funding, a simplified model would be that the organization should prioritize activities that have 100x or higher ROI before bothering with activities that have say 50x, or 5x. And from the perspective of an individual deciding between working on meta activities themselves or taking on a high-paying job and donating to GWWC or other meta orgs, this calculation might be helpful.
" I don't see any reason to think that it is distributed with a wide right tail except that it tells a better story for counterfactuals" Thanks for catching that: I actually got my directions wrong! I meant to say it is likely distributed with a LONG right tail, not high. I think this is defensible. The (like you said, probably very few) people who came across the pledge drive through serendipity or random happenstance rather than a close EA social network might counterfactually have taken years to hear about the pledge, and you don't need that many of them to change the average considerably.
Thanks a lot for your detailed comments!!
That doesn't seem fair to me. I don't know any evidence that suggests EAs outperform professional non-EA fundraisers at fundraising.
-
Yeah, you'd have to elaborate on that. ;)
-
I'm unimpressed by the haste consideration. You'd have to make it more concrete -- what additional benefits do you expect to get beyond having one more year of donations? How do you expect those benefits to compound? Do they compound that way in practice?
-
Well somebody needs to actually do the direct work -- you can't have 100% earn-to-give and 0% direct work or nothing actually gets done. So not everyone can use this argument. I got at this using a crank analogy earlier.
So the consideration for you is not what GWWC's absolute ROI on activities is, but rather...
(a) is working on this GWWC project mean that a higher ROI project does not get done?
(b) could you earn to give and pay for someone to do your GWWC work for you and then some? If so, how much better are you at that GWWC work?
Even if the project has enormous ROI, I imagine that it is the case you could have paid someone else to do it and come out on a profit. But it's possible that GWWC's budget is tight and/or there isn't anyone around who would do the work nearly as well.
-
I agree that's defensible and possibly 2x-10x's the total ROI of the project. But I'd also be concerned that these pledgers are lower quality on average and unlikely to return the full $60K estimated value.
"That doesn't seem fair to me. I don't know any evidence that suggests EAs outperform professional non-EA fundraisers at fundraising."
It's easier to get a long-term pledge for 'whatever charity looks best at the time' than for a particular charity you are working for at the time.
So it's no surprise normal fundraisers, who are paid to raise money for their single present employer, will have left GWWC's approach alone, even if it is much more efficient - which it seems to be.
That strikes me as overconfident. Normal fundraisers ask for regular, life-long giving all the time -- just in the form of a small, monthly donation. They could be asking for a monthly donation of 10% of your income, but they don't.
Though, of course, there are lots of reasons why a good idea might be ignored by the mainstream.
The thing they don't get you to do is pledge to give for the rest of your life. Because if you're committed to just one organisation (their employer) almost nobody would say yes. But GWWC has shown people do say yes at a reasonable rate if the recipient is flexible.
"I don't know any evidence that suggests EAs outperform professional non-EA fundraisers at fundraising."
GWWC's experience make me think it's above a 10:1 ratio, which is apparently typical for commercial fundraisers.
I will think about the other points later. But I definitely agree that there's a distinct possibility that people join because of the pledge drive are less likely to be committed than people who join normally, and that this worry, if true, will be a significant concern to the validity of any impact measurement for the drive.
That said, as a separate point I actually think (~60% confidence) that people who join from hearing about the drive through the media, rather than from a social network initially high in EAs, are more likely to be committed rather than less. This is because they had less initial contact with EAs before being convinced of the idea, suggesting that they had strong innate urges in that direction that did not require a lot of "push" from a social environment conducive to EAness. Does that make sense?
Another data point is how a sizeable minority of EAs (over-represented here and on the FB groups) barely need any convincing for EA...it just made sense as soon as they heard about it.
However, even if my theory about media pledgers being more committed than pledgers from a peer group high in EAs, this is not necessarily evidence against the worry that the pledge drive's pledgers are on average less committed than normal, since like you said, most people who joined from the pledge drive are not due to media outreach.
Yeah, I could buy that. If there's a certain amount of people that will just get EA immediately (which seems plausible based on what we know so far), then it makes sense that the pledge drive -- like any other media campaign -- would uncover some of those people.