Idk if there are best practices for reporting study results in a forum post, but I’ve decided to keep most the nerdy stuff out. My write-up, along with my methods and data are available here: (https://osf.io/bkgvr/). I spent the first semester of my psychology undergrad thesis trying to better understand how to introduce people to EA.
Background/Summary: There are mixed results in the literature about whether giving people information about cost-effectiveness differences between charities will increase their donations to effective charities. Recent work from Lucius Caviola and others, using cost-effectiveness differences of 100x per expert opinion, seems promising. I wanted to see if such information would increase people’s interest in effective altruism. It didn’t, and I also found no significant effect of such information on Effective Donations (though read below for more details), contrary to expectations. Possible takeaway: I’ve sometimes used cost-effectiveness variance information in my EA pitches – this pilot study suggests that such information is not effective at increasing interest in EA. Evidence for selection effect regarding interest in EA and political beliefs. More to explore: People were highly interested in EA, but were not very enthusiastic about getting involved in EA – how do we bridge this gap? Stronger manipulation, larger sample, etc.
Hypotheses: H1: Correcting misconceptions around charity effectiveness will increase people’s interest in the EA movement. H2: Correcting misconceptions around charity effectiveness will increase donations to effective charities. Various exploratory hypotheses.
Participants: 44 undergrad students at a small liberal arts college in the US, majority female, majority white, majority politically left/liberal, mostly intro psych students. Participants had not learned about EA for more than 5 minutes.
Methods: Recruited by email, participants completed an online Qualtrics survey lasting 6.5 minutes. I used a 2x2 factorial design.
Participants either did an exercise where they were asked to estimate the difference in cost-effectiveness between average and highly effective charities – or they had a control task. Then, participants either were told that the difference in cost-effectiveness is 100x – or they were told control, irrelevant information. All participants received an explanation of why charities can vary in cost-effectiveness (different problems, different methods, or different efficiencies/overhead). Participants then split a donation (real money, but not theirs) between an average cost-effectiveness charity and a highly effective charity.
All participants had the same procedure for the rest of the survey. All read a short description of EA
“Effective Altruism is community and a social movement dedicated to answering the question “how can we do the most good with the resources we have?” Effective Altruism advises that before jumping in to trying to improve the world, we should carefully consider how to do so most effectively both so that we don’t mistakenly make things worse, and so that we can help as many people as possible. By using evidence and reasoning we can figure out which methods of doing good are the most impactful, and then we can focus our resources – be that our time, money, or careers – on these methods of improving the world.”
Then, participants answered 6 questions asking about how interesting EA sounds and generally their favorability toward it. Next, participants answered 5 questions about whether they wanted to do certain things to get involved (while this was not numeric, I made a numeric scale out of it for analysis purposes). They next answered a comprehension check question to ensure they had read the description of EA (almost all participants answered correctly, and those who did not were excluded).
Participants answered demographic questions including the following question (response 1-7): “Here is a 7-point scale on which the political views that people might hold are arranged from extremely liberal (left) to extremely conservative (right). Where would you place yourself on this scale?”
All participants were then informed about the nature of the study and the fact that not everybody had been given complete information. All then saw that the most cost-effective charities save 100x more lives than average charities for the same amount of money. Participants then made a Final Effective Donation in which they did the same donation task as before – but with full information for everybody.
Measures: % donated to highly effective charity (Effective Donation). % donated to highly effective charity at end of survey (Final Effective Donation). Final Effective Donation – Effective Donation (Donation Change). Score based on 6 questions asking about interest/favorability toward EA, 2 reverse scored (Interest Score). Score based on 5 questions asking about getting involved in EA (Involvement Score).
Primary Hypothesis Results: H1: No main effects and no interaction – neither cost-effectiveness information nor doing the exercise about cost-effectiveness increased interest in EA. H2: No main effects and no interaction – neither cost-effectiveness information nor doing the exercise about cost-effectiveness increased Effective Donations.
Interesting other results:
Political identification was associated with EA where being more left or liberal was associated with greater interest in EA (no effect on Effective Donations or Involvement Score). I interpret this result as evidence for a selection effect regarding the significant political left demographics of the EA community; that is, people on the left are more interested in EA initially, rather than our community having a treatment effect of making people left (though I didn’t study this). Caveats: participants were generally left, average of 1.5 on a scale of 0-6 (converted from 1-7) explained above; sample size was small; exploratory analysis; question about political beliefs was unidimensional whereas actual beliefs are complex.
Donation Change was associated with Info Condition. This change was larger for participants who did not receive the info at the time of their first donation but did by the time of their second. These participants had larger increases in effective donations, which is what we would expect in line with H2.
I’ve heard, but can’t track down the source, that less than 10% of Oxford students hear about EA before graduating (or this used to be the case). This was not the case in my study. I screened participants for previous experience with EA. 28 (52%) participants had never heard of EA before. 21 (39%) had heard of EA but learned about it for less than 5 minutes, and 4 (7%) had learned about EA for more than 5 minutes so were excluded from participating. Note: this includes people who did not complete the study and/or failed comprehension check, which is why n > 44. Caveat: some people who had learned about EA for more than 5 mins may not have clicked on the survey at all due to the exclusion criteria in recruiting materials. Big caveat: People who know me are probably more likely to participate in my study than random students; most of these people have also heard me talk about EA and therefore have heard of it, making this (likely) a biased sample. These results should be taken with caution, but they indicate that the % of students at my school who have heard about EA is above 10% (but probably not as high as 46%), although the % of students who have actually learned about EA is still quite low (anecdotal).
Less interesting other results:
Final Effective Donation was higher than initial Effective Donation for almost all participants. Exercise responses were in line with other studies. Overall, high interest/favorability toward EA (~5.5/Scale of 1-7). Participants were not as inclined to get involved in EA, as the average answer on most questions was 0.64, or about halfway between “No thanks” (0) and “Maybe” (1), where the other option was “Yes!” (2).
Sorry I didn't include graphics in this post, LaTeX is confusing, and they don't seem necessary. Some are in the Presentation on OSF. Feel free to ask questions. For more about everything, check out all my materials, my full write-up, and data (stripped of most demographic information to preserve anonymity): https://osf.io/bkgvr/
Another consideration here: participants knew they were in an experiment, and probably had a good sense of what you were aiming at.
The difference in treatment and control was whether people
and whether (2x2 here)
If either of these increased their stated interest in EA or their giving behavior, it would be informative, but we still might want to be careful in making inferences to the impact of these activities and this 'de-biasing' in real world contexts.
Either of these tasks might have heightened the 'desirability bias' or the extent to which people considered their choices in a particular analytical way that they might not have done had they not known they were in an experiment.