So my first reaction to the Youth Ministry Adherence data was the basically the opposite of yours, in that I looked at it and thought 'seems like they are doing a (slightly) better job of retention'. Reviewing where we disagree, I think there's a tricky thing here about distinguishing between 'dropout' rates and 'decreased engagement' rates. Ben Todd's estimates which you quote are explicitly trying to estimate the former, but when you compare to:
those listed as “engaged disciples” who continue to self-report as “high involvement”
...I think you might end up estimating the latter. 'High involvement' was the highest of four possible levels, and 'Engaged disciple' was also the highest of four possible levels. By default I'd look at the number who are neither moderately nor highly involved, i.e. 8% rather than 42%.
More generally, my understanding is that Ben was counting someone as not having dropped out if they were still doing something as engaged as fulfulling as GWWC pledge, based on quote below. So if you start at a much higher level than that (like...attending the Weekend Away), there's a lot of room to decrease engagement substantially while still being above the bar and not having 'dropped out'. Which in turn means I'd generally be aiming to have similar leeway for 'regression to the mean' in the comparisons. Or you can compare everything to Ben's GWWC dropout number of 40%, which has no such leeway, as you do with the similarly-no-leeway case of vegetarianism.
I appreciate this is a highly subjective call though, this is very much just my two cents. I could easily imagine changing my mind if I looked at the Youth Ministry information more closely and decided that 'Engaged Disciple' actually constituted some kind of 'super-high-involvement' category.
The list contains 69 names. Several of the team went over the names checking who is still involved. We started with our own knowledge, and then checked our own engagement data for ambiguous cases.
We counted someone as ‘still involved’ if they were doing something as engaged as fulfilling a GWWC pledge.
On this basis, we counted about 10 people we think have ‘dropped out’, which is 14% of the total in 6 years.
Interesting, thanks! Something which probably isn’t obvious without reading the methods (pages 125-127) is that study participants were recruited through church mailing lists and Facebook groups. So the interpretation of that statistic is “of the people who answer surveys from their church, 92% report at least moderate engagement”.
“Moderate engagement” is defined as an average of a bunch of questions, but roughly it means someone who attends church at least once per month.
I think that definition of “moderate engagement” is a bit higher than “willing to answer surveys from my church” (as evidenced by the people who answered the survey but did not report moderate engagement), but it’s not a ton higher, so I’m hesitant to read too much into the percentage who report moderate engagement.
I felt like “high engagement” was enough above “willing to answer a survey” that some value could be gotten from the statistic, but even there I’m hesitant to conclude too much, and wouldn’t blame someone who discounted the entire result because of the research method (or interpreted the result in a pretty different way from me).
If we want to compare it to Ben’s EA estimates: I guess one analog would be to look at people who attended that weekend away but also answered the EA survey five years later. I’m not sure if such a data set exists.
Thanks for writing this! I found it a really interesting series and kudos to you for sharing earlier-stage thinking publicly. I definitely know that I can find sharing such thinking pretty daunting!
Small point:
>ACE estimates that the average vegetarian stays vegetarian for 3.9-7.2 years, implying a five-year dropout rate of 53%-77%.
I think that 3.9-7.2 is their estimate for (i) the averagevegetarian adherence length, but you might be interpreting that here as more like (ii) the medianlength of vegetarian adherence?
From that ACE report:
> We can see in the Guesstimate model that after multiplying, the length of adherence for current vegetarians is 32–49 years.
And then they calculate the length for former vegetarians using the following as a basis:
I’d say the underlying distribution here is pretty skewed, so the difference between that average vegetarian length and median vegetarian length might be pretty significant.
So I guess my pretty quick sense is that the median vegetarian adherence length may be a fair bit shorter than 3.9 - 7.2 years. And if you are interpreting that 3.9-7.2 as being more about the median length, and it was in fact a fair bit shorter, then that could meaningfully change some of your conclusions here.
I think that is all somewhat nitpicky though, and I could certainly be wrong about it! Regardless, thanks again for sharing all of this. :)
Thanks Kieren! I was interpreting it to be the expectation of a geometric distribution (i.e. mean length assuming a constant annual probability of leaving), which I think is the correct way to interpret that number? Let me know if that’s wrong though!
The assumption that length is geometrically distributed might not be warranted, I'm not sure.
Thanks, Ben! That all seems fair enough for these purposes.
Fwiw, I think that number might be more the arithmetic mean of some observations. Interpreting it as a geometric mean seems like it doesn’t strongly violate much, but I think the geometric mean is going to be a little bit lower.
But yeah, I doubt it makes much of a difference in the scheme of things!
Cool series, thanks for sharing on the forum. One nitpick:
ACE estimates that the average vegetarian stays vegetarian for 3.9-7.2 years, implying a five-year dropout rate of 14-26%.
I'm not sure how your rate is being calculated from ACE's figures here, but at first pass it seems wrong? Since 5 years is within but slightly towards the lower end of the range given for how long the average vegetarian stays vegetarian, I'd assume we'd end up with something more like a ~45% five-year dropout rate. By contrast, a 14-26% five-year dropout rate would suggest that >50% of vegetarians are still vegetarian after two such periods, i.e. 10 years.
If I'm misunderstanding either stat, just let me know and will happily retract.
This is the third in a sequence of posts taken from my recent report: Why Did Environmentalism Become Partisan?
Summary
Rising partisanship did not make environmentalism more popular or politically effective. Instead, it saw flat or falling overall public opinion, fewer major legislative achievements, and fluctuating executive actions.
Public Opinion...
I think right now EAs might be making a significant mistake by paying insufficient attention to the political realm. As EAs we tend to figure out what’s most impactful for us to work on and focus hard. That’s great! But there are various actions that are ‘non-delegatable’ - the extent to which an individual can do the action is limited (like voting, going to a protest, making hard money contributions to particular campaigns). It might be useful if we were all more in the habit of doing variou...
This post presents the executive summary from Giving What We Can’s impact evaluation for 2025. At the end of this post we share links to more information, including the full report and...
I spent one day trying to compare retention in EA to other movements. I was unable to find anything that I was very happy with, but I’m sharing these comparisons in case they are helpful to others. Hopefully Cunningham’s law will kick in and someone will be spurred to do a better analysis.
I found semi-comparable estimates of retention for Southern Baptist megachurch youth ministry participants, vegetarians, and civil rights activists who participated in Freedom Summer. While none of these data sources seemed precise or perfectly comparable, my best guess is that EA is slightly better at retention than they are, and unlikely to be substantially worse.
How does EA compare to other movements?
Most interviewees remarked that, once someone gets to the point where their career is driven by EA considerations, it’s unlikely that they will leave. This might suggest that CEA should not focus on retention, as it is already going well.
For some insight on this, I tried to compare Ben Toddd’s estimates with estimates from other movements. My best guess is that EA has better retention than Southern Baptist megachurches and Freedom Summer participants, and similar or slightly worse retention to vegetarianism. Most of my uncertainty comes from how reliable these estimates are and how comparable they are to each other.
Because of how wide my error bars are, it’s hard to say too much with confidence. I don’t think EA has a severe retention problem (compared to e.g. Baptist churches). We also don’t seem substantially better at retention than others (e.g. vegetarianism). So my guess is that EA is slightly above average at retaining highly engaged members (compared to other movements), which makes me think it’s worth investing a moderate amount of resources into retention.
Five-year dropout rate
Ben Todd’s estimate for top 2000 most engaged EA’s
Dropout rate (don’t self-report as moderate or high involvement)
Dropout rate (don’t self-report as high involvement)
Disengaged prospect
100%
100%
Normative attender
26%
58%
Enthusiastic follower
21%
65%
Engaged disciple
8%
42%
I think we are mostly interested in those listed as “engaged disciples” who continue to self-report as “high involvement”, so the 42% number is what I’m going to focus on.
This study did not control for the fact that people who were older at the time of surveying were more likely to have dropped out.[2] Also note that the survey was retrospective and presumably suffers from response bias.
To try to handle the fact they didn’t control for age, I tried to find the parameters for a geometric distribution which fit their overall results of 42% dropout. A constant 14.5% annual dropout rate results in 42% of respondents having dropped out:
Age
Elapsed years
Percent of respondents
Expected
dropout rate
Percent of total sample who
dropped out from this age bracket
18
0
0.4
0.00
0.00
19
1
10
0.15
1.45
20
2
24.4
0.27
6.56
21
3
20.1
0.37
7.54
22
4
14
0.47
6.52
23
5
7.2
0.54
3.91
24
6
8.6
0.61
5.24
25
7
4.7
0.67
3.13
26
8
6.1
0.71
4.36
27
9
4.7
0.76
3.55
Total
45
100.2
42.26
This assumes that “engaged disciples” are evenly distributed across the ages, which is probably not completely true but I would guess mostly true. A 14.5% annual dropout rate is a 54% five year dropout rate.
I’m not very confident in the 54% number, but I feel relatively confident that the number is higher than Ben Todd’s estimate of 20%. The paper cites another survey which found that 70-90% of teenagers stop attending church within two years of high school graduation. (One wrinkle is that there is some evidence people reengage with the church in their late 20s/30s, meaning that it’s a little hard to define what “dropout” actually means.)
Vegetarian adherence
ACE estimates that the average vegetarian stays vegetarian for 3.9-7.2 years,[3] implying a five-year dropout rate of 53%-77%.[4] My guess is that the average vegetarian is equivalent to Ben’s lowest level of engagement (GWWC member but not otherwise involved) or even less engaged, and being vegetarian seems about as big of a commitment as keeping the GWWC pledge. This implies a dropout rate higher than what Ben estimated for EA.
US Civil Rights Activists - Freedom Summer
Freedom Summer was a 1964 project with extremely high asks of its participants: they were mostly college students who gave up their summer to be unpaid volunteers facing severe violence (several were murdered). Participants needed to apply beforehand, and those who applied but withdrew from the project before it started give some estimate of adherence amongst civil rights activists, split by engagement level.
I think that attending Freedom Summer is a sign of involvement in the civil rights movement very roughly comparable to how having attended early EA retreats is a sign of being at one of Ben’s two highest levels of EA involvement. So that’s what I’m going to compare it to, even though the comparison is quite inexact.[5]
In this and the rest of this section I’m going to assume a constant annual dropout rate, i.e. a geometric distribution, unless otherwise specified.
In the section where the author analyzes demographic variables, they say that only educational ones had a significant impact. Taken at face value, this implies that age does not correlate with dropout rate, which seems unlikely to me, but possible.
Taken from their “final estimate” but excluding the effect of cultured meat alternatives, death, and GCRs, since those do not seem relevant to the question I’m trying to answer here.
Assuming a geometric distribution whose expectation is 1/p
There are a lot of ways in which Freedom Summer is different from EA, the most notable being the violence participants experienced. Another major concern is that the Civil Rights Act of 1968 is seen by some as the culmination of the civil rights movement, meaning that many participants perhaps should have been expected to move to other causes after 1968 (e.g. anti-Vietnam War). But I think we would have expected many of them to be active in at least some social movement, if they were truly “retained”.
This is the percentage who were active in civil rights at some point within the five years following Freedom Summer, but I am simplifying this to assume it’s an annual dropout rate. My justification for this is a) there probably aren’t many people who didn’t engage for 4 years and then became active again in the fifth, and b) to the extent that these people do exist, it’s arguably more accurate to classify them as having disengaged and then reengaged instead of having been retained.
Don't forget the 2018 EA Survey analysis that suggests a ~40% EA drop out rate after 4-5 years.