I believe Angelina left a note in the post where she shared this graph, but right before that cycle we changed the quiz (only 4/10 of the questions are the same). We knew these new questions were quite a bit harder, although candidly we were not expecting score drops this dramatic. We're doing some investigation of the questions, and my guess is we'll iterate a bit more since I think some of the new questions were difficult in ways that are not actually about understanding of the content.
With all that said, I wouldn't focus too much on changes in this score right now, since they are almost certainly tracking changes in evaluation criteria and not changes in student understanding.
I work on the Virtual Programs team at CEA, and we're actually thinking of making some updates to the handbook in the coming months. I've noted down your recommendations and we'll definitely consider adding some of the resources you shared. In particular, I'd be excited to add the empirical data point about cause prio, and maybe something discussing deference and groupthink dynamics.
I do want to mention that some of these resources, or similar ones, already exist within the EA Handbook intro curriculum. To note a few:
Also I want to mention that while we are taking another look at the curriculum - and we will apply this lens when we do - my guess is that a lot of the issue here (as you point out!) actually happens through interpersonal dynamics, and is not informed by the curriculum itself, and hence requires different solutions.
Note: I'm writing this comment in my capacity as an individual, not as a representative of CEA, although I do work there. I wouldn’t be surprised if others at CEA disagree with the characterization I’m making in this comment.
I want to provide one counterexample to the conception that most of mainstream EA is leaning “cause-first” in the status quo. CEA is a large organization (by EA standards) and we definitely invest substantial resources in “member-first” style ways.[1]
To be specific, here is a sampling of major programs we run:
Groups
University Groups (mostly focused on the University Group Accelerator Program currently, which is a scaled program targeting a broad range of mostly non-top unis)
City & National Groups (most of the funds go towards our top 15 city-national groups, but we also fund a long tail of other groups all around the world.)
Virtual Programs (designed to be accessible, available globally, focused on EA fundamentals principles, although it also covers causes.)
Events
EAGs and EAGx are designed to help members coordinate, and EAGx in particularly is held around the world and fairly "big tent"
Community Health / Online
Services offered by these programs (e.g. this forum) are basically infrastructure for community members
Some important caveats: there’s other things we do, we think seriously about trying to capture the heavy-tail and directing people towards specific cause areas (including encouraging groups we support to do the same), and we definitely shifted some content (like the handbook) to be more cause-area oriented. CEA is also only one piece of the ecosystem.
Overall though, I do think much of CEA's work currently represents investment that intuitively seems more "member-first", (whether or not this is the correct strategy), and we're a reasonably large part of the CB ecosystem.
Also, although I think the member/cause distinction is useful, it's also sufficiently vague and "vibes-y" enough that many programs and organizations, like CEA, could probably be construed as focusing on either one.
To push back a little: I don't think this is true for all success stories, and although this is true for some core organizers, many eventual organizers were participants in our discussion group/community events first, and then eventually became organizers (which in turn resulted in the outcomes described).
You're definitely correct that some of the folks though are definitively NOT counterfactual (e.g. Will, you, me) and were already taking EA taking actions without the group's influence.
To clarify, I don't think that events are low impact - they may very well have the harder-to-measure forms of impact you're describing here!
Mostly I'm trying to draw a comparison to some of our activities, like our discussion group, which were lower effort to setup and had clear, measurable positive outcomes.
Hi Defacto! I work on Virtual Programs at CEA.
I believe Angelina left a note in the post where she shared this graph, but right before that cycle we changed the quiz (only 4/10 of the questions are the same). We knew these new questions were quite a bit harder, although candidly we were not expecting score drops this dramatic. We're doing some investigation of the questions, and my guess is we'll iterate a bit more since I think some of the new questions were difficult in ways that are not actually about understanding of the content.
With all that said, I wouldn't focus too much on changes in this score right now, since they are almost certainly tracking changes in evaluation criteria and not changes in student understanding.
Hey - thanks for the suggestions!
I work on the Virtual Programs team at CEA, and we're actually thinking of making some updates to the handbook in the coming months. I've noted down your recommendations and we'll definitely consider adding some of the resources you shared. In particular, I'd be excited to add the empirical data point about cause prio, and maybe something discussing deference and groupthink dynamics.
I do want to mention that some of these resources, or similar ones, already exist within the EA Handbook intro curriculum. To note a few:
- Moral Progress & Cause X, Week 3
- Crucial Conversations, Week 4 (I think this gets at some similar ideas, although not exactly the same content as anything you listed)
- Big List of Cause Candidates, Week 7
Also I want to mention that while we are taking another look at the curriculum - and we will apply this lens when we do - my guess is that a lot of the issue here (as you point out!) actually happens through interpersonal dynamics, and is not informed by the curriculum itself, and hence requires different solutions.
Note: I'm writing this comment in my capacity as an individual, not as a representative of CEA, although I do work there. I wouldn’t be surprised if others at CEA disagree with the characterization I’m making in this comment.
I want to provide one counterexample to the conception that most of mainstream EA is leaning “cause-first” in the status quo. CEA is a large organization (by EA standards) and we definitely invest substantial resources in “member-first” style ways.[1]
To be specific, here is a sampling of major programs we run:
Some important caveats: there’s other things we do, we think seriously about trying to capture the heavy-tail and directing people towards specific cause areas (including encouraging groups we support to do the same), and we definitely shifted some content (like the handbook) to be more cause-area oriented. CEA is also only one piece of the ecosystem.
Overall though, I do think much of CEA's work currently represents investment that intuitively seems more "member-first", (whether or not this is the correct strategy), and we're a reasonably large part of the CB ecosystem.
Also, although I think the member/cause distinction is useful, it's also sufficiently vague and "vibes-y" enough that many programs and organizations, like CEA, could probably be construed as focusing on either one.
To push back a little: I don't think this is true for all success stories, and although this is true for some core organizers, many eventual organizers were participants in our discussion group/community events first, and then eventually became organizers (which in turn resulted in the outcomes described).
You're definitely correct that some of the folks though are definitively NOT counterfactual (e.g. Will, you, me) and were already taking EA taking actions without the group's influence.
Definitely agreed!
To clarify, I don't think that events are low impact - they may very well have the harder-to-measure forms of impact you're describing here!
Mostly I'm trying to draw a comparison to some of our activities, like our discussion group, which were lower effort to setup and had clear, measurable positive outcomes.