CEA: still doing CEA things

Thanks for your comment and feedback. I work on the events team, so just responding to a few things:

Asking people at the end of a conference how many people they’d hypothetically feel comfortable asking for a favor seems akin to asking kids at the end of summer camp how many friends they made that they plan to stay in touch with; if you asked even a month later you’d probably get a much lower number

We do ask attendees about the number of new connections they made at an event a few months later and FYI it's (surprisingly, for the reasons you describe) roughly the same. LTR dips slightly lower a few months after the event, but not by much.

I agree it's an imperfect metric though, and we're exploring whether different survey questions can give us better info. As you note, we also supplement this with the LTR score as well as qualitative analysis of written answers and user interviews. You can look at some more thorough analyses in my recent sequence, I'd love more feedback there.

I found it interesting that the scores for EAG and EAGX look pretty similar.

Just to note that I don't think LTR closely tracks the overall value of the event either. We find that events that reach a lot of new people often get high LTRs (but not always) so a high LTR score at an EAGx event could be due to first-timers feeling excited/grateful to attend their first event which I think is a meaningfully good thing but perhaps worth mentioning.

What percentage of EAG/x attendees completed a survey?

It's usually 20 - 50% of attendees in the surveys referenced by the data you see. Thanks for the suggestion of including no. of respondents.

Hi! Just responding on the groups team side :)

This is a good observation. As we mentioned in this retrospective from last Fall, we decided to restrict UGAP to only new university groups to keep the program focused. In the past, we had more leeway and admitted some university groups that had been around longer. I think we have hit a ~ plateau on the number of new groups we expect to pop up each semester (around 20-40) so I don't expect this program to keep growing.

We piloted a new program for organizers from existing groups in the winter alongside the most recent round of UGAP. However, since this was a fairly scrappy pilot we didn't include it on the dashboard. We are now running the full version of the program and have accepted >60 organizers to participate. This may scale even more in the future but we are more focused on improving quality than quantity at this time. It is plausible that we combine this program and UGAP into a single program with separate tracks but we are still exploring.

We are also experimenting with some higher-touch support for top groups which is less scalable (such as our organizer summit). This type of support also lends itself less well to dashboards but we are hoping to produce some shareable data in the future.
 

Thanks, I’m glad you like the dashboard and are finding it a productive way to engage with our work! That's great feedback for us :) As a quick aside, we actually launched this in late December during our end of year updates, this version just includes some fresh data + a few new metrics.

I’m not sure if all our program owners will have the capacity to address all of your program-specific thoughts, but I wanted to provide some high level context on the metrics we included for now.

When selecting metrics to present, we had to make trade offs in terms of:

• How decision guiding a particular metric is internally
• How much signal we think a metric is tracking on our goals for a particular project
• How legible this is to an external audience
• How useful is this as an additional community-known data point to further outside research
  • E.g. Metrics like “number of EAG attendees” and “number of monthly active Forum users” seem potentially valuable as a benchmark for the size of various EA-oriented products, but aren’t necessarily the core KPIs we are trying to improve for a given project or time period – for one thing, the number of monthly active Forum users only tracks logged in users!
• The quality of the underlying data
  • E.g. We’ve tried to include as much historical data as we could to make it easier to understand how CEA has changed over time, but our program data from many years ago is harder for current staff to verify and I expect is generally less reliable.
• Whether something is tracking confidential information
• Time constraints
  • Updating this dashboard was meant as a ~2 week sprint for me, and imposes a bunch of coordination costs on program owners to align on the data + presentation details.

So as a general note, while we generally try to present metrics that meet many of the above criteria, not all of these are numbers we are trying to blindly make go up over time. My colleague Ollie’s comment gives an example of how the metrics presented here (e.g. # total connections, LTR) are contextualized and supplemented with other evidence when we evaluate how our programs are performing.

Also thanks for your feature requests / really detailed feedback. This is a WIP project not (yet) intended to be comprehensive, so we’ll keep a note of these suggestions for if and when we make the next round of updates!

But this just drove home to me how counter-intuitive the cost-effectiveness numbers for AI-safety are compared to other cause areas.

E.g. I expect Dominion to have cost 10-100x less per hour of engagement, and the amount of investment on this forum to not make sense for animal welfare concerns.

Your first sentence might well be true. But the comparison with the Dominion documentary doesn't seem very relevant. If I consider the primary uses of the forum for animal welfare folks, I expect them to be some combination of a) sharing and critiquing animal-welfare related research and strategy, b) fundraising, and c) hiring.^[1]

So the comparison with an animal welfare-related documentary for mass consumption is less apt, and perhaps you should compare with an hour of engagement for animal-welfare related journals (for the research angle) or fundraisers with fairly wealthy^[2] animal welfare-interested people.

1. ^{^}

    Looking at the animal welfare posts on the forum sorted by relevance, I think my summary is roughly true. Feel free to correct me!

2. ^{^}

   Though I suppose normal fundraisers are with very wealthy people, and not just fairly wealthy ones? But I imagine EAF onlookers to have a long tail, wealth-wise.

Thanks for the comment!

1. I understand your intuition behind dividing the hours by expenses, but I don't think those numbers are correct; this comment has a bit more detail.
2. I also agree with Linch that Dominion is probably not the right comparison; a conference like CARE might be a better reference class of "highly dedicated people seriously engaging with complex animal welfare topics"
3. That being said, I think you have an underlying point which is underappreciated: we sometimes talk about "cause prioritization" as though one cause would be universally better than another, but it's entirely consistent to be willing to pay more for someone to read a forum post about AI safety than about animal welfare, while simultaneously being willing to pay more for an animal welfare documentary than an AI safety one. I think it is, in fact, true that many people I talk to would pay more for an hour of someone reading about AI safety than an hour of reading about animal welfare, and that is a relevant fact for figuring out how we should prioritize our work.
   1. (This argument also should make us aware of bias in the users: people who use the EA Forum are disproportionately interested in causes where the EA Forum is useful. My guess, though I'm not sure, is that the EA Forum is more oriented towards existential risk than EA overall is, for this reason. Thanks to AGB for pointing this out to me.)
       

Given the recent increase in Forum expenditures, it'd probably better to use some measure that incorporates historical cost if it were available to compute hourly rate -- we wouldnt expect to see much of the effects of recent budget expansion (much of which went to hiring developers) to reflect in the 2023 level of engagement.

Thanks for raising this. I agree that the dial has shifted somewhat toward funding constraints for non-mysterious reasons, and I’d note that this is true across different cause areas. CEA is funding constrained too! Our Interim MD has framed this as the start of EA’s Third Wave.

I’ve been leading internally on how we might unconstrain CEA itself, and (in its very early stages) thinking about how we might coordinate fundraising activities between EA meta (~movement building) orgs. But I want to be explicit that there are a couple nearby things CEA is not currently doing and hasn’t been in recent history:

1. Fundraising for non-CEA projects
   • Giving What We Can used to be part of CEA, but was spun out in 2020, and this has been outside our remit since then
   • Luke and Sjir recently wrote about funding constraints from a GWWC perspective
2. Grantmaking to non-CEA projects
   • Similarly, EA Funds was spun out of CEA in 2020
   • The bulk of our current grantmaking is to the groups and EAGx conferences we directly support and their organizers

The dashboard is accurately capturing the fact that we have not (yet) made any big strategic changes to our programs intended to address funding bottlenecks. But it does disguise the fact that we’ve done some things within existing projects, including launching an effective giving subforum (later shut down) and supporting an Effective Giving Summit. I’m confident that there are more things we could do to move the needle here, although I’m less confident that they would be more impactful than our existing programs, even in the new funding environment.

There is important work to be done figuring out the costs and benefits of a more money-oriented approach, but I’d caution that it’s unlikely CEA makes any major changes to our current strategy until we appoint a new CEO. Worth noting, though, that we’re open to pretty major changes as part of that process.

Glad you like the dashboard! Credit for that is due to @Angelina Li .

I think we are unlikely to share names of the people we talk to (mostly because it would be tedious to create a list of everyone and get approval from them, and presumably some of them also wouldn’t want to be named) but part of the motivation behind EA Strategy Fortnight was to discuss ideas about the future of EA (see my post Third Wave Effective Altruism), which heavily overlaps with the questions you asked. This includes:

• Rob (CEA’s Head of Groups) and Jessica (Uni Groups Team Lead) wrote about some trade-offs between EA and AI community building. 
• Multiple non-CEA staff also discussed this, e.g. Kuhanj and Ardenlk 
• Regarding “negative impacts to non-AI causes”: part of why I was excited about the EA’s success no one cares about post (apart from trolling Shakeel) was to get more discussion about how animal welfare has been benefited from being part of EA. Unfortunately, the other animal welfare posts that were solicited for the Fortnight have had delays in publishing (as have some GH&WB posts), but hopefully they will eventually be published.
  • My guess is that you commenting about why having these posts being public would be useful to you might increase the likelihood that they eventually get published, so I would encourage you to share that, if you indeed do think it would be useful.
  • I also personally feel unclear about how useful it is to share our thinking in public; if you had concrete stories of how us doing this in the past made your work more impactful I would be interested to hear them. The two retrospectives make me feel like Strategy Fortnight was nice, but I don't have concrete stories of how it made the world better.
• There were also posts about effective giving, AI welfare, and other potential priorities for EA.

On comms and brand, Shakeel (CEA’s Head of Comms) is out at the moment, but I will prompt him to respond when he returns.

On the list of focus areas: thanks for flagging this! We’ve added a note with a link to this post stating that we are exploring more AI things.

Thanks for the suggestion! This comment from Lizka contains substantial discussion of community versus noncommunity; for convenience, I will reproduce one of the major graphs here:

You can see that there was a major spike around the FTX collapse, and another around the TIME and Bostrom articles, which might make the "average" unintuitive, since it's heavily determined by tail events.

Thanks for engaging with this!

To chime in with a quick data note: "hours of engagement on the Forum" also includes time spent not viewing posts (e.g. time spent viewing events, local groups, and other non-post content on the Forum), so comparing those two numbers will give you a slightly misleading picture of the Community v. non-Community engagement breakdown.

The Forum saw a total of 39,965 hours of engagement with Community posts up until the end of Q2 in 2023, so the actual ratio between Community:non-Community post engagement is more like 2:3. I'll edit the top level post to make this clear (thanks for the nudge).

I can only imagine how bad this ratio was before the redesign if it's still worse than 1:1

FYI, the dashboard we published has this metric broken out over time, so you don't have to imagine this! :) (I don't work on the Online team, but I built this dashboard.)

Not sure why you unendorsed this.

I run Training for Good & also found this wild!

Thanks Vasco!

You have a graph for the "Number of EA Forum posts with 2+ upvotes (per month)". I think it would be better to have graphs for the number of posts per month and their mean karma.

Curious why — what insights do you hope to gain from this / are there specific questions you have about the Forum that you're hoping to answer with this data?

You have a graph for the "Overall EA Newsletter subscriber count (at the start of each month)". Have you considered having one for the number of subscribers of the EA Forum Digest?

Thanks, we've considered this but haven't prioritized so far. I'd be interested to know if there was a particular reason you think this would be useful, or if it's more that generally having this sort of data available to people would be nice? (Both seem like reasonable motivations ofc!)

University Group Accelerator Program - I'll update the top level post for clarity too, thanks!

Hey Vasco! Sorry for such a late response. I imagine you've already seen the data we have on CEA's 2023 budget (although note that we expect actual expenditures in 2023 to be lower than the budget) — hope that helps.

Thanks for suggesting the idea of adding some sort of cost data to our dashboard, I'll keep it in mind for the future :)

Thanks! @Angelina Li deserves the credit :)