tl;dr: AI Safety became the biggest thing in EA Forum discussion last year, and overall Forum engagement trended downards. What that means for the wider EA movement/philosophy is up for interpretation. If you have your own questions, let me know and I'll dive in (or try to share the data)

1. Introduction

This is a follow-up to my previous post using the EA Forum API to analyse trends in the Forum. Whereas that last post was a zoomed-out look at Forum use for as far back as the API goes, this is a specific look at the Forum aggregations and trends in 2023 alone.

Needless to say, last year was a tumultous year for EA, and the Forum is one of (if not the) primary online hubs for the Community to discuss issues and self-organise. I hoped to see if any of these trends could be spotted in the data available, and also see where the data led me on some more general questions.

I'm sharing this now in a not-quite-perfect state but I'd rather post and see the discussion it promotes than have it languishing in my drafts for much longer, and as noted in section 3.4.2 if you have a query that I can dive into, just ask!

2. Methodology

(For more detail in the general method, see the previous post)

On Monday 6th May I ran two major queries to the EA Forum API:

    1) The first scraped all posts in Forum history. I then subselected these to find only posts that were in the 2023 calender year.

    2) I ran a secondary query for all of these postIds to find all comments on these posts, and again filtered to only count comments made in 2023.

Any discrepancy with ground truth might be because of mistakes of my part during doing the data collection. Furthermore my data is snapshot as of how the 2023 Forum looked at May 6th this year, so any Forum engagement that was deleted (or users who deleted their account) at the point of collection will not be sampled. I'll leave more specific methods to the relevant graphs and tables below.

I used Python entirely for this, and am happy to talk about the method in more coding-detail for those interested. I'm trying to resussicate my moribund GitHub profile this summer, and this code may make its way up there.

3. Results

3.1 - Overall Trends in 2023

3.1.1 - Posts and Comments over time

This graph shows the a rolling 21 day mean of total posts and comments made in 2023, indexed to 1.0 at the start,^[1] so be aware it is a lagging indicator. 

Both types of engagment show a decline over the course of the year, though the beginning of 2023 was when the Community was still reeling from the FTX scandal, and the Forum seemed to be the primary online place to discuss this. This was causing so much discussion that the Forum team decided to move Community discussions off the front page, so while I've indexed to 1.0 at the beginning for the graph, it's worth noting that January/February 2023 were very unusual times for the Forum.

There is also a different story to be told for the individual engagement types. Posts seem to drop from the beginning of the year, tick up in the spring (due to April Fools'), and then drop away towards the end of the year. Comments, on the other hand, rapidly drop away, presumably as a result of engagement burning out after the FTX-Bostrom-Doing EA Better-FLI-Sexual Harrassment-OCB perfect storm. They then settle to some sort of baseline around May, and then pick up again sometimes in spurts due to highly-engaging posts. I think the September-October one is due to the Nonlinear controversy, the December Spike is the response from Tracing and Nonlinear themselves. There didn't seem to be any clear candidate for the spikes in the Summer though.^[2]

3.1.2 - Which topics were popular

This is just an overview, I have more topic results to share in 3.2

(To be specific, this is simply a count of how many times a topic tag appeared on a post in 2023, or how often a comment was made on a post with that tag. The vast majority of posts usually have multiple tags, and a small group actually have none.)

Topics which only make it on one list have been highlighted, otherwise it is not surprising that there is a significant amount of overlap between these topics. The biggest difference is the presence of explicit 'Criticism of...' topics in the Most Comments list, but otherwise there's a lot of stability in which topics are most commonly discussed on the Forum.

3.1.3 - Who were the most active users

This is just an overview, I have more user results to share in 3.3

(In this case, for both posts and comments I grouped by user and took the 10 most frequent results. I combined both DataFrames and summed by the karma to get a combined result for the third column)

This time, I've only highlighted those Users who have made it onto more than 1 list. At a brief glance, it seems users who gained high Karma seem to be those who make the most comments, rather than the most Posts. This was slightly surprising to me given my intuition that Posts have a higher score than Comments, but it seems volume of Comments outweigh this.^[3]

Shout out to @Nathan Young for being in the top 10 for posts, comments, and karma in 2023 (which makes his turn away from the Forum in 2024 rather sad, but so it goes). Secondary shoutout to @titotal for being top 10 in Comments and Karma despite taking an explicitly critical stance of many EA causes/arguments.

3.2 - Topics in 2023

The primary reason I set out to do this was to look at the main topics that the Forum discussed over 2023, how that changed over time, and what this can tell us about the state and trajectory of the movement.

3.2.1 - Top 10 Topics of 2023, Rolling Post Frequency

Below is the rolling 21 day mean of proportion of posts that included a given topic, subselected to the 10 most frequent topics over the course of 2023.

I've split out the above chart into 1-5 and 6-10 (open in a new tab for a higher quality image)

We can see some major trends here:

First the 'Building Effective Altruism' and 'Community' topics were dominant at the beginning of February, and most likely continuing a trend from the previous year, and then steadily decreased over the year.

Second - AI Safety takes over as the dominant Forum topic from March onwards (likely due to the release of GPT-4 and the FLI Open Letter) and doesn't relinquish the lead. Note that it is still behind Community and Building EA in terms of Comments and Post Karma. Still, the trend in the Community toward AI Safety continues.

Third - Animal Welfare has slowly risen from around 7.9% of posts at the beginning of the trend to 13.9% at the end. For GH&D the numbers are 9.9% -> 14.7%. Whether this is a return to those topics after Community/AI posts crowded them out at the beginning of the year, or simply a boost driven by Giving Season remains to be seen.^[4]

3.2.2 - Which Topics had the highest average karma

Sub-selected to only those topics which were tagged to 20 or more posts

This table, however, more measures 'Engagement' rather than 'Reception'. High Karma posts might be ones that generate a lot more engagement rather than universal acclaim, and Low Karma posts are more likely to have no captured Community interest as much as other topics.

3.2.3 - Risers and Fallers in Topic Popularity

This table was calculated by ranking the Topics by frequency, grouped by January-June and then July-December. Taking the difference between the two lists showed which Topics were highly prominent early in the year and then fell away, along with which ones emerged later on in the year. Top 100 topics only.

For the first column of topics which fell away, I think that these correspond to the Community dramas of early 2023 (see 'Diversity and inclusion' and 'Criticism of effective altruism'). For the second column, I think we see the broadening of content types on the Forum ('Opinion', 'Audio', 'Video').

3.3 - Users in 2023

3.3.1 - Engagement across Users

Forum use is quite unequal between users. In fact, it is so unequal that to make a useable plot below, I had to convert number of Posts and Comments to a log scale:

In all cases, however, there are more users making comments than posts. This 'power-law' pattern in Forum behaviour appeared in mutliple different plots, though I've focused on the User results here. Still, at a first guess, if you had to describe how the EA Forum worked, a power law of engagement is a good first approximation.

3.3.2 - Engagement and Karma

This trend continues when we add Karma into the mix. Most Forum uses engage a small number of times, and accumulate a small amount of Karma. Some, however, post extensively and gain lots of karma as well:

However, the average Karma story is more interesting, as it shows that even as a user keeps engaging, the trend of average Karma decays to sit between between 10 and 20.^[5]

Shout out to @Rockwell for an exceedingly high 47.65 average Karma despite 75 combined posts and comments over the year, you can see her dot breaking the trendline just before the '100' mark on the chart above!

3.3.3 - Risers and Fallers in User Engagement

Finally, I wanted to look at who posted more in the first 6 months compared to the last 6 (all within the 100 most-active accounts), similar to the analysis done for Topics in 3.2.3, to see which high volume accounts left the Forum over the year and which have started recently.

Additionally, I had some code to look at user behaviour and find 'similar users' for the top candidates, but this had the unintentional side effect of showing which accounts or most similar to Anonymous accounts - so I've decided not to share this for now. (Similar concerns noted by @Jeff Kaufman here).

3.4 - Miscellaneous Findings

3.4.1 - The most 'x'

Given I had the data which gave me the score (proxy for well received), total votes (weak proxy for controversial), and wordcount (very weak proxy for effort), I thought 'why not'. So,^[6] here are some posts and comments to note of 2023:

• Post with the highest score - 10 Years of Earning to Give (❤️) 
• Post with the most votes - Doing EA Better
• Post with the longest wordcount - Podcast/video/transcript: Eliezer Yudkowsky - Why AI Will Kill Us, Aligning LLMs, Nature of Intelligence, SciFi, & Rationality
• Comment with the highest score - Dustin's comment on The EA community does not own its donors' money
• Comment with the most votes - Habryka's reaction to CEA statement on Nick Bostrom's email
• Comment with the longest wordcount - Joel's treatise on Assessment of Happier Lives Institute’s Cost-Effectiveness Analysis of StrongMinds

3.4.2 - Your queries

If you've read it this far, I feel like you found this post interesting. In which case, you probably have questions of your own! If you have a quick query I can put to the DataFrames, then put it in a comment below and I will see what comes up.

4. Interpretations and Takeaways

It's been a lot already! So, to focus on what's important, I'll make 2 points on interpretation and 2 on my personal takeaways.

4.1.1 - Be careful with interpretation

1. The EA Forum is not the movement, and is only one slice of how it organises itself and connects to world. I know of various EAs who are influential, impressive, and doing good who hardly post on the Forum at all (though they might read it). Do not generalise from my spare-time-data-analysis to the state of EA-as-a-whole
2. Even on the Forum, there's probably a slight 'bikeshedding' effect. It's not easy to give a worked out opinion on what exactly the value of Cohen's d for a recent RCT in nutritional interventions in the third world, for instance. It's very easy to shoot off a scorching hot take about AI from first principles. A Forum naturally trends towards engagement, not impact on the world.

4.1.2 - My takeaways

1. Is the downward trend in the Forum temporary, or permanent? I noticed this in the last analysis I did, and it seems to have continued across 2023. Of course, the lead up to FTX was a period of rapid growth, and the fallout a period of retrenchment for EA. Has Community engagement stabilised, or is there still evaporative cooling?
2. AI Safety is the leading current cause, and that makes me feel uncomfortable. I've never bought the simple criticism of "EA 1.0 = Bednets = Good. EA 2.0 = AI = Bad." Nevertheless, I feel like the turn towards AI Safety as overwhelmingly important^[7] is a risky one for the future of EA, and I wish that the community's discussions and engagements with the wider world were more varied.

5. Future Work

As always, the planning fallacy hit me with this post. I had the idea but kept procrastinating, and then when I finally ran the initial data scrape I didn't dive into the details, and then when I did each question simply unlock many more to explore! Here are some of the main ideas that could be pursued next:

• Actually look at the content people post rather than the titles and topics. This will require more in-depth NLP methods to extract signal from that noise.
• More sophisticated network analysis and/or visualisations of users/topics to see if there really are clearly differentiable EA 'schools of thought' in the data.
• Look for dates where there are the biggest discontinuities in trends, or changes in directions of trends, and then see what events in the real world seemed to have those effects on the Forum.
• Track Forum data throughout 2024 to see if it is recovering a lower base, or is continuing its decrease after the frenzy of late 2022/early 2023.

Thanks for reading this post, and I appreciate any feedback or discussion on any of the points I made. If you want to collaborate on any similar future work, then please reach out, I'd love to hear from you :) 

1. ^{^}

   Hence the x-axis starts at January 21st, and the value for January 21st is indexed at 1.0 for both Posts and Comments

2. ^{^}

   If you have potential candidates, let me know

3. ^{^}

   To be precise, the average post has 4.55x more Karma than the average comment. On the other hand, there were 6.94x more Comments than Posts in 2023

^{^}

But should be answerable, with 2024 data available

^{^}

Note this is counting 2023 only, so (Total Karma Earned in 2023)/(Total Forum Engagements in 2023)

^{^}

(at the time of my download and subject to any data-cleaning mistakes I made)

^{^}

I think one can make a case for it to be important, but being overwhelmingly the top cause seems a suspect claim, at least to me

Show all footnotes