Comments
Thanks! I really appreciate this level of transparency and thoughtfulness.
CEA Mid-year update (2020)
Thanks! I really appreciate this level of transparency and thoughtfulness.
This is very helpful. Did you do any benchmarking for retention? Do you know what the convention is for counting part-time versus full-time, interns, student employees, etc.? Voluntary versus involuntary turnover?
Did you do any benchmarking for retention?
We did some light googling and also got some data from other EA orgs. Retention varies a lot by industry, but broadly I think our retention rates are now comparable or a bit better than the relevant average.
Do you know what the convention is for counting part-time versus full-time, interns, student employees, etc.?
Not sure what the convention is, but we looked at full-time employees.
Voluntary versus involuntary turnover?
Again, not sure what the convention is, but we're including both. I think that voluntary turnover is also a mildly negative sign (of bad hiring or training).
I'm impressed with the success of Virtual EAGx. Do you have a measure of how successful that was for a comparable population to EAG London 2019? Or, say, comparing the success for people who have been at 2 previous EAGs?
Also, I'm curious, what CRM are you using and for what purpose?
Thanks Edo! We use a number of metrics to track the success of our events. I’ve pulled together a few charts for you that show success on four measures:
You’ll see that our likelihood to recommend score was much lower than usual for the EA Global: Virtual in March, but we learned lessons from that event which contributed to the EAGxVirtual score being comparable with previous in-person events.
In terms of valuable experiences, EAGxVirtual was much more successful at changing the minds of attendees, particularly first-time attendees. However, the number of people making important connections was lower than EA Global: London 2019.
The main difference we’ve noticed between virtual and in-person events is the number of new connections made (see slides 4 and 5). This feels significant because attendees consistently report one-on-one meetings and new connections as their most valuable experience at our events. In-person events seem to be much more effective at generating new connections but there isn't much of a difference in terms of generating at least one important connection. Perhaps we could make some tweaks to the virtual format to increase the number of new connections at future events.
You’ll also see that our virtual events have been very successful at maintaining the motivation of attendees, more so than EA Global: San Francisco 2019. In particular, we received lots of positive feedback about the virtual format from community members who wouldn’t normally be able to attend an in-person EA conference.
> Also, I'm curious, what CRM are you using and for what purpose?
We are using Zoho CRM. The major use case for it right now is processing event applications: it's integrated with our application forms, ticketing systems, bulk mailing software etc. to provide a unified overview of everyone who has applied for an event. (This is briefly mentioned above in the "Systems Improvements" section of this post.) We also use it to identify attendees who are good fits for specific parts of the event, for example creating a list of first-time attendees who might benefit from our "Guides" program. Other CEA projects sporadically use the CRM to identify specific people who match some criteria (e.g. if we are trying to brainstorm potential group organizers in some location) but this is not the main use case.
Thanks!
I find it interesting that first-time attendees make fewer connections than returning attendees. Some reasons that it might be the case - less focus on scheduling 1-1s, less interest in them than in more experienced people, fewer referrals, more interest in the content, less time spent on the conference.
Generally, it is amazing that there are many plan changes and increased motivation within first-timers and returning attendees. Again, amazing work!
Thanks for the information on the CRM :) We in EA Israel are trying out a costume Airtable database for our needs (which are mostly managing connections and mapping people and organizations of interest).
We'd like to share an update on our recent progress.
In a previous post, we set out five goals for 2020:
Overall, I think we're making good progress toward our goals for the year.
Specific progress:
I also feel that we've fixed a lot of internal issues over the last 18 months and that we continue to gradually improve. We have 13 months of runway. Staff performance, morale, and retention appear to be solid and improving. We've improved our accounting systems, grantmaking records, and HR compliance/onboarding, and invested our reserves to protect against inflation.
We’re in the early stages of improving our support for groups. This is a major focus at the moment.
We regranted a total of $325,000, including grants to the following:
We collaborated with the current NYC organisers to run an application round for an EA NYC Community Director. We were excited by the quality of the applications we received, and decided to make offers to Arushi Gupta and Aaron Mayer to be EA NYC Co-Directors (rather than just making one offer). This is the first time that we’ve collaborated with a group to run an application round for a specific location. We’re pleased with how the process went, and it helped us further develop our processes for evaluating grant applications more generally. We may want to run similar application rounds in the future — though it took a relatively long time, so we don’t expect to be able to do this for many locations.
In response to COVID-19, we replaced EA Global: San Francisco 2020 with a virtual event.
As the event was scheduled for March, just as the extent of the coronavirus epidemic was becoming clear, there was not much time to plan this transition. I think the team did a good job anticipating the change and adapting.
We met about half of our goals for the event.
Some key stats:
Goals we achieved:
Goals we didn’t achieve:
See more details in the wrap report.
In June, we collaborated with EAGx organisers from around the world to run the largest ever EA conference, with over 1,400 attendees from 57 countries. For many community members without a local group nearby, this was their first chance to attend an EA event and meet other EAs from across the world. The event resulted in 10,456 connections and 2,495 meetings within the Grip networking app, and over 3,000 hours of content views.
Some key stats:
This event’s “likelihood to recommend” score was a big improvement over our first virtual event, and was closer to the usual score for an in-person event (see chart above).
Although the proportion of attendees making important connections was lower than at previous events, 50% of survey respondents reported making a minor plan change, and over a third changed their mind about something.
Goals:
We achieved almost all of our goals for the event, including:
We shifted our admissions system to new software, began batching responses to applications, and hired extra capacity to help with the admissions process. This seems to be saving time on admissions.
It seems that virtual events are worth running — they are normally not quite as good as in-person events, but cheaper to do and easier for people to access. We’re tentatively thinking of continuing to run virtual events in addition to in-person events in the future.
The Forum's key metric (number of views of good posts) has more than doubled since the start of the year.
While we've spent a lot of time fine-tuning and validating this metric, we want to flag that the definition of a "good post" is inherently subjective. Experienced EAs outside of CEA have generally agreed with our assessments when we've asked them to evaluate posts, but we want to avoid drawing strong evidence of the Forum's success from this metric alone.
(Some examples of "good posts" from this year include writings on randomista development, personal cruxes on AI safety, and considering a wider range of longtermist jobs.)
We've also seen many other metrics of Forum activity increase since the start of this year. For example, views of all posts by logged-in users have roughly doubled, and the number of posts with 2+ karma has increased by ~80%.
We continue to port features across from LessWrong. We recently imported tags, pingbacks and link previews. (Tags are our most requested feature ever.) We also contribute to LW’s codebase.
We also launched a sequence of posts introducing people to EA (this is a first draft — we welcome feedback via this form). We’ve begun to do more outreach to potential top posters, including promotion at EAGxVirtual, which resulted in our biggest spike in new users ever. We’ve also been trying to increase the Forum’s visibility on search engines (though we are struggling to make improvements, despite engaging SEO consultants).
We continue to offer editing for Forum posts, and to run the Forum prize, EA Newsletter, EA/CEA social media, etc.
Low satisfaction scores: We asked users: "On a scale of 0 to 10, how likely would you be to recommend the Forum to someone who recently became interested in effective altruism?" The average response was 6.9. We think that this low score was partly because people felt that the Forum is relatively insider-y, and so not suitable for new people. However, we'd like to get those scores up.
Demographics: According to Google Analytics, 41% of Forum viewers are female (higher than the proportion of community members who are female), but we believe* that males make up a higher proportion of authors than their proportion in the community. We are considering how we could encourage more women to post.
*Many authors are anonymous, so we aren't certain of this.
The community health team tries to enable EA's long-term success by responding to media enquiries, mediating community disputes and complaints, and providing advice and support to local organisers who are facing particularly difficult problems.
One of our goals is to be available to the community to help with our areas of specialty. Here's the breakdown of cases we've assisted with during 2020 so far:
Some of our proactive work:
Jonas Vollmer has joined CEA as Head of EA Funds. Jonas was previously Co-Executive Director of the Center on Long-Term Risk (previously the EA Foundation). We were impressed by Jonas' analysis of the strategic options for Funds, and by his plans for active grantmaking (that is, seeking out potential grantees more proactively rather than only granting to people who applied for funding).
If we think that Funds is making good progress, we'll consider spinning off the project in the next year or so. Jonas would report to our board, but continue to be supported operationally by our team (like 80,000 Hours is).
Although we have been mostly focused on strategy and hiring so far this year, cumulative donations YTD to the four main funds are up about 36% (~$4.5m vs. ~$3.3m). Including pass-through donations (e.g. people donating to MIRI), we're at $5.9m compared to $4.1m last year and $5.6m the year before. Note that donations are very lumpy, so this could partly be random deviation.
Luke Freeman just joined us to lead GWWC, with a view to spinning off the project from CEA.
Luke is a long-standing GWWC member with a background in startups, digital marketing, and local community building in Australia. We've been impressed by his supportive and welcoming approach to community building, and plans for spreading awareness of the pledge.
If we think that GWWC is making good progress, we'll consider spinning off the project in the next six months or so. Luke would report to our board, but continue to be supported operationally by our team (like 80,000 Hours is).
New pledge rates have held relatively steady (around 10 per week).
As detailed in the announcement post, Luke is looking for feedback, volunteers, member stories, and introductions, and has been holding some online events for members.
At the mid-year point, we are 36% below budget, due to cancelling in-person events such as EA Global, slower hiring, lower-than-anticipated salary increases, and community-building grantmaking falling more in the second half of the year. Reforecasting our spend to the year end suggests we will be 15% under budget for 2020 overall.
Taking the above into account, we currently have 13 months of runway, providing financial security for CEA until around halfway through 2021.
We’ve been tidying up our financial systems and improving our internal reporting to allow more accurate and faster forecasting. We’ve moved some of our funds to interest-bearing accounts, which yields around $3k/week in interest payments. We also implemented a grant management system, which should allow us to automate more of our grantmaking, increase compliance, and improve monitoring and evaluation.
We continue to test and improve our strategy for the organisation, including developing metrics that can track our progress. This is still a work in progress, so we’re not ready to share more publicly at the moment. I’m overall slightly disappointed with the rate of progress here, and I’m aiming to focus more on this in the final six months of the year.
I feel that staff morale is now solid and improving. The data support this view, but I think it’s extremely hard to measure this reliably. (The linear regression over the last year is for self-reported morale to increase by an average of 0.71 points per year on a 10-point scale. This could partly be explained by changes in the question we ask staff, or how people were approaching the question, but I think this is at least partly a real effect.)
Retention is the percentage of staff at the beginning of the period who still work for the org at the end of the period.
For historical reasons, we report this from February to February. Note that retention and turnover figures for 2020 might change over the course of the year.
As I discussed above, I think we’re making good progress toward our goals for the year. We plan to review our progress again at the end of the year.
For what it's worth EA Survey 2019 data suggests this too. Obviously classifying Forum posts directly gives a more comprehensive sample, but as you note has the issue of authors being of indeterminate gender. The chi square tests are, however, not significant for posting on the EA Forum (p=0.052) and writing about EA (not on the EA Forum) (p=0.057), while the others were p<0.001.
Thanks for sharing this! I found it somewhat surprising that the scale of effect looks like it's bigger for comments vs. posts. (I imagine that the difference in significance is also partly that the sample size for posters is much smaller, so it's harder to reach a significance threshhold.)