Announcing EA Survey 2022

Thanks!

I'd say it's pretty uncertain when we'll start publishing the main series of posts. For example, we might work on completing a large part of the series, before we start releasing individual posts, and we may use a format this year where we put more results on a general dashboard, and then include a smaller set of analyses in the main series of posts. But best guess is early in the new year.

That said, we'll be able to provide results/analyses for specific questions you might want to ask about essentially immediately after the survey closes. 

Thanks for the suggestion! We can certainly add something about this to the landing page. [And have now done so]

I would also note that this text is usually also already included where the survey is distributed. i.e. when the survey is distributed through the EA Newsletter or CEA social media, it will go out with a message like "If you think of yourself, however loosely, as an “effective altruist,” please consider taking the survey — even if you’re very new to EA! Every response helps us get a clearer picture" before people see the survey.  That kind of message didn't seem so necessary on the EA Forum announcement, since this is already a relatively highly engaged audience.

Thanks! This is useful feedback.

We'd like to include more questions in the extra credit section, and I agree it would be useful to ask more about the topics you suggest. 

Unfortunately, we don't find that adding more questions to the extra credit section is completely 'free'. Even though it's explicitly optional, we still find people sometimes complain about the length including the optional extra credit section. And there's still a tradeoff in terms of how many people complete all or part of the extra credit section. We'll continue to keep track of how many people complete the survey (and different sections of it) over time to try to optimise the number of extra questions we can include. For example, last year about 50% of respondents started the extra credit section and about 25% finished it.

Notably we do have an opt-in extra survey, sent out some time after the main EA Survey. Previously we've used this to include questions requested by EA (usually academic) researchers, whose questions we couldn't prioritise including in the main survey (even in extra credit). Because this is completely opt-in and separate from the EA Survey, we're more liberal about including more questions, though there are still length constraints. Last year about 60% (900) people opted in to receive this, though a smaller number actually completed the survey when it was sent out. 

We've previously included questions on some of the topics which you mention, though of course not all of them are exact matches:

• Moral views: We previously asked about normative moral philosophy, metaethics, and population ethics
• Identification with EA label: up until 2018, we had distinct questions asking whether people could be "described as "an effective altruist"" and whether they "subscribe to the basic ideas behind effective altruism". Now we just have the self-report engagement scale. I agree that more about self-identification with the EA label could be interesting.
• Best and worst interactions with EA: we've definitely asked about negative interactions or experiences in a number of different questions over the years. We've not asked about best interactions, but we have asked people to name which individuals (if any) have been most helpful to them on their EA journey.
• Community building preferences: we've asked a few different open-ended questions about ways in which people would like to see the community improve or suggestions for how it could be improved. I agree there's more that would be interesting to do about this.

Thanks for asking! We would definitely encourage community builders to share it with their groups. Indeed, in previous years, CEA has contacted group organizers directly about this. We would also encourage EAs to share it with others EAs (e.g. on their Facebook page) using the sharing link. I would not be concerned about you 'skewing the results' by sharing and encouraging people to take the survey in this way, in general, so long as you don't go to unusual lengths to encourage people (e.g. multiple reminders, offering additional incentives to complete it etc.). 

Thanks! We think about this a lot. We have previously discussed this and conducted some sensitivity testing in this dynamic document.

I wonder whether it could be useful to survey a sample from an actually random frame to just get some approximate idea of the size and demographics of the EA community.

The difficulty here is that it doesn't seem to be possible to actually randomly sample from the EA population. At best, we could randomly sample from some narrower frame (e.g. people on main newsletter mailing list, EA Forum users), but these groups are not likely to be representative of the broader community. In the earliest surveys, we actually did also report results from a random sample drawn from the main EA Facebook group. However, these days the population of the EA Facebook group seems quite clearly not representative of the broader community, so the value of replicating this seems lower.

The more general challenge is that no-one knows what a representative sample of the EA community should look like (i.e. what is the true composition of the EA population). This is in contrast to general population samples where we can weight results relative to the composition found in the US census. I think the EA Survey itself represents the closest we have to such a source of information.

That said, I don't think we are simply completely in the dark when it comes to assessing representativeness. We can test some particular concerns about potential sources of unrepresentativeness in the sample (and have done this since the first EA Survey). For example, if one is concerned that the survey samples a disproportionate number of respondents from particular sources (e.g. LessWrong), then we can assess how the samples drawn from those sources differ and how the results for the respondents drawn from those sources differ. Last year, for example, we examined how the reported importance of 80,000 Hours differed if we excluded all respondents referred to the survey from 80,000 Hours, and still found it to be very high. We can do more complex/sophisticated robustness/sensitivity checks on request. I'd encourage people to reach out if they have an interest in particular results, to see what we can do on a case by case basis.

Thanks for taking the time to comment!

All of the questions you had substantive comments about the wording of were external requests which we included verbatim.

Re. the order of the dates: when testing, we found some people thought this was more intuitive left to right and some top to bottom (for context, it's not specifically designed as 'grid', it just happens that the columns are similar lengths to the rows). It could be put in a single column, though not without requiring people to scroll to see the full questions, or changing the style so that it doesn't match other questions. Exactly how you see the questions will vary depending on whether you're viewing on PC, phone or tablet though.

I'm curious to read some of the reasoning of those who disagreed with this, as I'm currently high-conviction on these recommendations but feel open to updating substantially (strong beliefs weakly held). 

WAY too many of the questions only allow checking a single box, or a limited number of boxes. I'm not sure why you've done this? From my perspective it almost never seems like the right thing, and it's going to significantly reduce the accuracy of the measurements you get, at least from me.

Thanks for your comment. A lot of the questions are verbatim requests from other orgs, so I can't speak to exactly what the reasons for different designs are. Another commenter is also correct to mention the rationale of keeping the questions the same across years (some of these date back to 2014), even if the phrasing isn't what we would use now. There are also some other practical considerations, like wanting to compare results to surveys that other orgs have already used themselves.

That said, I'm happy to defend the claim that allowing respondents to select only a single option is often better than allowing people to select any number of boxes. People (i.e. research users) are often primarily interested in the _most_ important or _primary_ factors for respondents, for a given question, rather than in all factors. With a 'select all' format, one loses the information about which are the most important. Of course, ideally, one could use a format which captures information about the relative importance of each selected factor, as well as which factors are selected. For example, in previous surveys we've asked respondents to rate the degree of importance of each factor , as well as which factors they did not have a significant interaction with. But the costs here are very high, as answering one of these questions takes longer and is more cognitively demanding than answering multiple simpler questions. So, given significant practical constraints (to keep the survey short, while including as many requests as possible), we often have to use simpler, quicker question formats.

Regarding politics specifically, I would note that asking about politics on a single scale is exceptionally common (I'd also add that single-select format for religion is very standard e.g. in the CES). I don't see this as implying a belief that individuals believe in a single "simple political identity or political theory." The one wrinkle in our measure is that 'libertarian' is also included a distinct category (which dates back to requests in 2014-2015 and, as mentioned above, the considerations in favour of keeping questions consistent across years are quite strong). Ideally we could definitely split this out so we have (at least) one scale, plus a distinct question which captures libertarian alignment or more fine-grained positions. But there are innumerable other questions which we'd prioritise over getting more nuanced political alignment data.

I think one consideration is that they want to make the surveys comparable year to year, and if people can select many categories, that would be make it difficult.

For adding multiple options, I think there's another sense of challenge, where if someone could select different political identities or religions, that would make the result difficult to interpret. It seems sort of "mainstream" for better or worse, that there is one category for some of the things you mentioned.

Zooming out, it seems that instead of seeing things like single/multiple as sort of a didactic/right or wrong choice or trying to impose a viewpoint,  these seem to be design decisions, that is sort of inherently imperfect in some sense, and part of some bigger vision or something.

I can confirm that we'll be releasing the results publicly and that there wasn't a 2021 EA Survey. And you can view all the EAS 2020 posts in this sequence.

Thanks for asking! This is a question requested by another org (in 2019), so I can't give you the definitive authorial intent. But we would interpret this as including virtual personal contact too (not just in-person contact).

Many thanks for checking in and sharing the survey! I can confirm that we're leaving the survey open until the end of the year now, like last year. Please also view this post about the extra questions about FTX that have been added now, and the separate survey people can take with questions about FTX, if they have already completed the survey.