Thanks!

Roughly speaking, there seem to be two main benefits and two main costs to making an anonymised dataset public. The main costs: i) time and ii) people being turned off of the EA Survey due to believing that their data will be available and identifiable. The main benefits: iii) the community being able to access information (which isn't included in our public reports) and iv) transparency and validation from people being able to replicate our results.

Unfortunately, the dataset is so heavily anonymised in order to try to reduce cost (ii) (while simultaneously increasing cost (i)), that it seems impossible for people to replicate many of our analyses (even with the public dataset), because the data is so heavily obscured, essentially vitiating (iv). We have considered, and are considering, other options like producing a simulated dataset for future surveys in order to allow people to complete their own analyses, if there were sufficient demand, but this would come at an even higher time cost. Conversely, it seems benefit (iii) can be attained, in the main, without releasing a public dataset, just by producing additional aggregate analyses on request (where possible).

Of course, we'll see how this system works this year and may revisit it in the future.

To add to that, if there are concerns about data being de-anonymized, there are statistical techniques to mitigate it.

unless I think that I'm at least as well informed than the average respondent about where this money should go

This applies if your ethics are very aligned with the average respondent, but if not, it is a decent incentive. I'd be surprised if almost all of EAs' disagreement on cause prioritization were strictly empirical.

Presumably part of the idea is that it is somewhat incentivising while also being very cheap: the money goes to places CEA would like to support anyway, and doesn't really motivate non-EAs to take the survey.

A different concern is it is not clear to me how counterfactually valid the donation is.

I agree this won't be an incentive to many EAs. So long as it serves as an incentive to some respondents, it still seems likely to be net positive though. (Of course, it's theoretically possible that offering the prize might crowd out altruistic motivations (1) (2) (3), but we don't have an easy way to test this and my intuition is that the overall effect would still be net positive).

I would hope that concerns about being less well placed to make the donation would not incentivise people to not take the EA Survey, just so that they don't risk winning the prize and making a sub-optimal donation. If the respondent doesn't feel comfortable just delegating the decision elsewhere, they could always decline the prize, in which case it could be given to another randomly selected respondent.

Thanks Alex! Yeh, due to the space constraints you mention, we're planning to run some questions (which mostly stay very similar across multiple years) only every other year. The same thing happened to politics and diet.

This is, of course, not ideal, since it means that we can't include these variables in our other models or examine, for example, differences in satisfaction with EA among people with different religious stances or politics, every other survey.

Thanks for explicitly mentioning that you found these variables useful. That should help inform discussion in future years about what questions to include.

Hi Denise,

Thanks for taking the survey!

The questions about positive/negative influences were CEA requests (although we did discuss them together): I believe the rationale is that for positive influences, they were interested in the most important influences (and wanted to set a higher bar by preventing people indicating that more than three things had the “largest” influence on them), whereas for the possible negative influences, they were interested in anything which had a negative influence, not merely the largest negative influences.

Regarding donations: historically, we have always asked about the previous year’s income and donations (because these are known quantities) and then planned donations for the year the survey is run (since people likely won’t know this for sure, but it’s still useful to know, for broader context. Now that we launch the survey right at the end of the year, the difference between past and planned donations is likely less acute. Naturally, it would be ideal if we could ask for income and donation data for both the previous year and the present year, but we constantly face pressure to include other questions, while trying to maintain survey length, so we have had to leave out a lot of things. (This also explains why we had to cut the questions we had in previous years asking about ‘individual’ and ‘household’ figures, given that many people’s earnings/donations are part of a unit).

The two questions were subtly different for other reasons. If I understand it correctly, the first question asks for positive influences on your impact, and the second question for negative influences on your EA involvement. 

So for example, being persuaded that less involvement in EA -> increasing your impact will lead you to select the same response for both.

Thanks for your question. The 2020 posts have already started coming out. You can see the first two here and here. And you can also find them on our website here. All the posts will also be under the Effective Altruism Survey tag.

Thanks!

If you are referring to the question I think you're referring to, then we really do mean that people should select up to one option in each column: one column for whichever of the options (if any) was the source of the most important thing you learned and one column for whichever of the options (if any) was the source of the most important new connection you made.

Hi Dale. Thanks for your comment.

The gender question and many of the other demographic questions were selected largely to ensure comparability with other surveys run by CEA.

That aside, I think your claim that open comment gender questions are "considered poor survey technique" is over-stated. The literature discusses pros and cons to both formats. From this recent article in the International Journal of Social Research Methodology:

One of the simplest ways to collect data on gender identity is to use an open text box (see Figure 2) which allows participants the freedom to describe their gender in whatever way they see fit while accommodating changing norms around acceptable terminology. Terms commonly used around gender evolve over time... It would therefore be misguided of researchers to attempt to find the most contemporary terminology and use it to the exclusion of all other terms. Research teams are also likely to find such a process difficult and frustrating (Herman et al., 2012). Thus, an open text box is certainly the most accommodating approach to a range of evolving terms to describe gender identity.

If open text boxes are used for research that intends to analyze by category, however, researchers will still ultimately be categorizing the gender identities in order to define groups for statistical analysis and groups to which the findings might be generalized... These decisions will also need to be made if researchers using a multiple-choice approach choose to provide a long list of as many gender identity terms as possible. This approach is a fine option, but researchers need to be cognisant that terminology that was in common use when a tool was published may no longer be current when research is conducted using that tool... Good arguments can be made for the value of participants being able to see the specific term for their gender identity among a list of possibilities, but even Herman’s and Kuper’s lists, published within the past decade, contain terms that are increasingly considered problematic and do not contain some terms that are more common today.

An approach which provides a smaller number of options for gender identity has benefits and drawbacks. Providing fewer categories inevitably forces gender minority participants to place themselves into categories that the researcher provides, but gives the advantage that the participant, not researcher, chooses the categories in which they will be included.