Thanks!

Roughly speaking, there seem to be two main benefits and two main costs to making an anonymised dataset public. The main costs: i) time and ii) people being turned off of the EA Survey due to believing that their data will be available and identifiable. The main benefits: iii) the community being able to access information (which isn't included in our public reports) and iv) transparency and validation from people being able to replicate our results.

Unfortunately, the dataset is so heavily anonymised in order to try to reduce cost (ii) (while simultaneously increasing cost (i)), that it seems impossible for people to replicate many of our analyses (even with the public dataset), because the data is so heavily obscured, essentially vitiating (iv). We have considered, and are considering, other options like producing a simulated dataset for future surveys in order to allow people to complete their own analyses, if there were sufficient demand, but this would come at an even higher time cost. Conversely, it seems benefit (iii) can be attained, in the main, without releasing a public dataset, just by producing additional aggregate analyses on request (where possible).

Of course, we'll see how this system works this year and may revisit it in the future.

Presumably part of the idea is that it is somewhat incentivising while also being very cheap: the money goes to places CEA would like to support anyway, and doesn't really motivate non-EAs to take the survey.

A different concern is it is not clear to me how counterfactually valid the donation is.

The two questions were subtly different for other reasons. If I understand it correctly, the first question asks for positive influences on your impact, and the second question for negative influences on your EA involvement. 

So for example, being persuaded that less involvement in EA -> increasing your impact will lead you to select the same response for both.

Thanks for your question. The 2020 posts have already started coming out. You can see the first two here and here. And you can also find them on our website here. All the posts will also be under the Effective Altruism Survey tag.

Thanks!

If you are referring to the question I think you're referring to, then we really do mean that people should select up to one option in each column: one column for whichever of the options (if any) was the source of the most important thing you learned and one column for whichever of the options (if any) was the source of the most important new connection you made.

Hi Dale. Thanks for your comment.

The gender question and many of the other demographic questions were selected largely to ensure comparability with other surveys run by CEA.

That aside, I think your claim that open comment gender questions are "considered poor survey technique" is over-stated. The literature discusses pros and cons to both formats. From this recent article in the International Journal of Social Research Methodology:

One of the simplest ways to collect data on gender identity is to use an open text box (see Figure 2) which allows participants the freedom to describe their gender in whatever way they see fit while accommodating changing norms around acceptable terminology. Terms commonly used around gender evolve over time... It would therefore be misguided of researchers to attempt to find the most contemporary terminology and use it to the exclusion of all other terms. Research teams are also likely to find such a process difficult and frustrating (Herman et al., 2012). Thus, an open text box is certainly the most accommodating approach to a range of evolving terms to describe gender identity.

If open text boxes are used for research that intends to analyze by category, however, researchers will still ultimately be categorizing the gender identities in order to define groups for statistical analysis and groups to which the findings might be generalized... These decisions will also need to be made if researchers using a multiple-choice approach choose to provide a long list of as many gender identity terms as possible. This approach is a fine option, but researchers need to be cognisant that terminology that was in common use when a tool was published may no longer be current when research is conducted using that tool... Good arguments can be made for the value of participants being able to see the specific term for their gender identity among a list of possibilities, but even Herman’s and Kuper’s lists, published within the past decade, contain terms that are increasingly considered problematic and do not contain some terms that are more common today.

An approach which provides a smaller number of options for gender identity has benefits and drawbacks. Providing fewer categories inevitably forces gender minority participants to place themselves into categories that the researcher provides, but gives the advantage that the participant, not researcher, chooses the categories in which they will be included.