Case-control survey of EAGx attendees finds no behavioural or attitudinal changes after six months

Prepared by James Fodor and Miles Tidmarsh
EAGxAustralia 2023 Committee

Abstract

EAGx conferences are an important component of the effective altruism community, and have proven a popular method for engaging EAs and spreading EA ideas around the world. However, to date relatively little publicly available empirical evidence has been collected regarding the long term impact of such conferences on attendees. In this observational study we aimed to assess the extent to which EAGx conferences bring about change by altering EA attitudes or behaviours. To this end, we collected survey responses from attendees of the EAGxAustralia 2023 conference both before and six months after the conference, providing a measure of changes in EA-related attitudes and behaviours over this time. As a control, we also collected responses to the same survey questions from individuals on the EA Australia mailing list who did not attend the 2023 conference. Across 20 numerical measures we collected, we did not find any statistically significant differences in the six-month changes across the two groups. Specifically, we are able to rule out effect sizes of larger than about 20% for most measures. In general, we found self-reported EA attitudes and behaviours were remarkably consistent across most individuals over this time period. We provide a discussion of these results in the context of developing better measures of the impact of EAGx conferences, and conclude with some specific recommendations for future conference organisers.

Background

‘EAGx’ is the branding used by the Centre for Effective Altruism (CEA) for centrally-supported but independently-organised conferences held around the world each year. The aim of these events is to communicate EA ideas, foster community growth and participation, and facilitate the formation of beneficial connections for EA projects. EAGx conferences have been organised in Australia every year since 2016 (with a hiatus in 2020 and 2021 due to the COVID-19pandemic), with the most recent event taking place in Melbourne in September 2023.

While EAGx conferences have proved popular with attendees, relatively little publicly available evidence has been collected regarding their impact or effectiveness. The main source of information can be found in the forum sequence by Ollie Base. Most conference retrospective reports give details about attendance and self-reported attendee satisfaction, but do not attempt to measure the impact of the conference in achieving any concrete goals. The limited range of publicly-available evaluations is surprising given the importance of these events to the EA community, and has prompted comment on the EA forum regarding the relative lack of evaluation of EA projects generally, and of EAGx conferences specifically.

For the past few years, the main form of EAGx evaluation has been a post-conference survey, along with a six-month follow-up, administered by CEA, in which attendees are asked to report the beneficial outcomes of the conference for them personally, including making new connections, starting new projects, or learning key information that informed major decisions. Of these, the number of new connections made is typically regarded as the most important, with the number of connections per dollar spent being used as a key metric by CEA in assessing effectiveness. These methods have a number of advantages, including ease of collection, ability to compare across locations and over time, and relative ease of interpretation.

A major limitation of these existing measures is that they require survey respondents to explicitly make value judgements about their experiences at the conference (e.g. ‘How many new connections did you make at this event?’ and ‘describe your most valuable experiences’). Such judgements may be liable to various biases, including an experimenter demand effect and belief-supportive bias of EAs who enjoyed the conference and are predisposed to believe such events are valuable. In addition, it is unclear to what extent such connections are maintained over time or actually result in valuable outcomes. Similarly, respondents are asked to make predictions about their future behaviour (e.g. ‘Which actions do you plan to take as a result of this event?’, and ‘what are your future plans for involvement in the EA community?’). Self-predicted future behaviour is often unreliable, and subject to cognitive biases such as optimism bias or the salience of a recent positive social interaction.

As a first step in developing additional evaluation methods, the organising committee of EAGxAustralia 2023 instigated a survey designed to measure EA-related behaviours and attitudes of conference attendees both before and six months after the conference. While still relying on self-reports, this methodology is designed to avoid explicit questions about experiences at the conference or predictions of future behaviour, instead relying entirely on reports of past behaviour and current attitudes. In addition, to provide a basis for comparison the survey also included a control group, consisting of respondents who had previous engagement with the EA community but did not attend the conference in 2023. By comparing the change in reported behaviours and attitudes between the treatment and control groups, the survey aimed to provide a measure of the impact of the conference on its attendees relative to non-attendees.

Methods

This study utilises a difference-in-differences design, in which we compare the change in self-reported attitudes and behaviours in a six month followup with the treatment group who attended the conference, to the same measurements collected in a control group of non-attendees. The control group was intended to match the treatment group in terms of overall levels of EA engagement at the initial measurement period (i.e. before the conference), while consisting of people who did not attend the EAGxAustralia 2023 conference. 

To measure EA attitudes and behaviours, a survey with a total of twenty-one questions was prepared, which were designed to encompass various types of impact the conference aimed to bring about. With the exception of question 5, all were numerical or multiple choice questions. The questions are presented and explained in the following tables.

The first two questions relate to engagement with the EA community. It was hypothesised that exposure to the conference would help attendees make friends and connections in the EA community, and also learn more about their local groups, both of which may lead to increased attendance at events. Greater attendence at EA events may be a proxy for greater engagement in EA ideas and community. The third question measured donations to EA causes, which was hypothesised to increase following the conference owing to a combination of increased knowledge of EA causes and increased motivation to donate. The fourth question asked respondents to self-report the importance of EA ideas to their life, which was hypothesised to increase following positive and affirming experiences at the EA conference. Question five was the only long answer question, asking respondents to report their current plans for improving the world. It was hypothesised that attendance at the conference would lead to changes in these plans at a higher rate than non-attendees.

Table 1. Questions 1-5 of the survey.

Questions 6-13 relate to attitudes about the relative importance of EA cause areas. The purpose was not to test for any particular type or direction of change, only to assess whether attendance at talks or networking at the EAGx conference led to higher rates of attitude change about EA causes than for non-attendees who did not have these experiences. Note that no restrictions were placed on how many cause areas each respondent identified as ‘top priority’.

Table 2. Questions 6-13 of the survey.

The final set of questions, numbered 14-21, all related to the frequency of various EA behaviours. The hypothesis was that attendance at the conference would increase motivation and opportunity for engaging in these behaviours, and so lead to an increase in reported frequency relative to non-attendees.

Table 3. Questions 14-21 of the survey.

The survey was prepared as a Google form, which was sent via email to potential respondents. The treatment group was sourced from all successful conference applicants (approximately 300), who were invited to complete in the weeks prior to the conference in September 2023. The cut-off for completion was the Friday night of the conference weekend (i.e. the day before the conference began on Saturday), ensuring that no responses were influenced by any experiences at the conference itself. The control group was sourced from the EAGx Mailchimp email list, along with posts on EA Facebook groups, personal invitation, and emails to those accepted to the EAGxAustralia 2023 conference but who did attend. These responses were collected slightly after the treatment group, during October and November of 2023.

Followup was conducted approximately six months afterwards, from March to May 2024. All respondents were sent an initial email, followed by two reminder emails at roughly weekly intervals, and finally a personalised email or Facebook message communication. These multiple followup efforts resulted in a dropout rate of around 42% for both treatment and control groups, a positive result as the target was to achieve no more than 50% dropout. Respondents were incentivised with a novel sticker (designed especially for this purpose) which was mailed after completing the followup survey, as well as a directed donation of $50 AUD to an effective charity of their choice. The monetary cost of the survey was approximately $4000 AUD, in addition to roughly 100 hours of labour spent on designing, administering, and analysing the survey data. 

The numbers of respondents in each group are summarised in the table below. Response rates for the initial survey are based on the number of attendees at the conference for the treatment group, and on an estimate of the research of email and Facebook outreach for the control group.

Table 4. Comparison of treatment and control groups.

During preprocessing of responses, it was observed that three respondents in the control group completed the followup survey twice. In these cases the second (final) set of responses was used for analysis. One participant in the control group recorded 200 friends in the initial survey and then 100 in followup. Since this variation is likely due to inaccurate estimation rather than a genuine change, we replaced both values with the followup mean in the control group of 5.2. In a few cases where respondents did not answer certain questions in the followup, we also excluded their initial responses to these questions for consistency. Finally, one respondent in the control group mentioned they only donate once per year. As such we excluded their donation question from analysis, so as to avoid a spurious appearance of change due to the change in the six month window. We also did not find any consistent or interpretable pattern in the long answer responses, and so do not discuss these any further.

Results

Conference attendees differ from non-attendees

In the data from the initial collection period (‘before’), the control group and treatment groups showed several significant differences. Those in the treatment group attended significantly more EA events (5.7 compared to 3.1, p=0.003), had more EA friends (12.9 compared to 4.9, p<0.001), and donated less money ($839 compared to $2430, p=0.002). Participants in the treatment group also reported engaging in significantly more volunteering (2.46 compared to 1.43, p<0.001), applying for grants (2.19 compared to 0.43, p<0.001), and applying for work (2.41 compared to 0.55, p<0.001). There were also small differences in the self-reported importance of EA ideas (3.27 treatment compared to 3.00 control, p=0.016), and of the priority accorded to movement-building (2.19 compared to 1.78, p=0.013). These differences likely indicate demographic discrepancies between the two groups. We consider this further in the discussion. The first set of figures given below show the difference in the ‘before’ condition between the treatment and control groups. Error bars show pooled standard error of the mean for the two groups.

Figures 1-4. Comparison of treatment and control groups (before).

Results from both the ‘before’ and ‘after’ surveys for the treatment and control groups are shown in the figures below. In all cases, error bars on the ‘after’ group show 95% confidence intervals for the difference between the ‘before’ and ‘after’ time periods.

Figures 5-12. Comparison of treatment and control groups (after).

EA attitudes and behaviours are highly stable over time

Considering now the difference-in-differences analysis, in which we compare the changes in the treatment group to the changes in the control group, the only significant result was a change in donations. In the treatment group donations showed a slight but insignificant increase of $43, while in the control group donations changed by -$1263 (i.e. a reduction in donations). Although this difference in differences of $1306 was highly significant (p=0.004), we suspect this result is spurious owing to the high positive skew of donations data. In particular, most of the effect disappears if the three largest donors in the control group are removed. Additionally, it seems implausible that non-attendees reduced their donations by half, but would have maintained their donations if they had attended. We are therefore reluctant to attribute much significance to this result.

For all other variables no significant changes are observed, with all p-values above 0.1. As shown in the figures above, this is largely due to both the treatment and control group showing very little change in the ‘after’ survey relative to the ‘before’ survey. The figures below show the difference between the ‘before’ and ‘after’ values, computed as treatment group difference minus the control group difference. Error bars show 95% confidence intervals for this difference in differences.

Figures 13-17. Difference in differences analysis.

Discussion

This six-month longitudinal survey of EAGx attendees and a control group of non-attendees found two major results. First, attendees and non-attendees differ significantly in several important aspects of EA behaviour and attitude. Second, these attitudes and behaviours are highly stable over time, and show no evidence of significant change in either group during the six months following the conference. In this section we consider these results and provide additional commentary on how they may be of value to the EA community at large.

One fairly natural explanation of the differences between the treatment and control groups is that the latter comprises an older demographic with a larger number of professionals, compared to the treatment group with a larger proportion of young students. Professionals have higher income, hence accounting for the higher donations, while also having less spare time for attending EA events and making friends. This would also explain the lower levels of engagement in volunteering, applying for grants, and applying for work in the control group compared to the treatment group. Unfortunately the survey did not include demographic questions such as age or gender, so we cannot directly test for age differences between the groups. We elected to exclude demographic questions to keep the survey as short as possible, and also because we did not have any specific hypotheses about the relevance of such variables to the effectiveness of the conference, though in retrospect simple questions about age and gender would likely have been useful. Another possibility is that some EA event or activity around the time of the conference could have influenced the views of the control group, though this seems relatively implausible.

The statistical significance and internal coherence of these initial differences between treatment and control groups yield two key lessons. First, the results provide a partial validation of the study design, showing that it is possible to detect meaningful and interpretable differences in EA behaviours and attitudes even with a fairly small sample size (n=78). This provides evidence that the survey design was capable of detecting impacts of the conference on attendees if these impacts existed. Second, the results indicate that the audience of EAGx conferences is likely only a non-representative subset of the wider EA community, with an overrepresentation of younger persons and students. This may indicate the potential for greater efforts to attract older professionals to EAGx conferences, or perhaps shows that such people are less likely to assess existing conferences as relevant to their interests.

The major finding of the present study is that, besides the (likely spurious) difference in donations, there were no significant changes in EA behaviours or attitudes following the conference in either the treatment or control groups. These results are inconsistent with the hypothesis that EAGx conferences elicit substantial and sustained changes in EA attitudes or behaviour through social encouragement, networking, or other forms of learning. They are also inconsistent with an alternative hypothesis that EAGx conferences help to maintain existing EA connections, knowledge, and motivation, which otherwise tend to diminish over time without conference attendance. In fact, it appears that EA attitudes and behaviours are very robust over time, with no significant changes observed after a six month follow up.

As noted, the only exception was the decline in donations in the control group. We suspect this was largely driven by the wording of the question, which asked only about donations during the past six months. As the initial survey was completed around September while the followup was completed around March, this meant that any respondents who donate annually near the end of the financial year (end of June in Australia) would be recorded as making large donations in the initial survey and no donations in the followup. One respondent explicitly noted this in their extended response, and we suspect this was also relevant for two other large donors who reported zero donations in the followup survey (and perhaps to other donors). Removing these observations does not eliminate the donation difference entirely, but reduces it to only $341 (p=0.09). As such, we regard this result as owing to the survey design rather than any genuine tendency for those who did not attend the conference to decrease their donations over time. Future surveys should avoid this limitation by rewording the question, for example by explicitly asking for annual donors to answer with half their annual donation.

One obvious limitation of the present study is that the followup results for the treatment group only represent about 15% of conference attendees, and we have no data on how representative respondents are relative to all attendees. Owing to the time and effort required to fill in the survey, it is likely that respondents were more highly engaged with EA than non-respondents, though it is unclear how this would be expected to affect interpretation of our results. On the one hand, less-engaged attendees may have had larger changes in attitudes or behaviours from attending the conference owing to starting at a lower baseline of engagement. On the other hand, less-engaged attendees may have been less strongly affected by the conference compared to those more engaged in the community. 

If true EA engagement is highly-volatile then treatment respondents would be on average more engaged in the initial survey than the follow-up (causing a decline in engagement). However this would apply more strongly to the control group (as the response rate was much lower), making the EAGx appear more effective than it really was. This would also invalidate the survey’s finding that engagement appears stable. 

The results of this survey are in tension with anecdotal reports and CEA analyses, both of which indicate that EAGx conferences have sizable effects on attendees. Partly this reflects differences in methodology - instead of directly asking respondents to report how the conference impacted them, we sought to measure such effects indirectly by comparing the differences in self-reported measures of EA attitudes and behaviours before and after conference attendance. Given that most attendees enjoy EAGx conferences and are likely to believe they have a positive effect on attendees, directly asking attendees is likely to overestimate impact. Conversely, our method is likely to underestimate impact, as by design it only measures impact using pre-specified narrowly defined metrics, which are unlikely to capture the full range of effects that conferences have on attendees (though it would be surprising if all measured questions show null effects while most unmeasured effects were positive and substantial). As such, we believe that both methods should be used in tandem to provide a more holistic and richer picture of the impact of EAGx conferences.

Another major challenge in evaluating the effectiveness of EAGx conferences is that a large portion of the impact may be concentrated in a small subset of attendees. It is plausible that the majority of conference attendees experience few long term effects, while a small minority are led to major life or career changes as a result of connections made or motivation obtained at the conference. If, for example, the latter group comprised only 10% of conference attendees, the 14% response rate (for both initial and final surveys) obtained in this study would include only a handful of such attendees, making it difficult to measure any statistically significant results. At the same time, we think it is important that such hypothetical possibilities are not allowed to detract from the fact that our results do not provide evidence to support a significant effect of conference attendance on EA attitudes or behaviours for most attendees.

Recommendations

Based on our results, we make the following recommendations to people organising and evaluating large EA conferences:

1. Seek opportunities for new methods of measurement and evaluation, such as using a similar longitudinal survey design with treatment and control groups.
2. Consider spending more time and budget on evaluation. The total cost of roughly $10,000AUD to run this evaluation represents a small fraction of the budget of most EAGx conferences. Given that CEA spends several million dollars on such conferences every year, it would be useful if there was reporting on the proportion invested in evaluation and the expected value of information. Many sources (see here for example) report a guideline of about 10% of total program costs spent on evaluation.
3. Consider independent evaluation. In addition to evaluation by CEA and conference organisers, it may be valuable to commission a third party to design and/or carry out an evaluation of the effectiveness of EAGx conferences. The EA Community Survey could perhaps serve as a model for this.
4. Consider additional efforts to incentive survey completion. One method would be to incorporate the initial survey into the conference registration process, and then provide a partial rebate to those who complete the followup. This would introduce additional administrative overhead, but could substantially increase the response rate.
5. Focus on identifying and targeting those most likely to experience large benefits from attending the conference. Better methods of identifying and directing resources to this small subset of attendees may deliver disproportionate impact.
6. Develop ways to combine types of measures. For instance, information could be collected regarding self-assessed sources of impact or plans made soon after the conference, and then a followup survey could evaluate whether these plans are carried out, or whether the same aspects of the conference are regarded as impactful.
7. Consider the use of randomisation. The EAGxAustralia conference is too small to randomly allocate tickets, but this is not necessarily true of all EAGx or EAG conferences. A simple design could involve a sample of around sixty potential attendees at the borderline of being accepted to the conference, which is then randomly divided into two, with half attending and the other half not attending the conference. Both groups would be asked to complete a six-month followup and could be paid for their participation as an incentive. This method would ameliorate most of the methodological limitations of the present study.