A Q1 update and 2018 in review

By Baxter Bullock and Catherine Low

Since launching in 2016, Students for High-Impact Charity (SHIC), a project of Rethink Charity, has focused on educational outreach for high school students (primarily ages 16-18) through our interactive content. In January 2018, we began implementing instructor-led workshops, mostly in the Greater Vancouver area. Below, we summarize our experiences of 2018 and explain why we are choosing to suspend our outreach operations.

Summary

• 2018 saw strong uptake, but difficulty securing long-term engagement - Within a year of instructor-led workshops, we presented 106 workshops, reaching 2,580 participants at 40 (mostly high school) institutions. We experienced strong student engagement and encouraging feedback from both teachers and students. However, we struggled in getting students to opt into advanced programming, which was our behavioral proxy for further engagement.
• By the end of April, SHIC outreach will function in minimal form, requiring very little staff time - Over the next two months, our team will gradually wind down delivered workshops at schools. We plan on maintaining a website with resources and fielding inquiries through a contact form for those who are looking for information on how best to implement EA education.
• The most promising elements of SHIC may be incorporated into other high-impact projects - The SHIC curriculum could likely be repurposed for other high-impact projects within the wider Rethink Charity umbrella. For example, it could be a tool for engaging potential high-net-worth donors, or as content to provide local group leaders.
• We believe in the potential of educational outreach and hope to revisit this in the future - While we acknowledge the possibility that poor attendance at advanced workshops is indicative of general interest level in our program and/or EA in general, it's also possible that the methods we used to facilitate long term engagement were inadequate. We think that under the right circumstances, educational outreach could be more fruitful.
• SHIC will release an exhaustive evaluation of our experience with educational outreach in the coming months.

2018 in review

In late 2017 we made a strategic shift towards a high-fidelity model of student engagement through instructor-led workshops. We tested this model throughout 2018, with our instructors visiting schools in Greater Vancouver, Canada[1].

Most students (56%) participated in a single-session workshop lasting approximately 80 minutes. These workshops consisted of a giving game[2], followed by an overview of the core ideas of effective altruism[3], including coverage of key cause areas. The remaining 44% of participants participated in multi-session (typically three), in-depth workshops which usually included a giving game, interactive explorations of the topics mentioned above, a cause prioritization activity, and a discussion of effective career paths.

Our goal for the second half of 2018 was to identify high-potential students from our school visits, and engage them further with supplementary advanced workshops at a central location in Vancouver. To gauge interest initially, we began with an opt-in approach for all interested students who provided an email address in order to obtain more information. We ran a workshop in November which primarily consisted of an in-depth activity on cause prioritization, and a workshop in December focused on effectively creating online fundraisers for the holidays.

Our results

The metrics we identified to gauge our success were:

1. Teachers and school uptake
2. Student survey results indicating shifts of opinion and/or behavior
3. The number of students who continue to engage with the material beyond our initial visit, and the extent to which they remain involved.
   
   

We exceeded our expectation with metric 1 - Demand from teachers was higher than initially expected, with 25% of the schools we contacted booking at least one workshop. We also saw some success through representation at local teacher conferences. All told, in 2018 we reached a total of 2,580 students at 40 institutions, mostly in high school Social Studies and Math classes throughout Greater Vancouver[4]. Qualitative feedback from teachers and students, which includes our reads on the reactions of workshop participants, suggested that students were engaged. Teachers generally thought the content and delivery was excellent and well-suited to their curriculum area. The number of teachers who asked us to return for additional classes and/or the for the autumn semester after our initial visit serves as further evidence of continued interest in the program.

We met our expectations for metric 2 - Qualitative and quantitative data gathered from pre- and post-workshop surveys showed small but significant improvements in knowledge, attitudes, and values for the majority of our questions, suggesting that students generally shift their perspectives as a result of our program. However, there is likely a social desirability effect to consider[5]. Additionally, it was unclear to what extent these students are likely to act upon these newfound perspectives.

We attempted to gauge whether these shifts were temporary, but only 37 out of the 388 students who were emailed a 3-month survey chose to start the survey, so we were unable to draw conclusions from such a small sample.

We did not meet our expectations for the third, and most important metric - Despite the success of our first two metrics, we struggled to further engage students with opt-in advanced workshops. We feel the turnout was poor enough to suggest that this was not an effective method for SHIC to engage students beyond the initial workshop. 1396 students were eligible to be informed about our advanced workshops, and we sent out 508 invitations[6]. 17 students expressed interest, and five ended up attending our first advanced workshop in November, and two attended the December workshop (both returning from the first). There may have been logistical reasons for why the December workshop was less attended by students. The decrease in attendance could be attributed to a change in the workshop time (moved to begin on a Sunday morning) or its proximity to the holidays, or it could have been the particularly discouraging weather (cold, raining heavily).

Based on these results, we’re left with one or more of the following three conclusions:

1. The students were engaged in the SHIC program in class, but our methods for engaging students beyond the classroom were ineffective.
2. Students had the will, but not the bandwidth to engage further with the SHIC program.
3. Students were not as engaged by the SHIC program as our post-program survey data and experience suggested, and therefore uninclined to participate in further programming.

Our best guess is that all three of these conclusions are true to some extent. High school-aged students in North America tend to engage in a plethora of extracurricular activities, often leaving them with little bandwidth. We also believe that using email as our primary method of contacting students was not the optimal form of correspondence, as email tends not to be a widely used form of communication with this age group[7].

Our reasoning for suspending workshop operations

Our decision to suspend SHIC’s operations was an unusually difficult one because, despite our failure to deliver on our key metric, we still believe educational outreach in some form could be effective. For example, we could have shifted our audience to more targeted groups beyond high schools, or used other methods of engagement within high schools. It’s also possible that SHIC workshops had more of an effect than we were able to measure, such as increasing the chances that students will donate or fundraise for effective charities in the future, or be more receptive next time they encounter EA ideas.

However, as a result of not reaching our goals, we came to believe SHIC is now less marketable to new and existing donors who were also interested in long-term engagement of students. Had this not been the case, we would have more strongly considered exploring the effectiveness of our program with new audiences, such as university students and adults.

Finally, the section below outlines our belief that organizational bandwidth and resources would be more valuable if reallocated to other promising projects, such as RC Forward and the Local Effective Altruism Network (LEAN).

We are nonetheless proud of what SHIC has accomplished, and believe there is a possibility that this program made an impact on many students’ lives, despite our difficulties in tracking that impact.

Looking Forward

A SHIC website containing our materials and a contact form will remain active - While active outreach will be suspended, we think it’s important to maintain accessibility to the user-friendly content SHIC has created. We would also like to put volunteer time and minimal staff time towards consulting those with inquiries about how best to implement EA education.

A detailed evaluation of SHIC and high school outreach is forthcoming - By May 1, we plan on releasing an in-depth look at not only our experiences with high school workshops, but all programs SHIC has conducted, as well as aggregate data from other similar projects related to EA educational outreach. It is our hope that this post will provide a foundation upon which experimentation with educational outreach can continue within EA.

Repurposed SHIC resources could be very valuable - A dissolution of one of our projects allows us to reallocate some of the more promising elements of that project towards other initiatives. In SHIC, we have the resources and staff to effectively make complex philosophical and mathematical concepts accessible to the public. We believe this could prove very valuable for other projects within Rethink Charity, such as educating high net worth donors in Canada about effective giving opportunities as part of RC Forward, and providing high quality materials and guidance for group leaders who wish to run workshops on EA concepts as part of the Local EA Network (LEAN).

A potential return to educational outreach in the future - We still believe that educational outreach could be very impactful, and are pleased that others within the EA community are continuing with their outreach programs. Pending clearance of several hurdles from a strategic standpoint, this could be worth returning to in the future. Those hurdles include reconsidering the optimal audience, methods of data collection, and methods of long-term engagement.

_______________________________________________

Endnotes

1. Prior to becoming SHIC Manager, Catherine Low tested SHIC materials in the classroom as a high school teacher in 2016 and 2017. Her experiences provided evidence that an instructor-based model may be more effective than student-led models.

2. SHIC’s giving game involves students analyzing 3 to 4 charities comparatively, then deciding which charity will receive a donation.

3. In most cases, the term “effective altruism” was not used as part of the SHIC program.

4. In Vancouver high schools specifically, we reached 2,228 students in 31 institutions.

5. We included a social desirability test in our survey to attempt to measure this bias, however the results have not been factored into our survey analysis.

6. For students who participated in the full SHIC program in the Winter and Spring terms of early 2018, providing a contact email was mandatory. In Summer and Fall 2018, 18% of the students voluntarily gave us a contact email. The former comprised 62% of the emails we obtained, the latter 38%.

7. We considered other (social media focused) methods of communication that ultimately felt inappropriate.