In March the members and friends of the Czech Association of Effective Altruism (CZEA) met for a weekend long intense retreat (you can read more about CZEA in this post). We would like to share our experience in case any EA group was interested in doing something similar.
We also believe that the difference between running a good retreat and an average retreat can be as big as between running an average retreat and no retreat at all, so it’s worth to invest the effort in developing the format.
Table of contents
GoalsGoals
Our goals for the retreat were, in order of importance
- Community building and networking. Local EAs should leave knowing each other better
- Introducing CZEA activities and engaging more people in our projects
- Analyzing the weekend impact and sharing our tips and insight
- Education in more advanced EA topics
Based on extensive questionnaires (EA people are willing to fill them up even if they are long), the event seems to have had impact on the goals.
- The average number of EAs participants know well grew from 3.7 to 9.6 (operationalized as “being able to describe in three sentences”)
- Average self-reported knowledge of CZEA grew from 4.6 to 6.9 on a scale 1 to 10 (perfect).
Expected number of hours participants plan to spend on EA activities grew from 11.8 to 13.8 per week. If part of the effect persists, it means the event had some leverage. - Average self-reported knowledge of effective altruism grew from 5.1 to 6.5 on a scale 1 to 10.
- We will consider the goal “Analyzing the weekend impact and sharing our tips and insight” fulfilled if someone in EA community running a weekend retreat reads this and actually utilizes some of the info shared, so if you do, please let us know! It is really important for our internal prioritization.
Format
The event ran from Friday evening to Sunday noon.
There were 25 participants in total (and 8 more cancelled applications), half of them already active in CZEA. Others were already familiar with EA basics.
Design principles
Establish epistemic standards. It helps to establish epistemic standard early. By epistemic standard we mean rational reasoning, cooperative solving of disagreements, asking about unclear things, and independent evaluation of evidence and arguments.
Mixing. The natural order of things in social groups tends to be assortative matching. In the case of an EA retreat, it may mean the more senior members talking more among themselves, and the new members also. This should be prevented. As a lot of things are best learned by osmosis and implicitly, it makes a lot of sense to arrange the activities in such a way that there are opportunities for this to happen.
Dramatic arc. People enjoy an event more if it fits human-compatible narrative structure. For example, if you imagine the intellectual intensity or the emotional intensity as the function of time, it can build up, have some climax, fall down, have some much small peak just before the end.
Rhythm. Hard to describe formally, but if activities are structured in a good way, people get closer to the state of flow, are less tired, learn better. Think about the state of mind you want the participants to be after an activity, and if the next activity fits.
Establish common topics and knowledge. Unstructured talk in coffee breaks is often the best part of scientific conferences, but the talks are indispensable for establishing common topics and knowledge about topics.
Do not be afraid of intensity.
Avoid classroom look&feel Classrooms tend to induce some unfortunate mind-states. (At least since 14th-century)
Program
Guided by the goals, we brainstormed an overabundance of activities. Guided by the design principles, we created a program. We re-checked if the program seemed to be aligned with the goals. (About ½ of the initial ideas did not get into the final program)
Here we list all the activities, our reasoning for their inclusion, their rating by the participants on 4 scales - total utility, fun, learning new ideas, and networking (U,F,L,N). Sometimes we comment on experience gained after the event from feedback form or our impressions. The graphs adjacent to the topics show rating on a scale from 1 (bad) to 5 (good), to emphasize the difference the scale of the plots is from 1.5 to 4.8.
We hope it will be inspirational, but definitely should be changed based on goals of your event, the audience, and unique opportunities arising from the people present. We hope the feedback scores could work as a rough guide how an activity may help a specific goal.
Friday
Dinner (18:20-19:20)
We paid for a high quality all vegan food. Also snacks and tea were available during the whole event. Investing in good quality catering seemed worth it. Good light food and constant availability of drinks allow people to keep concentrated. Snack & tea bar is a natural Schelling point for meeting people.
Opening Talk (19:30-19:50)
Operations info. Also useful to introduce rules like “Pacman” (a.k.a. “Open your circles”), “To be continued”, “This is not a classroom”
Double Crux(19:50-20:50)
Double Crux is a CFAR rationality technique for resolving disagreements. There were a few CFAR alumni attending the retreat who were able to teach it.
Reasons for inclusion: Establish epistemic standards early. Technique useful for internal CZEA communication.
Reflection: We believe that knowledge of this technique improved the quality of conversation during the whole weekend.
Organizational Structure of the Czech Association for EA (20:50-21:20)
Reasons for inclusion: Directly helps the goal of people understanding CZEA structure.
Reflection: Part of the info was unnecessarily duplicated in other talks.
Exercise in Giving Feedback (21:30-22:20)
Giving feedback was mainly interactive discussion moderated by two CZEA members who are HR professionals.
Reasons for inclusion: Establish epistemic standards. Induce people to give feedback. Making the program more interactive.
Tea Time (22:20-)
Saturday
Breakfast (9:00-9:30)
Explaining Concepts (9:30-10:40)
Reasons for inclusion: Networking, mixing, raising the level of common knowledge.
The participants were asked to pick from a list of interesting concepts (e.g. tragedy of commons, toxoplasma of rage, full list here) which ones they would like to have explained to them or explain to others. We matched small groups and had three runs of fifteen minutes long peer to peer explaining.
Experience: From the feedback this was the most popular activity, scoring very high in all the criteria, contributing to all goals.
Talk on Development of the EA Movement in the Last Five Years (10:40-11:40)
Reasons for inclusion: Contributing toward the goal of raising knowledge about more advanced EA topics.
Experience: In practice, it was a useful opportunity to clear some misconceptions spread in early years of effective altruism, like “it’s about earning to give”, “it’s mostly about pledges”, “it’s mostly about fundraising to GiveWell charities”
How to Use Slack, Trello, Gdrive etc. (11:45-12:15)
Reasons for inclusion: Rhythm, something not so demanding. Also at CZEA we use a whole stack of collaborative tools and we wanted everyone to be able to use them effectively.
Experience: For power users it was boring.
Lunch (12:20-13:20)
Trio Walks (13:30-14:50)
Practicing Double Crux and Mutual Debugging in Trios. Inspired by CFAR.
Reasons for inclusion: Networking, mixing, improving rationality. Make people walk. May be socially more demanding.
Experience: The feedback scores have bimodal distribution - for some trios it was one of the highlights of the weekend, for some it did not work at all.
Presentation of Czech and Slovak EA Projects (15:00-16:00)
Reasons for inclusion: Networking, getting everybody updated, explaining what we do for new members.
Experience: Would be better to join this with “Job fair” which we had on Sunday.
Group Photo (16:00-16:20)
Annual General Assembly of the Czech Association for EA (16:20-18:00)
Reasons for inclusion: Legal. We have to do it once a year.
Experience: Boring as expected.
Dinner (18:10-19:50)
EA Themed Pub Quiz (19:30-20:30)
Reasons for inclusion: Fun. Have something not-so-serious. Networking.
Common Mistakes of EA Students (20:30-21:00)
Talk by Jiří and Anna on topics such as “How to not become a depressive altruist”. Topics covered seems important for the wellbeing of many EAs (like “how not to become completely obsessed with x-risk”)
Reasons for inclusion: Learning from mistakes. The reason why to do it later in the course of the weekend rather than earlier: it costs fewer weirdness points.
Experience: It was quite funny, people love to learn from mistakes of others.
Split evening block of Career Planning or EA Future Plans Brainstorming or CFAR Techniques Training (21:00-)
In this bloc, we split the group to three parts, based on different topics resonating in the group - career planning, brainstorming future plans for Czech effective altruism, and training of CFAR style applied rationality. Detailed descriptions would be too long, but generally, all the options were attractive.
Sunday
Breakfast (9:00-9:30)
On Heuristics (9:40-10:00)
Talk by Aleš. Explaining some more advanced EA topics.
Reasons for inclusion: Directly aimed at one of the goals.
Interactive Theatre: How to Explain and Talk About EA (10:05-10:45)
Reasons for inclusion: Networking, mixing, fun. Development of social skills and ability to explain EA concepts.
Activity based on Forum theatre form. One person acts the role of someone having some serious misconception about EA, the other as an effective altruist trying to explain EA ideas. Members of the audience can suggest different actions for the actors, or come to “stage” and perform their actions.
Job Fair (10:50-11:50)
“Job fair” for volunteering in projects of CZEA
Reasons for inclusion: This seemed the right point where people should make actionable plans for the future.
Experience: This would have been better joined with project introductions.
Lunch
It makes sense to make this lunch break longer, as a space for informal discussion.
Lightning Talks by participants
Reasons for inclusion: Energizing activity, small peak before the end. Allows remembering people. Allows adding what was unintentionally omitted.
Closing remarks
Experience: Some people travelled from far locations and were leaving during Sunday and didn’t stay for the retreat closing.
Feedback (30min)
Filling forms online, bring your own laptop. Super-important for analyzing the weekend impact and improving the next run.
Reasons for inclusion: It is hard to make people fill a >30min form at home.
Promotion: How people learned about the event
Almost all participants learned about the event through one of these “via Facebook event”, “via CZEA”, “through friends”.
Decisive reasons for actually going, as reported by participants, were:
- Wanted to get to know the Czech EAs better (9x)
- Participation on the program of the event (4x)
- Good format, program (4x)
- Veg food, location (2x)
- Reasonable price
Logistics, costs and various tips and ideas
Total expenses were around €75 per person.
Estimated time spent on planning, activity development and feedback analysis was about 200h, or about 1.5 months FTE.
Vegan food and good tea were appreciated (both rated 9/10).
A gong is a useful tool for organizers.
A ‘networking spreadsheet’ where people could write something about themselves before the retreat was a good thing to have.
Participants liked “the pacman rule”: When you are standing in a group, try to leave a place for one more person so other people can join in more easily. (rated 4.4/5) “To be continued”, “This is not a classroom” , “Interesting discussion > talk” were somewhat positive (around 3.2/5)
It helped us to have clearly defined goals.
Our mistakes and things to do better
We didn’t prepare an icebreaking activity.
We had only one lecture room with no place to go for people who preferred discussion instead of listening to a lecture.
The level of talks and discussion was sometimes chosen to fit the least informed person. We could include more advanced topics. (The plot show answers to question Was the program easy or difficult? 1.. very easy 10… very difficult)
We didn't account for the "fun peak" at Saturday evening, leading to somewhat unexpected mood.
People were leaving during Sunday and didn’t stay for the retreat closing.
We didn’t prepare how to immediately involve some of the newly motivated participants after the retreat ended. Some may have lost interest (we will investigate it further).
Impact evaluation and feedback
The responses were generally positive. At the end of the retreat, participants reported a better understanding of the local organization and its projects and were determined to spend more time on EA-related activities. Some of them already joined our ongoing projects (e.g. planning of a Prague AI Safety conference) and we see more engagement in our community. Also, communities in Brno and Bratislava have become more active after their members attended the retreat.
We will try to track medium-term impact by a follow-up survey after 6 months.
To give voice to participants directly, these were some of the answers from feedback forms :)
What is the most important thing about which I have changed my mind?
- the nature of effective altruists
- EA Bratislava
- I believe in my knowledge even less
- I will explore more
- I think about people, and I think a lot
- Career planning
- I appreciate the usefulness of self-development and CFAR techniques
- I thought I was just an observer, and in the end I was determined to get involved quite a lot.
- I'm wondering if EA charity contributions are the best I can do for EA.
- Earning to give
- Content - I found it important to discuss the priorities for a long time and actively seek to find facts that would help me change my mind - eg via double-crux.
- How to advise others with your career
- Perhaps exploration / exploitation.
- The area of AI safety is actually worth attention
What surprised me the most?
- The number of actively involved people
- How great was the food
- Environment, food
- Unexpectedly non-autistic
- That I felt good among unknown people.
- EA Theater
- Intensity of the program
Which parts of the program were least useful...?
Most common answers included “organizational info! and talk about Trello etc.
Credits
The main organizers of the retreat were Jiří Nádvorník and Anna Gajdová, with help from Kristýna Němcová and Jan Kulveit. Talks and activities also by Aleš Flídr, Naďa Bednárová, Přemek Paška, Marika Řežábková, Veronika Portešová and Lenka Raymanová. The event took place at Ekocentrum Louti, catering was by Momos and Amitaya Tea.
We took a lot of inspiration and some content from CFAR.
We also took inspiration from the methodology of PSL and Velky Vuz, Czech organizations developing Experiential education techniques.
Future plans
We plan to do a weekend retreat at least once a year for CZEA.
If you would like to organize something similar we would be happy to help you, describe any of the activities in more detail, share materials, etc.
We also have a few activities we really wanted to try but were not able to fit in the schedule
- Play cooperative board games about saving the world (e.g. Mansions of Madness)
- An AI Safety themed LARP
(Note: Feedback data analysis was done and this post was written cooperatively by Anna Gajdová, Jiří Nádvorník and J.K.)
I and some fellow university group (UW–Madison, U Chicago, Northwestern, Michigan) leaders in the USA's midwest are in process of organizing a career retreat next semester. We are using this post!
EA Bristol (UK) totally stole most of what you wrote in here so I would say that's another success for you!
Hey Jan, this is really great - thanks for writing it up! Do you have a doc you could share with the pub quiz questions you used?
Thanks for this, great info and presentation and a very well planned event! That said, I'm in general rather skeptical of the impact such events have on anything but the fun of the participants :) I don't have any empirical data to back this claim (so I might as well be completely wrong), but I have an impression that while such events help like-minded people to get to know each other, in terms of an actual, long-term impact on the goals of EA they don't do much. And here is why: those who're enthusiastic about EA and/or willing to contribute in a certain way will do so anyway. For them online information, or a single talk may even be enough. And the other way around: those who aren't much into it will rarely become so via such an event.
I am aware that this may be quite an unpopular view, but I think it would be great to have some empirical evidence to show if it's really wrong.
My guess is that events organized for an effective knowledge-building in the given domain (including concrete skills required for a very concrete tasks in the given community, some of which were a part of your event) would be those that would make more of a difference. Say, an EA community realizes they lack the knowledge of gathering empirical data or the knowledge of spreading their ideas and attracting new members. In that case, one could invite experts on these issues to provide concrete intensive crash-courses, equipping the given community so that it can afterwards put these skills to action. This means a hard-working event, without much extra-entertainment activities, but with a high knowledge gain. I think networking and getting to know others is nice, but not as essential as the know-how and the willingness to apply it (which may then spontaneously result in a well networked community).
(Edit: I once again checked the primary goal of your event and indeed, if you want to provide a space for people to get to know one another, this kind of retreat certainly makes a lot of sense. So maybe my worries were misplaced given this goal, since I rather had in mind the goal of expanding the EA community and attracting new members).
Hi Dunja,
Actually there is empirical research on this! LEAN interviewed EA group organisers as part of the 2017 LEAN Impact Assessment, and actually face to face, in person experiences such as retreats and EAG frequently came up as the most significant, landmark influence for a fair few successful organisers in actually kick starting them into getting something going, and also giving them the confidence, reassurance and optimism to see it as a worthwhile investment of their time.
I was slightly surprised by how much of a big factor this was for people, but the evidence seems pretty strongly supportive at this stage.
https://rtcharity.org/2017-lean-impact-assessment-qualitative-findings/
Hi Richenda, great stuff, thanks for sharing the link! that's indeed a big impact and it's valuable to know for future events. It fits very well with what Evan and Jan have written below :)
I partially agree with you but I'll focus on what I disagree with :)
Personally, hanging out with EAs makes me A LOT more enthusiastic about EA and I work on my EA projects much more as a result. I basically forget about EA when I’m away from the community for long periods of time. I might be an outlier here but I’m sure that the same is true for others to a lesser degree. And it’s these kind of events that not only energise me but also help me find EA friends with whom I can hang out, co-work or even live. Which, by the way, makes such events more valuable when they are for people from one city.
Also, I know from first-hand experience that online information is not enough for cause prioritisation, making career decisions or deciding where to donate. I read a lot but when I started going to EA meetups some gaps in my knowledge and flaws in my thinking were soon exposed. Discussions hit diminishing returns after a while though.
But maybe both goals can be achieved with simple socials at a lesser cost.
OK, I have to admit these are really good points :) I don't work in any direct way in the EA sector, but I can imagine that just like with any job, communication with others can kick-start new enthusiasm and even new projects.
Effective altruism is kind of emotionally hard. Compared to some other forms of altruism It often does not give you the "warm glow" feelings, smiling child's eyes, etc.
This emotional feedback has to be supplied by the community. If it is missing, it has some unfortunate effects
This was true for me until hanging out with EAs worked so well I became an EA community organizer, and now I can't stop thinking about EA, because I'm always in touch with the community and am trying to constantly facilitate EA events.
Are you skeptical of the impact of these retreat-style events, or large congregations of effective altruists in general? Conferences like EAG faced similar skepticism of their impact early on. While there is a sense of hype at EAG events, especially if they're primarily attended by fresher community members, the expected value of EAG events is high. The value per person is high-variance, and the median impact per attendee may be low, but the mean impact of EAG per attendee is high. This is because for those high-impact individuals who weren't connected to EA before attending an EAG conference who then got highly involved, they transform the movement. For example, I know several EAs who within a few months of attending EAG, and hardly having heard of EA before then, became key staffers at EA organizations. I expect several EA organizations could attest the same. Given surveys of EA organizations place the value of identifying and hiring the best candidate for a new position as equivalent to >$100k USD; each EAG event likely results in multiple such outcomes; and EAG events don't themselves cost that much, at the least I expect every EAG conference breaks even in expected value. That's not an impressive outcome, but the networking and coordination value EAGs add to the EA movement are a crucial institution. We haven't found an alternative process which satisfies the same goals as well. We can't predict very well in advance exactly which individuals will generate the greatest value by attending EAG events, but that from the pool of people who attend EAG tons of value is undeniably generated justifies the costs of organizing and hosting these events.
Of course EAG events are professional conferences the Centre for Effective Altruism and others optimize for maximizing impact as a result of networking between EAs. Local/regional EA retreats are a different kind of event: more casual, and less goal-oriented. I don't think local EA groups without the assistance of professional EA organizations often have the capacity to organize an event as big and impactful as an EAG conference.
So that leaves the question of the expected value of retreats, i.e., whether its worth local EA communities trying to host them. The direct financial costs (assuming the 75 Euro/person/weekend price is typical, that isn't high) aren't high, as the time spent organizing these events probably primarily comes from volunteers. I figure the primary cost of organizing EA retreats would be the opportunity cost. That is, could the time spent organizing a local EA retreat by all EAs involved be better spent on something else?
While I think local EA groups could identify better uses of their time, that's not the same as saying they will identify better uses of their time. If that's free time that'd otherwise go unused, I expect organizing retreats is worthwhile.
I hope these thoughts help frame thinking about evaluating the EV of EA retreats.
As an aside, I've wanted to do more things like this with EAs in Canada, but there hasn't been a good opportunity to do so. A highly focused, goal-oriented retreat is something I hadn't thought of trying, but might be a great idea. Thanks for the inspiration.
Thanks for this, Evan, I was primarily referring to smaller events which aren't primarily targeted at attracting new people. Though now that you mention it, I find the bigger events even worse haha! I was at one bigger EA event and while I perfectly understand it can introduce many people into the topic, and make people passionate about the cause, I haven't experienced the same mainly because I haven't learned much really. But this probably depends on personality traits, expectations etc. :) In general, your argument makes very much sense: if sufficiently many people are around, for some of them this will work (and the above post by Richenda shows there is even some empirical evidence for that). For me, forums like this are e.g. way more interesting ;) At the end of the day, it's probably the best if there is a variety of venues/platforms for different kind of people and interests.
Actually I would love to have some RCT on the effects of retreats on EA groups :)
We don't have that, so we have to go by models, guesstimates, anecdotal personal experience, and expert opinion.
As the topic seems broad, I'll just state some points where our models possibly differ and we can try to find out where the crux is
I.
II. "events organized for an effective knowledge-building in the given domain"
There is some relevant social research on it: https://rtcharity.org/2017-lean-impact-assessment-qualitative-findings/
RCTs, in my view would be unsuited to measuring anything actually useful about groups, however tempting the idea is. There are so many variables muddying the water for such assessment that you would end up just fabricating without realising.
Thanks for writing this up,
For your impact review this seems likely to have some impact on the program of future years EA: Cambridge retreats. (In particular it seems likely we will include a version of the 'Explaining Concepts' activity, which we would not have done otherwise, as well as being an additional point in favour of CFAR stuff, and another call to think carefully about the space/mood we create).
I am also interested in the breakdown of how you spend the 200h planning time since i would estimate the EA: Cam retreat (which had around 45 attendees, and typically had 2 talks on at the same time) took me <100h (probably <2 weeks FTE). Part of this is likely efficiency gains since I worked on it alone, and I expect a large factor to be I put much much less effort into the program (<10 hours seems very likely).
Thanks for the feedback.
Judging from this and some private feedback I think it would actually make sense to create some kind of database of activities, containing not only descriptions, but info like how intellectually/emotionally/knowledge demanding it is, what materials you need, what are prerequisites, the best practices... and ideally also a data about the presentations and feedbacks.
My rough estimate of time costs is 20h general team meetups, 10h syncing between the team and CZEA board, 70h individual time spent planning and preparations, 50h activity developement, 50h survey design, playing with data, writing this, etc. It guess in your case you are not counting the time cost of the people giving the talks preparing them?
Some reservations I would have about the usefulness of a database vs lots of write-ups 'in context' like these is that I think how well activities work can depend heavily on the wider structure and atmosphere of the retreat, as well as the events that have come before. I would probably be happier with a classification of 2 or 3 different types of retreat, and the activities that seem to work best in each. (However we should not let perfect be the enemy of good here, and there is probably a number of things that work well across different retreat styles).
Your time costs seem largely similar to mine then (on the things we both did), I had not anticipated the large amount of time you spent on survey design etc. I don't think my time cost would change much if I included the talk prep, since I would be surprise if it totaled >10 hours.
Thanks for writing this up, lots of interesting ideas for retreat activities which I hadn't previously seen/ though of!
This sounds interesting as a model of both community building and fostering collective action. I wonder if there's a MED (minimally effective dose) that can happen in-town, rather than a retreat. I can imagine having a hard time getting commitment for people in NYC (where I'm based) to do this, but perhaps we could do a minimally effective version in 6-8 hours in town. Anyone tried something similar but shorter?
EA London is planning something like this next month, but over a weekend, so people will meet up on both days at the same venue in London.
I'm excited by the creative modules you ran during the retreat. I'm not surprised as the sort of lateral thinking which lends itself to thinking of common solutions to common problems in EA is something I'd expect after the CFAR training a lot of you went through. Some of the talks you gave seem unique among ones developed by EAs, so if you have a video recording or transcript of those individual talks available, I'd encourage you to post them on the EA Forum as well.
We do not have recordings, and all talks were given in czech language so even if we would have recordings they would not be very useful ;-).
I'd be very curious about what the HR professionals thought about giving feedback.
A little bit more info about explaining concepts activity.
I believe you meant to link to this comment of yours :)
True :-)
I think I know what my next birthday party is: