Hide table of contents

Comment Permalink

1) >"The numbers on how useful things are seem quite low to me..."

On the scale 1 was "Useless" and 10 was "Life-transforming". But just before asking for feedback, I made a change in the slides and added this meaning to the ratings of the events:

"3 - £100, 5 - £1,000, 8 - £10,000, 10 - £100,000 (e. g. career change)"

I explained it to people as well. This was... not smart. Because of this, some respondents gave low scores to all the events. E.g. someone said that the weekend was "Far more valuable (10-30x the counterfactual)" but did not gave any event a rating that is higher than 4. Others ignored the point and gave high ratings for all events.

That's why I weighted and normalised the ratings. If someone said that the weekend was "Vastly more valuable (>30x counterfactual)", I multiplied all their ratings by a constant so that their highest rating would be 10. If they rated the weekend as "Far more valuable (10-30x the counterfactual)", I multiplied all their ratings so that the highest rating would be 9. 8 for "Much more valuable", 7 for "Somewhat more valuable", and 6 for "About as valuable".

MichaelPlantJun 12 20180

Yeah, I thought the ends of the scales might have been more extreme than we'd normally use. It's probably quite hard to get people to sensibly answer unfamiliar, tricky questions.

See in context

A lesson from an EA weekend in London: pairing people up to talk 1 on 1 for 30 mins seems to be very useful

by saulius

Jun 12 20186 min read 17

16

Building effective altruismEvent strategy

Frontpage

A lesson from an EA weekend in London: pairing people up to talk 1 on 1 for 30 mins seems to be very useful

We organised an EA weekend in London for people who are already familiar with EA. A review of CZEA "Intense EA Weekend" retreat was helpful to us so I am sharing the lessons we learned as well.

Table of contents

Main lesson: 1 on 1s seem to be very useful

Overall feedback

Schedule

Event descriptions and lessons from running them

One-minute introductions

Explaining concepts

Productive disagreements workshop

Discussing various topics in small groups

"How to talk about EA" workshop

Application forms

Main lesson: 1 on 1s seem to be very useful

We experimented with various event formats. In a feedback form we asked to rate how useful each event was (for having a greater positive impact) on a scale of 1 to 10. According to the feedback, the most valuable event was pairing people up to talk 1 on 1 for 30 minutes:

Activity	Weighted and normalised*	Unweighted	Vote count
1-1s	7.3	6.8	24
Career Club	6.6	5.9	16
Murphyjitsu (next steps): how useful?	6.3	5.4	17
Hamming Circles	6.2	5.6	20
Socialising after the events	6.1	5.6	21
Decision Making workshop	6	5.3	14
“How to talk about EA” workshop	5.5	4.8	17
Lightning talks	5.4	4.9	20
Productivity workshop	5.4	4.6	8
Productive disagreements workshop	5.3	4.8	19
1 minute introductions	5.1	4.6	19
Discussion about where to donate	5	4.7	7
Explaining Concepts	5	4.7	20
Discussion about diversity within EA	4.4	4.2	10

*see this comment for explanation

This is consistent with feedback from group leaders' retreat where 1 on 1s were voted to be the most valuable activity on most days. What is more, when asked what the most enjoyable part of the weekend was, six respondents answered that it was 1 on 1s (the most common answer).

We gave each of the participants a piece of paper with a list of questions to ask each other to jump-start the conversations. I think we told them that asking the questions was optional. The questions may have helped to avoid the fear of running out of topics to talk about, even if they weren't used directly. We tried to pair people who don’t know each other well and have something in common.

The rest of this post does not contain anything important but could be useful to people who are organising similar events.

Overall feedback

30 people attended on at least one of the days (excluding 4 organisers). 25 attendees filled the feedback form. One of the questions was:

Here are the results:

Answer	Count
About as valuable (roughly the same as counterfactual)	1
Somewhat more valuable (1-3x the counterfactual)	8
Much more valuable (3-10x the counterfactual)	9
Far more valuable (10-30x the counterfactual)	5
Vastly more valuable (>30x counterfactual)	2

It’s unclear how good people are at answering such a difficult question but overall, the event seems to have been successful.

You can see a full summary of responses to the feedback form here. It includes answers to some free text questions that are not discussed in this post.

Schedule

Saturday

9:30 - 10:30 Arrivals & registration

10:30 - 10:45 Overview & schedule

10:45 - 11:45 One-minute introductions

11:50 - 13:00 Explaining concepts

13:00 - 14:00 Lunch

14:00 - 15:30 Productive disagreements workshop

15:40 - 16:40 Decision-making workshop or circling in a park

16:50 - 18:00 1-1s

18:00 - 23:00 Socialising, dinner

Sunday

9:30- 10:25 Arrivals

10:25 - 11:00 Group loving-kindness meditation or discussion in another room

11:00 - 11:15 Intro to the day

11:15 - 12:20 Hamming circles/Planning for the year

12:25 - 13:00 1-1

13:00 - 14:00 Lunch

14:00 - 14:30 1-1

14:30 - 14:40 Group photo

14:40 - 15:40 Career club or discussion about where to donate in another room

15:50 - 16:15 Lightning talks (Moral Circle Expansion and a short talk about tax-efficiency)

16:15 - 16:50 Discussing various topics in small groups

16:50 - 17:25 "How to talk about EA" workshop

17:30 - 18:15 Problem-solving circle: next steps

18:15 - 18:30 Feedback

18:30 - 22:00 Socialising, laughter yoga

Slides from the event can be seen here.

Event descriptions and lessons from running them

One-minute introductions

Everyone had 1 minute to introduce themselves. The purpose was for people who have something to talk about to know about each other. The following slide was shown to give people ideas on what to talk about:

Questions were ordered and picked in a way to make it less intimidating to talk. However, it caused many of the answers to be less informative. It would’ve probably been better to put questions like “What’s your plan for improving the world?” first. It was also difficult to hear what many people said partly due to bad acoustics. Two people wrote that they were uncomfortable introducing themselves. Overall, some online document or a sheet on the wall would have probably been better.

Explaining concepts

This is an idea we took from CZEA "Intense EA Weekend" retreat because this was rated as the most useful event at that retreat. The participants were asked to pick from a list of interesting concepts (e.g. scope insensitivity, optimiser’s curse) which ones they would like to have explained to them or explain to others. We used a different sheet from CZEA. Our sheet can be seen here. We asked people to divide into pairs or small groups and find different people when they hear a gong (every 15 minutes). We told that it is ok to skip topics if they don’t want to explain them or have them explain. According to the feedback, this was not a successful event.

Productive disagreements workshop

This idea was taken from an event that was ran in New York and adapted for EA purposes. First, I gave a short talk on how to disagree productively. I then asked everyone to rate whether they "agree", "disagree" or are "unsure" on various controversial EA topics and give their opinions on the importance of various causes. A sheet of questions can be seen here (note that there are two tabs). We then asked participants to pair up, compare sheets, find disagreements and discuss. Every 15 minutes we asked everyone to find a different pair.

One piece of feedback about the event was: “Hard to find disagreement. Many statements were too hard to have a confident belief on.” I think another issue was that many people didn’t know enough about some of the topics to have meaningful discussions.

One idea was that explaining concepts and productive disagreements workshops could help people to have more productive discussions during 1 on 1s because it sets a tone of talking about EA topics. What is more, people could continue one of these activities during the 1 on 1s. Some people told me that events were successful in this regard. However, it still seems like overall they weren’t very successful.

Discussing various topics in small groups

The idea was:

People suggest many topics
I read the list of topics and everyone votes on the ones they want to discuss (multiple votes allowed)
People separate into groups based on which topic they want to discuss the most.

This is the way discussions are organised at a philosophy meetup I occasionally attend. One mistake I made is not giving people a minute to think of topics to discuss. The only topics suggested were

Diversity within EA community
Value drift
Productivity and self-improvement (this wasn’t really a topic, it had prepared materials)

We separated into three groups to discuss these. Discussions did not work too well, possibly because the groups were too big.

"How to talk about EA" workshop

We separated into 4 groups based on which subtopic we wanted to discuss. Subtopics were:

How to talk about EA with people interested in you
How to talk about EA with people interested in the ideas
How to talk about EA with people you want to be interested in the ideas
How to talk about EA with people already on board

In the feedback form, two people mentioned that they’d rather this was a workshop (possibly a one that includes roleplay) rather than a discussion.

Application forms

We publicly advertised an expression of interest google form. Purposes were:

See how many people are interested
Filter out the people who are not yet familiar with EA (because the event was not aimed at newcomers). This required quite a lot of effort. I don't think it was worth it.
See which events people are most interested in attending

Out of 57 people who filled the form, only 14 attended the event. This is partly because we publicized the form 2 months before the event but sent tickets only 2 weeks before the event because that’s when the availability of the venue was confirmed. Some people made other plans for the weekend because they weren’t sure if the event is going to happen. We had to send the tickets sooner with no venue specified. Many of the attendees came because we invited them personally 2 weeks before the weekend. Some of them also invited other people. There could have been many more attendees if we sent invitations sooner. However, it’s unclear whether it would’ve been easy to manage many more people.

16 Reactions

More posts like this

Comments17

Sorted by

New & upvoted

Click to highlight new comments since: Today at 10:46 AM

MichaelPlantJun 12 20184

Thanks for writing this up. Three questions

The numbers on how useful things are seem quite low to me. What did you write as the ends of the scale? I'm thinking in terms of net promoter scores where anything below a 9 or a 10 is considered neutral or bad.

Can you explain Hamming circles? I couldn't find out how they worked even after a quick google.

Did you ask people if there was anything they wanted to do on the weekend but didn't do? I'd be curious to see if people came up with anything.

sauliusJun 12 20182

3) Because of your question I added a link to the summary of the responses to the feedback form to the article. Here are responses I see that suggest different activities:

Perhaps a little more discussion about actual causes e.g. what should be done about AI
In addition to the 1-1s(which were really useful), the could have been some more structured socialising (eg having people come together in groups of 3 and talk about what they care about, what their problems are, etc and switch groups in eg 5 minute intervals)
Maybe one minute intros could be on a post it on a wall so that people coming a day late still know who does what.

We also had some event ideas ourselves that we decided not to include in the weekend:

Community health session (a discussion of ways to improve the health of the EA community, be that through more diversity, more mutual support etc.)
Self-care for the altruistic (discussion and support session)
Discussions/anti-debate on cause prioritisation
Watching a video of a talk together and then discussing it (vote on a talk to watch during the event)
Values session: discussing why we have certain moral values
Problem-solving circle (participants raise problems they have and others try to help them)
The Humane League Work Party (Encouraging companies to commit to improving conditions on farms - writing letters & emails, phone calls, petitions. Would have been an option throughout the event.)
Gathering and interpreting data relating to the impact of charities for SoGive

sauliusJun 12 20182

2) I will ask the person who ran Hamming circles to explain it, I only participated in half of the event myself. You can get some idea from the slides (slides 21- 28).

sauliusJun 12 20182

1) >"The numbers on how useful things are seem quite low to me..."

On the scale 1 was "Useless" and 10 was "Life-transforming". But just before asking for feedback, I made a change in the slides and added this meaning to the ratings of the events:

"3 - £100, 5 - £1,000, 8 - £10,000, 10 - £100,000 (e. g. career change)"

MichaelPlantJun 12 20180

Yeah, I thought the ends of the scales might have been more extreme than we'd normally use. It's probably quite hard to get people to sensibly answer unfamiliar, tricky questions.

markov_userJun 19 20181

I guess this very much depends on how individual activities are executed. We had our 2.5 day retreat in Dortmund, Germany about a month ago, and while I didn't see the evaluation results, I got a strong impression that most people agreed on these points (still, take this with a grain of salt):

career discussion in small groups (~3-5) was quite useful; we had about 1 hour per group, and more would probably have been better.
double crux (I guess similar to productive disagreement?) was a cool concept, but a bit difficult to execute under the given circumstances (although it worked great for me), for similar reasons as mentioned by you
discussion about where to donate - this was, to some degree, what this weekend was primarily about for us, as we raised money on the first evening and then had to figure out where to send it. And while it started very slowly, we ended up spending many hours on Sunday on this (very open) discussion, and it was tremendously valuable. I really didn't expect this, but ultimately, judging from how engaged everybody was, how interesting our conversations were in the end, and how often each of us changed their mind over the course of the discussion, this was a great way to spend our time

Jiri_NadvornikJun 13 20181

Hi Saulius. CZEA retreat organizer here.

Thank you for this report.

We did "concept explaining somewhat differently:

a) people could fill out form w/ list of concepts they can/want explain several days earlier. They also had links to recommended articles for many of concepts offered.
b) during activity we projected on a screen list of groups (~3 people learning, one teaching) and what is their concept for this session.
c) after 15 minutes we hit the gong and projected list of new groups with new topics.

It seems to me that Circling and Hamming circles need the people to feel close and trusting each other. We thought about doing Hamming circles on Saturday evening but decided to do something else because the atmosphere was not right. How well it worked for you?

KirstenJun 13 20181

I didn't attend, but someone told me the circling thing was awkward.

sauliusJun 13 20181

I see, that is very different.

I only know about the feedback I already presented and the one hamming circle I myself participated in. In my circle all 3 of us were quite connected already. It may have helped but I'm not sure it was necessary. Both of them didn't know what to do with their careers so it was like a mini career-coaching session. It felt productive.

sauliusJun 13 20180

oh, I didn't notice that there was a question about circling as well. There are circling meetups where I did circling with strangers and it was great. So it's not necessary to be close with people before, though the experience is quite different if you are. I imagine that many people will find circling awkward no matter what but in this case I think we simply did not have enough time to get into it. We only allocated 1 hour for it (and I'm not sure even that is enough) and then we decided to do it in the park but walking to the park took much more time than expected. It was the lowest rated activity (3.7) but I didn't include it in the table of the article because I believe that it wasn't given a fair chance.

Mojmir_duplicate0.24929786764781214Jul 11 20180

Thanks for the write-up, I was wondering how concretely did you run the 1-1s. I saw the list of questions in the google slides, did you just ask people to discuss any of these? Also, did you pair people up or did they pair up themselves?

sauliusJul 12 20181

People were free to talk about anything they want. I'm pretty sure we told that using the questions was optional. I changed the post slightly to make it more clear. Personally, I didn't use the questions in any of my 1 on 1s. I know some people used the questions and they said that the first two questions ("When did you know that you wanted to be altruistic and why?" and "When did you realise you care about effectiveness and why?") resulted in long and not very productive conversations. So I moved them to the end in the questions document. I think that the list of questions has a lot of room for improvement. Maybe next time I will say that if they want to use the questions, they should look through the list and decide which question they are the most curious about.

We paired people up ourselves. We tried to pair up people who don't know each other well and have something in common (e.g. both are excited about the same cause area). We considered letting people pair themselves up but had reservations because some people might get upset if no one wants to talk with them. However, at the end of the first day we told people that if there is someone they'd like to be paired up with on the second day, they can write it on a piece of paper and put it in the box (so that only we could see their preference). But very few (2-4) people did that.

Btw, we will run an event that is all about 1 on 1s (https://www.facebook.com/events/330349444169839/) to see if it is a good stand-alone event (because it could be that it works only in the context of a weekend/conference/retreat). I'll write here about how it went, if I'll remember to do it :)

Mojmir_duplicate0.24929786764781214Jul 24 20180

Those are really useful comments, thanks. I wonder how did the event go?

sauliusJul 24 20181

It went well, I asked ~8 people for feedback and it was all positive, almost everyone said that they liked it better than a regular social. One person said that it made them less socially anxious than regular socials. I think we'll organise these every other month or something like that (in addition to socials).

We first quickly presented what topics each of us would most want to talk about, then we paired people up, based on that (though a lot of pairings were random). Each person had three one on ones and then we all had a picnic. I'm not sure how well it would work if the weather was less nice and we couldn't do it in a park (or a building with many breakout rooms like at the EA weekend).

SiebeRozendalJun 13 20180

What did you think about the length of the retreat? Would people have liked to stay longer? We're planning to organize a retreat from Thursday to Sunday in The Netherlands in between Christmas and New Year's.

sauliusJun 13 20181

I don't know, we simply didn't talk about that at all. My guess is that 4 days is not too long. EA globals sometimes last 3 days, if you include the social on Friday. I believe that a recent group organisers' retreat lasted an entire week. An AI camp lasted 10 days. These latter two events are not quite the same, but I guess you could ask Remmelt Ellen whether they felt too long, I believe he was present in both of them. Hmm, the fact that your event is during winter could matter a bit though, because going outside is usually a refreshing change of atmoshpere during such things.

By the way, this was not a retreat, we did it in an office in London and people slept elsewhere.

SiebeRozendalJun 17 20180

Alright thanks! :) Remmelt is also organizing this retreat, so we have that info!