Great post! I think you're one of the first uni groups I've seen who's particularly selective with their fellowship - I wouldn't have initially agreed with that strategy, but you give convincing reasons for doing so and it sounds like it paid off. :) Congrats on a successful fellowship round!
**Update: In this post we recommend a selective fellowship using our selection criteria. Since then, we discovered that our selection process was not predictive of eventual engagement and have changed our recommendation. We still stand by the recommendations in the other sections of this post.
During the Summer of 2020 I helped organize a virtual version of the Yale Fellowship with co-organizers from McGill, Northeastern, and Yale. Since McGill and Northeastern are new EA groups we hoped this could serve both as a recruitment mechanism and as practice for running their own Fellowships in the fall. We had unexpectedly good results with 87% of fellows giving 9/10 or 10/10 on the question “Would you recommend the Fellowship to a friend” (NPS - 87), an average attendance of 95% and 100%* completion. 5/6 Northeastern, 1/1 Yale, and 4/5 McGill fellows have joined their respective boards and one fellow from Northwestern is starting a Fellowship there as well. These are a lot better results than I normally get when I run the Fellowship at Yale** so I am writing this retrospective with my guesses on what went well.
My guesses are:
For Retention Specifically
*We had one person who was accepted end up deciding to not do the Fellowship. In this retrospective we focus entirely on the 15 Fellows who participated.
**Normally we have around 1 or 2 students drop out or miss more than 2 meetings and we usually only get around 2-3 fellows to join the board. In Spring 2020 (when we started calculating NPS) we got an NPS of 54.
This fellowship could not have been possible without my exceptional co-organizers Anna Mouland, Kaleem Ahmid, and Thomas Woodside. They are all great organizers and I was lucky to work with them. We also greatly appreciate help from Catherine Low in helping create the initial vision for the fellowship and helping to recruit organizers and advertise. The content of our fellowship is mostly based off of Joshua Monrad’s version of the Yale EA Fellowship. I am extremely grateful for everything he taught me about running Fellowships, moderating discussions, and creating inclusive welcoming environments.
This model of the fellowship had similar goals as most others but had the added goal of seeding new EA groups.
Edit: We are currently not recommending this process since we discovered it was not predictive of engagement.
We received 80 applications for 16 spots. The large number of applicants likely had to do with the fact that people were actively looking for things to do over the summer after many programs and internships were cancelled due to COVID. There is also the possibility that branding the Fellowship as the “Yale Effective Altruism Fellowship” made it more desirable to non-Yale students who wanted to include it on their resumes.
We would love to take as many students as possible but have found in previous years that by limiting it to a smaller group:
There is always a risk that someone might be permanently turned-off from EA if they are rejected from a Fellowship program. Additionally, being selective could have a negative image at schools where group organizations being selective is not a norm (unlike Yale). We take several steps to try and reduce risks.
We just want to let you know that this interview is not in any way to try and judge your competence or altruistic-ness or anything like that. But rather we are trying to figure out your personal fit within the Fellowship and how much we think you will get out of it.
We say this little disclaimer at the start of every interview because we don’t want you to think that we are saying it because of your interview performance or anything like that but we unfortunately did have a lot more applications than we can take. We would really love to take everyone and everyone who made it to interviews is really impressive but we only have the capacity to take 15 a semester.
However, we do run this fellowship every semester and people who re-apply get priority. Additionally, if the fellowship isn’t right for you we have a lot of other ways to get involved with the group and a lot of our members get involved through ways besides the Fellowship such as our intro workshop, speaker events, and socials.
Do you have any questions about that? Okay - now with that disclaimer out of the way we can get into the interview :)
“Note: Young people take interviews very seriously and may be a lot more nervous than you think they are. It’s incredibly important to be kind, patient, and encouraging. Intimidating them will be bad for them, for your group, and for the reputation of EA as a whole. Also, be mindful that people have different styles of thinking and talking, and it’s important not to be dismissive of those styles which are different than yours.” (From Joshua Monrad’s guide to running Fellowships)
2. Give rejected but good applicants a guaranteed spot in the next Fellowship as long as they re-apply.
3. Write personalized rejection emails encouraging the applicants to stay involved
Edit: We are currently not recommending this process since we discovered it was not predictive of engagement.
When evaluating applicants for the Summer 2020 cohort, we rated them on the basis of a set of criteria that had also been outlined on our application:
The questions we used to judge these criteria can be found in our application
We rated the applicants from 1-5 on each of these criteria except for Experience which we ranked on a scale of 1-3 with 1 being most experienced. This method is based on the interviewing recommendations laid out in Thinking Fast and Slow.
Due to there being so many applicants we decided to automatically reject applicants from schools that had their own Fellowship programs and defer those to their own school.
We created a sheet where we blinded the names of the applicants to mitigate biases and conflicts of interest. In this sheet we made tabs for each organizer to score the applications based on the selection criteria above. These were separate tabs so that we were not influenced by each other’s scores.
We first went through a few applicants all together, reading their application and giving our own scores. Then we compared our scores to calibrate with each other. After we were sufficiently calibrated we had two organizers go over one half of all the applications and the other two go over the other half. We each ended up reading 40 applications.
After everyone had graded their portion, we ranked the applicants from highest to lowest mean score. We checked each organizer's average scores and compared between organizers grading the same applicants to make sure that we remained calibrated. We then proceeded to remove the least competitive applicants.
This meeting took 3.8 hours to complete. This was particularly long since we had so many applicants and a fairly long application. (Afterwards we decided to add a word limit to the application questions, and add a new question, “Why are you applying for this fellowship?, to aid in the scoring of the “Fit/enthusiasm” criterion.)
We made a spreadsheet of organizer availability (with numbers assigned to each organizer rather than their name to mitigate the chance that applicants would choose familiar interviewers) and sent the sheet to 44 applicants to sign up for 30 minute interviews (many interviews ran between 30-45 minutes). Luckily we had a lot of time on our hands at the start of the summer and were able to do this.
For the interview, we asked applicants to read the Introduction to EA from CEA and asked them to “come prepared to share your reactions, disagreements or uncertainties.”
During the interview, we discussed the article and asked questions which we thought might help us rate them according to our selection criteria. You can read a sample interview here.
We had an experienced organizer join the first interviews of new organizers for calibration and feedback. After each interview the interviewer scored the applicants on the same scale and took notes before starting the next interview. We did this in a central google doc that had places to put reviews for each applicant
We strongly encourage groups to do interviews if they have the capacity, even if they don’t have that many applicants. For one, it gives groups a much better picture of the applicant pool and how much they would gain from the fellowship. Second, it sends a strong signal to prospective fellows that the fellowship is a serious thing with high standards.
In this meeting, given that we had 4 organizers, and 16 spaces available, each organizer picked their top 4 or 5 applicants that they interviewed. Organizers used their scores to inform this but not to determine it. We should have been better about making sure to take averages of organizers’ interview scores for calibration but it turned out okay.
We have had in the past where one organizer just gets many of the best applicants in their interviews. This sometimes happens and is why each applicant should be discussed before admitting or rejecting.
This only took about 1 hour this time around but has gone much longer in the past when we were really stuck between a few applicants for the last spot.
We then sent out acceptance emails and personalized rejection emails.
Other considerations on diversity and experience can be found in Joshua Monrad’s Fellowship guide. We highly recommend reading these.
Fellows had weekly 75 minute discussions. Fellows were also strongly encouraged to sign up for 1-1s with at least one of the organizers. Each organizer made a calendly and we put them at the bottom of every email to make scheduling easier. There was a getting-to-know-you social and a concepts workshop at the start of the fellowship. In the middle of the fellowship we had an optional discussion on decision theory. At the end of the fellowship we had a final social.
We structured our fellowship as a synchronous Zoom meeting of 15 fellows and 4 organizers. We would all first meet together and do announcements and a brief intro on the topic. This served as a convenient time to give disclaimers on sensitive topics, remind people to sign up for 1-1s, and to publicize any extra events.
After about 5 minutes of announcements we split Fellows randomly into four breakout rooms with one organizer in each. Every week we would start our breakout sessions checking in with each other. We would each go around saying a “happy” and a “crappy” from the week. This is a nice way to see how everyone is doing and create more of a sense of community.
Then, we had each fellow type in the chat a question they would like to discuss. Then Fellows could vote on which to focus on. For some weeks there were questions we as organizers wanted to make sure were discussed and we would just go over those first. This method gives the Fellows more control over the discussion and makes them feel more involved. However, this can lead to important questions not getting asked and unrelated questions possibly being upvoted.
At the end of the discussion we would all come back together in the main Zoom room. I would randomly call on someone from each group to either share something from their discussion or defer to another person in their group by saying “x had a really good point I will let them explain it”. This keeps fellows on their toes and paying attention because they might be called on but gives an easy out if the fellow does not want to answer. Meetings would usually end with podcast and/or book recommendations pertaining to topics that seemed to be of interest during group discussions, which was something that was appreciated by a few of the fellows.
You can find our readings directory here.
A benefit of this structure is that Fellows are never ‘stuck’ with one moderator or one group. Some groups of people are much quieter than other groups of people. Some Fellows tend to like certain organizer’s moderation style better than others or just get to experience more than one. Randomly mixing up the group each week gives variety and gets rid of the risk of a Fellow being stuck in a quiet group or with a bad moderator. (Our moderators did a good job but this a concern especially amongst those who are just learning how to lead discussions)
Another benefit of this structure is that it allows all of the fellows to get to interact with each other and know each other. We wanted to create a sense of community within our cohort which we thought would most effectively be done if everyone met at the same time. This does limit the audience of the Fellowship due to time-zones, however so that decision has to be weighed out.
We opened 1-1s as a place where fellows could further discuss topics from the fellowship, ask questions about EA, find out how to get more involved in the global community, discuss career plans, and get resources and connections. It is also a time where organizers can get to know the Fellows better. Afterwards the organizer can send an email with helpful resources from places such as the EA Hub, EA forum, or various panlists and Facebook groups. Since Yale EA has been running Fellowships for several years now I was able to connect Fellows with Fellowship Alums who were now working in impactful careers. 14/15 Fellows signed up for at least one 1-1. We did not set a limit to the number of 1-1s a fellow could have. Seven Fellows had more than one 1-1 but none had more than four.
In the middle of the fellowship we had an optional discussion with Mahendra Prasad, a PhD candidate at UC Berkeley. This discussion focused on decision theory and its implications on things such as voting and AI. We had fellows first watch this talk he gave at EAG and read this article. The discussion ended up mostly consisting of a short intro and Q&A.
We did two zoom socials that turned out better than I expected. Both had a similar structure:
Effective altruism includes a lot of sensitive topics. Not handling these topics in a delicate way can make EA feel unwelcoming to different groups of people. Additionally, it is important to note the privilege we have to be able to have these choices when it comes to things like donations and careers.
Note that people may be from countries you talk about in the global health and development portion of the Fellowship or come from cultures and backgrounds that heavily influence their approach to topics such as animal welfare. Careers, donations, and diets are all very personal topics that can be hard to discuss.
Additionally, note that all of these topics are extremely complex and this is just the beginning of us thinking about them.
Not addressing these topics sensitively can lead to Fellows dropping out and/or leaving with a negative image of EA. One year we had a Fellow drop out after another Fellow was insensitive during a giving exercise. Now we make sure to always have a moderator in break-out groups to make sure something like this does not happen again.
I highly recommend these notes on moderating
Oxford has also created an extremely helpful Fellowship Facilitator Guide and Discussion Norms Template
I also recommend checking out the disclaimers we always say and the intentional community building activities for certain weeks in the appendix.
I expect much of the value of this fellowship to come from the strengthening of the McGill and Northeastern groups and the number of Fellows who went on to join the board. 5/6 Northeastern, 1/1 Yale, and 4/5 McGill fellows joined their respective boards and as of a few weeks into the semester are still engaged. This is particularly impactful since it improves the sustainability of the two new groups.
We ran a pre-fellowship survey and a post-fellowship survey to measure changes in Fellows plans and opinions. Unfortunately, we realized when analyzing the results* that by asking so many questions we made it very difficult to achieve statistical significance with an N of 15. The only metric where we achieved statistical significance was for the question “To what extent do you agree that biosecurity is a cause that more people should be working towards improving?” Where more fellows chose that they “Strongly Agree” after the Fellowship.
However, we thought the graphs might still be helpful for people to see so we collated them into this results document of comparative data.
Some fellows gave us permission to share their written responses to the post-fellowship survey which we have added in this document which contains feedback and descriptions of impact.
1st week (Intro) -
Note this takes a while but is important. It is important to try and build a foundation of community and take time for people to get to know each other
3rd week (Giving Exercise)
We do not think that these organizations could end poverty on their own. We know that ending poverty will require a lot of policy decisions in addition to philanthropic giving. However, right now we have some money and want to help the best we can.
4th week (Careers)
5th week (Animal Welfare)
8th week (Conclusion)
Thanks for the post! This definitely isn't addressed at you specifically (I think this applies to all EA groups and orgs), so I hope this doesn't seem like unfairly singling you out over a very small part of your post, but I think EAs should stop calculating and reporting the 'NPS score' when they ask NPS or NPS-style questions.
I assume you calculated the NPS score in the 'standard' way i.e. asking people “Would you recommend the Fellowship to a friend?” on a 0-10 or 1-10 scale, and subtracting the percentage of people who answered with a 6 or lower ("Detractors") from the percentage of people who answered with a 9 or 10 "Promoters"). The claim behind the NPS system is that people who give responses within these ranges are qualitatively different 'clusters' (and also the people responding with a 7-8 are also a distinct cluster "Passives" who basically don't matter and so who don't figure in the NPS scores at all) and that just subtracting the percentages of one cluster from another is the "easiest-to-understand, most effective summary of how a company [is] performing in this context."
Unfortunately, it does not seem to me that there's a sound empirical basis for analysing an NPS style scale in this way (and the company behind it are quite untransparent about this basis (see discussion here). This way of analysing responses to a scale is pretty unusual and obscures most of the information about the distribution of responses, which it seems like it would be pretty easy for an EA audience to understand. For example, it seems like it would be pretty easy to depict the distribution of responses, as we did in the EA Survey Community information post.
And it seems like calculating the mean and median response would also give a more informative, but equally easy to understand summary of performance on this measure (more so than the NPS score, which for example, completely ignores whether people respond with a 0 or a 6). This would also allow easy significance testing of the differences between events/groups.