notice confusion, ask dumb questions and encourage others when they ask questions (to decrease deferring of the type "This doesn't make sense to me but the other person seems so confident so it must be true")
regularly think about solving the whole problem (to keep the big picture in mind and stay on track)
reflect on their experience. We had a reflection time each morning with a sheet to fill out which had prompts like "what are key confusions I would like to resolve" "What is my priority for today" etc
Content Quality: It depends on what you mean by content quality, but I think having a high bar for session content can actually be a bit detrimental. For example one of the sessions I learned most from wasn't "high quality" - the person wasn't an expert and also hadn't prepared a polished presentation. But they knew more than I did and because the group was small I could just ask them lots of questions and learn a lot that way. We also encouraged people to leave sessions when they don't find them the best use of their time in order to ensure that people only listen to content they find valuable. We got the feedback that people found that norm really helpful.
I think part of it is that networking is relatively hits-based: Most new connections don't make much difference, but meeting someone who is a good fit for collaboration or who gives you access to an opportunity can be really valuable. I don't think most researchers have hit diminishing returns on networking yet. Even if you already know 200 people in the field, getting to know 10 more out of which one is an "impactful connection" can make a big difference. For me it has also often made a difference to meet someone I already know again and get to know them better, and that ended up making a big difference for me.
or can easily connect to them
I think it's actually (relatively) hard to connect to people even if you have a decent network. If you want to meet someone new you can get an intro and then have a call with them, but I think there is a lot of friction to that, and it feels like you need a reason to do so. Meeting people at an event has a lot less "social overhead". If you want to talk to someone you just walk over and see how it goes, and if it's not productive you walk off again.
especially with a lot of junior people (who might have a lot of terrible/obvious ideas)
I'm not sure if "finding it useless to talk to junior people who have terrible or obvious ideas" is a problem people have? I didn't find anyone at the retreat "too junior" to be interesting to talk to (even though with different people the conversational dynamic would be more skewed towards me giving vs receiving advice depending on the other's experience level, but that seems fine to me). And if someone does find it uninteresting to talk to someone with less experience, they can still just end the conversation. Actually we had an explicit norm that it is perfectly okay to end conversations or leave sessions if you don't think they are the best use of your time at the moment.
As a temporary measure in response to uncertainty about our future funding levels, we’ve put the bottom ~40% of grants above our current funding bar on hold.
Did you notify these applicants about their applications being put on hold?
(I just talked to an LTFF applicant who didn't know about this measure, and they suspected that their application might be one of those which have been put on hold.)
Update: We did a follow-up survey 4 months after the retreat, asking participants what impact the retreat had on them. Here is a summary of the responses:
36 people responded (~54% of participants)
Personal prioritization
7 people reported their prioritization which area to work in was influenced by the retreat
One person quit their previous job to do a career transition to AI safety
One person decided to keep working on their current topic rather than switch as they had considered before
One person started to work on a research topic they wouldn’t have focused on otherwise
Connections
26 people mentioned they made connections they found useful, e.g. for research discussions.
7 people started research collaborations with people from the retreat
Applying for training programs and funding
5 people applied to SERI MATS as a result of the retreat
4 people got accepted into the online training program
2 people got accepted into the in-person research program
4 people applied for funding (1 accepted, 1 rejected, 2 unclear)
3 people wrote a post, and one person created a youtube channel
One is unsure whether their post was a result of the retreat
For one it was their first ever post
Other/Miscellaneous
3 people reported the retreat didn’t change much for them personally, but they think they were able to help others
All three were mentioned by others in the feedback as being particularly helpful
13 people said they got a better picture of the AI safety landscape in general
6 people reported they got a motivation boost from feeling part of a community of like-minded people
3 people mentioned they went to EAG London because of the retreat
One person shifted focus within their area of work
One person got useful experimental results they think they wouldn’t have gotten without the input from the retreat
(Note that the survey consisted of a free text field, and the categories I made up for summarizing the results are pretty subjective. Also note that some people mentioned that they started projects or applied for things, but would have done so even without the retreat. I did not include those in my count.)
I think these results are really helpful to improve the picture of how such a retreat is valuable for people!
The biggest surprise for me is the amount of tangible results (research collaborations, getting into SERI MATS, concrete projects, career changes) that came out of the retreat. There were 9 people who said in the feedback form that they want to start a project as a result of the retreat (the form they filled out right at the end of the retreat). I would have guessed that maybe 5 would actually follow through after the initial motivation directly after the retreat has worn off. And I think that would have been a very good result already. Instead, it was more than 9 people who started projects after the retreat!
I still have the intuition that the more vague effects of the retreat (such as feeling part of a community) are really important, potentially more so than the tangible outcomes. And these vague things are probably not measured very well by a survey that asks “What impact did the retreat have on you?”, because it is just easier to remember concrete than vague things.
Overall, these survey results make me even more excited about this kind of AI safety retreat!
Hi JJ, just wanted to say a big thank you for all your work!
I used the fiscal sponsorship, the free health coaching, the AISS newsletter, and the alignment slack, and found all of them super helpful! And probably I've used even more things which of I'm not even aware that they are coming from AISS.
I particularly appreciate your lots of links page. It's one of my favorite resources for an overview of which things exist in the AI safety ecosystem.
participants made 6.7 new connections on average (Question in the feedback form was "How many additional people working on AI safety would you feel comfortable asking for a small professional favor after this retreat? E.g. having a call to discuss research ideas, giving feedback on a draft, or other tasks that take around 30 minutes...").
18 people mentioned their plans to reach out to people from the retreat for feedback on posts or career plans, for research discussions or for collaborations.
motivation boost
projects/collaborations started
9 people mentioned projects (Writing posts, doing experiments, and field building projects) they would start as a result of the retreat.
participants getting a better map of the space
which people exist, what kinds of things they are good at, what they have worked on, what you can ask them about.
What projects and what research exist.
A general sense of "getting more pieces of the puzzle".
I think this is hard to measure, but really valuable.
I find it hard to say which of these is most important, and they are also highly entangled with each other
Epistemic standards: We encouraged people to
Content Quality: It depends on what you mean by content quality, but I think having a high bar for session content can actually be a bit detrimental. For example one of the sessions I learned most from wasn't "high quality" - the person wasn't an expert and also hadn't prepared a polished presentation. But they knew more than I did and because the group was small I could just ask them lots of questions and learn a lot that way.
We also encouraged people to leave sessions when they don't find them the best use of their time in order to ensure that people only listen to content they find valuable. We got the feedback that people found that norm really helpful.
I think part of it is that networking is relatively hits-based: Most new connections don't make much difference, but meeting someone who is a good fit for collaboration or who gives you access to an opportunity can be really valuable. I don't think most researchers have hit diminishing returns on networking yet. Even if you already know 200 people in the field, getting to know 10 more out of which one is an "impactful connection" can make a big difference. For me it has also often made a difference to meet someone I already know again and get to know them better, and that ended up making a big difference for me.
I think it's actually (relatively) hard to connect to people even if you have a decent network. If you want to meet someone new you can get an intro and then have a call with them, but I think there is a lot of friction to that, and it feels like you need a reason to do so. Meeting people at an event has a lot less "social overhead". If you want to talk to someone you just walk over and see how it goes, and if it's not productive you walk off again.
I'm not sure if "finding it useless to talk to junior people who have terrible or obvious ideas" is a problem people have? I didn't find anyone at the retreat "too junior" to be interesting to talk to (even though with different people the conversational dynamic would be more skewed towards me giving vs receiving advice depending on the other's experience level, but that seems fine to me). And if someone does find it uninteresting to talk to someone with less experience, they can still just end the conversation. Actually we had an explicit norm that it is perfectly okay to end conversations or leave sessions if you don't think they are the best use of your time at the moment.
I just checked and under https://www.safe.ai/about these three people are listed:
I just saw that 80000hours' technical AI safety research career review was based on this post!
I have found this post super valuable when it came out, and I have sent it to lots of people who are starting out in technical AI safety since then.
Do you think now it makes more sense to send them the career review instead? Or does it still make sense for them to read both?
Thanks a lot for writing this!
I read this post back in 2020 and I've been using FVP regularly since then, so you've had a decent bit of impact on my life!
Did you notify these applicants about their applications being put on hold?
(I just talked to an LTFF applicant who didn't know about this measure, and they suspected that their application might be one of those which have been put on hold.)
Update: We did a follow-up survey 4 months after the retreat, asking participants what impact the retreat had on them. Here is a summary of the responses:
(Note that the survey consisted of a free text field, and the categories I made up for summarizing the results are pretty subjective. Also note that some people mentioned that they started projects or applied for things, but would have done so even without the retreat. I did not include those in my count.)
I think these results are really helpful to improve the picture of how such a retreat is valuable for people!
The biggest surprise for me is the amount of tangible results (research collaborations, getting into SERI MATS, concrete projects, career changes) that came out of the retreat. There were 9 people who said in the feedback form that they want to start a project as a result of the retreat (the form they filled out right at the end of the retreat). I would have guessed that maybe 5 would actually follow through after the initial motivation directly after the retreat has worn off. And I think that would have been a very good result already. Instead, it was more than 9 people who started projects after the retreat!
I still have the intuition that the more vague effects of the retreat (such as feeling part of a community) are really important, potentially more so than the tangible outcomes. And these vague things are probably not measured very well by a survey that asks “What impact did the retreat have on you?”, because it is just easier to remember concrete than vague things.
Overall, these survey results make me even more excited about this kind of AI safety retreat!
Hi JJ, just wanted to say a big thank you for all your work!
I used the fiscal sponsorship, the free health coaching, the AISS newsletter, and the alignment slack, and found all of them super helpful! And probably I've used even more things which of I'm not even aware that they are coming from AISS.
I particularly appreciate your lots of links page. It's one of my favorite resources for an overview of which things exist in the AI safety ecosystem.
I hope whatever you are up to next goes well!
I think the value comes from:
I find it hard to say which of these is most important, and they are also highly entangled with each other