During my first month working in AI safety, someone told me, "When you enter AI safety, you have to work out for the first time what tier of god-tier you are." At the time, I interpreted this as "just because you have a physics PhD and led a successful EA group for three years, this doesn't mean you are qualified/capable to do AI safety research." Many times, working in this space, I have been forced to confront my limitations. I hope that AI safety can be/become a much broader church than the comment I heard implies. I do not believe that a PhD is necessary to do AI safety research. I do think the field is very small and growth is constrained by strong founders, field-builders, and grantmakers, which has resulted in a highly elite set of programs and job market. I hope we can increasingly move to a place of abundance and enable more people to contribute their skills and dedication.

What's your evidence that "cG regularly turns away competent, overqualified applicants"?

I think it's deeply unfortunate if a lot of people are getting turned off AI safety as a cause area because they can't break into the field. I also think the calls for urgency are justified on impact grounds. Maybe there are low-cost ways to help rejected applicants feel better? I like the idea of publishing application statistics or offering tailored advice to rejected applicants, though the latter is really expensive 

It seems like a lot of people are rejected by Harvard or Google, but are able to pivot to other Ivy League colleges or FAANG companies. At worst, there are second-tier colleges and companies to apply to. I'm not sure this is the case in AI safety, as the field is still very small (2-4k FTEs). It seems like the best thing many rejected applicants can do is apply for a CS PhD or work in a regular tech company to build their skills, which can be deeply unsatisfying for impact-driven people who think AGI is near (as I do).

Do you think the marketing is false because:

• The roles are getting more than enough qualified, competent applicants?
• The cause area of AI safety is saturated with talent?
• AI safety is not impactful enough to justify claims like "this cause area is particularly important to scale"?

I'm not sure the analogies to Google or Harvard are useful here. These organizations are driven by profits/enrollments, not by impact. They likely are far less picky than AI safety orgs because they are very big and don't require the incredibly niche skill-sets and mission alignment of AI safety roles. Google can also pay a ton and both Harvard and Google are conventionally high status, attracting a ton of applicants naturally. One might argue that there is a moral imperative to scale AI safety fast in a way that doesn't apply to Harvard or Google on the whole. Note that the GDM interpretability team and Harvard AI safety team have called for urgency, independent of Google's general messaging.

None of this is to justify the hiring practices at AI safety organizations. I think plenty of operations roles (e.g., HR, finance, legal, marketing, facilities) don't require deep mission alignment or specialist AI safety skills. At MATS, the hiring rate for ops roles is ~2%; there are just a ton of applicants! Obviously, roles that depend on specialist AI safety skills, knowledge, or connections will be more selective, as well as leadership or independently operating roles, which greatly benefit from deep mission alignment.

In defence of the call for urgency in combination with the low acceptance rates:

• AI safety seems really impactful and urgent;
• Marginally better talent can have outsized impacts in a fast-growing field;
• Some roles (e.g., grantmakers) are very hard to hire for as the necessary skills are very rare;
• Emphasizing the important and urgency of AI safety will allow more samples from the distribution of talent, which means better-staffed roles and higher likelihood of finding rare talent.

Ah, makes sense. In the world where MATS randomizes over applicants for a typical 100+ fellow cohort and we learn after 6-18 months how impactful MATS is, what are the main upsides? My guess:

• Might help understand how useful fellowship programs are vs. other AI safety interventions, like academic grants or mass-messaging (most of which are also saturated with funding at the moment).
• Might help better price how cost-effective AI safety is vs. other EA cause areas.
• Might help MATS staff (e.g., me) work on higher-impact interventions.

Valid points, all. Re. your last point, if one thinks the default case is doom, once might want to fund a lot of blue sky interventions as the upside potential is stronger than the downside risk.

Steelmanning the case for RCTs at MATS:

• Maybe we are actively causing harm by selecting fellows suboptimally and could improve fast and cheaply.
• Maybe we can show mentors convincing evidence that helps them select fellows better without risking their support of MATS.
• Maybe there are strong near-term proxies for impact, like a panel of experts assessing a fellow's end-of-program research output.
• Maybe we could just assemble a talent database of everyone working in AI safety and chart their journeys in a way that usefully indicates whether MATS is reliably counterfactual.

(Note that challenge trials is probably the wrong framing because the risk is not to MATS participants, but to broader civilization.)

Just noting that MATS is hiring an Impact Analyst (deadline: Sep 21). Please apply if this sort of thing interests you!