Thank you to all the wonderful people who've taken the time to share their thoughts with me. All opinions are my own: Will Aldred, Jonah Boucher, Deena Englander, Dewi Erwan, Bella Forristal, Patrick Gruban, William Gunn, Tobias Häberli, James Herbert, Adam Jones, Michael Kerrison, Schäfer Kleinert, Chris Leong, Cheryl Luo, Sobanan Narenthiran, Alicia Pollard, Will Saunter, Nate Simmons, Sam Smith, Chengcheng Tan, Simon Taylor, Ben West, Peter Wildeford, Jian Xin.

Executive Summary

There is broad consensus that research is not the most neglected career in AI Safety, but almost all entry programs are targeted at researchers. This creates a number of problems:

• People who are tail-case at research are unlikely to be tail-case in other careers.
• Researchers have a bias in demonstrating ‘value alignment’ in hiring rounds.
• Young people trying to choose careers have a bias towards aiming for research.

Introduction

When I finished the Non-Trivial Fellowship, I was excited to go out and do good in the world. The impression I got from general EA resources out there was that I could progress through to the ‘next stage’ relatively easily^[9]. Non-Trivial is a highly selective pre-uni fellowship, so I expected to be within the talent pool for the next steps. But I spent the next 6 months floundering; I thought and thought about cause prioritisation, I read lots of 80k and I applied to fellowship after fellowship without success.

The majority of AI Safety talent pipelines are optimised for selecting and producing researchers. But research is not the most neglected talent in AI Safety. I believe this is leading to people with research-specific talent being over-represented in the community because:

1. Most supporting programs into AI Safety strongly select for research skills.
2. Alumni of these research programs are much better able to demonstrate value alignment.

This is leading to a much smaller talent pool for non-research roles, including advocacy and running organisations. And those non-research roles have a bias towards selecting former researchers.

From the people I talked to, I got the impression that this is broadly agreed among leaders of AI Safety organisations^[10]. But these are a very small number of people thinking about this - and they’re often thinking about this completely independently of each other!

My main goal of this post is to get more people outside of high-level AI Safety organisations to study the ecosystem itself. With limited competitive pressures on the system to force it to become more streamlined, I believe having more people actively helping the movement coordinate could magnify the impact of others.

If you are interested in working on any aspect of the AI Safety Pipeline, please consider getting in touch; I’m actively looking for collaborators.

We Need Non-Research AI Safety Talent

Epistemic status: Medium

After about 30 hours of research into the area, it is clear to me that we have very little data or consensus on what’s needed in AI Safety; but the most recent and comprehensive survey of 25 leaders in EA found that leadership, policy expertise and media engagement were the most valuable skills to recruit for. In fact, all of the top 7 most sought-after skills were related to management or communications.

We potentially need more government and policy people to engage directly with politicians; More public advocates to ensure the public are on board. More founder types to ensure that the opportunities for people to work in this field are there. And potentially more organisational talent to ensure that coordination happens a bit better between the communities^[11].

From the people I talked to, which of these roles precisely is most important to recruit for varies a lot. But there was a strong consensus that non-research roles are more important to recruit for at this time.

On a more meta-level, there also appears to be gaps in generalist skills in hiring rounds. And in particular when hiring for non-research roles, a challenge in discerning applicants with value alignment.

I believe these gaps aren’t accidental, but a direct consequence of a research-focused pipeline.

Most Talent Pipelines are for AI Safety Research

The vast majority of training programmes within AI Safety are explicitly geared towards developing researchers. There are over 20 full-time fellowship programs out there for upskilling in AI Safety. As far as I’m aware only 3 are for non-researchers^[12].

Ironically, these non-research fellowships are the most competitive, with none having an acceptance rate more than 2%.

This intense focus on research-based upskilling creates a cascade of four systemic problems:

1. It is a potentially unnecessary filter for people who could be top performers without research skills.
2. It creates an impression that research is the main path to impact.
3. It is harder for non-research people to demonstrate ‘Value Alignment’.

This Creates the Wrong Filter for Non-Talent Roles

Epistemic status: Medium/Low

The focus on research presents a large, unnecessary filter for the talent pool in AI Safety.

Because research fellowships are so competitive (<5% acceptance rate), applicants must demonstrate that they are both exceptionally good at research and are passionate about doing research as their career.

People who get into research fellowships are much more motivated to stay in AI Safety. This makes them a very outsized proportion of the talent pool.

Many of the people doing direct policy work, advocacy work and organisation running at the moment were initially selected as researchers. They had to demonstrate they were in the top 5th percentile of researchers. This means work they are currently doing is likely their secondary skillset.

I don’t think it’s unreasonable to expect a person with 94th percentile research skills, who would not make it into a research fellowship, to be 99th percentile at government or advocacy work.

The filter not only blocks people, but makes them think that research is the only way in.

This Creates a Feedback Loop of Status

Epistemic status: Medium

If the main paths you see when you enter AI Safety are research fellowships, there’s a strong incentive to work towards these ‘established markers of success’ over pursuing your own path for impact.

Note how the research fellowships get thousands of applications every year, but a tiny fraction are actively sending out cold emails to work in other areas.

By giving people ‘permission’, we’re signalling to them that research is the path of least resistance to impact. But given how competitive research is, this is unlikely to be true for most people.

There may also be a subconscious bias for hiring managers to think along similar lines.

Research Fellowships have a Bias in Hiring

Epistemic status: Medium

Consider two applicants for a senior people management job:

Applicant A started out wanting to be a researcher. They did MATS before becoming an AI Safety researcher. By gaining enough research experience they were promoted to a research manager.

Applicant B always wanted to be a manager. They got an MBA from a competitive business school and worked their way into becoming a people manager in a tech company. Midway through their career they discover AI Safety and decide they want to make a career transition.

Even if Applicant B put in a significant amount of effort outside of work (eg by signing the 10% pledge, posting on the EA Forum or even starting an AI Safety group), in many employers’ eyes they would not look as value aligned as someone who did MATS, something which is part of a researcher’s career path anyway.

I believe that this disproportionately increases the competitive advantage of a person who did a research fellowship, even if they choose to go into a non-research role.

Conclusion

The AI Safety pipeline is overoptimised for researchers, and this is leading to gaps in the talent pool for AI Safety careers.

Most of the people I’ve talked to have agreed with me; in fact for many of them it’s obvious. Their next question is almost always ‘what do you propose to do about this?’ over ‘is this actually a problem?’.

But to many people who do not look at the field from a meta-level, I don’t believe this is obvious.

If you’re interested in continuing this work and you would like collaborators - particularly for this Autumn or the Summer of ‘26 -  please get in touch with me!

FAQ

• Maybe AI Safety has enough researchers, but isn’t the field still bottlenecked on extreme tail-case research talent?
  • Yes, but I don’t expect more research programmes to attract people who aren’t already attracted to existing high-profile programmes (such as MATS). And whilst everyone in these programmes is in the top 5%, we’re still far from saturated with truly top talent.
• How do timelines play into this?
  • I think for longer timelines, the case for large-scale restructuring is very clear. We’re missing out on potentially huge cumulative gains.
  • For shorter timelines, it’s more of an open question. My intuition is that some form of reshifting should not take too long, and that AI Safety needs to have its act together in the coming months before transformative AI arrives.        

Further Questions

I think there are an enormous number of directions further work could take. These are further questions that I’ve come up with across the course of my conversations that I think it would be valuable to have answers to:

1. What guidance should be given to the enormous bycatch (95%) who aren’t yet suited for research or similar programs?
2. Would individual mentorship instead of research programs be more cost-effective for direct policy and governance students?
3. Would short, 6 hour bootcamps (for instance Bluedot's 6 hour operations bootcamp experiment) be an 80:20 solution to the signalling issues?
4. Does organisational structure need to improve to accept more new hires? What could the role of Consultants be in improving organisational structure?

Bycatch; Addendum

Epistemic Status: Low. These are some personal reflections I've had whilst working on this.

With acceptance rates of <5%, the AI Safety fellowships cumulatively receive more applications every year than the number of people working in AI Safety altogether. This has been referred to as ‘Bycatch’^[13].

Perhaps selective non-research pipelines would not solve this, but it seems crazy to me that in such a neglected space, we would not be trying to incorporate as much talent as possible.

A chosen few get the privilege of working on potentially the most important problem in history.

When I raised this point, one person said to me, ‘Well there’s simply not enough space for everyone; and there’s always Earning to Give’.

When I hear the term ‘Bycatch’, I’m reminded of the famous Albert Einstein quote:

If you judge a fish by its ability to climb a tree, it will live its whole life believing that it is stupid.

1. ^{^}

    For this article, I’m defining AI Safety as including both Technical AI Safety and AI Governance.

2. ^{^}

    Tarbell, Talos and IAPS.

3. ^{^}

    For this article, I’m defining AI Safety as including both Technical AI Safety and AI Governance.

^{^}

 Tarbell, Talos and IAPS.

^{^}

 For this article, I’m defining AI Safety as including both Technical AI Safety and AI Governance.

^{^}

 Tarbell, Talos and IAPS. You could include Horizon and RAND’s programs in these buckets, but these are not explicitly focused on X-risk, nor do they identify with the AI Safety movement.

^{^}

 For this article, I’m defining AI Safety as including both Technical AI Safety and AI Governance.

^{^}

 Tarbell, Talos and IAPS. You could include Horizon and RAND’s programs in these buckets, but these are not explicitly focused on X-risk, nor do they identify with the AI Safety movement.

^{^}

 For this article, I’m defining AI Safety as including both Technical AI Safety and AI Governance.

^{^}

 A small number say that AI Safety still needs more extremely-tail case researchers. But in terms of numbers of researchers, very few said that this was the biggest bottleneck.

^{^}

 Less confident about this one.

^{^}

 Tarbell, Talos and IAPS. You could include Horizon and RAND’s programs in these buckets, but these are not explicitly focused on X-risk, nor do they identify with the AI Safety movement.

^{^}

 https://forum.effectivealtruism.org/posts/2BEecjksNZNHQmdyM/don-t-be-bycatch

Show all footnotes