Comments
Hi, Forum! We just published this career review on some of the most important and hard-to-fill roles at fast-growing organisations working to make AI go well. If you have suggestions to improve the piece, we'd be happy to hear them.
Read this career review on the 80,000 Hours website.
In a nutshell: Many organisations working on making AI go well plan to grow rapidly in the coming years, driven by the urgency of the problems humanity faces and the significant increase in philanthropic funding available to solve them. These organisations often struggle to fill senior management and operations roles like chief operating officers, chiefs of staff, recruiting leads, and programme managers. If you have experience managing teams or complex projects, especially at a growing startup or nonprofit, you could lead and execute projects to bring about a better future.
If you have strategy and management experience in a fast-growing startup or nonprofit, or in consulting, finance, tech, or recruiting — and you have deep AI safety knowledge and motivation — you might be a great fit. To help scale these organisations you need to understand their goals, reason in a transparent and analytical way about how they can achieve them, and be open to others’ viewpoints on how to solve problems.
Frontier AI companies have scaled their revenues, headcounts, and AI capabilities at a staggering rate in recent years. Since 2023, OpenAI and Anthropic have grown their staff by roughly 5–10x, revenue by over 30x, and AI models from simple chatbots to autonomous agents capable of expert-level cyberattacks and software engineering.
If progress continues at this pace — or is accelerated by powerful AI systems automating AI research and development itself — the world will not be prepared. Few government, nonprofit, or private-sector institutions are equipped to handle the risks of transformative AI.
To keep up, many AI safety, security, and governance organisations will have to scale at a similar pace. Some philanthropists hope to deploy tens of billions of dollars in the coming years to address these urgent threats. To use this much funding well, organisations currently working on these issues will need to plan and execute ambitious projects — which in turn means professionalising and growing their teams.
However, these organisations’ growth is already bottlenecked by a lack of senior management and operational leadership talent. In two AI safety talent surveys (one by us and another by a grantmaker), the positions organisations said they most struggled to fill were in senior management and operations: chief operating officers, chiefs of staff, directors of operations, and senior programme or people managers.
For the AI safety field to rise to the challenge humanity is facing, it will need seasoned executors with knowledge of the field and good strategic judgement. If you have experience in a senior position scaling nonprofits or startups, we strongly encourage you to apply for advising or to jump straight to the following sections:
This career review covers a broad category of jobs focused on scaling the size and impact of AI safety organisations. Experts we’ve interviewed identified these jobs as significant bottlenecks right now: they report needing applicants who can manage people, build scalable systems, multiply others’ outputs, and reason well about strategy. Senior generalists need to think in a transparent and open-minded way to help make and execute organisational strategy decisions.
Here are a few examples of positions that use this skill set:
Chief operating officer
A COO works alongside other senior leaders to make an organisation’s strategic decisions, then builds the systems that turn those decisions into reality. They’re typically responsible for internal management structures, establishing a team’s culture, and setting up processes for hiring, compensation, events, etc. In short, a COO multiplies the output of the entire organisation and makes growth possible.
This job requires especially strong analytical reasoning and strategic judgement. As one expert told us, it often resembles what strategy consultants are brought in to do in other industries — figure out what’s happening, how things work, and how to improve processes — but the COO stays to lead the implementation.
What this looks like in practice depends heavily on the organisation’s size. In smaller teams, COOs may take on hands-on projects like leading hiring or running events — though ‘hard ops’ like payroll and taxes are almost always delegated to staff or external consultants. As an organisation grows, the COO might hire a director of operations or other direct reports to manage and oversee specific functions.
Chief of staff
A chief of staff’s responsibilities depend heavily on the principal they work for (usually a CEO or other executive) and the needs of the organisation. However, all chiefs of staff work to enable leaders to do more and higher quality work.
Some chiefs of staff work across their principal’s portfolio — sitting in on most meetings and decisions, owning their inbox, and acting as an executor or thought partner across many projects. Others specialise, freeing up their principal’s time by owning a section of the portfolio themselves. This might involve directly managing some employees, running a hiring round, leading the CEO’s communications, etc. A chief of staff aims to eventually lead these projects with minimal supervision, deeply understanding the way their principal thinks and how they’d handle a given situation.
On either end of this delegation spectrum, the value is the same: the opportunity cost of leadership time is enormous, and a good chief of staff can dramatically free up a leader’s time or multiply their output. One chief of staff told us they felt their work was impactful not just because they led an important hiring round and drafted public communications for their CEO, but because this allowed their CEO to focus on more urgent strategic issues. All the roles described in this career review have this ‘multiplier’ mindset, but it’s a particularly important quality for chiefs of staff.
Talent director or recruiting lead
As the thousands of roles we post on our job board each year demonstrate, finding new talent is a big task for the field of AI safety. Several leaders in the field told us that they need expert recruiters to help grow their work. A strong recruiter builds deep networks in AI safety, develops good taste about who would excel in a role and why, oversees active outreach to promising candidates, and serves as the project manager for hiring rounds.
In AI safety, a recruiting lead needs to be especially knowledgeable about the organisation’s work and its priorities. They need to know how to identify talented candidates, including how their views line up with the organisation’s mission. They also need to notice when an otherwise promising candidate might not be a good fit, which can be hard to recognise and requires working closely with staff to design hiring criteria.
This work also requires applied strategic decision-making. Skilled recruiters are constantly building mental models of candidates and the field as a whole, mapping out where someone’s skills could be useful and how to acquire the information they need about potential candidates (such as designing and running a work test). Recruiters can also help many organisations outside of their own, by referring second-place candidates or noticing when a candidate might be well-suited for a different role elsewhere. If you care a lot about AI safety and have strong interpersonal skills, you could make a sizable impact by helping an organisation build their team.
Programme or research manager
Programme managers design and run structured programmes like events, workshops, publications, or fellowships. They research and scope out new projects end-to-end, proposing a strategy and success criteria, as well as how they’ll track their impact over time. Programme managers often work with in-house or external experts to design new initiatives. Then, they execute. One expert told us they have a much harder time finding good programme managers than researchers because fewer candidates are interested.
As we’ve covered elsewhere, the growth of the AI safety field has led to an increased demand for research managers to support full-time researchers and fellows. They plan and oversee research projects, set goals, motivate others, assist in logistics and project management, review papers, and network with researchers across the field. Research managers typically need some knowledge of the research behind the projects they oversee, but not to the same depth as the researchers themselves.
There is significant and growing demand for these roles.
As mentioned above, in two recent surveys of hiring managers in AI safety, senior management and operations roles were identified as the most important bottleneck to growth. We suspect this will be even more true in the future, as philanthropists hope to move tens of billions of dollars towards efforts to make the future of AI go well for humanity, presenting organisations with many opportunities to scale. We know of at least one AI safety nonprofit that plans to double its headcount over the next year (from about 40 to about 80 staff).
These roles can multiply an entire team’s output.
Strategic decisions like salary or hiring policies can shape an organisation’s ability to recruit and retain top talent. A great manager can bring out the best in their reports, helping them meet clear goals and get unstuck. A well-run organisation can absorb new resources and turn them into results, while a poorly run organisation squanders them. Even very talented people sometimes get little done because they work somewhere that isn’t well-organised or pursuing sensible goals. One AI safety expert told us that the field has entered its “mid-game,” where the ability to execute on specific projects has become more urgent.
You’ll likely find intellectual stimulation and growth opportunities.
These roles put you close to senior leadership, letting you strategise alongside interesting thinkers and develop a deep understanding of the field — one you can use to shape AI’s chaotic and transformative impact on the world. Success in these positions requires making judgement calls under uncertainty, reasoning about tradeoffs, and quickly updating on the real-world results of your actions.
And because these roles are relatively neglected, you can also build a world-class skill set very quickly — “how do I run one of the world’s top AI safety events?” or “how do I incorporate AI progress forecasts into our annual growth plans?” are paths to mastery that few others are considering.
We often hear from readers asking how much AI safety knowledge they need for these roles, and why it’s so essential for getting hired.
Experienced professionals are (understandably) sometimes confused or frustrated when they apply for an AI safety job that fits their background, but are rejected early in the process. They may have written a few short answer responses in the application or had an interview that tested their knowledge of and interest in AI safety, only to get a vague rejection. Often, they insufficiently communicated an understanding of the organisation’s priorities, or knowledge about AI safety in general. This is sometimes referred to as having a lack of ‘context.’
We think there are (at least) three reasons why hiring managers often prioritise preexisting AI safety knowledge and motivation in their senior operations and strategy hires:
In practice, applying this knowledge could look like: deciding which organisations or media outlets to partner with to achieve a shared goal; designing and executing a fundraising strategy for potential donors interested in AI safety; leading high-touch hiring rounds for top talent; or determining what the organisation’s risk tolerance should be for concerns like public relations, opportunity costs of moving too slowly, budgeting given your organisation’s estimates around AI progress, etc.
Of course, the kind of knowledge and motivation required will differ across organisations. Some organisations have detailed worldviews and theories of change that affect all of their work (MIRI, for example), while others work across many research areas and have less of a defined ‘house view’ (CSET, for example). The organisations on our job board have different approaches to making AI go well, so we’d encourage you to read their websites carefully before applying. In addition, the more well-scoped a position with an organisation is, such as building HR or IT systems, the lower the bar will typically be for this.
Beyond knowledge and motivation, many AI safety organisations also highly value candidates with strong analytical reasoning skills. This is partly a general skill that exists in other industries — one expert recommended Jeff Bezos’ annual shareholder letters, the New England Journal of Medicine‘s “Clinical Problem Solving” series, or books like Good Strategy, Bad Strategy as examples of sharp analytical reasoning.
But AI safety organisations often prioritise a particular kind of analytical reasoning. You may be asked to express arguments in a probabilistic, transparent, and open-minded way. For example, this post by Trevor Levin demonstrates a method of probabilistic and analytical reasoning to plan for (part of) AI’s future impact. AI safety organisations often work on neglected problems with few feedback loops, which makes it harder to learn from similar past work or iterate, so doing this analysis early and throughout a project is important.
Otherwise, the required skills are:
Hiring senior generalists in AI safety has somewhat of a ‘chicken and egg’ problem. Recruiters want to find candidates who can demonstrate their ability to make good strategic decisions in applied AI safety contexts, but there are few ways to practice and showcase this skill from outside the field.
For this reason, trusted referrals and in-person connections at conferences are often used to identify candidates. But experienced professionals also have a high financial opportunity cost for their time, and can be busy with existing careers, families, or other responsibilities — making it difficult to invest time in conferences, part-time side projects, courses, etc.
Experienced professionals often need an efficient way to learn about the field and get connected in a few hours, rather than a few weeks. For this reason, we offer career advising calls to help you fast-track your AI safety career. Our team is particularly excited to speak with people who have the backgrounds described in this post. We’ve helped many mid-career professionals pivot into AI safety over the years.
This review has mostly focused on professionals with several years of management and strategy experience. However, junior readers should also be optimistic about considering this path.
Because AI safety organisations struggle to hire seasoned professionals who deeply understand their worldview and thinking, many have prioritised training and promoting their early-career employees into senior positions rather than hiring externally. One expert told us they would sacrifice several years of management experience for a candidate that could clearly stand in for their organisation’s leaders in matters of strategic taste, like designing a budget given short AI timelines, or communicating ideas to important stakeholders.
Great junior operations staff can often “eat their boss’s job,” as MIRI’s CEO Malo Bourgon said in our previous operations management career review. That review is a great place for more junior operations candidates to start learning about how to get into and excel in the field.
Our job board features opportunities for scaling AI safety organisations.
If you think this path might be a great option for you, but you need help deciding or thinking about what to do next, our advising team might be able to help you compare options, make connections, and possibly even help you find jobs or funding opportunities.
Our advisors sometimes describe this as an “impact sabbatical,” where you agree to take a lower salary for a few years before returning to the private sector. If you believe the development of transformative AI models is possible in the next few years, this might be one of the most important times in human history to work directly on these problems. We’d hate to see great talent sitting at their desks watching the world change outside their window, rather than working to positively influence it themselves.
