Context ought to be defined in terms of clear knowledge requirements. Things that can be documented, taught and assessed if needed. I expect much of what someone needs to know to do their job well could probably be learned in anywhere from a few hours to a few weeks of focused onboarding.
If that's not true, and orgs really do need people with years of movement-specific experience, they should say so. Instead, there’s a huge ecosystem of career advice, fellowships, introductory programmes and recruitment campaigns encouraging outsiders to consider AI safety careers.
I think these programmes are creating expectations that the actual hiring market cannot satisfy. Better to leave it to the headhunters if you ask me, and re-allocate the funding behind those programmes to GHD. Hard to justify why another talent-sourcing agency for AI safety is needed, from a cost effectiveness standpoint, when there are still bednets need distributing.
'Context' seems to be the latest iteration of 'culture fit' or 'mission alignment'. Another nebulous term that EA orgs use to explain why they keep reeling in hundreds, sometimes thousands, of applicants who basically don’t stand a chance.
The burden of proof should be on orgs invoking context as a hiring requirement, not on applicants to demonstrate possession of that indefinable, je ne sais quoi quality of context.
Thanks Gergo, this and your previous post / guide on building context are both really valuable.
I agree with your Venn diagram - there's plenty of very talented and experienced people around, with varying levels of context. Developing parts of that context is possible to do autonomously - eg following the steps in your earlier guide. But developing the networks, the sense of the field (who's doing what / where) is not easy, especially for people who are not located in a hub.
I agree that hiring organisations seem to want high-context for most roles, including operations roles, whether or not this is needed. It seems like this limits your pool of potential candidates - if you want a rock star head of finance, who also has high context, who is also available to work from your location or be hired from your location. When organisations cant find these unicorns, they're either leaving the roles unfilled, or hiring a person with less experience but 'potential to grow into the role'. I've previously commented that there's an opportunity cost to both of these options.
I've been on both sides of recruiting and job-seeking, so I know it's not easy for anyone. I would like to see more programmes like the Successif organisation builder's programme, which provide a way for experienced people to build context through paid part-time work, while also helping organisations that are trying to scale and deliver.
Not my area, but a well written concise post, with great explanation. How right you are I do not know (Not my area again) but I appreciate clear arguments that can easily be engaged with.
Hi Gergő! I believe there is an important misunderstanding when we say AIS talent needs more context.
When hearing this, most candidates believe they need more "technical" or "existential-risk" context (leading them to apply for more BlueDot style courses or workshops). But, is it that so? Or do they need more "field" context?
Are organizations looking for candidates to signal more awareness about AIS urgency? Or rather, they are expecting talent to signal values alignment and knowledge of AIS and adjacent-fields (EA frameworks, network connections, current AIS organizations needs and bottlenecks, etc.)?
In other words, are AIS organizations looking for capable and informed talent about AIS dangers, or are they prioritising talent they can trust and informed about the field particular needs?
These are very different context requirements, and both can be important. But maybe, we are failing to properly inform and guide talent about the latter.
I also believe an important constraint in AIS is the scarcity of organizations to absorbe promising talent and funding available. Maybe that is also the reason why is getting harder to enter the field as an outsider. Should we focus on building more high-impact AIS organizations instead of demanding and increasing higher bar for talent?
I have experienced both sides of the candidate and hiring manager equation, and I know there is not a clear answer to all this. But just telling promising talent to have more "context" without proper guidance and nuances seems not being enough.
Thanks for sharing this! I didn't think of people misunderstanding what "we" mean by context, but if it leads to taking technical courses unnecessarily, that sounds like a problem indeed. What I mean by it is outlined here, which is where I usually send people, including in this post. It gives people quite a lot of pointers on what to do!
I think there's two things that make candidates more attractive that genuinely do take time to develop:
Network: at this point, a lot of the value I bring to any organisation is my network. I know a lot about people and organisations. And when I onboard new people, I spend a lot of time just explaining the "who's who" and facilitating intros.
Language / comms: AI safety/ EA has a particular way of communicating, and for people coming across from other sectors, they do kinda need to learn ways of communicating to be taken seriously. E.g. simple things like using the term "org" rather than "company" / "business". I remember the transition when I moved from government to work in an EA org, e.g. I was confused when "costings" were just referred to as a "budget"
Talent bottleneck:
I think it's incorrect to say that there's a shortage of talented people: by definition, there is a shortage. E.g. We often joke that the best thing for AI safety would be a cloning machine to clone the best people.
The talent shortage is particularly relevant when you account for the fact that a majority of roles are only available in two locations: London and the Bay Area. This excludes a lot of candidates.
I've noticed that the bar for being hired into an EA org in London is way lower when I compare it to Melbourne/Sydney, because the demand for talent is so much higher in London.
As someone who is currently hiring (and to put this bluntly), it can be difficult to find candidates who can follow instructions and/or produce a document that wasn't written by Claude. The bar is not high and I am somewhat confused about how the economy functions at this point.
Particularly for the first 3-5 hires, where you need to have a very high bar for talent in order to set the right trajectory for the organisation ... it's just not easy.
I think we’re mostly using “talent” differently here. If it means the whole package such as skills, experience, context, networks and being in the right location: then yes, there’s a shortage of people who tick all those boxes. My point is that “AI safety is talent-constrained” can sound to newcomers like broadly capable people are in short supply. I also don't envy you on the hiring side, though with all the AI-applications one gets these days:)
Great post, Gergo! I have noticed the same thing about people not being able to get into a BlueDot course, which used to be the lowest bar to clear into AI safety. Interestingly, despite having context and facilitation experience, I also wasn't accepted as a facilitator, which you would expect is the main bottleneck to getting more course participants (although this could just be that my availability didn't match the demand of participants).
I will also say that, as someone who's full time into AI safety, it's still hard for me to keep up with the context! I know this could be solved by asking any AI agent to get me a summary of the latest news, but sometimes I don't have time even for that (:
Quick typo note on the last paragraph: "...which makes them hard to fill even when fill even when funding..." should probably read "...which makes them hard to fill even when fill even when funding..."
At this point, it feels like the bottleneck being "talent" vs. "context" or anything else is just semantics.
Talent that has years of experience but doesn't have the specific knowledge I want is not the talent I need. I need a different talent. Therefore, it's still a talent bottleneck.
Maybe it would be better if AI safety orgs could be extremely more exact about what talent means to them, but that also leaves the field fairly open to a type of gaming a lot of people seem to want to avoid.
The field does act like a false positive hire is immensely worse than a false negative, though, which might not be the case.
I don't have a point, these are just random observations.
I agree that what you describe is technically still a talent constraint, but this nuance is mainly lost in translation, which is why we have this issue.
Maybe it would be better if AI safety orgs could be extremely more exact about what talent means to them, but that also leaves the field fairly open to a type of gaming a lot of people seem to want to avoid.
Terms of reference should say what experience and competencies are needed for a role. I dont see a downside with being more precise about this - ie these are mandatory things we're looking for, and these are desirable competencies. The more general a job advertisement is, the more candidates will apply, which creates unnecessary workload for the hiring organisation, and wastes the time of candidates who realistically were never competitive.
The field does act like a false positive hire is immensely worse than a false negative, though, which might not be the case.
Agreed. There's a real aversion to hiring 'the wrong person'. But if you hire enough people, you'll eventually make a bad hire. Hiring / onboarding / probation / performance management should be seen as an ongoing process to get the best candidate and make sure they're working effectively, rather than relying 100% on the hiring stage.
As I've mentioned above, there are opportunity costs when leaving a position vacant, or hiring a more trusted but less experienced candidate and hoping they will 'grow into the role'. It seems like these costs are frequently underweighted.
This post was inspired by my friend Jen’s quick take. I’m repeating some of her points, but I wanted to add my 2 cents. I’m also going to toot my own horn and refer you to a previous post of mine about the importance of context and how to build it. I recommend reading that first if you haven’t already.
The Bottleneck Paradox
We have been hearing a lot about the talent bottleneck in AI Safety. In some ways this is true, but in others it’s misleading and leaves many people disillusioned and doubting whether they have anything to contribute to AI Safety.
Meanwhile, the field is becoming harder to penetrate. Some reasons for this are:
Organisations are better funded than they were a few years ago, which means they are less keen on working with volunteers and just want to hire full-time staff.
Many of the “oldtimers” who could give the best career advice are now quite senior and oversee a larger number of people, which significantly raises the bar for talking to newcomers.
The field has grown a lot, and many roles require more specialised AIS knowledge. There is more to read, and if you are new, it’s not easy to figure out which sub-area you should focus on.
Hell, I run into people with pretty impressive backgrounds, but they can't even get into a BlueDot course.
Organisations are desperate to hire, but mainly for roles that require both many years of work experience and the high levels of context, which you build by having been around for years in the space. Here is a comment from Tom, who is a senior advisor at Probably Good.
A slight qualifier here is that getting to the level of context required for some jobs - especially senior ones that experienced professionals might be applying to - can take (sometimes much) longer, so it's important to have realistic expectations there. For instance, if you want to work in AI safety, and have a background (e.g. quantitative finance, venture capital) that could give you great skills to be a grantmaker, you'll likely still need to know more than just the high-level concepts and the landscape of organisations working on it; you might need to know the strengths and weaknesses of different theories of change, and have a sense of the wider funding landscape.
The people who can fill those roles are few and far between, and usually have already been working in AI Safety for years. If they become available, the headhunters are on their tails.
Some roles don’t need as much context, but:
If there are more openings (e.g. operations[1]), they get a large number of skilled applicants
If they require a more niche background (e.g. marketing), then there aren’t as many opportunities available
Conclusion
Instead of saying AIS is bottlenecked by talent, I would like us to offer a more nuanced explanation, which looks something like this:
Some AI Safety roles demand a specific skill set, deep context, and years of work experience, which makes them hard to fill even when funding and talent aren’t constraints. Other roles are much more competitive, drawing many applicants since they don't require candidates to already have been around in AI Safety for a long time.
This is an insider reference to an article widely read in the EA movement. Check it out if you want to build some context. :)
Crossposted to Less Wrong
‘Like most of life’s dangers, the dangers of artificial intelligence are not easily identified from the armchair.’
J Dmitri Gallow
Epistemic status
I’m not an expert in this field, and herein I make some sweeping generalisations educated guesses about the overall...
Note: This post was crossposted from Planned Obsolescence by the Forum team, with the author's permission. The author may not see or respond to comments on this post.
Subtitle: If acceleration compounds, the naive extrapolation implies superintelligence within a year
This post discusses a public METR report on AI R&D acceler...
TL;DR:
* When we started Rethink Priorities (RP) in 2018, our mission was, simply put, to close the knowledge gaps that stopped people from improving the world as much as they could. That hasn’t changed much, but the landscape we’re doing it in is shifting.
* In 2027, we plan to grow our core GHD, Animal Welfare, and Worldview Investigations teams to take on emerging priorities across causes, including those that the forthcoming transformative AI is creating.
...
Context ought to be defined in terms of clear knowledge requirements. Things that can be documented, taught and assessed if needed. I expect much of what someone needs to know to do their job well could probably be learned in anywhere from a few hours to a few weeks of focused onboarding.
If that's not true, and orgs really do need people with years of movement-specific experience, they should say so. Instead, there’s a huge ecosystem of career advice, fellowships, introductory programmes and recruitment campaigns encouraging outsiders to consider AI safety careers.
I think these programmes are creating expectations that the actual hiring market cannot satisfy. Better to leave it to the headhunters if you ask me, and re-allocate the funding behind those programmes to GHD. Hard to justify why another talent-sourcing agency for AI safety is needed, from a cost effectiveness standpoint, when there are still bednets need distributing.
'Context' seems to be the latest iteration of 'culture fit' or 'mission alignment'. Another nebulous term that EA orgs use to explain why they keep reeling in hundreds, sometimes thousands, of applicants who basically don’t stand a chance.
The burden of proof should be on orgs invoking context as a hiring requirement, not on applicants to demonstrate possession of that indefinable, je ne sais quoi quality of context.