Thanks Gergo, this and your previous post / guide on building context are both really valuable.
I agree with your Venn diagram - there's plenty of very talented and experienced people around, with varying levels of context. Developing parts of that context is possible to do autonomously - eg following the steps in your earlier guide. But developing the networks, the sense of the field (who's doing what / where) is not easy, especially for people who are not located in a hub.
I agree that hiring organisations seem to want high-context for most roles, including operations roles, whether or not this is needed. It seems like this limits your pool of potential candidates - if you want a rock star head of finance, who also has high context, who is also available to work from your location or be hired from your location. When organisations cant find these unicorns, they're either leaving the roles unfilled, or hiring a person with less experience but 'potential to grow into the role'. I've previously commented that there's an opportunity cost to both of these options.
I've been on both sides of recruiting and job-seeking, so I know it's not easy for anyone. I would like to see more programmes like the Successif organisation builder's programme, which provide a way for experienced people to build context through paid part-time work, while also helping organisations that are trying to scale and deliver.
Context ought to be defined in terms of clear knowledge requirements. Things that can be documented, taught and assessed if needed. I expect much of what someone needs to know to do their job well could probably be learned in anywhere from a few hours to a few weeks of focused onboarding.
If that's not true, and orgs really do need people with years of movement-specific experience, they should say so. Instead, there’s a huge ecosystem of career advice, fellowships, introductory programmes and recruitment campaigns encouraging outsiders to consider AI safety careers.
I think these programmes are creating expectations that the actual hiring market cannot satisfy. Better to leave it to the headhunters if you ask me, and re-allocate the funding behind those programmes to GHD. Hard to justify why another talent-sourcing agency for AI safety is needed, from a cost effectiveness standpoint, when there are still bednets need distributing.
'Context' seems to be the latest iteration of 'culture fit' or 'mission alignment'. Another nebulous term that EA orgs use to explain why they keep reeling in hundreds, sometimes thousands, of applicants who basically don’t stand a chance.
The burden of proof should be on orgs invoking context as a hiring requirement, not on applicants to demonstrate possession of that indefinable, je ne sais quoi quality of context.
Agree, there is a mismatch in communication between what AIS organizations actually need and what they are telling candidates. And although I consider necessary the talent infrastructure AIS is building, it is quite inefficient and problematic as I discussed here.
The main concern I see: fellowships and most training programs are focused on "low-hanging fruits". They are advertised as ways to bring experienced talent into the field, but rather they tend to select people who either would have less problems to enter a job, or people who were already in the field (I analysed the fellows backgrounds of famous fellowships such as GovAI or MATS and I can confidently say this is the case).
And if we are focused on just draining the talent pool of already best-suitable talent for the sake of short term cost-effectiveness or immediate appealing metrics... AIS is going to have a chunk of a problem in a few years without good talent available.
Strong disagree. A language is a way of life, not a set of memorised answers. I agree that it would be nice if there was some more honesty about what exactly talent-constrained means. I like how this post clarifies it.
One of the specific items that seems to be attracting some heat is the "if you're not sure this is for you, apply anyway" type lines on applications. That seems like a cheap one to fix - my interpretation of this one has historically been about credentialling: 'if you don't have the credential you think you might need, you might be surprised'. So maybe we could just come up with a new boilerplate paragraph that better communicates what is actually meant here.
Not my area, but a well written concise post, with great explanation. How right you are I do not know (Not my area again) but I appreciate clear arguments that can easily be engaged with.
I don't think AI Safety fails is the ideal kind of infrastructure we should have in place for people who are getting rejected, but I'm also not suprised someone eventually made it.
I wish there was a place where rejectees got to interact and provide peer support to each other, whether or not they are looking to puruse ai safety career further.
(Assuming the rejections are real, which I assumed they were because they sound familiar, but the poster below made me question it) This is brilliant! Every person who's thinking of attempting the pivot should see this first and think on how much time/effort they want to invest in it. It tells a completely different story to the 'we desperately need talent, please, apply even if you think it isn't you' claptrap.
• the rejections are split between comments taken from places like this very forum(we had to start somewhere), and increasingly as time progresses, people submitting them individually. the only reason it hasn't gotten more traction is because we'd rather it grow on it's own than grow through us marketing it super heavily.
• they are vetted about as much as anyone posting the same issues on this very forum are vetted. highly doubt people are taking the time required to fill out the forums to list a bunch of made up feaux-denials/lack of response stories. it's a lot of work and to what end? the purpose is to make a point and it seems like it's doing okay in that regard.
• it is not parody nor meant as a parody. the definition, or one definition of parody, is "a feeble or ridiculous imitation". nothing is being imitated here. everything presented is legitimate and meant to call out the absurdity of the space, the 'dance monkey dance' processes they put people through, with no leg up given to help anyone who can't dance. much less people who can but just aren't quite professionals at it yet.
• theoretically, one could call it satirical. satire is the greatest form of protest. it is the same form of protest that got kimmel and colbert taken off air.
"A satirist is a man profoundly revolted by the society in which he lives. His rage takes the form of wit, ridicule, mockery."
• to say the (very, in some cases over-the-top) qualified people trying to get into this space and not succeeding, are becoming increasingly outraged, would be an understatement.
• not sure we're trying to help support candidates/applicants. what more can possibly be done? this site was created in a fit of frustration and a need to express it somehow someway and this is what came of it.
• *with all respect that can possibly be expressed in the confines of an anonymous forum post* the fact that someone of your caliber(active CEO in EA/AIS space) is able to look through these, and wishfully think / surmise / make an assumption that they are a parody, speaks volumes in and of itself, about how self-aware the space is about how they treat candidates.
respectfully,
an individual who applied to manifund, got an autoreply-rejection, and tbqh, in thinking about it, that didn't even pass the bar to be submitted to the site once it was running lol :)
hey! thanks for clarifying. once again, I'd recommend making the sources of these things clear, because otherwise it might seem plausible to a reader that this is completely just a joke site. (i think there's still some strength in being a joke/satire site, but your purpose is more evident if it conveys that these are real people, imo.)
you could just start by linking to the original EA Forum comment for the ones that came from there!
re: "parody", I'm not sure I used this term correctly -- you're probably right that "satire" is a better fit for what I had in mind, apologies for my imprecision). to be clear, even before your message I was expecting that these were originating from real examples, as I stated in my original comment.
I was trying to point out that your site's design makes this unclear, eg because of the slop-iness of the specific language used in the website copy, as well as the "quotes" which to my eye, also seem like summarized LLM-ese.
finally re: Manifund, I'm curious if you can say more about autoreply-rejection? Manifund, unlike traditional funds like CG/LTFF/etc, is just a platform where people list their proposals; we make no guarantees about reviewing or replying to any listed proposal. (in this way, it's similar to the EA Forum, and also similar to aisafety.fail itself). so I'm not sure what you meant by an autoreply-rejection.
(we do flag LLM-written proposals and hide them from the feed until they receive a certain amount of funding. is this perhaps what you meant?)
Hi Gergő! I believe there is an important misunderstanding when we say AIS talent needs more context.
When hearing this, most candidates believe they need more "technical" or "existential-risk" context (leading them to apply for more BlueDot style courses or workshops). But, is it that so? Or do they need more "field" context?
Are organizations looking for candidates to signal more awareness about AIS urgency? Or rather, they are expecting talent to signal values alignment and knowledge of AIS and adjacent-fields (EA frameworks, network connections, current AIS organizations needs and bottlenecks, etc.)?
In other words, are AIS organizations looking for capable and informed talent about AIS dangers, or are they prioritising talent they can trust and informed about the field particular needs?
These are very different context requirements, and both can be important. But maybe, we are failing to properly inform and guide talent about the latter.
I also believe an important constraint in AIS is the scarcity of organizations to absorbe promising talent and funding available. Maybe that is also the reason why is getting harder to enter the field as an outsider. Should we focus on building more high-impact AIS organizations instead of demanding and increasing higher bar for talent?
I have experienced both sides of the candidate and hiring manager equation, and I know there is not a clear answer to all this. But just telling promising talent to have more "context" without proper guidance and nuances seems not being enough.
Thanks for sharing this! I didn't think of people misunderstanding what "we" mean by context, but if it leads to taking technical courses unnecessarily, that sounds like a problem indeed. What I mean by it is outlined here, which is where I usually send people, including in this post. It gives people quite a lot of pointers on what to do!
At this point, it feels like the bottleneck being "talent" vs. "context" or anything else is just semantics.
Talent that has years of experience but doesn't have the specific knowledge I want is not the talent I need. I need a different talent. Therefore, it's still a talent bottleneck.
Maybe it would be better if AI safety orgs could be extremely more exact about what talent means to them, but that also leaves the field fairly open to a type of gaming a lot of people seem to want to avoid.
The field does act like a false positive hire is immensely worse than a false negative, though, which might not be the case.
I don't have a point, these are just random observations.
I agree that what you describe is technically still a talent constraint, but this nuance is mainly lost in translation, which is why we have this issue.
Maybe it would be better if AI safety orgs could be extremely more exact about what talent means to them, but that also leaves the field fairly open to a type of gaming a lot of people seem to want to avoid.
Terms of reference should say what experience and competencies are needed for a role. I dont see a downside with being more precise about this - ie these are mandatory things we're looking for, and these are desirable competencies. The more general a job advertisement is, the more candidates will apply, which creates unnecessary workload for the hiring organisation, and wastes the time of candidates who realistically were never competitive.
The field does act like a false positive hire is immensely worse than a false negative, though, which might not be the case.
Agreed. There's a real aversion to hiring 'the wrong person'. But if you hire enough people, you'll eventually make a bad hire. Hiring / onboarding / probation / performance management should be seen as an ongoing process to get the best candidate and make sure they're working effectively, rather than relying 100% on the hiring stage.
As I've mentioned above, there are opportunity costs when leaving a position vacant, or hiring a more trusted but less experienced candidate and hoping they will 'grow into the role'. It seems like these costs are frequently underweighted.
Mmmm I think orgs should probably just be more honest with themselves as well - have the very specific criteria, sure, but just be honest about them. Like: if you're really only going to hire a unicorn, then just say unicorns only - that seems fine to me? And then we can just move on to arguing about the object-level question of do you actually need a unicorn.
I think one of the challenges as organisations scale rapidly is that organisations are hiring for positions that they'd never had before - eg they are hiring for a Head of talent, Head of Finance, Head of Operations. Since they've not had those roles before, there are two issues:
It's hard to know what the role should do, and related to this, what are the mandatory and desirable requirements.
The organisation doesnt necessarily have much experience in sceening and assessing candidates with expertise in that role. So the organisation might overweight or underweight certain selection criteria or candidate responses; or might default to the established methods and criteria of assessing candidates that they are comfortable with, even if this is not a strong fit for for the specialised role.
I think there's two things that make candidates more attractive that genuinely do take time to develop:
Network: at this point, a lot of the value I bring to any organisation is my network. I know a lot about people and organisations. And when I onboard new people, I spend a lot of time just explaining the "who's who" and facilitating intros.
Language / comms: AI safety/ EA has a particular way of communicating, and for people coming across from other sectors, they do kinda need to learn ways of communicating to be taken seriously. E.g. simple things like using the term "org" rather than "company" / "business". I remember the transition when I moved from government to work in an EA org, e.g. I was confused when "costings" were just referred to as a "budget"
Talent bottleneck:
I think it's incorrect to say that there's a shortage of talented people: by definition, there is a shortage. E.g. We often joke that the best thing for AI safety would be a cloning machine to clone the best people.
The talent shortage is particularly relevant when you account for the fact that a majority of roles are only available in two locations: London and the Bay Area. This excludes a lot of candidates.
I've noticed that the bar for being hired into an EA org in London is way lower when I compare it to Melbourne/Sydney, because the demand for talent is so much higher in London.
As someone who is currently hiring (and to put this bluntly), it can be difficult to find candidates who can follow instructions and/or produce a document that wasn't written by Claude. The bar is not high and I am somewhat confused about how the economy functions at this point.
Particularly for the first 3-5 hires, where you need to have a very high bar for talent in order to set the right trajectory for the organisation ... it's just not easy.
I think we’re mostly using “talent” differently here. If it means the whole package such as skills, experience, context, networks and being in the right location: then yes, there’s a shortage of people who tick all those boxes. My point is that “AI safety is talent-constrained” can sound to newcomers like broadly capable people are in short supply. I also don't envy you on the hiring side, though with all the AI-applications one gets these days:)
Great post, Gergo! I have noticed the same thing about people not being able to get into a BlueDot course, which used to be the lowest bar to clear into AI safety. Interestingly, despite having context and facilitation experience, I also wasn't accepted as a facilitator, which you would expect is the main bottleneck to getting more course participants (although this could just be that my availability didn't match the demand of participants).
I will also say that, as someone who's full time into AI safety, it's still hard for me to keep up with the context! I know this could be solved by asking any AI agent to get me a summary of the latest news, but sometimes I don't have time even for that (:
Quick typo note on the last paragraph: "...which makes them hard to fill even when fill even when funding..." should probably read "...which makes them hard to fill even when fill even when funding..."
An important point, Gergo. I also agree with the comments about unpacking “context”. As someone making this transition, I’d find it useful to know what kind of context a role needs, to what depth, and what can be learned after joining.
Understanding AI risks and the landscape is one layer; deeper knowledge of stakeholders, research agendas or ethical questions will depend on the role and organisation. Clearer expectations would help people direct their effort. It’s understandable that a growing field is still working this out, but losing capable people to confusion and disillusionment is a real risk.
Recently when talking to new people interested in "AI Safety" I’ve been feeling there’s less “alignment” in the sense of caring about x-risk or extinction level issues and working on them as obvious much higher than anything else priority.
This is related to context issue, it’s harder to understand these issues and come to care about them, takes context and time. I’d focus on that a lot re the talent bottleneck: talent that doesn’t get the motivation can’t upskill or work on the right things autonomously, so we need to get them understanding and caring about existential risks.
AI Safety is bottlenecked by context, not talent (BC #19)
This post was inspired by my friend Jen’s quick take. I’m repeating some of her points, but I wanted to add my 2 cents. I’m also going to toot my own horn and refer you to a previous post of mine about the importance of context and how to build it. I recommend reading that first if you haven’t already.
The Bottleneck Paradox
We have been hearing a lot about the talent bottleneck in AI Safety. In some ways this is true, but in others it’s misleading and leaves many people disillusioned and doubting whether they have anything to contribute to AI Safety.
Meanwhile, the field is becoming harder to penetrate. Some reasons for this are:
Organisations are better funded than they were a few years ago, which means they are less keen on working with volunteers and just want to hire full-time staff.
Many of the “oldtimers” who could give the best career advice are now quite senior and oversee a larger number of people, which significantly raises the bar for talking to newcomers.
The field has grown a lot, and many roles require more specialised AIS knowledge. There is more to read, and if you are new, it’s not easy to figure out which sub-area you should focus on.
Hell, I run into people with pretty impressive backgrounds, but they can't even get into a BlueDot course.
Organisations are desperate to hire, but mainly for roles that require both many years of work experience and the high levels of context, which you build by having been around for years in the space. Here is a comment from Tom, who is a senior advisor at Probably Good.
A slight qualifier here is that getting to the level of context required for some jobs - especially senior ones that experienced professionals might be applying to - can take (sometimes much) longer, so it's important to have realistic expectations there. For instance, if you want to work in AI safety, and have a background (e.g. quantitative finance, venture capital) that could give you great skills to be a grantmaker, you'll likely still need to know more than just the high-level concepts and the landscape of organisations working on it; you might need to know the strengths and weaknesses of different theories of change, and have a sense of the wider funding landscape.
The people who can fill those roles are few and far between, and usually have already been working in AI Safety for years. If they become available, the headhunters are on their tails.
Some roles don’t need as much context, but:
If there are more openings (e.g. operations[1]), they get a large number of skilled applicants
If they require a more niche background (e.g. marketing), then there aren’t as many opportunities available
Conclusion
Instead of saying AIS is bottlenecked by talent, I would like us to offer a more nuanced explanation, which looks something like this:
Some AI Safety roles demand a specific skill set, deep context, and years of work experience, which makes them hard to fill even when funding and talent aren’t constraints. Other roles are much more competitive, drawing many applicants since they don't require candidates to already have been around in AI Safety for a long time.
This is an insider reference to an article widely read in the EA movement. Check it out if you want to build some context. :)
Note: This post was crossposted from Planned Obsolescence by the Forum team, with the author's permission. The author may not see or respond to comments on this post.
Subtitle: If acceleration compounds, the naive extrapolation implies superintelligence within a year
This post discusses a public METR report on AI R&D acceler...
TL;DR:
* When we started Rethink Priorities (RP) in 2018, our mission was, simply put, to close the knowledge gaps that stopped people from improving the world as much as they could. That hasn’t changed much, but the landscape we’re doing it in is shifting.
* In 2027, we plan to grow our core GHD, Animal Welfare, and Worldview Investigations teams to take on emerging priorities across causes, including those that the forthcoming transformative AI is creating.
...
This post is part of the Rethink Priorities Cross-Cause Fund (CCF) series on how to allocate funding across causes when it is uncertain what matters most.
...
Thanks Gergo, this and your previous post / guide on building context are both really valuable.
I agree with your Venn diagram - there's plenty of very talented and experienced people around, with varying levels of context. Developing parts of that context is possible to do autonomously - eg following the steps in your earlier guide. But developing the networks, the sense of the field (who's doing what / where) is not easy, especially for people who are not located in a hub.
I agree that hiring organisations seem to want high-context for most roles, including operations roles, whether or not this is needed. It seems like this limits your pool of potential candidates - if you want a rock star head of finance, who also has high context, who is also available to work from your location or be hired from your location. When organisations cant find these unicorns, they're either leaving the roles unfilled, or hiring a person with less experience but 'potential to grow into the role'. I've previously commented that there's an opportunity cost to both of these options.
I've been on both sides of recruiting and job-seeking, so I know it's not easy for anyone. I would like to see more programmes like the Successif organisation builder's programme, which provide a way for experienced people to build context through paid part-time work, while also helping organisations that are trying to scale and deliver.
Great point, and I agree that there should be more orgs such as SuccessIf!