AI Safety Needs Great Engineers

Top line: If you think you could write a substantial pull request for a major machine learning library, then major AI safety labs want to interview you today.

I work for Anthropic, an industrial AI research lab focussed on safety. We are bottlenecked on aligned engineering talent. Specifically engineering talent. While we'd always like more ops folk and more researchers, our safety work is limited by a shortage of great engineers.

I've spoken to several other AI safety research organisations who feel the same.

Why engineers?

May last year, OpenAI released GPT-3, a system that did surprisingly well at a surprisingly broad range of tasks. While limited in many important ways, a lot of AI safety folk sat up and noticed. Systems like GPT-3 might not themselves be the existential threat that many of us are worried about, but it's plausible that some of the issues that will be found in such future systems might already be present in GPT-3, and it's plausible to think solving those issues in GPT-3 will help us solve equivalent issues in those future systems that we are worried about.

As such, AI safety has suddenly developed an empirical subfield. While before we could only make predictions about what might go wrong and how we might fix those things, now we can actually run experiments! Experiments are not and should never be the entirety of the field, but it's a new and promising direction that leverages a different skill set to more 'classic' AI safety.

In particular, the different skill set it leverages is engineering. Running experiments on a real - if weak - AI system requires a substantial stack of custom software, with projects running from hundreds of thousands to millions of lines of code. Dealing with these projects is not a skillset that many folks in AI safety had invested in prior to the last 18 months, and it shows in our recruitment.

What kind of engineers?

Looking at the engineers at Anthropic right now, every one of them was a great software engineer prior to joining AI safety. Every one of them is also easy to get on with. Beyond that, common traits are

• experience with distributed systems
• experience with numerical systems
• caring about, and thinking a lot about, about AI safety
• comfortable reading contemporary ML research papers
• expertise in security, infrastructure, data, numerics, social science, or one of a dozen other hard-to-find specialities.

This is not a requirements list though. Based on the people working here already, 'great software engineer' and 'easy to get on with' are hard requirements, but the things in the list above are very much nice-to-haves, with several folks having just one or none of them. 

Right now our job listings are bucketed into 'security engineer', 'infrastructure engineer', 'research engineer' and the like because these are the noun phrases that a lot of the people we like identify themselves with. But what we're actually most concerned about are generally-great software engineers who - ideally - have some extra bit of deep experience that we lack. 

How does engineering compare to research?

At Anthropic there is no hard distinction between researchers and engineers. Some other organisations retain the distinction, but the increasing reliance of research on substantial, custom infrastructure is dissolving the boundary at every industrial lab I'm familiar with. 

This might be hard to believe. I think the archetypal research-and-engineering organisation is one where the researchers come up with the fun prototypes, and then toss them over the wall to the engineers to clean up and implement. I think the archetype is common enough that it dissuades a lot of engineers from applying to engineering roles, instead applying to research positions where they - when evaluated on a different set of metrics than the ones they're best at - underperform.

What's changed in modern AI safety is that the prototypes now require serious engineering, and so prototyping and experimenting is now an engineering problem from the get-go. A thousand-line nested for-loop does not carry research as far as it once did.  

I think this might be a hard sell to folks who have endured those older kinds of research organisations, so here are some anecdotes:

• The first two authors on GPT-3 are both engineers.
• Some of the most pure engineers at Anthropic spend weeks staring at learning curves and experimenting with architectural variants.
• One of the most pure researchers at Anthropic has spent a week rewriting an RPC protocol.
• The most excited I've ever seen Anthropic folk for a new hire was for an engineer who builds academic clusters as a hobby.

Should I apply?

It's hard to judge sight-unseen whether a specific person would suit AI safety engineering, but a good litmus test is the one given at the top of this post:

With a few weeks' work, could you - hypothetically! - write a new feature or fix a serious bug in a major ML library? 

Are you already there? Could you get there with a month or two of effort? 

I like this as a litmus test because it's very close to what my colleagues and I do all day. If you're a strong enough engineer to make a successful pull request to PyTorch, you're likely a strong enough engineer to make a successful pull request to our internal repos. 

Actually, the litmus test above is only one half of the actual litmus test I give folk that I meet out and about. The other half is 

Tell me your thoughts on AI and the future.

with a pass being a nuanced, well-thought-out response. 

Should I skill up?

This post is aimed at folks who already can pass the litmus test. I originally intended to pair it with another post on skilling up to the point of being able to pass the test, but that has turned out to be a much more difficult topic than I expected. For now, I'd recommend starting with 80k's software engineering guide.

Take homes

We want more great engineers.

If you could write a pull request for a major ML library, you should apply to one of the groups working on empirical AI safety: Anthropic, DeepMind Safety, OpenAI Safety, Apollo and Redwood Research. 

If that's not you but you know one or more great engineers, ask them if they could write a pull request for a major ML library. If yes, tell them to apply to Anthropic. 

If that's not you but you'd like it to be, watch this space - we're working on skilling up advice.

This post is twinned with the same one on LessWrong