Thank you for your advice, Ben! I especially appreciated the document with your reasoning about how you chose between those two job offers. It was also quite a chance since I am currently applying for Kairos Generator residency and a couple of their open job roles.
How is Kairos culture working from the inside? If you want to share your experience so far I would be eager to hear it!
I shared a lot of advice at EA Global. It felt a bit like playing a character. It was my first EAG, and only six months ago I’d never attended an off campus EA/AI Safety event. But it also felt quite natural. I ran Brown’s EA group, then did an AI safety research fellowship, and now I work at Kairos, a field-building organization whose mission is to bring more talent into AI safety. I’m a textbook case! There were suggestions I repeated over and over as I discussed next steps with college students hungry for impact [1] so I’ve collected them here. The advice is presented in no particular order.
Last spring, I applied for AI safety jobs with a vague belief that AI risk was very important. When it came to explaining why, I only understood the most basic heuristics. In my applications I emphasized my interest in mechanistic interpretability, the importance of guardrails against misuse, and my acumen as a theorist. Those are, almost to a T, opposite to my current beliefs about AI safety.
I think it’s wise to be worried about existential risk from AI. I also think it’s wise to know—this is the crucial bit—why you are worried about AI. Here are some questions I think that someone working in AI safety should be able to answer: [2]
If you are planning to spend most of your working hours on some issue, it’s natural to want to understand that issue pretty deeply. Personally, I’ve found knowing what I’m worried about helpful not only for strategy and prioritization, but also for my mental health and comfort in my identity. I’m still pretty uncertain about some of these questions, and that occasionally makes me confused or upset, but I felt much worse when I had gaping holes in my understanding. It’s motivating to have a good sense of why you’re doing what you’re doing, to have a tactile sense of the connection between your immediate goals and terminal goals.
Having strong threat models is also just useful. If nobody had good threat models, everyone in AI safety would be running around doing useless nonsense. That means the people with strong threat models are doing something useful (unless you think that everyone in AI is running around doing useless nonsense). At the macro scale this might look like writing AI 2040 or running an organization with a specific mission, such as METR, but good threat models are also useful for all sorts of more micro decisions. If you do technical work, they inform what research you do or how you frame it. If you do policy work, they inform how you brief legislators. If you do field-building, they inform where you direct new talent. In some cases, these good threat models need not come from you—you can always defer. On some things, you’ll have to. I haven’t read the evidence and formed my own view on whether hardware verification is a tractable engineering problem. Still, the more you personally understand what you’re worried about, the easier it is to take on responsibility and pursue creative projects.
Earlier this autumn I decided between two roles, both in AI safety. It took a week. I spent 5 days peppering my could-be employers with questions, taking notes and writing down 13 pages of comparison. At the time I felt a little embarrassed, even frivolous, but it was obviously the right move—I ended up changing my mind on the second-to-last day. If you are applying to a hardcore EA or AI safety organization, they’ll probably support you taking the time to do this, or even be impressed.
Doing the right sort of work is a huge multiplier on your impact. As an immediate corollary, you should put real effort into deciding what the right sort of work for you is.
Many of the young EAGers I spoke to said they cared a lot about finding the most impactful role, but they were confused about how to contribute. Should they do operations work for a large research org, or start their own thing, or perhaps look into grantmaking? All of these people had thought about heuristic arguments for each, and worried over the question, and asked experienced people for advice. All of that is good.
But you can try a lot harder, with only a few hours of effort! Take out a pen and a piece of paper (or MacBook, as the case may be) and write out all of the relevant considerations. Try to find evidence of how impactful each role is. Think about how much you care about immediate impact and how much you care about your growth. Compare your options on equal terms, like by converting all values to their worth in dollars, or hours of AIS labor generated.
Explicitly comparing career options has the added benefit that after you decide what to do, you know why you’re doing it and can more easily notice if things are going wrong. It helps you keep track of your real goals. For example, I realized I care a lot about my growth as an executor and manager, and now I’m searching for hands-on projects to own at Kairos.
This summer, I worked out of the London Initiative for Safe AI (LISA) on the Pivotal Fellowship. I was paired with a mentor from SecureBio on research that involved a little bit of statistical theory and a lot of implementation, research taste, and collecting empirical results. Overall, I think I did well; I got positive feedback from mentors and peers. The project was also a natural fit for my personal identity—a mathy guy who likes theory. And yet, I noticed, my work wasn’t a home run. It was more like a solid double. There were plenty of great research projects at Pivotal, at least by the standards of novelty and academic merit, and I don’t think mine stood out.
But I really excelled at having interesting conversations over lunch. Seriously! I asked questions about what other AI safety employees at LISA were doing, and why it was important. I read forum posts and tried to square them with what I’d heard. Pretty soon I was in recurrent arguments with grant-makers, research managers, and a professional blogger (dumb job, smart guy). The program was nearly over when I realized the significance of these conversations.
There’s a classic piece of EA career wisdom: ask not where you excel now, but instead where you improve swiftly. Whatever you actually do in your job, you’ll probably get more practice within a few months of full-time work than you have in the rest of your life combined. Moreover, most of the expected impact of your career is almost certainly more than six months away (even if you have short timelines). This means that being skilled now isn’t very important. What’s very important is your learning curve, or skill ceiling, or whatever you want to call getting good faster than the people around you until you are really, really excellent.
To be clear, the best way to tell that you have a fast learning curve in some topic is to just do the thing. It’s tempting to stop after an early success, for fear that you’ll shatter the feeling that you are a person with potential. That is wrong. The feeling of being a person with potential is not very helpful; in reality, you’ll be varying levels of good at varying things, and you just don’t know which is which yet. So, if you think you might be really good at something, try it. Talk to people who do it. Find a quest, cheap test, you’ve heard it before. It is true.
Hopefully, this will be fun! If it’s not, that might be a sign that the skill you’re testing isn’t a good fit for you. Maybe this is wishful thinking, but I believe that most people are energized by the work where they most excel.
Below are some documents I wrote in the few months before I got a job:
Number 5. directly connected me to the job I have now, 3. led to a fascinating in-person work trial, 4. started interesting conversations with AIS people I look up to and would not otherwise have met: it was all useful.
In general, the ability to reason under uncertainty and write transparently is super valuable to AIS orgs. AI is insane. Weird shit happens all the time. It interacts with everything and it’s hard to reason about. Being able to think through hairy problems and arrive at educated guesses with quantified uncertainty will be useful in almost any sort of AI safety work you do, and the way to communicate and practice this reasoning is by writing things down.
Writing things down exposes your reasoning to other people. I cannot emphasize enough how useful this is. When you ask someone important for advice, you can at best hope to get their canned advice for your whole cohort. If you send them a one-page document with some bullet points, on the other hand, they can quickly identify your uncertainties and chime in, or correct errors you’ve made, or direct you to relevant resources you haven’t found. Your writing allows them to give specific, forceful advice. That in turn is much more likely to change your beliefs and decision-making for the better.
Also in favor of writing—thinking is often fake. I sometimes feel like I have a good idea, and I “let it ruminate,” and then I try to write down my thoughts and find that there were only two sentences of actual substance, which had just been running laps around my brain for days. Better to find that out sooner by writing things down.
This section is just a list of cheap tests you can consider, if you yourself are a student or new grad looking into AI safety.
And that’s all I have for you. If you want to, become a new guy. If you don’t, don’t. Remember to write things down.
This is advice to the impact optimizer, who may or may not be you or some part of you. I did not do most of this stuff in college, because I had a hazy blob of goals in which “reduce existential risk from AI” was not ranked in the top 3.
If you don’t know where to get started on these questions, I recommend BlueDot’s AGI Strategy curriculum. For more on how loss of control might look, you can see this article by Paul Christiano. You can read AI 2027 for some predictions of how the world might look right before things go wrong, and what winning might look like.
Great post Ben!