A Conversation on the final day of Inkhaven
A lot of money is about to flood into AI safety philanthropy and almost nobody knows how to give it away well.
On their final day in the sunny Bay Area, three Inkhaven residents - let’s call them Adam, James, and Sam - were reflecting on the state of grantmaking after lunch. Sam has spent a lot of time investigating the upcoming IPOs of Anthropic and OpenAI and wants to grow the field of grantmaking while breaking into it himself. We discussed the bottlenecks in grantmaker supply and how a fellowship might help.
Adam: There are two things. The first is that a grantmaking fellowship seems like a great idea. When I heard people talking about it, I was like, “why isn’t this already a thing?” At the current margin, we don't need more policy research fellowships. We have plenty of those already. AI policy research, to be fair, varies across contexts and does require some level of AI security/safety-specific context, but policy research in general is a skill. So is AI. Whether it gets automated away in 2027 or not, you can learn that elsewhere. You can't learn grantmaking anywhere else.
There are lots of fellowships that can help you test your fit for that outside of this space. Grantmaking is not the kind of thing you can just drop into. It’s less fungible. Yes, there are general skills you’re using, but a lot of it is taste, judgment, and intuition built up within a very specific context — it's a lot more like cultural assimilation. It takes several years of bouncing around fellowships just to get into the space and understand what's going on. That's basically assimilating into a culture.
The main thesis is that OpenPhil is basically Japan right now. Japan is facing a massive drop in population/birthrates. But they’re also really committed to their own culture, and don’t typically embrace outsiders. They don’t generally want to take people in; if they have to, they want them to assimilate.
James: But aren’t the Effective Altruists the high-context recruitment pool for OpenPhil?
Adam: You have people who've been EAs for 15 years who will come in and do three years' worth of work trials. I had a friend at one point who was in university and made more money from OpenPhil work trials than from his actual job.
What's happening is they need to test for this culture — this very specific thing that takes a long time to develop. What you want is to be able to test whether someone'd be a good person to assimilate into the culture. That's the immigration test.
James: By the way, I have a funny story. When I was an undergrad, I recommended a friend to apply to OpenPhil. I think they mixed up the emails, so they sent me an invite to an OpenPhil work trial. I was like, "Maybe you meant my friend, but I'm also happy to take the work trial." In the end, it was quite fun. I completed it, and while I wasn't hired, it was still interesting. It was different examples, like forecasting, where you need to reason through and give a numerical score.
Good writing or blogging is a good indicator for grantmaking. Full research experience is probably not required, but reasoning, transparency, and clarity can be demonstrated through blog posts, since it’s tough to show a grantmaking record. Clear reasoning in public writing - like we tried to do at Inkhaven - is the next best thing.
He got hired, and I got like $5,000 from the work trial too. The OpenPhil work trial gig economy.
It would be better if it were more regular, because if what they're doing is basically testing for cultural fit, that's also what fellowships are for. A fellowship is an opportunity not just to test but actually to develop the skill if you have the raw abilities. It's hard to develop the skill while you're also doing work tests.
Then we talked about the Anthropic IPO. The pledges from many Anthropic employees suggest that a growing amount of money will soon be available for AI safety grants, with only a few people to distribute it.
Sam: What's the chance of Anthropic IPO-ing in the next two years? People are saying 60–70%.
James: I know people who moved to New Hampshire to use their IPO proceeds to pay lower capital gains taxes than in California, to donate, and cash out.
Sam: That's the thing — that's a real preference revelation. On Adam's point about grantmaking being less fungible: there's enough publicly available data on AI policy research from the last three to four years that you could theoretically put it into an AI system and generate a knowledge base to help you learn by yourself. But how many internal memos does a funder disclose for you to know what good reasoning and transparency actually look like?
James: That’s a strong point. Policy research is really public, while grantmaker reasoning lives in Google Docs.
Sam: At the objective level — what good grantmaking looks like — and then also at the taste level, getting a sense of what their preferences actually are. Right now, the only thing we have externally is the RFPs they put out from time to time. Open Phil just started a blog, and after all these years of existence, they have barely three posts.
These are all the kinds of things you don't want to have on the internet in case some journalist is combing through everything. But if you have someone in the fellowship, it's like using a Google Doc with them — you could share that context. You can't get it via the internet.
James: Tell me if this is right. What you're saying is: policy research stuff is publicly available. They want to publish it. In principle, you can find it all, and you can access it from the outside. It's a lot harder to understand what's going on with grantmaking in general because all the information is in Google Drive and Google Doc comments.
Sam: The reason it's not public isn't that the information is the crown jewels. If you're three weeks into a grantmaking fellowship, they'll happily share it with you. They just don't want to put it on the internet in case a journalist writes "Open Phil strikes again." It's the same reason the Heritage Foundation or Schmidt Futures don't put all this on the internet either. That's not how philanthropy works. But if you're an internal person with the contacts, of course, they'll share with you. That's how you can get new people.
The fellows would get access to internal documents and assist mentors in their day-to-day work. They would also learn active grantmaking - like writing requests for proposals, because that’s the best way for grantmakers to change the field actively.
That’s why I believe we need a grantmaking fellowship. Candidates need to learn judgment skills now before the Anthropic IPO.
Sam had clearly been thinking about this for a while.
One angle I’d add is that there may be a complementary, more immediate approach: bringing in experienced grantmakers from adjacent fields and helping them transition into AI safety.
I think many of the core skills (evaluating proposals, making judgment calls with uncertain information, communicating decisions, managing portfolios, etc) are quite transferable. The main gap is context, for the reasons you've described. I think that'd be faster to remedy than building grantmakers from scratch.
Do you think this kind of approach (getting proven grantmakers into AI safety) could help solve the expected bottleneck? Perhaps I’m underestimating the barriers.
Sounds reasonable, so for active grantmaking (e.g. requests for proposals, making sure new , get builtorganizations), I believe domain expertise (e.g. AI safety experience) is more important