Non-AI Grantmakers Should Be Pricing In AGI Timelines More Than We Are

Epistemic status: Wrote this pretty fast, probably ~70% confident some version of the thesis claim is true, would welcome being proven demonstrably wrong.

Context: I’m a grantmaker focused on alt-protein and am fairly well-connected/moderately in the loop of what animal welfare grantmakers are doing.

This post was significantly inspired by “You (Yes You) Should Have A Feb 2020 Checklist For AI Policy,” which is more geared for the existing AI safety crowd but directionally a good push in general.

Thesis: I think EA non-GCR grantmakers should probably be doing more to prepare for AGI/TAI than we are, and might be on track to get caught with our pants around our legs within the next few years. We are not acting as if we’ve priced it in.

TAI Soon

Short timelines to AGI/TAI seem very plausible (like, at least 50% chance of arrival in the next 1-6 years). Points of evidence include: 

1. The general shortening of timelines by top forecasters across the last 10 months (Kokotajlo/AI Futures Project, Greenblatt, Cotra);
2. The OpenAI HuggingFace warning shot;
3. Various public statements by frontier labs, their employees, and their executives warning about imminent recursive self improvement in 2027 or 2028;
4. Mythos saturating the METR time horizons evaluation months ago such that they are now building a time horizons 2.0 eval.
5. Various anecdata about the coding agents causing a significant progress speedup at frontier labs, consistent with what would happen if we were in the early on-ramp to full automation of AI R&D.
6. The general sense that capabilities progress has gone quite fast, closer to the most bullish forecasts a few years ago (e.g. AI 2027, Situational Awareness) than forecasts that expected longer timelines.

The Good

My rough qualitative sense is that the thing we’re doing best at is preparing/hoping for a very crazy amount of funding to come online for all EA cause areas post-Anthropic IPO.

1. Various blog posts are being written to soft-pitch various megaprojects, mainstream news profiles about meta-fundraising, people are moving to SF, etc. Even if most funding is expected to go to AI safety, a trickle down effect is still plausibly enough money to double funding for animal welfare or global health.
2. Some grantmakers are deliberately increasing their spend this year to try to make the jump to crazy funding amounts less discontinuous than it otherwise would be.
3. As a consequence of 2, people are trying to set up new projects that can become megaprojects if the third wave is truly a wave and not a dribble. We probably are doing this too slowly and could be moving faster than we are; we may have as little as 9 months to get a lot of stuff from zero to one.

The Less Good

The things we are doing worse at is everything that is not encompassed by “ooh shiny what if we all have tons of money in a year.” There’s probably some point here about people’s epistemics, like obviously this type of discourse gets more people excited and has more memetic spread as a result. I think there’s probably also some general cognitive dissonance and motivated reasoning among non-AI people to stay in denial/not wanting to really believe that things could get completely crazy and scary. 

What are we not doing that we should be? The one sentence version is: we are still mostly not pricing in the possibility that AGI will radically change the underlying facts of our non-GCR cause areas and updating our strategy accordingly.

Basically this is an expected value argument. If you seriously think there’s a meaningful chance (25-60%) of TAI within the next five years, there’s almost certainly some very high EV things you can do now to lay the groundwork better for those worlds.

The idea that we can forecast events that totally shift the ground under us (for good or bad) has mundane precedent for it. Global health and climate philanthropy collectively across 2023-2024 did not accurately price in the ~50% chance of a Trump 2024 victory and what Trump 2.0 would mean. As a result, these communities mostly (with some exceptions) got caught with their pants around their ankles when USAID and most of the IRA got deleted (though I do mostly buy FP’s glass half full take about the IRA).

What To Do

Here are some very rough but concrete suggestions for non-AI grantmakers based on all of this (note, all of these assume you’re not planning to pivot to working on AI safety in the next 6-12 months):

1. Grantmaking strategy:
   1. At bare minimum, you should probably make more grants within your existing sub-strategies that are robust to worlds with fairly short timelines. If we end up with very fast science/innovation progress, how does that change the biggest bottleneck for your focus area? (A rough example is policy and regulatory advocacy looks better for alt protein if you think TAI will accelerate scientific/R&D progress on cultivated meat). You should probably fund more stuff pointed at the bottlenecks that bite more in worlds with TAI soon.
   2. Your focus area almost certainly has some obvious low-hanging fruit that falls outside your existing sub-strategies, but that has potentially enormous payoff conditional on TAI arriving soon-ish, plausibly 1-2 OOMs above your benchmark or bar. The types of things that could make big, discontinuous progress in unexpected ways if scientific discovery and innovation go haywire. I think for the most part we aren’t currently funding these types of new projects at the proportional level that their EV implies. I’ve seen some good early public (and private) signs that people are starting to wake up to this, but we could probably be moving faster.
2. You should probably start reading more about AI and generally paying more attention to it so that you have more context and can notice when really crucial inflection points happen. There are plenty of good blogs, organizations, foundational long-reads, podcasts, and people to follow on Twitter to generally stay abreast of things. Imo it really does help to just inhale a lot of content until you have a sort of tacit sense of what things matter.
   1. If we end up in a March 2020 COVID moment for AI policy and it becomes clear that really, everyone in EA needs to drop what we’re doing and pivot and not sleep for a year, you’ll have less catching up to do if you start reading stuff now (and doing #3 below).
   2. Also, doing #2 will give you better intuitions and ways of reasoning about what the super high payoff things in #1b are likely to be, and how to make your existing grantmaking more robust in #1a.
   3. Doing this also obviously makes your meta-fundraising plays for your own cause area more likely to succeed because you'll come up with a better pitch.
3. You should be using Claude Code/Codex to be more productive, e.g. do 10x more desk research per grant or prioritization investigation compared to previously. Models and products (harnesses, UI surfaces like Cowork) will only keep getting better and more capable of doing big, long-running tasks on the way to TAI, and you should probably invest some fraction of your time in reaping the benefits for your own work. The capability frontier for what ordinary knowledge work AI can do has moved dramatically with Fable/Sol relative to where it was 10 months ago and we could probably all do a bit better at acting accordingly.
4. Non-AI and AI people should probably talk and interact more socially so that we can all help each other and make a March 2020 moment for AI policy go well (e.g. by leveraging each other's networks), if and when it happens.