Existential riskExistential risk

This is a linkpost for https://heresjonny.substack.com/p/paul-christianos-barista. 

When I left my previous job in April of last year, I took a long break from thinking about AI safety. This was nice. Thinking about catastrophic risks all the time wears you down. Imagining oneself as Cassandra, baffled that nobody around you is taking this sufficiently seriously, is not good for the soul^[1]. I was angry at the community that had got me interested in AI risk in the first place. I felt powerless to help, and that I had earned more selfishness anyway.

Of course, avoiding AI altogether wasn’t an option. I was worried that coding would soon be automated, and my most marketable skill would disappear. I wanted to broaden my skillset and position myself well in case I couldn’t go back to a regular software engineering job. I started writing posts about how I was going to found a company, after having been heavily influenced by reading The Intelligence Curse^[2]. I got really into thinking about the intersection of AI and psychiatry. I started talking about the importance of defensive acceleration and how effective altruists systematically underrate comparative advantage. This was motivated reasoning; I wanted to work on something other than AI safety and not feel guilty about it^[3]. I figured that all these super smart, status-y types had it under control already.

The EMH is trivially false across all domains

In April of this year, I had various conversations that led to me feeling excited about being part of the EA community again. I went to EAG London and primarily talked about things other than AI safety^[4]. I started doing some contract work for an EA-flavoured global health and development charity. I was managing to feel like I was part of the community without getting sucked into AI safety’s gravity well.

I ended up applying for a job at an EA organisation. I managed to get through to the work trial stage, which involved giving people advice. It was noteworthy that diverse groups of people often seemed to need to hear relatively similar advice. I kept repeating something along the lines of:

Build a model of the world. Try to falsify the components of that model. Seriously think through the implications of this model, and then act accordingly.

And then of course something deeply upsetting happened. I realised that I was not following my own advice.

So I went back to basics. Claude and I went back-and-forth over my litany of questions and objections to the idea that advanced AI systems could pose an existential threat. I wrote a post where I attempted to reason through what I thought was likely to happen over the next two years. It started to seem likely that, at the very least, things are going to get weird, soon.

Somewhere during the process it dawned on me that vanishingly few people do this kind of thing; most were LARPing, just like me. When considering something complex, at some point, it’s all too easy to let rigour lapse and hand-wave something away. Thinking rigorously is hard, but in a particularly subtle way that makes this kind of hand-waving easily done without being consciously noticeable.

I felt a sense of clarity. It might be that this clarity is entirely fake and I’m totally wrong, but still, it persists. I was ready to say it with roon:

experts are fake, smart generalists rule the world, everything is designed by people no smarter than you, courage is in shorter supply than genius

I’ve long felt like I’m trapped in a midwit meme, painfully aware of the importance of AI safety but simply lacking the intelligence or skills to actually do anything about it. When I read Scott Alexander’s Parable of the Talents, I imagined myself as Paul Christiano’s personal barista. But I don’t think any of it is true. I’ve sufficient genius. What I was lacking was the courage to actually do the hard work of thinking seriously about things for long enough to reach a conclusion, even when that conclusion is an unpleasant one.

Having sat down and made a good-faith effort to think it through, I do not think we are on track to develop AGI in a safe way. I think I have enough power and resources that there’s some chance I could make a difference. And so I’m going to try.

1. ^{^}

   Although Michael Burry seems to be making it work for him

2. ^{^}

   I feel weird about this. In some sense, it is true; I am the sole director of a company. The word “found” has various implications that aren’t accurate in my case, but sometimes it is useful to pretend that they are. Sometimes “founding” is quite high-status, and so there’s a temptation to play it up. I think that’s pretty clearly what was happening in this post. The whole thing seems naïve too – although I think that finding success with a startup probably does require some amount of naïveté.

3. ^{^}

   Despite being motivated, I do think there’s some legitimacy to the claim regardless.

^{^}

It was my sixth EAG, and the other five had all been dominated by the topic.

Show all footnotes

Anthropic just released its S-1 form, which contains the official financial information it must report to the SEC in order to go public. From the S-1, we now know that Anthropic's revenue grew more than 100x (specifically, 112x) from 2023 to 2025. Revenue was:

• $41 million in 2023
• $386 million in 2024
• $4.59 billion in 2025

A few people in the EA community apparently believe this revenue growth rate will continue for the next several years at least. However, I am confident we won't see Anthropic bring in $514 billion in revenue in 2027, which is what extrapolation would imply.

Indeed, I'm confident that Anthropic won't hit half of what extrapolation would imply, which is $257 billion in 2027.

I'm willing to bet as much and would be delighted to wager $20 on it, to be donated to the charity of the winner's choice.

I also have an open offer for a more general AI bubble bet.

Update (Friday, October 30th at 5:45pm Eastern):

Here's a good article on this topic. Two economists look at Anthropic's S-1 and ask whether a $2 trillion valuation is reasonable: https://econjared.substack.com/p/the-fantastical-promises-of-a-2-trillion

Would it be a good idea to have AI safety experts be put under a lie detector to make it more plausible to the doubters (who believe AI risk is a marketing stunt) that AI risk is real. Some skeptics may be swayed if they believe AI safety experts are Genuinely concerned. I am aware that lie detectors are not very reliable and can give false positive or negative results, but this may work. I intuitively think this is not a good idea, and think I have likely overlooked a key flaw. 

Help The Unjournal prioritize AI economics and governance research

(https://uj-prioritization-dashboard.netlify.app/ai-governance/)

The Unjournal (https://info.unjournal.org/) is prioritizing research on AI economics, governance, social and economic impacts, and risks from increasingly capable AI. (NB: We're not covering technical/CS/ML/AI-safety research.). We want to commission expert public evaluations and help synthesize, disseminate, and curate this work.

Much of this research seems 1) important/impactful/influential, (2) involves considerable 'firepower' from prominent researchers and practitioners, (3) not 'obviously true', and (4) time-sensitive: relevant for funding, policy, and research-steering now, but not after the "six months to six years" required for the traditional (economics) journal process

Our preliminary shortlist page (https://uj-prioritization-dashboard.netlify.app/ai-governance/) includes paper summaries, AI-generated audio walkthroughs, and an interface for giving feedback. We also invite suggestions fr other research (please check the broader list/interface to avoid duplication).

Especially looking for your input on:

• Usefulness and accuracy:* Is this useful? How could we improve it? What have we got wrong?
• Impact: Which research do you use, or think others use or should use?
• Scrutiny What claims and approaches need careful experts and practitioners critique? What sorts of experts? If you're that expert, please dm and join our pool.
• Participation: Who should be part of this discussion (nominate and nudge)?
• Gaps: What's missing (research, claims, etc.)?

Timing and next steps (our plan)

• Finalize an initial shortlist of at least three research outputs before the end of October, with more depending on interest and funding.
• Commission rapid, public, impact-focused evaluations/reviews by experts and practitioners, aiming for a first public evaluation within about two weeks of commissioning.
• Over the following two to four weeks, invite author and stakeholder responses and follow-ups, and facilitate private and public discussion

If it this works well, we hope to scale up and continue in the following months.

Alongside the evaluations, we want to help people find and use this research, and to help shape its direction. I hope the prioritization itself will be useful to students, researchers, practitioners, and other stakeholders. (NB: the "prioritization" here is mainly about "potential for impact and value for evaluation"; assessing the research credibility is for the evaluation part).

I wanted to share this quickly; planning a fuller post soon. (Feedback on this post itself is helpful too.)

Related: We’re also exploring sponsorship of evaluations/themes/questions (https://uj-prioritization-dashboard.netlify.app/sponsor/). Looking for feedback: Would this be useful, would you sponsor or recommend others to do so? Would sponsorship affect your trust in the evaluations (given the rules and safeguards mentioned in the link)?

Humanity's inability to coordinate an AI slowdown may itself be early evidence that we are starting to lose control. Especially given all the recent omens around cyber and bio of frontier models, with open weight models just a few months away.

It seems like the most common objection to an international AI treaty with China is that it’s pointless as they will just defect. But there’s a strong case to be made that mutual verification of compliance is a tractable problem, so I suspect raising salience of this in AI policy discussions would make an international agreement more politically feasible. Presumably it would be a matter of only committing to what we can mutually verify now, then investing in stronger verification techniques as they’re developed. I find IFP’s policy recommendations for accelerating verification R&D to be a good starting point for thinking about this, especially in a near-term practical-to-policymakers way.^[1] Ideally we would have some policymaker-legible brief memo/executive summary for verification proposals to point to that gets frequently brought up where it matters, i.e. Get members of Congress to write open letters about it, get journalists to ask about it, emphasize it in interviews that AI safety experts go onto, emphasize it at AI dialogues with the White House by those who can attend, etc. Granted, many of them are likely using motivated reasoning to dismiss this regardless of any evidence presented as a result of industry lobbying, but I reckon it would help to put more pressure on them to address it. From what I've seen thus far, the response from international slowdown advocates has mostly been "it's also in China's interest to slowdown, superintelligence kills everyone!" Sure, but I think it helps to make our proposals more skeptic-robust.

Tentative thesis: China is unlikely to accept a subordinate position in AI capabilities to the US, just as the US is unlikely to accept a subordinate position in AI capabilities to China or anyone else. 

This thesis suggests that there are two likely paths for the future international AI (non)regulation:

1) a continuing race between the US and China, perhaps joined by some late entrants, possibly with some rules established (e.g. mutual ban on autonomous weapons or something like that), or

2) a “pause or stop deal” that will establish some sort of ceiling on capabilities, which will be equal for China and the US.

What I think is far less likely is a deal in which China would accept lower AI capabilities than the US.

I simply don’t see a good reason why Chinese leaders would accept subordinate position. They know that China has the ability to catch-up to the US in various technological domains, as evidenced by the fact that in, like, 1990, China was behind in more or less everything, and now they are pushing the technological frontier in many areas. 

Some estimates floating around the web suggest that, if the US would stop AI development now, China could reach current US capabilities within months. That is maybe overly optimistic/pessimistic depending on where you stand, but I very much doubt that catch-up would take more than a decade. And Chinese government, being patriotic about the abilities of the Chinese nation, probably will not have an absurdly pessimistic estimate.

Moreover, the Chinese regime is in many ways oriented around this idea of catching up (this is a big difference between China and EU). It is a central plank of the official historical narrative of the People’s Republic of China (see for example preamble to their constitution: https://english.www.gov.cn/archive/lawsregulations/201911/20/content_WS5ed8856ec6d0b3f0e9499913.html) that the century between First Opium War and the Communist victory in the Chinese civil war in 1949 was the century of humiliation, during which various imperialist powers exploited China due to the fact that it was technologically backward country unable to stand up to them. I happen to think this narrative is largely correct, but even if it isn’t, what matters is that Chinese government believes it and is determined to never again allow China to be put in the position where a technologically superior power could walk over it.

This has some implications:

A) Since China is currently behind, any sort of “pause or stop deal” between the US and China would likely have to involve either the US stopping further development and allowing China to catch up, or the US somehow crippling its own capabilities to a current Chinese level. Of course, for either option, the US would have to place a lot of trust in the Chinese government.

 B) As long as China is behind, export controls on hardware and other attempts to limit Chinese capabilities are at best of doubtful utility as a way to get to a "pause or stop" agreement. In fact, imho there are making a deal less likely. They antagonize Chinese leaders, make them even more paranoid and less inclined to a constructive agreement. But also, to the extent they are successful in limiting Chinese capabilities, they make it less likely to that the US government would agree to an equal deal, since that would mean sacrificing larger lead, which is harder to politically accept for the US. Perhaps this is a reason why nuclear arms treaties in the Cold war were agreed only after the USSR more or less caught up to US nuclear arsenal?

Now, attempts to limit Chinese capabilities can still be seen as useful if you think it is important for the US to “win” the competition, for example because there are reasons to believe US AI will be safer. Or because you think the world would be a better place if the US “wins”, even though both countries would develop similarly dangerous systems. Or of course if you think that the US victory would be preferable even if American AI would be more dangerous than Chinese AI.

Btw. for a well written snapshot of Chinese opinion/vibes, I recommend this: https://www.chinatalk.media/p/how-anthropic-became-chinas-goliath

I'm not convinced the recent capabilities advancements (highly concentrated in pure math, the section which by definition can't be applied) support the forecasts that misaligned superintelligence is on balance harmful. In general, I think there is not enough evidence to draw the conclusion that multi-agent alignment (which was far from inevitable ex-ante in comparison to the sometimes monotheist conceptions of ASI) is impossible, especially with AI eventually automating and enforcing the institutional mechanisms. 

However, this is not guaranteed either. The labs are making the biggest bet that humanity has taken by far; that RSI will automate and solve the alignment problem, and the ensuing intelligence explosion will solve most of humanity's suffering. This explains much of the race dynamics until now, where the balance of tail-risks are now undoubtedly on the upside. Note also the competition with China, which plays particular relevance for open-source.

My p(doom) probably sits somewhere just over halfway (to account for the right-tailed distribution here) between my pre and (initial) post Hugging-Face likelihoods: perhaps 8%. It would be unwise not to update since this saga unfolded. 

Some of the latest macroeconomic models from the likes of Anthropic and Acemoglu suggest rather grim futures for employment relative to what economists were previously predicting, hence its fair to say our methods lean overly conservative towards excess rigour here. Perhaps the recent backlash from the mathematicians, and some of the sentiment behind the anti data-centre movement, reflect these (increasingly justified) anxieties. 

Nonetheless, this is orders of magnitude more dangerous than nuclear weapons. That's enough for me to take AI-safety incredibly seriously!