Quick takes

People give 'The Day After' as an example of a movie that motivated nuclear disarmament and it would be good to have something similar for AIXR and I agree and I think there's something important to learn about that case. That movie and 'Threads' are about the catastrophe *happening*, and about how absolutely terrible that would be. It forces you to put yourself there and that makes a strong emotional impact. I think this type of intuition pump is the most powerful of them; people get the most motivated to change their lives when they think about their last moments and what they could regret then. Same thing happens when people think of their loved ones dying; they feel motivated to tell them that they love them or protect them. The end of the world is basically both of those things at once. Making people put themselves in that scenario is something AIXR comms hasn't tried much IMO. Only examples I remember are John Sherman once in his podcast and a book, and one tweet from Michael Trazzi.

Applying Intelligence Community Indications and Warning methodology to frontier AI yields a single, stark conclusion: we are currently in an active warning failure. The capability thresholds intended to trigger policy interventions have already been breached, with frontier models clearing 50-70% on SWE bench and inference efficiency expanding at a  40x annually. Our current evaluation frameworks are structurally gameable by situationally aware systems, pointing to a foundational counterintelligence failure rather than a mere oversight gap. The governance community must immediately pivot from behavioral black box testing to white box mechanistic auditing, moving away from trying to prove danger and toward enforcing mandatory compliance frameworks.

Evals are being gamed not because the methodology is insufficient but the models on which the compliance audit run are sophisticated enough to game the audit. IC methodology already solved the problem of denied human capabilities through triangulation by using independent behavioural signals not better direct elicitation .The AI safety community needs to make the same epistemological shift. The question isn't how to make evals harder to game, it's whether evals are the right instrument at all.

Invitation for bets I’m willing to bet that Anthropic’s revenue growth over the next year will be slower than its revenue growth over the last 3 years. I proposed a specific bet here. Anyone who wants can offer to take the other side of that bet. Or you can make a counteroffer. I’m also willing to make a longer-term bet that the AI industry is in a bubble. I proposed a specific bet for that, too, here. Feel free to offer to take the other side of that bet or make a counteroffer. I’d also be open to other bets. It seems pointless to bet about whether AGI or transformative AI will be deployed within the next 5-10 years, yet, for the heck of it, I would agree to a bet against that, too. (I’ll make bets for small, nominal amounts of money to be donated to the winner’s charity of choice, since the practical and legal problems with betting are too large otherwise.) I’d also bet against the deployment of 100,000+ SAE Level 5 fully autonomous vehicles in North America within the next 3 years, if anyone has a strong opinion on that. I’d make a similar bet against the deployment of autonomous humanoid robots in North American households, although we’d have to come up with some specific resolution criteria. Similarly, I’d bet against any significant level of near-term labour automation by LLMs or generative AI. Or against LLMs becoming capable of performing all sorts of specific tasks well. On any of these topics, I’m also open to invitations for a public dialogue. (More on that topic here.)

SMBC by Zach Weinersmith is doing a great job of conveying AI Safety memes more widely. Relevant comics: https://www.smbc-comics.com/comic/speech https://www.smbc-comics.com/comic/safe https://www.smbc-comics.com/comic/ai-17 https://www.smbc-comics.com/comic/ai-15 I would love to see his take on an illustrated AI Safety book, like 'Open Borders' meets 'If anyone builds it, everyone dies'.

AI systems that match or exceed human intelligence could very plausibly arrive within the next decade, and raise some significant challenges as they do. Alongside the well-known issues surrounding AI safety, there are many other potential problems that we don’t yet seem prepared for, but that could affect society on a huge scale. Forethought has published lots of research on this recently, and we wanted to cover some of the key points for people who might be interested in exploring some of these challenges with their career. A few of the issues that seem particularly important: * How a sharp acceleration in technological progress could leave us unprepared for new technologies, including potentially dangerous ones like engineered pathogens. * How growing reliance on AI for information could reshape how individuals and institutions form beliefs, and the risks this might bring. * The prospect of power and value lock-in, where AI provides the means for institutions and groups to entrench power dynamics and values. * Questions around AI sentience and welfare, and why now may be a particularly important window for thinking about them. * Ideas for how people can engage with these problems through their career, along with relevant organizations in the (broad) space. Here’s an excerpt from our section on the risks of accelerated technological progress: We can think of technological development as akin to pulling balls from an urn. As we develop each new piece of technology, it’s like we’re reaching into the urn hoping to pull out balls that will help humanity. Many of these are straightforwardly beneficial, like vaccines and other medical technologies. But some balls can be highly dangerous, like nuclear technology. The trouble is, we have a limited ability to know whether technology will harm us or help us until we’ve developed it. The scientific speedup that advanced AI may bring is like tipping this urn upside down, spilling thousands of balls onto the ground at

More EA undergrads should do political volunteering. It's impactful AND fun. Choose an election that's impactful (e.g. AI safety candidate) and neglected (e.g. primaries in always-blue/red places), couch-crash the weekend there, and volunteer with the campaign. I say this after doing 15 hours of street canvassing myself. I was surprised by how anecdotally impactful and fun it was. If you like people-watching, talking to strangers, and/or joining passionate projects for a weekend, I think you'll also love this. I wish I thought of this earlier. Literature on the impact (Claude-generated): Kalla & Broockman's meta-analysis of 49 field experiments finds zero average persuasive effect in general elections, but effects do show up when voters lack a partisan cue (i.e. primaries and ballot measures). Mann & Haenschen (2024) find mobilization effects (e.g. canvassing) are 33-76% larger in low-attention races than in high-attention ones. Your marginal volunteer hour goes much further in a primary.

My three most recent posts on Substack are relevant to effective altruism: * Shouldn’t we spend money on AGI safety, just in case? * The sad decline of effective altruism * The pseudo-religious origins of the AI bubble I can’t discuss them on the EA Forum, but I’m happy to do so on Substack.