On the occasions that I’ve talked to people from the AI safety community about the prospect of pausing AI development, the major theme has been that a pause isn’t politically feasible. Rather than pushing a radical pipe dream that would probably alienate potential supporters, we should be willing to bow to the prevailing sociopolitical forces and be content with nudges in the right direction.

And then SB 1047 happened.

AI safety advocates proposed California Senate Bill 1047 back in 2024. All things considered, the Bill would have had a fairly light touch: it only applied to very large and/or costly models, it merely clarified how “standard of reasonable care” applies to frontier AI models, and the other things it required were pretty straightforward: creating and being transparent about safety and security protocols, undergoing annual third-party audits, and whistleblower protections. Even the “kill switch” bit was just to make sure that in an emergency, there were mechanisms to force a shutdown. It was exactly the kind of moderate and reasonable proposal AI safety people championed. Nevertheless, the Bill was heavily opposed by the frontier AI companies and eventually vetoed by CA Gov. Gavin Newsom.

I wonder if AI safety policymakers felt like Chamberlain did when he heard about the Nazi invasion of Poland.

For what SB 1047 made clear was that we were not being negotiated with. We were being played. Like Hitler, the frontier companies made assurances meant to placate us, to give the illusion that if we made concessions to them, they would make concessions to us. Like Chamberlain, we had numerous examples of their previous duplicity, yet we embraced appeasement due to insufficient armament and willful denial, and we were shocked and humiliated by the enemy’s blitzkrieg. But unlike Chamberlain, even 2 years on, we have failed to recognize the necessity and reality of war (in our case, metaphorical war). If even SB 1047 provoked such opposition and ultimately failed to pass, we have to ask ourselves: is there any AI safety policy worth its salt that they willlet us pass? What, if anything, do we stand to gain by continuing our policy of appeasement?

This isn’t to say that policy isn’t a vital part of AI safety. It’s more important every day. But it’s to say that “policy” by itself isn’t enough. Even if we came up with and passed the most ingenious policy under the current sociopolitical constraints, it would be about as effective as a spiderweb trying to stop a Boeing 747.

That’s why I think the solution lies in the other direction. Rather than working within the status quo, we need to push the Overton window in the direction of safety. Even if a pause isn’t the ultimate policy we choose, it should be a serious consideration that is on the table. Only once a pause is politically plausible will I feel that we have pushed the window far enough.

We have no more time or use for appeasement. We must come with not peace, but a sword (metaphorically).

Going To The People

Why isn’t there a mass movement already?

It’s a question worth taking seriously. If AI is so dangerous, then why aren’t people taking to the streets in the millions? Doesn’t this show that people are unconcerned and movement-building is futile?

I think it’s worth putting things in perspective here. ChatGPT came out in 2022—less than 4 years ago. Even many highly educated people who use LLMs regularly in their jobs haven’t really had time to get used to it, and for most people, LLMs are probably still seen as those chatbots you can have funny little conversations with. Existential risk from AI is, to many of us, still firmly in the realm of science fiction.

It’s especially challenging because despite the furor within the AI safety community, nothing has really happened… yet. Even something as momentous as Mythos is still very abstract: it’s a story, not a “real” event. And even for someone like me, someone who has a fair bit of knowledge about AI (even if it’s nothing compared to your average AI safety person), it’s difficult to have a good intuitive sense of AI capabilities development. I see all the graphs go up, but I don’t feel it.

Does all this mean building an AI safety movement is impractical?

I don’t think so.

To be honest, I don’t think that the EA and AI safety communities have really tried very hard to engage with the public more broadly. There are many prominent technical and governance organizations like MIRI, MATS, BlueDot, etc., and there are huge AI safety communities at universities like UW–Madison and CMU which have done a lot of great work around technical and policy AI safety. As a result, we have a mound of fairly academic books and a mountain of papers on AI safety. But we still have a relative dearth of clear, public-facing communication around the risks of AI. Only recently have I seen any media about AI safety genuinely meant for a lay audience, including the book If Anyone Builds It Everyone Dies (2025), the AI Doc (2026), and Bernie Sanders’s public warnings about the existential risks of AI (2026). And to my knowledge, almost no major AI safety organization has done any work on coalition-building or activism; until recently, PauseAI was the only exception I knew of.

Speaking of not trying very hard: last I checked, there were only a little over 1,000 people who’ve committed to the If Anyone Builds It, Everyone Dies March on Washington. I know there’s more people in AI safety than that. So if you’re one of the AI safety people who hasn’t committed already, go do that now.

The thing is, the seeds of a grassroots AI governance movement seem to be present: recent polls in 2023 and 2024 found that 77% of Americans trust businesses and government to use AI responsibly either not at all or not much, and I’d be surprised if the numbers have dropped since then. The potential is there, just waiting to emerge. Now is our best opportunity to guide and sharpen popular energy to achieve meaningful AI governance.

But we can’t just engage—we need to engage mindfully. If there’s one thing people hate, it’s being flat-out told what to think. For example, I think the “well actuallys” in response to concerns about data centers, while well-intentioned and truth-seeking, are much more costly to AI safety than they are beneficial to our epistemics. However these correctives are meant, they come off as “once again, the uneducated mass is wrong and stupid; good thing they have us to guide them.” That’s not to say that we shouldn’t aim to correct popular misconceptions; it’s an important part of our role. But we need to do so in a way where we make it clear that we’re not ignoring the broader concerns, e.g., that the companies building the data centers have absolutely zero concern for the communities they build them in and are currently bound by very few rules of conduct.

While we want to spread the message around the existential risks of AI, there are plenty of other reasons which make people concerned about AI which converge on similar policy goals. Perhaps the biggest one is power and wealth concentration in the hands of a tiny elite. Even if our reasons for being concerned about such concentration don’t fully overlap, it’s an area where we can find common ground upon which we can build a coalition.

It’s not that we don’t have an agenda; we do. We just need to make clear to people that our agenda is the well-being of all humanity and sentient life, and the way we do that is by including people, talking to them as equals, and meeting them where they are.

But that requires more than just talking the talk: it’s not enough to include people in theory. At least some wing of the movement has to go out and engage with real people in real life. People at town halls, in unions, at churches, in discussion groups. People in movements with potentially convergent goals like No Kings. It’s not easy, but I think it’s necessary.

But What If We’re Bad At Activism?

One of the concerns I got when I was discussing this with an AI safety person was that AI safety people are bad at movement building, and that doing movement building badly enough could be net negative.

I think backfire effects are fairly overrated in that unless people are doing something violent or destructive, the worst I’d expect is that the action is roughly neutral. But I agree that AI safety people are probably generally at a unique disadvantage when it comes to movement-building. My guess is they don’t have a great activist tradition, much knowledge about social movements, or involvement in social networks that connect them with the broader population.

But I don’t think any of this is fatal. I wasn’t raised an activist, and I still have a ton to learn about the strategies and tactics that work for social movements. And I’m an introvert with a small social network composed mostly of highly-educated non-activist social elites. I might not be the best activist, but I’m participating and improving, and I’d go so far as to say I’ve had a modest impact.

I think the biggest barrier is that in the AI safety community, it’s something of an uncharted path. It’s frightening, uncertain, unstructured, and demanding of skillsets AI safety people have generally not invested as much in.

Luckily, we don’t have to reinvent the wheel; we just have to copy it. We can learn what we need from activists and organizers in other movements. Even within the EA community, we have a source of activist know-how and resources, e.g., in many parts of animal advocacy.

I propose that we create a program that facilitates exactly this: matching up AI safety folks who are interested in activism and movement-building with experienced activists and organizers who can guide and mentor them. And having these activists guide strategy and tactics for activism would go a long way to mitigating the concerns about skill issues.

But What If Timelines Are Short?

Then movement-building is pointless. But then, I’d also think pretty much everything is pretty much pointless, including technical research and policy, since I think at this point both are only able to offer band-aid solutions/marginal improvements. In such a world, the highest EV actions would be very extreme ones.

But I don’t think we live in such a world. It’s worth noting that even pessimistic forecasts like AI 2027 have a distribution that places substantial probability on ASI coming substantially later. And I think it’s reasonable to place substantial probability on ASI not arriving for several decades even; there is just a lot that we don’t know. I think this makes it reasonable for a portfolio approach to invest some resources into building an activist wing, just in case it ends up being useful.

Pillars of the Movement

What should be the goals of AI governance?

One major goal should be establishing effective democratization of AI, something that Bernie Sanders has publicly championed in recent months. Critically, such democratization must be complete and holistic. It cannot be a mere promise to redistribute the profits of AI or, even more vapidly, to democratize the use of AI. Rather, it must be the establishment of public oversight over the development, deployment, and distribution of benefits/costs of AI. If AI is really an existential threat, then it should be the public, and not billionaires or technocrats, who make the high-level decisions about AI development and deployment.

I anticipate objections from technocrats who believe that people are, to put it bluntly, too stupid and/or lazy to be fit to govern the future of AI. For one thing, I would remind them that the public is overall wary about AI, and some evidence suggests they are substantially more wary than AI experts. More broadly, the purpose of democratization is not just that of aggregating expertise; it is also about checks and balances. Especially with a technology as tantalizing as AI, we cannot rely on the largesse and character of those in power to manage it wisely for the greater good. As we have seen again and again over the last several years, those who have power over the development and deployment of AI are inevitably driven either to corruption or exile. Even those with good intentions, like the people at Anthropic, have ultimately reneged on their commitments to safety and transparency, plunging headlong into a reckless race to the bottom. It’s very hard for me to see how public governance could do worse than that.

I also anticipate the objection that we are not sufficiently good at democracy. While it still might not be as bad as our current situation, I’m not thrilled with the prospect of the US government (especially under the current administration) nationalizing AI. My counter is this: until we as a species have developed and implemented the kinds of real deliberative and participatory democracy worthy of the name democracy, we are not mature or skillful enough to safely develop and deploy AI. If you can’t govern yourself, how the hell are you gonna govern somebody else? Thus, true democratic control over AI should be seen as a precondition for the development and deployment of transformative AI.

A related major goal should be the establishment of regulation with teeth. No “voluntary commitments” or “self-governance.” No, AI research, development, and deployment should be regulated with a strictness on par with R&D on biohazards and nuclear weapons. If it’s genuinely an existential risk, it should be regulated as if it’s an existential risk. I don’t know exactly what policies would really achieve this; that’s the job of AI policy people to figure out. Again, the job of the movement is to make such policies politically viable.

Finally, and in some ways most immediately, a major goal should be an international treaty on AI. The Moloch argument is the most real argument that accelerationists have, and the only one I have any real respect for. Eliminating the race dynamic and establishing bilateral communication and commitment mechanisms between the US and China is therefore critical to the success and efficacy of other AI governance endeavors. And while I have no love for the PRC and no wish for their values to be the ones which prevail, we are the ones who hold the lead and most of the cards in this game. Some have even made the case that we, and not China, are the bottleneck when it comes to the AI race dilemma. So really, it’s on us in the US to escape this trap.

What is the aim behind these pillars? It is to give ourselves the greatest quality andquantity of real-world shots. As Eliezer Yudkowsky argued back in 2023, “trying to get anything right on the first really critical try is an extraordinary ask.” No matter how hard our smartest minds think about alignment, no matter how many tests are run in the lab, there will always be the risk that something is missed, that some mistake is made. This is why it’s important for us to slow way down and be patient, as the only way to mitigate this kind of risk is to iterate gradually in the real world. Even so, there will be great pain and crises, but the hope is that at least we will only face a series of manageable such occurrences, rather than a single, irrecoverable one. Here, time is on our side, for even a few hundred years of a wait would be nothing compared to the millions of years our species may be able to keep.

Coda: The First Shoots

As a true intellectual does, I write in the abstract and theoretical. Is there anything more real?

The answer is yes. I want to highlight an organization that my friend Max founded earlier this year: the Pittsburgh AI Safety Coalition (PAISC). PAISC is a fledgling organization that aims to do exactly the things I’ve been arguing for: building a coalition that bridges the gap between AI safety communities and the lay public with the purpose of AI safety broadly and the establishment of a bilateral international treaty more specifically. So far, its member organizations include CMU’s Effective Altruism chapter, the Pitt Democrats, and 2 unions (each with approximately 1,000 members), and a handful of faculty and AI safety students at CMU support the organization in a private capacity. Max is the kind of ideal leader I’ve been talking about: one who is an experienced and competent activist/organizer for other causes and who has personal and professional connections to organizations outside the EA and AI safety spheres.

Of course, PAISC isn’t going to change much by itself—maybe it could get Pennsylvania’s Senators to be a bit more safety-oriented. But imagine if there were AISCs all across the country, or even just in every city where there’s currently an AI safety group. Imagine the kind of social and political shifts we could make if we had that.

Again, I think it’s critical for AI safety that we have a grassroots movement. Because if we don’t, we won’t be able to pass the policies we need to ensure safe and responsible development of AI. Our enemies will make sure of that.