I can't get over this sentence:
We cautiously support some technical safety work.
Yeah wouldn't want to fullthroatedly support technical safety work - in the worst case scenario, we might figure out how to build AGI safely, which would greatly advance AI capabilities.
The broader context, as I understand it, was that many forms of technical safety work either (a) are capabilities masquerading as safety or (b) are too dependent on the labs for compute and model access and thus vulnerable to industry capture. Without wholly endorsing this frame, I do think it is a reasonable concern to have and it accurately describes at least some work labeled as safety.
I understand that people undermining alignment work will attempt to justify it with the concern that it may advance capabilities and may not advance safety. I just think that's a terribly mistaken drunk mormon hypothesis.
It's proceeding from the false premise that somehow alignment and capabilities are two equal concerns that we trade off against each other, and if you can imagine a way that it ends up advancing capabilities and not advancing alignment, you shouldn't do it. But the tradeoff is not anywhere close to equal, the more accurate perspective would be that if you can imagine it advancing alignment, you should do it.
You appear to be arguing against a position I do not actually hold. I support many forms of alignment work. I expect most avenues to fail, yet still be worth exploring. I expect many such avenues to advance capabilities very little or not at all. Much of interpretability seems to be like this. And even given that some forms of interpretability may be used to train smarter models, it still (probably) seems worth the price to understand those models better.
A subset of work labeled "alignment" or "safety" seems fundamentally flawed and at best a waste of resources. It might be aimed at the wrong things, like Elon Musk's plan to make AI that "seeks truth"; or it might be falsely labeled "alignment" while actually doing something else, like attempts to get AI to refuse to talk about [insert politicized issue here]. Here I am deliberately picking obvious examples as an existence proof of this category; one may reasonably disagree as to what other forms of research qualify.
I do think there is a further kind of research which is just advancing capabilities and doing little for alignment, e.g. the attempts to automate AI research itself. In this case the tradeoff seems overwhelmingly lopsided against proceeding. The faster you build it, the sooner you die.
Alignment and capabilities are no more equal than poison and medicine. It is still a bad idea to drink a cup containing mostly poison.
I of course cannot speak for the reasoning of PauseAI leadership, and can only make an informed guess as to their stance on the tradeoffs involved.
You appear to be arguing against a position I do not actually hold
Yes, I was arguing against Pause AI's position (both summarized by you in the piece and as expressed by their leadership elsewhere), I'm aware that you hadn't expressed a strong pro- or con- stance on it either way.
My comments were a little overly combative, which wasn't meant to be directed at you in particular - my frustration is that I think a lot of people consider "We cautiously support some technical safety work" to be a moderate position while I believe it is insane.
I agree that it's theoretically possible for there to be bad alignment-capabilities tradeoffs, just like it's theoretically possible for there to be bad tradeoffs of anything vs anything.
But in the actual world we live in today, there is widespread skepticism of the existence of AI risk, including among AI researchers. Many people are happy to work full steam ahead on developing AGI with no thought to potential existential risks, and declare this publicly. The issue has low salience among the general public and even less so abroad. And the difficulty of building AGI falls automatically year by year due to the advance of Moore's law.
With all of these factors in play, I think AGI is happening. More to the point, if it doesn't happen, it won't be because a bunch of individual people who were worried about AI risk refrained from doing things that might advance capabilities; it would be because we somehow got an international agreement. Individual people withdrawing from the field doesn't have anything to do with an AI pause; all it can do is slow things down. And there is a hard limit on how much individual withdrawal can slow things down, because there is a large core of people who believe AI poses no risk who want to advance it as much as possible who will not be dissuaded, and the more you slow down the more they're aided by Moore's law.
In this environment, it is really important to try to figure out alignment. That is the thing to focus on. The problem with all the examples of bad alignment research you gave above is not that they advance capabilities, it is that they are bad alignment research.
I can't speak to Pause AI's stance, but in my view, short timelines are one reason for statements like "we cautiously support some technical safety research". Capabilities are so far ahead of alignment that it seems unlikely alignment would be solved in time, even with a lot more resources.
Yesterday I described an experience that impressed upon fifteen-year-old me the importance of speaking with urgency and courage when something awful is happening.
I lived a fresh reminder of the importance of courage last week at PauseCon, a first-of-its-kind conference in Washington, DC run by PauseAI US.[1]
I was there in a personal capacity, and the opinions in this post are my own. Those opinions mostly boil down to: It was really, really good. I’m impressed and I want to see more work like this.
PauseCon's main programming consisted of an informal sign-making gathering, several presentations by local organizers, a lobbying workshop, scheduled meetings with Congressional offices, several social events, and a protest in front of the Capitol on Monday. They were all pretty fun and productive, and I’m dedicating a section to talk about each.
I made a sign! Or tried to, anyway. I am not very good at making signs yet, but maybe one day.
The local presentations were inspiring, and included an impressive geographic diversity. The obvious places like New York and California were represented, but so were Boise, Idaho and Anchorage, Alaska. If any of y’all are reading this, thanks for making the trip!
One local organizer described going to a music festival to talk to people waiting in line, which many thought was brilliant and immediately started making plans to copy.
In another anecdote, an organizer was doing some tabling work with a petition to sign. People would see the “AI” on the banner and approach, asking “pro or anti”? And when the organizer said “pro-human” or “anti-AI” or similar, many would say “GIVE ME THAT” and sign the petition immediately. (Lots of people really hate AI.)
Another organizer described a long campaign of patiently but stubbornly following up with his representative’s office for weeks until they got on board.
Perhaps most inspiring, though, was the slide which had in great big handwritten letters:
“Where there’s life, there’s hope!”
I was pleasantly surprised by the PauseAI US strategy. A sampling of their talking points from the workshop they provided:
Other talking points:
I noticed some dissonance here—“moralizing, confrontational advocacy” sure is a way to describe your messaging strategy—but it was brief, and I also noticed that during the workshop they did not encourage volunteers to do anything like yell at people on social media. The workshop was focused on polite, professional conversations with policymakers. It was hard to find fault with much of what the leadership actually advocated even when adopting a cynical stance.
Connor Leahy showed up as a guest speaker. His advice differed slightly from that of PauseAI, but it was things like “say ‘multilateral agreement’ instead of ‘treaty’ because a treaty is a specific thing that has to be ratified by the Senate”. (My take on this, which I later told Felix, was that this was the kind of wonkish inside baseball you’d want for a formal meeting as a think tank expert but not necessarily a concerned constituent. And ‘treaty’ fits on a sign.)
He recounted speaking with a famously rude staffer and responding to a contemptuous “are you the idiots who want China to win?” with a disarming “well obviously a unilateral pause would be dumb, we need to treat this like the Cold War and negotiate an international deal”, and that reportedly made the staffer go “huh.” He advised treating our proposals as obvious and common-sense; of course you don’t want to build something smarter than you, of course when you’re in an arms race you sit down to negotiate about it…
My main takeaway was that I agreed with PauseAI’s actual platform more than I expected from the online arguments, even after attempting to correct for the fact that people are meaner online. Insofar as I may have disagreements with PauseAI US and Connor Leahy, it’s mostly not the sort of thing that affects the 10,000-foot view people express in a strategy talk. Maybe others already knew this in their bones, but I appreciated the chance to calibrate in person.
Again I was impressed by the degree to which the PauseAI talking points said almost the exact things I hope to communicate to policymakers.
The central asks for each office were:
The first two bullets were the asks they told everyone to lead with. The third bullet was a “compromise” or “moderate” option and references a bipartisan bill introduced by Senators Hawley (R-MO) and Blumenthal (D-CT) last September, which PauseAI US thinks is good enough to officially endorse with their limited lobbying budget.
They also had separate asks for members on specific committees. Foreign Affairs, China, and Foreign Relations would be asked to hold a hearing on extinction risk and a possible treaty; Commerce committees would be asked to hold a hearing on extinction risk and domestic regulation. Senate Commerce folks would also be asked to push a floor vote on the Evaluation Act, which has been stuck there for a while.
(I had mixed feelings about the Evaluation Act, which some said was a messaging bill and others described as solid domestic transparency regulation. I did bring it up in my solo meeting, though. It seems net good to boost in any case, with some nuance as to whether it’s endorsed as a message or as serious policy. I will not be going down that rabbit hole here.)
The messaging on the treaty was also impressively tight:
Thus prepared, we set out on our mission.
Felix De Simone, organizing director at PauseAI US, did a great job scheduling meetings with staffers for (checks notes) at least sixty people plus cancellations. We couldn’t get a meeting with every relevant office, but I still met more staffers than I expected to; when we couldn’t get on the schedule we dropped in anyway to leave material and get contact info for followups.
On Monday I tagged along for a meeting with the office of NJ rep Robert Menendez (no, not the infamous one, the other one). I’m not a constituent, so the person who was took point. Then we reconvened with several other New Jersey folks to drop in on the office of Senator Cory Booker. (No one was available, but we got contact info for a followup.)
On Tuesday, several of us met with a staffer in the office of Senator Andy Kim, and gave the pitch. Afterwards, I rushed across the Hill to the House offices for a solo meeting with the office of Donald Norcross. The meeting had been moved up at the staffer’s request, so I had to hustle. I got a two-for-one deal on staffers, though! I hope this means they were intrigued.
It’s a little hard to say how well the meetings went; staffers can be difficult to read and it’s their job to be polite and friendly and make people feel heard. Still, I thought we made progress. The staffers asked good questions, too; one asked what my timelines were and I brought up the AI 2027 forecasts. Another asked about our engagement with the labs; PauseAI’s official answer is “we don’t” but I took over and channelled some of the conversations I’ve had or witnessed at Lighthaven or on LessWrong.
My Tuesday afternoon was freed up by the moved meeting, so I got lunch with a group of PauseAI folks including Felix. He had one final meeting on behalf of a constituent who couldn’t attend, and after chatting for a while I offered to help. He was excited to have a MIRI person who worked on If Anyone Builds It to tag team with, and we hashed out a plan where I’d cover the scary AI stuff and he’d lean on his practice talking policy.
He was also really excited by the bipartisan statements graphic. “I wish I’d seen this when I was making our binders!” We printed a copy. This was a great moment for me, and updated me towards proactively sharing our best material with allies.
I think the resulting tag-team in the office of El Paso rep Veronica Escobar might have been my favorite meeting. We were scheduled to meet with Escobar herself, but she was reportedly stuck in a vote and could only drop in briefly to say hello. (We weren’t the only ones this happened to; apparently things are often chaotic right after a recess.)
Since all of our reps were Democrats, they didn’t have much pull on the Republican-controlled committees for secondary asks. But we still covered the treaty and x-risk asks for everyone. A couple of my peers from New Jersey reported a good meeting on Monday with their rep’s office that got them a potential lead on at least two more offices.
Marginal progress!
Takeaways:
We were encouraged to treat these meetings as a beginning, rather than a one-and-done, and to follow up. I intend to.
I’m a little sad that I didn’t get much practice in arranging meetings, which feels like one of the hardest steps to me, but I still feel grateful that Felix handled that step for PauseCon and I think it was the obviously correct move.
Tag teaming works great. I got to play both sides this week; in one meeting I was the policy wonk while an ML engineer talked about AI, and in another I provided the tech context for Felix’s policy proposals. I could do the whole script by myself, but it still feels better to pair up and specialize.
I appreciated the chance to unwind and chat more informally. I talked to someone who was contemplating driving around the country starting PauseAI groups, a really impressive dedication to the cause, and even some international folks, in the US temporarily to work on something or participate in PauseCon.
Overall, it was great to meet a bunch of folks earnestly and enthusiastically trying to save the world.
The protest at the Capitol was my first one ever! I think it was well-organized, and it encouraged rather than discouraged me about attending more. The organizers had plenty of supplies and signs (shoutout to TJ, I believe it was, for arranging to lug hundreds of pounds of stuff to and from the event, with some help, and to Anthony Fleming, who runs PauseAI DC and organized the protest itself.)
I didn’t notice the protest attracting much attention outside itself, though some passerby did stop to talk with participants. I get the sense DC sees a lot of these and is largely inoculated against them. I nonetheless think the protest did what it set out to do, rallying a large and visible group of people to make their voices heard, and perhaps more importantly helping to cement an identity as the kind of people who come together to bravely and stridently stand up and tell Congress to get their act together on AI.
There were a lot of speeches, and some were quite good. I didn’t agree with all of them, but I hardly expected to. I think that’s part of the point of the movement; whatever our reasons, we’re all here, and we’re all pulling in the same direction. For my part, I recorded a couple of short videos for later sharing.
One talk at PauseCon described an “avalanche of outrage” around AI. It’s not entirely controllable, but it can sort of be aimed. To a first approximation, I model PauseAI US as trying to channel the avalanche, trying to draw together widely divergent views and reasons for disliking AI and get them pointed in productive directions. Lining up this many voices behind an international treaty is an impressive accomplishment, and I hope PauseAI meets their goal of roughly doubling their local group count from ~30 to ~75 this year. If you want to help them accomplish this, see here.
My experience at PauseCon filled me with hope and pride.
I listened as the leadership of PauseAI US addressed a roomful of volunteers from across America, from New York to Idaho to Texas to California to Alaska, young and old alike, and bid them speak their true concerns with courage and frankness in the halls of Congress.
I watched a young man with trembling voice and shaking hands speak truth to power and not falter. From a meeting with the office of his representative he emerged, gushing and proud, and in my heart a young boy cheered.
I met with my Congressional offices, asking them to join the growing bipartisan list of their colleagues who have acknowledged the risk of extinction from superhuman AI, and to push for an international agreement halting the race.
It is my earnest and sincere hope that members of Congress concerned about AI can find within themselves the courage of a fifteen-year-old boy, and stand, and say, “Enough.”
If you, too, are concerned about the path the world now treads; if you can find within yourself the courage of a fifteen-year-old boy; and if you wish to add your voice to the growing chorus of those who say “Enough”; you can do so here.
There is a global PauseAI as well, largely unaffiliated with the American nonprofit. I will sometimes use “PauseAI” for brevity in referring to the US organization.
Executive summary: The author argues that PauseCon 2026 was an impressively effective and encouraging example of grassroots AI pause advocacy, updating them toward supporting PauseAI’s approach and emphasizing the value of courageous, coordinated political engagement.
Key points:
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.