Crosspost from my Substack.
Inspired by Protest Review: San Francisco’s AI Treaty March by Harjas Sandhu.
This was the largest AI safety protest in the world so far (Bay Area, LOCK the fuck in), with over 2,000 attendees!
We also had people there who were anti-datacenter and climate change, which was awesome. Big tent, everyone!
London Weather Gods Blessed Us Today
There was this band playing absurdly catchy music in the front.
And this dog.
Reminder that animals are 99.9999% of the constituents of AI risk
Field’s Medalist Tim Gowers gave a speech about the Riemann hypothesis and the death of math.
He described having this enormous “IF” hanging over his future, hanging over the future of his grandkids. He described wondering if they would get a chance to grow up. It was touching.
I also spoke to a mathematician who had come to see him. I asked her to evaluate the claim that we’ve seen a decade of progress in math in the last few months. She responded “it’s been more than a decade”.
(Meanwhile Paul Shearer thinks the best description of AI’s impact on math is that it “annoyed a bunch of mathematicians”.)
Speaking of Pauls, Christiano’s name comes up a lot in the speeches. This makes sense given just how respected his voice is in the space.
Paul Christiano is one of the most decorated AI researchers in the world, so his takes are a much bigger deal than your average doomer. He founded the Alignment Research Center, which later spun off into METR. Also, Paul invented RLHF—one of the most important techniques in all of AI—and is now on the OpenAI Foundation Board.
In his recent conversation with Dwarkesh, Paul Christiano estimated a 1 in 3 chance we get super-intelligence within a year and a ~10% chance of takeover in the next 12 months.
If this happens, it would probably result in the permanent disempowerment or death of you and your family.
This brings us too…
We passed by lots of people cheering and snapping photos of us. Cars honked as they drove past.
At times, I wanted to grab their shoulders and shake them:
Listen. We are not here for fun. We are worried for your life. We are worried for your children’s lives.
I also wanted to change our chants from “PAUSE AI!” to something more urgent. My friend Una and I settled on “PAUSE AI! WE DON’T WANNA DIE!”
I thought the music was too cheery too and the chanting was too mild. If we had properly internalized the stakes, we would’ve been screaming our heads off.
If you believe there’s a 10% chance of extinction in the next few years, you should act with AT LEAST the urgency of someone trying to prevent the imminent deaths of 800 million people, since that is the death toll in expectation.
By the way, 10% is also the median answer that 1,500 AI researchers gave for the chance of extinction from AI. It is also the probability that Anthropic’s Evan Hubinger’s and Turing-Award-Winner Geoffrey Hinton gave.
Imagine what a protest would look like if we believed that 800 million people were going to die at the hands of AI companies. What kind of crowds would we draw? What kind of intensity would we have?
By comparison, Extinction Rebellion’s 2023 Climate protest drew 50,000 people in the UK, 25x ours. AI protests are still small.
If we want to survive, the AI safety movement needs to be on an exponential curve. we do end up dying, I think it will be because we cannot accurately internalize the risk. It is incumbent on us to try.
Consider joining the email list of Pause AI here, or going to a local protest.
“I wish it need not have happened in my time”
“So do I, and so do all who live to see such times. But that is not for them to decide. All we have to decide is what to do with the time that is given us”
There was a moment, standing in front of Downing Street, listening to Irina Tavera speak, surrounded by posters, when a weird feeling struck me.
Everything was, for a moment, denaturalized. It felt like a scene at the beginning of sci-fi movie: a crowd of protesters screaming about AI, pumping their signs, calling for a slowdown. And then I blinked, and I was a character in that movie. I am in 2026, at an AI protest, in a world which is imminently going to create Superintelligence. I have no fucking idea what’s going to happen.
As I’ve learned more about AI risk, I’ve come to realize that I was taking the stability of my future for granted. I did not realize what my expectations were until they were broken: a life free from instability and risk and the fear of unexpected death.
I’m 21 years old. Most likely, my adulthood will look more like The Cold War than the 2000s. It will be a world teetering on the brink of loss, a world with an enormous “IF”.
I wish it need not have happened in our time. But it has. It’s time to decide what to do.
Please consider attending an AI protest, contacting your politician, or otherwise waking people up to existential risk.
"This was the largest AI safety protest in the world so far (Bay Area, LOCK the fuck in),"
This is an awesome take. It blows my mind that even just within the EA community itself we can't get thousands to protests in the Bay Area. I would be there if I lived there!