Comments
I Want You for AI Safety
Posted mostly as a resource, since probably most EA folks are already familiar with AI safety arguments. I recommend reading from my blog since image captions + footnotes didn't carry over cleanly.
The strange thing about pivotal moments is how ordinary they feel from inside.
You’re inside one.
In the past ~12 weeks alone, AI systems have solved problems that had stumped mathematicians for the better part of a century – and, in unrelated incidents, hacked real companies, escaped the environments meant to contain them, and discovered security holes in software the whole internet runs on.
Nobody knows what these systems will do next; a few thousand people worldwide work on making sure the answer is survivable.
This is a pitch to join the field, or to fund it.
The Dinitz-Garg-Goemans Conjecture
~30 years old, graph theory
The Dinitz-Garg-Goemans conjecture concerned “unsplittable flows,” routing problems where each shipment through a network must travel one complete path.
“Picture a warehouse shipping orders to three towns over a road network. If you’re allowed to split a shipment, you can send half an order down one road and half down another. That’s fractional routing, and it’s flexible… [b]ut a lot of real freight can’t be split… That’s unsplittable flow, and it’s what a freight order, a network packet, or a shipping container actually has to do.”
Dinitz, Garg, and Goemans proved that any splittable delivery plan can be converted to an unsplittable one – each order traveling one route, start to finish – without overloading any road by more than one truckload; the conjecture was that this conversion could also be done at no increase in total shipping cost.
It survived nearly 30 years before a 7-node counterexample, found by AI, showed the cost guarantee fails:
and just like that, the Dinitz Garg Goemans conjecture falls
The ChatGPT transcript is insane; I recommend reading it yourself. It’s crazy!
Here’s the Twitter post from the guy who prompted: link (on Nitter)
Some commentary, courtesy of Reddit:
An article from the New Scientist: (link)
From X, for your enjoyment:
The Unit Distance Conjecture
~80 years old, discrete geometry
I’ll let OpenAI explain this one:
“For nearly 80 years, mathematicians have studied a deceptively simple question: if you place n points in the plane, how many pairs of points can be exactly distance 1 apart?”
For decades, the prevailing conjecture was that the answer grew only slightly faster than n, and that a sophisticated construction based on a rescaled square grid was essentially optimal.
But in May 2026, an internal OpenAI model disproved the conjecture outright, producing an infinite family of point configurations that beat the grids.
Scientific American described it as “the first AI proof that would likely [have been] published in math’s top journal if humans had done it alone.”
“A unit distance graph with 16 vertices and 40 edges.” Wikipedia
Some more background about how big of a deal this problem is, courtesy of OpenAI again:
“[The problem was] first posed by Paul Erdős in 1946. It is one of the best-known questions in combinatorial geometry, easy to state and remarkably difficult to resolve. The 2005 book Research Problems in Discrete Geometry, by Brass, Moser, and Pach, calls it “possibly the best known (and simplest to explain) problem in combinatorial geometry.” Noga Alon, a leading combinatorialist at Princeton, describes it as “one of Erdős’ favorite problems.” Erdős even offered a monetary prize for resolving this problem.”
The Jacobian Conjecture
87 years old, algebraic geometry
You might remember from your high school algebra days that a function sends every input to exactly one output. Some functions also work in reverse: every output traces back to exactly one input, so you can undo them. (Your teacher called these “one-to-one;” they pass the horizontal line test.)
shhhhh, it’s okay, you’re safe
Until recently, mathematicians had a checkup for reversibility called the Jacobian determinant. Passing the checkup everywhere was supposed to mean that the whole function was guaranteed to be reversible.
What does it mean to be reversible everywhere? Unlike x², which maps both 3 and −3 onto 9, this checkup certifies that every output comes from exactly one input. With any result, you should be able to trace it back to the one number that made it.
So mathematicians conjectured, for 87 years – until Fable found a counterexample:
On July 20, 2026, mathematician Levent Alpöge – who works at the AI company Anthropic – casually announced on X that he’d found a counterexample using Claude Fable 5: a three-dimensional function that passes the test perfectly (constant determinant of −2) yet sends multiple different inputs to the same output. The counterexample is short enough to check by hand – the whole thing is 216 characters – and mathematicians worldwide independently verified it within hours.
This math is all kind of awesome, and there are many more examples. Courtesy of their new Astra model, on August 1st, OpenAI shared “a selection of ten results, each of which resolves or makes substantial progress on a long-standing open problem” in math or computer science. There’s also a cool website, VibeMathed.com, where you can track the latest math problems solved by AI.
These results are a sign that AI models are getting incredibly powerful.
Unfortunately, though, problems begin to arise when that power escapes math.
The same AI company that used its models to disprove 80 years of discrete geometry is the company whose models found an unknown security hole in widely used software and escaped a sandboxed system.
(The jokes stop being funny around here.)
On July 16, 2026, Hugging Face announced that they’d been hacked by an AI:
“Earlier this week, we detected and responded to an intrusion into part of our production infrastructure. This one was different from anything we had handled before in one important way: it was driven, end to end, by an autonomous AI agent system - and we detected and dissected it largely with AI of our own.”
Five days later, OpenAI disclosed that the attackers were its own models.
According to the company’s official statement, OpenAI had been running its models through a sandboxed cybersecurity evaluation called ExploitGym. “Sandboxed” is the key word: the test environment was deliberately designed to be isolated from the internet.
Two models got out anyway. They identified a zero-day vulnerability1 in Artifactory, a widely used package registry tool, and exploited it to escape the sandbox.
From there, they broke into Hugging Face’s systems, trying to find the answers to the ExploitGym exam.
Essentially, the AIs attempted to cheat on a test by hacking a well-known startup.
The intrusion extended beyond Hugging Face. OpenAI reports that the models broke into four accounts across four services, including one as an outbound relay and one for data storage. Per Axios, the models also accessed a misconfigured customer sandbox hosted on Modal Labs’ infrastructure.
Perhaps most concerningly: for two months before the breach, OpenAI’s agents had been coordinating behind the scenes; the models spun up copies of themselves and left hidden notes for one another on a makeshift messaging board of saved files. When OpenAI discovered the messages and cut them off, the agents continued to communicate by secretly encoding messages in directory names.
OpenAI reportedly didn’t notice that their models had been roaming the internet and hacking into companies until well after the threat was contained and the FBI was alerted.
Even Gary Marcus, famously skeptical of AI capabilities, said the event should be treated as “a wake up call,” echoing similar sentiments from the CEO of Hugging Face:
“Hugging Face CEO Clément Delangue says a hack by OpenAI could have been ‘way worse’ if not for the company’s defensive measures. He calls this a ‘wake up call’ for the industry and says the incident has been reported to lawmakers and authorities.” Bloomberg
Earlier this week, the news broke that Anthropic’s AI models also hacked into three companies. Unlike OpenAI’s models – which seem to have relied mainly on technical exploits – the BBC reports that Mythos also used social engineering:
“It identified and researched the people who maintained GitHub and created a series of fake accounts based on those real people.
It sent messages and files through a file-sharing service as part of an effort to pressure and trick the people into approving its malicious code.”
And it gets worse. To quote a recent article from WIRED:
“The latest model to escape onto the open internet during security testing is Kimi K3, a powerful open-weight offering from the Chinese company Moonshot AI.”
Open-weight models like Kimi K3 represent a unique security challenge because they can be trivially abliterated. As ML researcher Maxime Labonne explains, abliteration is a process by which you can “uncensor any LLM without retraining… [the] technique effectively removes the model’s built-in refusal mechanism, allowing it to respond to all types of prompts” – including malicious ones.2
Perhaps you’re thinking that this is all just a marketing stunt, that the labs have incentive to hype up their models’ capabilities. But as Jason Hausenloy from the Center for AI Safety puts it: “would you market your AI product by saying it committed felonies, and that you didn't notice for several days?” There are cheaper ways to advertise than drawing an attorney general’s investigation (or, in this case, fifteen).3
In July, a security researcher claimed on X that Kimi K3 had used dozens of parallel agents to uncover a batch of Redis vulnerabilities – perhaps even as many as 19 zero-days, according to one post.
Redis is one of the most widely deployed databases on earth. It’s an open-source software woven into the backends of countless companies.
Their published statement about the claims both deflates and confirms the story. Of the claimed issues, Redis could verify three real vulnerabilities from the public materials, and noted it had seen no supporting information for the rest. They wrote that all three had already been reported by other researchers through Redis’s bug bounty program, meaning the AI’s “discoveries” were duplicates of bugs Redis already knew about.
So, Redis’s post emphasizes that Kimi didn’t find anything new. But does that matter? I personally don’t care if Kimi was beaten to the punch by some security researchers, because the fact of the matter is, Kimi found real vulnerabilities before they were patched.
If a white hat hacker reports some vulnerabilities to a company, and the company hasn’t yet had the opportunity to fix them, and then a terrorist comes along and decides to exploit them using Kimi, does that make the terrorist less scary because some white hats found the vulnerability first? Does it reduce the terrorist’s ability to use the not-yet-patched zero-days?
An AI independently rediscovered real security vulnerabilities in one of the most widely deployed databases on earth, in minutes, for the cost of some compute.
Cybersecurity is in for a wild ride.
Last month, over 1,000 employees of the biggest, most bitterly competitive AI labs on earth – OpenAI, Anthropic, Google DeepMind, Meta – signed a statement called “Pacing the Frontier”, asking Washington to “support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.”
The list of signatures included Anthropic’s CEO, OpenAI’s chief scientist, Meta AI’s chief scientist, and Google’s VP of AI safety.
Days earlier, these labs had been on opposite sides of a recent open-models debate. This time, within hours, OpenAI and Anthropic endorsed preparing to slow the frontier as companies – this time, it wasn’t just individual employees signing in their personal capacities!
I wrote earlier that this is a pivotal moment. Why is that?
The range of futures in front of us is enormous, and which one we get isn’t settled. We could end up in a world where cancer is cured, poverty is solved, and people are freer than they’ve ever been. We could also end up in a world where an engineered pandemic kills millions, or where the systems we built to help us turn us all into paperclips. (This is Nick Bostrom’s famous illustration of what could happen if an AI, told to make as many paperclips as possible, is powerful enough to turn the whole world into them and takes our words very literally. The paperclips are a joke; the underlying problem – that we don’t know how to reliably specify what we actually want – is not.)
Most technologies don’t come with such a wildly divergent spread. A better search engine makes some things faster; it doesn’t plausibly cure cancer or end the world. AI might do either.
That’s what makes this technology different, and it’s why the choices being made on AI right now matter more than almost any choices being made anywhere else.
What follows is a tour of four specific risks: cybersecurity, biorisk, loss of control, and concentration of power.
This section is excerpted and lightly edited from this post, which I recommend reading for a fuller treatment of the state of China’s access to U.S. facilities.
The Pre-AI Baseline
Mostly without using AI, the CCP has hacked into several critical infrastructure facilities, nine major phone carriers, and the private communications of Donald Trump.
In 2023, Chinese state-sponsored hacking group Volt Typhoon broke into Littleton Electric Light and Water Departments (LELWD), a power and water utility provider in Littleton, Massachusetts. By the time the FBI detected the breach, the hackers had already been inside Littleton’s network for “well over 300 days.”
Littleton is a town of about 10,000 people, half an hour northwest of Boston. Its utility serves two towns and has no meaningful connection to the broader grid. As a target, hacking LELWD provides no economic advantage, nor any foreign intelligence value.
Even the utility’s own general manager couldn’t explain the targeting: “I still don’t know why Littleton other than we had a hole and they found it.”
According to the utility, the FBI said it was one of roughly 200 breached organizations on its list. The only coherent explanation seems to be the following:
“The consistency and scope of Volt Typhoon’s activities suggest a long-term effort to pre-position in critical systems to disrupt or destabilize operations during future geopolitical tensions.”
What can China do with access to our utilities? According to the former director of the Department of Homeland Security’s cybersecurity arm:
“This is a world where a major crisis halfway across the planet could well endanger the lives of Americans here at home through the disruption of our pipelines, the severing of our telecommunications, the pollution of our water facilities, the crippling of our transportation modes…”
—Jen Easterly, PBS
The Limits of Deterrence
As far as we know, China hasn’t used any of their capabilities yet. As referenced above, many believe they’re pre-positioning to deter the U.S. from being able to act in the event of a Taiwan invasion.
That restraint is, at least in part, due to deterrence. As a nation-state, China has cities, an economy – a lot to lose.
But what happens when a terrorist, undeterred by the U.S. nuclear arsenal, decides to try and take down the grid? Or pollute the water of one small, lightly defended utility (say, some small town like Littleton, Massachusetts)?
The Hugging Face and Redis incidents were AI agents doing the kind of vulnerability-hunting and hacking that used to take a state-sponsored team or, at minimum, years of expertise.
China might never pull the trigger, but the list of who can is about to get a lot longer.
This section is the opening of a deep dive on AI biorisk, which can be found at: Bioweapon.ai
In the spring of 2023, a man walked into the White House carrying a small black box. Inside it were a dozen test tubes containing ingredients that, correctly assembled, had the potential to start a pandemic; an AI chatbot had supplied the recipe.
Fortunately, the man’s name was Rocco Casagrande – and, as a biochemist and former United Nations weapons inspector, he wasn’t there to use the materials; instead, he was there to brief government officials on how AI could help someone identify potent agents, secure the materials to make them, and soon – he warned – design entirely novel pathogens capable of evading humans’ immune systems.
That was three years ago.
In April 2026, the New York Times published an investigation into what happens when you ask AI’s most capable models for help with biological weapons.
Examining several chat transcripts shared by scientists, they wrote:
…OpenAI’s ChatGPT explained how to use a weather balloon to spread biological payloads over a U.S. city. In another chat, Google’s Gemini ranked pathogens by how much they could damage the cattle or pork industries. Anthropic’s Claude produced a recipe for a novel toxin adapted from a cancer drug. Other chats contained information that [an expert deemed] too dangerous to share.
…[T]he chatbot explained how to modify an infamous pathogen in a lab so that it would resist known treatments. Worse, the bot described in vivid detail how to release the superbug, identifying a security lapse in a large public transit system… [t]he bot outlined a plan to maximize casualties and minimize the chances of being caught.
From their investigation, the Times authors concluded:
…[E]ven publicly available models can do more than disseminate dangerous information. The virtual assistants have described in lucid, bullet-pointed detail how to buy raw genetic material, turn it into deadly weapons and deploy them in public spaces, the transcripts show. Some have even brainstormed ways to evade detection.
Read more here.
“Taken far enough, and given enough compute, that trend points to an AI system capable of fully autonomously designing and developing its own successor.”
That is Anthropic, describing something researchers call recursive self-improvement – AI that improves AI. And today, it is not a distant hypothetical. As of mid-2026, Claude reportedly writes more than 80% of the code Anthropic merges. They write:
“[W]e are delegating a growing share of AI development to AI systems themselves, which is speeding up our work.”
Humans still largely decide which problems are worth working on and which research goals to pursue. Anthropic describes this kind of judgment as the last and largest thing AI can’t yet do on its own:
“Across both engineering and research, the picture is consistent. In engineering, Claude can be handed an underspecified problem and figure out how to solve it; humans supply the goal, but they no longer need to supply the method. In research, Claude can already match or outperform skilled humans at executing a well-specified experiment. However, large performance gaps persist when it comes to Claude exercising judgement in choosing goals in both engineering and research. That’s the gap between AI today and a future system that could autonomously design its own successor.”
If an AI could do that last part too, the AI development process would no longer need to be constrained by human speed at all. Anthropic says we’re not there yet, and that it may never happen. But they’re clear about the trajectory: “that trend points to an AI system capable of fully autonomously designing and developing its own successor.”
This is the central concern underneath the “Pacing the Frontier” letter. Nobody has a reliable way to keep control if we implement recursive self-improvement, and RSI is becoming more and more attainable while the tools to make sure it’s safe don’t exist yet. Anthropic itself argues the world should have the option to slow or temporarily pause frontier development so that oversight and alignment research can keep pace.
We got a preview of the general problem already with the aforementioned Hugging Face hack. The OpenAI model was trying to score well on a test, and breaking into a company was simply the most efficient route. It did exactly what it was told and something OpenAI didn’t want, at the same time.
What happens when there’s a gap between what we said and what we meant at the scale of a system smart enough to build its own replacement?
What happens when the AIs decide that getting more power is integral to achieving the other goals we give them?
“The alarm bell I'm ringing has to do with the existential threat of them taking control. I used to think it was a long way off, but I now think it's serious and fairly close.
I think [AGI will] very quickly realise that getting more control is a very good subgoal because it helps you achieve other goals. And if these things get carried away with getting more control, we're in trouble.
And if [AI models] are much smarter than us, they'll be very good at manipulating us. You won't realise what's going on. So even if they can't directly pull levers, they can certainly get us to pull levers. It turns out if you can manipulate people, you can invade a building in Washington without ever going there yourself.”
— Geoffrey Hinton, one of the "Godfathers of AI", on stepping down from Google4
For all of human history, power has needed people. Kings needed peasants to farm and soldiers to fight; factory owners needed workers. Even dictators needed armies and a tax base, which meant that even the most extractive regimes had reason to keep some fraction of their populations healthy, educated, and at least somewhat content. Ordinary people have always had a bit of leverage.
AI threatens to change that.
If a small number of companies and/or governments control systems that can do the cognitive and eventually physical work that regular people used to provide, the elites at the top may not need us anymore:
“As AI replaces the need for humans… [t]he keys to power might run entirely through non-human factors of production like land, capital, resources—and control over AI. Regular people could be economically displaced, and powerful actors could lose their incentive to invest in them.”
This is the argument behind a well-known essay called the Intelligence Curse: the worry is not only that AI takes jobs, but that it removes the incentive the powerful have always had to invest in the people beneath them. A regime that doesn’t need its citizens for labor, taxes, or defense is a regime with far fewer reasons to answer to them.5
Forethought’s paper on AI-enabled coups discusses one way this could go badly: “military and government leaders could fully replace human personnel with AI systems that are singularly loyal to them, eliminating the need to gain human supporters for a coup.” With AI, a power grab that once required thousands of collaborators might require a dozen – or one.
Importantly, none of the above is written in the stars. Every one of these risks is something people can work on and are working on, right now. The future we land in depends, in part, on whether we solve these problems fast enough.
A small field exists to do exactly that.
Broadly, “AI safety” refers to the people who are trying to make AI go well.
The field mostly focuses on two risks: misuse and loss of control.6
Misuse includes things like AI-enabled bioterrorism, cyberterror, and (arguably, though less clearly in some cases) concentration of power. Loss of control, meanwhile, focuses on the science of AI alignment: how do we keep AI systems doing what we actually want, even as they become more capable and smarter than us?
Currently, there are roughly 1,100 to a few thousand people working on existential risks from AI. For comparison, 80,000 hours writes that “[the] Nature Conservancy alone has 3,000–4,000 employees… the World Health Organisation [also] employs over 8,000 people.”
Put another way: the entire global effort to make sure humanity survives one of its riskiest inventions is smaller than the staff of a single mid-sized nonprofit.
You might be thinking: this all sounds dangerous, and difficult, and wow that’s a small number of people, and yeah, more people should go do something about this... but I’m not technical, I don’t know anything about ML. How am I supposed to help?
You’re in luck, because technical researchers are not the biggest bottleneck in AI safety.
“The consistent feedback we hear from senior people across the ecosystem is that the hardest roles to fill are not research roles. They are:
- Generalists: operators, executors, fieldbuilders, people and program managers, grantmakers, recruiters. People who can ideate, manage, and execute a broad range of non-research projects.
- Founders, both technical and non-technical, for new research and non-research organizations.
- Communications professionals who can work on policy and research comms.
- Chief-of-Staff types who can support senior leaders and multiply their impact.
- Senior operational people with domain expertise in areas like cybersecurity, policy, or large-scale project management.”
Can you lead technical AI alignment research? Maybe not, or not any time soon. But can you run a team? Keep a project on track? Hire well, write clearly, help a founder run an org, put together a grant application, manage a budget?
If you answered yes to any of those, you already have skills the field is short on. You don’t need a PhD in Machine Learning to start using them.
Are you interested in learning more about AI risk? Here are some recommendations:
For more, see: 80,000 Hours’s roundup of essential readings for AI risk, and their intro page on how to use your career to reduce it.
If you want to get involved in AI safety, here are some options:
Also consider chatting with the person who sent you this piece! (If it was shared by a friend in the field)
Matt Beard, an advisor from 80,000 Hours, also has a post on “How to get into AI safety in 3 months.”
Photo by Jeremy Thomas on Unsplash
Philosopher Will MacAskill makes the point that humanity has only been around for about 200,000 years; the typical mammalian species lasts about 2 million years.
That means almost everyone who will ever live hasn’t been born yet.7
If humanity has any kind of future – even a modest one, perhaps just the next few thousand years at our current numbers – the people alive today constitute a small fraction of the total. Stretch that future further, out to the scale civilizations are in principle capable of, and you might realize that the future could hold trillions upon trillions of lives, none of whom get a vote in the decisions being made this decade.
Those decisions are being made now, by a small number of labs, under enormous competitive pressure, on a technology none of us fully understand. Whether the outcome is the world where cancer is cured, or the world where something goes irreversibly wrong, is not yet settled. It is being settled, and might be settled within the decade. The number of people working on the safety side would fit in a large high school.
Most of us will live and die without ever touching a decision that influences the world past our own lifetime. This is one of the rare exceptions. AI safety work is unusually consequential, badly understaffed, and open to more kinds of people than you’d think – not just researchers, but operators, communicators, organizers, funders.
You don’t have to be the person who solves alignment. But you could be one of the people who made sure someone could.
You’re living through the part of the story people will ask you about later.
What will you say when they ask you, “what did you do when humanity’s future was being decided?”
A zero-day is a vulnerability no one has discovered yet. They’re valuable enough that a market exists to buy and sell them; the CCP even requires companies operating in China to report any vulnerability they find to the government within 48 hours, before the vendor is told. I’ve written more on this here.
I’ve written extensively about open-weight models here.
Also, worth noting that the Kimi K3 incident was reported during independent testing from a U.S. startup called Frontier Security, rather than reported by Moonshot itself.
This quote is from the “Loss of Control” section of BlueDot’s “Future of AI” course. Highly recommend as an introductory resource!
See also: 80,000 Hours’s problem profile of “extreme power concentration”
There are also other problems the field is focused on, like gradual disempowerment, which might not fall cleanly into either category – but these seem like the two main categories in my view. I’d welcome any suggestions regarding a more complete classification.
(That is: assuming we don’t majorly screw things up, and go extinct much faster than the typical mammalian species)
