Comments
This post is co-authored with Ben Garfinkel. It is cross-posted from the CEA blog. A PDF version can be found here.
Summary: Some strategic decisions available to the effective altruism m...
TLDR: So there has been recent discourse on 𝕏, and recent news of major cyber attacks that were done with the help of AI. The missing frame here is the dual-use gap: as AI models become more capable, they create more upside for defenders and more downside for attackers. The gap between the benefits and the harmful effects is getting wider. I know that sounds obvious, but I think people are underestimating the second-order effects of this. The dual-use gap makes small failures feel less small. A compromised account, a bad package, or one missed vulnerability can suddenly have a much larger blast radius, because attackers can use AI to move faster, automate more, and chain together mistakes that would have been harder to exploit before. And the usual response is "good AI will defend against bad AI," but who is actually guaranteeing that the defensive AI finds every path before the attacking AI does? And also who is guaranteeing that it defends everyone?
This is cross-posted on LessWrong; I’ll try to keep an eye on discussion in both places.
So there has been recent discourse on 𝕏, and recent news of major cyber attacks that were done with the help of AI.
I think a lot of the discourse is getting stuck on the wrong question. The debate is usually something like:
- One side says we need more safety precautions around open source and closed source model releases, because cyber capable models make vulnerability discovery and exploit development cheaper.
- The other side says this is fine because cyber is dual-use. As long as the "good" AI is better than the "bad" AI, defenders will win, like how Jensen Huang mentions in this podcast clip.
I get why the second view is attractive. It's optimistic. The defender does have advantages sometimes. Mozilla's blog post on Mythos finding a ton of bugs is probably the most supporting of this view, and the conclusion was basically that maybe the defects are finite and we can finally find them all with AI. But my worry is that this view subtly turns cybersecurity into "does the good model beat the bad model?". And I don't think that is right. It's missing the center of the problem. The dual-use gap.
The Dual-Use Gap
The way I think about it is like this. More capable models have beneficial effects. They can help defenders review code, find vulnerabilities, write patches, understand codebases that are way too large for one person to keep in their head etc. This is the green line.
But the same capabilities create harmful effects. They can help attackers understand unfamiliar systems, scale phishing, exploit fresh vulnerabilities faster, write malware variants etc. This is the red line.
As models get better, both lines move. The gap gets wider. The green line goes up. The red line goes down.
The mistake is having the perception that these effects cancel out. In reality they actually do not.
If AI makes defenders n-times better and attackers n-times better, the world is not the same. The whole game is just faster now by the scale of n-times. Mistakes matter more. Weak systems get punished harder. And anyone who does not get the defensive upside is now living in a more dangerous environment. The defense of having "defending AI" only helps if the defender actually has access to them, the right people, and enough time to respond.
That is a lot of assumptions.
"Good AI beats Bad AI" is not enough
I keep seeing a version of the argument that is basically:
Cyber is dual-use, but defense can also use AI, so we should accelerate the good AI.
The first half, I do agree with. Cyber is dual-use. I also agree that defensive AI is necessary. But "Good AI will defend us" only works if you are able to answer a bunch of annoying questions: Who is "us"? Do regular people get defended? Do small startups get defended? Do open source maintainers get defended? Do local schools, hospitals, and random mid-sized companies get defended? Or only frontier labs, and large enterprises get the best defensive AI?
Because if the answer is mostly the last group, then we are not closing the dual-use gap. We are just giving the best defended institutions better defenses while everyone else in the world gets dropped into that faster threat environment. That is the part that doesn't feel discussed to me in my opinion.
The recent incidents feel like dual-use-gap events
The recent Google blog post is an example of a recent red line signal. The interesting part is not that someone asked a agent to write a script. It's that attackers are building workflows around agents: vulnerability exploitation, augmented operations, malware work, and more scalable use of these models.
Then OpenAI’s TanStack/npm supply-chain incident is another version of the same thing. OpenAI said a widely used npm package was compromised as part of a broader supply-chain attack. The broader lesson is that even frontier labs are still inside the normal software supply chain. They depend on packages, developer devices, repos, etc.
The green line is real too
To be clear, I am not saying AI in cyber is all bad. That would be dumb.
The defensive upside is real. OpenAI’s Daybreak proposal makes sense to me. If attackers are moving faster with AI, defenders probably need AI inside code review, threat modeling, detection engineering etc. You cannot fight against AI-enabled offense with quarterly security reviews. This is a step in the right direction, but again still not enough, I bring my point back up again: Who is "us" that is being defended, this proposal still doesn't cover everyone to be defended.
My main actual worry
The worry is not only that bad AI might beat good AI. The worry is that the dual-use gap widens faster than our ability to distribute defense to everyone.
A frontier lab can hire incident responders, build internal monitoring, write evals, and even have access to their most frontier internal model with full infinite access. A three person startup cannot do or have all of that. Neither can a solo maintainer or a regular person.
So when people say "AI will defend us," I want to know who "us" is.
If "us" means the richest and most competent institutions, then sure, maybe. But the world is not only those institutions/companies. It is also messy SaaS integrations, abandoned packages, random extensions, small teams, and regular people with phones and laptops.
Offense scales across weak targets. Defense has to work in each specific messy environment. That asymmetry matters more as this dual-use gap expands. It's not "AI makes cyber better." Not "AI makes cyber worse." It's both. And that is exactly the problem.
I think this is a useful frame. When discussing AI and cybersecurity, we're often faced with the question of whether AI benefits attackers more than defenders (or vice versa), but perhaps the more interesting question is who actually receives the defensive benefits.
Even if defensive AI advances more quickly than offensive AI, many vulnerable targets, small organizations, open-source maintainers, local governments, and individuals, may not have access to the same tools, expertise, or response capacity. In that world, the overall security may improve while the risk is more concentrated with weaker actors.
This suggests from an impact standpoint that deploying defensive capabilities may be as important as upgrading them. But a world where only a few institutions can effectively defend themselves may still be a much riskier world overall.
The “dual-use gap” is an appealing notion because it shifts the attention from average outcomes to the differential distribution of protection and risk.