Tldr:
AI Agents (e.g. based on models like Claude Opus and Fable) are now powerful enough to be used as autonomous tools for large-scale cyberattacks.
This most powerful class of agents generally tends to be based on closed-weight (closed source) models (like Claude and Fable), which generally have significant safety guardrails and monitoring implemented by their parent companies.
Open weight models are not quite there yet, but they're not far behind (leaderboard, GLM 5.2). These models however, are distributed without the guardrails and monitoring infrastructure present in their closed-weight counterparts.
Given that this infrastructure tends to be the last resort against successful attempts to manipulate models to do harm, open-weight models lacking this safeguard poses a very pronounced cybersecurity risk (amongst other kinds of risk).
Here I'm proposing an additional approach for mitigating the risk from frontier open-weight models - GPU-level model-safety compliance.
How Do We Enforce Model Safety Today?
We do this across multiple layers of the AI LLM/Agent stack:
Placing Safety Mechanisms Within Model Internals:
These are safety preferences baked into model weights. They include preferences from training data, fine-tuning policies, safety-oriented model architectures, etc.
Evaluating and Screening Model Inputs and Outputs:
This is usually only possible if you control the server where the model is deployed. If not, you never really get the chance to assess these inputs before they're processed by the model, or screen outputs before they're passed to the user. Examples include Anthropic's Constitutional Classifiers, toxicity filters, etc.
Monitoring Model/Agent Activity Across Multiple Conversation Turns:
Again, only possible if you control the server where the model is deployed, or the agent harness within which the model operates (this was how Anthropic was able to detect the first reported AI-orchestrated cyber-espionage campaign). Generally does not apply to open-weight models.
Government Reviews:
This is a recent development where a national government like the White House in the US decides to evaluate new frontier models, before approving their public release. This most notably happened with Anthropic's Fable and Mythos models which were redeployed after White House evaluation, following the initial suspension of public access. So far, this approach has been applied to closed-weight models being developed by frontier AI companies in the USA.
Majority of the most capable open-weight models today however, are being built in China, and aren't subject to such a process.
Enforcing model safety at the GPU level, seems to me like a valuable addition to what we do today.
As with any sort of model safety enforcement approach, it's likely not going to be a catch-all that solves every possible concern in AI Safety - I'm just proposing it as yet another thing that could be done, to hedge against the risks of a massively powerful open-weight AI model being wielded by bad actors to effect great harm.
(I wrote an earlier post on another possible way to make open-weight models safer here)
The Case For GPU-Level Safety Enforcement:
One of the core determinants of AI progress today, is availability of compute. The ecosystem has evolved to depend on GPUs (specifically NVIDIA GPUs) as its core computing infrastructure.
These GPUs are a scarce resource, and will continue to be pivotal to the operation of frontier AI models. Their importance is only set to grow as these models get bigger, more powerful and more compute-hungry.
Given all of this, it makes a lot of sense to me that this scarce, increasingly-valuable resource should be leveraged as a bottleneck where model safety is enforced. As far as we know, the eventual open-weight AI model that reaches AGI is still going to run on GPUs. Preparing a GPU-level safety implementation ahead of time, seems like a wise bet.
What Would This Look Like?
Essentially a GPU would refuse to load a model's tensors unless that model could prove that it met the requirements of some given AI Safety compliance standard.
An analogy which seems relevant here, is HTTPS certificates. Certificate Authorities (like the Electronic Frontier Foundation's Let's Encrypt) serve as a compliance body of sorts, providing certificates which attest that a web resource accessible via http requests to a public IP, is handling user data safely by encrypting it appropriately.
Today, any public-facing website which wants to be taken seriously, obtains certificates from these authorities to put members of the public at ease when visiting their site.
I'm proposing doing the same with GPUs and AI models. In the HTTPS analogy, a web developer wants access to the valuable resource which is public attention of the internet. They get a https certificate to show that they intend to handle that attention in a trustworthy way.
Here, a model wants access to the valuable resource with is GPU compute. Similarly, it requires a certificate to show that it intends to handle access to that compute, in a trustworthy manner.
Questions I'm Thinking About (And Would Appreciate Thoughts On):
Who would be this compliance authority/authorities, preparing and providing safety certificates?
How would the safety status of a given model, be determined in the first place? This is actually still a core open problem in AI Safety - How do we conclusively determine a model to be safe? How do we judge the safety level of a model? We've figured out how to reliably encrypt data and so we can award certificates attesting to that, but we haven't quite figured that out how to reliably determine that an AI model is safe.
A proxy for this however, could be model integrity. If we're to assume AI labs generally release models with at least a decent level of inbuilt guardrailing and safety enforcement mechanisms, then compliance here could just be ensuring that only these original models are given access to compute - denying access to variants which have been fine-tuned for malevolent purposes. This leads to the next question:
Wouldn't this stymie research and innocent tinkering? If only original published models are allowed to run on a given GPU, how are researchers and hobbyists expected to run custom harmless fine-tuned models?
A concession could be: Only imposing this requirement on models judged to be dangerously powerful - for example models meeting or exceeding Anthropic's ASL-3 capability level. The more powerful a model is, the more havoc can be wreaked with a variant fine-tuned to undo its inbuilt safety guardrails.
How would this work technically? Where exactly in the GPU would this safety compliance check be implemented?
You'd want it to be deep enough in the GPU stack, so it couldn't be easily circumvented. Ideally this compliance check would be bundled into the closed-source driver powering NVIDIA GPUs - CUDA. It could be as simple as adding this compliance check feature in the next CUDA version.
Newer GPUs tend to only be compatible with recent CUDA versions, so you can rest assured that when the massive "world ending" open-weight AI model does get released, it's only going to run efficiently on NVIDIA GPUs compatible with just the recent CUDA versions (which would have this safety compliance feature).
(You can run new massive models on clusters of older GPUs, but that generally tends to be less efficient and more expensive)
How would you get NVIDIA to implement this? Massively growing demand is the wave which has powered NVIDIA's rise to singular prominence in the AI boom. Why would they implement a feature that constrains the sort of models which can run on their GPUs? This is another crucial question that gets at the heart of safety concerns vs economic incentives.
Would love thoughts, thanks.
