Early last month, Leopold Aschenbrenner released a long essay and podcast outlining his projections for the future of AI. Both of these sources are full of interesting arguments and evidence, for a comprehensive summary see Zvi’s post here. Rather than going point by point I will instead accept the major premises of Leopold’s essay but contest some of his conclusions.

So what are the major premises of his piece?

1. There will be several orders of magnitude increase in investment into AI. 100x more spending, 100x more compute, 100x more efficient algorithms, and an order of magnitude or two gains from some form of “learning by doing” or “unhobbling” on top.
2. This investment scale up will be sufficient to achieve AGI. This means the models on the other side of the predicted compute scale up will be able to automate all cognitive jobs with vast scale and speed.
3. These capabilities will be essential to international military competition.

All of these premises are believable to me and well-argued for in Leopold’s piece.

Leopold contends that these premises imply that the national security state will take over AI research and the major data centers, locking down national secrets in a race against China, akin to the Manhattan project.

Ultimately, my main claim here is descriptive: whether we like it or not, superintelligence won’t look like an SF startup, and in some way will be primarily in the domain of national security.

By late 26/27/28 … the core AGI research team (a few hundred researchers) will move to a secure location; the trillion-dollar cluster will be built in record-speed; The Project will be on.  

The main problem is that Leopold’s premises can be applied to conclude that other technologies will also inevitably lead to a Manhattan project, but these projects never arrived. Consider electricity. It's an incredibly powerful technology with rapid scale up, sufficient to empower those who have it far beyond those who don’t and it is essential to military competition. Every tank and missile and all the tech to manufacture them relies on electricity. But there was never a Manhattan project for this technology. It’s initial invention and spread was private and decentralized. The current sources of production and use are mostly private.

This is true of most other technologies with military uses: explosives, steel, computing, the internet, etc. All of these technologies are essential in the government’s monopoly on violence and it’s ability to exert power over other nations and prevent coups from internal actors. But the government remains a mere customer of these technologies and often not even the largest one.

Why is this? Large scale nationalization is costly and unnecessary for maintaining national secrets and technological superiority. Electricity and jet engines are essential for B-2 bombers, but if you don't have the particular engineers and blueprints, you can't build it. So, the government doesn’t need to worry about locking down the secrets of electricity production and sending all of the engineers to Los Alamos. They can keep the first several steps of the production process completely open and mix the outputs with a final few steps that are easier to keep secret.

To be clear, I am confident that governments and militaries will be extremely interested in AI. They will be important customers for many AI firms, they will create internal AI tools, and AI will become an important input into every major military. But this does not mean that most or all of the AI supply chain, from semi-conductors to data-centers to AI research, must be controlled by governments.

Nuclear weapons are outliers among weapons technology in terms of the proportion of the supply chain and final demand directly overseen by governments. Most military technologies rely on an open industrial base mixed with some secret knowledge in the final few production steps.

So should we expect AGI to be more like nuclear weapons or like a new form of industrial capacity? This depends on how much extra scaffolding you need on top of the base model computation that’s piped out of data centers to achieve militarily relevant goals.

Leopold’s unhobbling story supports a view where the intelligence produced by massive datacenters is more like the raw input of electricity, which needs to be combined with other materials and processes to make a weapon, than a nuclear bomb which is a weapon and only a weapon right out the box.

Leopold on base models says:

“out of the box, they’re hobbled: they’re using their incredible internal representations merely to predict the next token in random internet text, and rather than applying them in the best way to actually try to solve your problem.”

“On SWE-Bench (a benchmark of solving real-world software engineering tasks), GPT4 can only solve ~2% correctly, while with Devin’s agent scaffolding it jumps to 14-23%. (Unlocking agency is only in its infancy though)”

Devin can have a product without a proprietary model because they have a scaffold. They can safely contract for and pipe in the raw resource latent model and put it through a production process to get something uniquely tooled out the other end, without needing to in-house and lock-down the base model to maintain a unique product.

Of current chatbots Leopold says:

“They’re mostly not personalized to you or your application (just a generic chatbot with a short prompt, rather than having all the relevant background on your company and your work)”

Context is that which is scarce! The national security state needn’t lock down the base models if the models are hobbled without the context of their secret applications. That context is already something they’re already extremely skilled at locking down, and it doesn’t require enlisting an entire industry.

“In a few years, it will be clear that the AGI secrets are the United States’ most important national defense secrets—deserving treatment on par with B-21 bomber or Columbia-class submarine blueprints”

Leopold is predicting the nationalization of an entire industrial base based on an analogy to submarines and bombers, but a large fraction of the supply chain for these vehicles are private and open. It’s not clear why he thinks military applications of AGI can’t be similarly protected without control over the majority of the supply chain and final demand.

If you imagine AGI as this single, powerful oracle and army that can complete any and all tasks on command, then Leopold is right: governments will fight hard to lock everyone else out. If instead, AGI is a sort of “intelligence on tap” which is an input to thousands of different production processes where it’s mixed with different infrastructure, context, and tools to create lots of different products, then governments don’t need to control the entire industrial base producing this intelligence to keep their secrets. Leopold leans hard on the Manhattan project as a close analogy to the first situation, but most military technologies are in the second camp.