Below is the executive summary of our new paper, The Manhattan Trap. Please visit the link above to see the full paper. We also encourage discussion and feedback in the comments here.

This paper examines the strategic dynamics of international competition to develop Artificial Superintelligence (ASI). We argue that the same assumptions that might motivate the US to race to develop ASI also imply that such a race is extremely dangerous.

A race to develop ASI is motivated by two assumptions: that ASI provides a decisive military advantage (DMA) to the first state that develops it, and that states are rational actors aware of ASI's strategic implications.

However, these same assumptions make racing catastrophically dangerous for three reasons.

First, an ASI race creates a threat to strategic stability that could trigger a war between the US and its adversaries, particularly China. If ASI could provide a decisive military advantage, states would rationally view their adversaries' ASI development as an existential threat justifying military intervention and espionage. A state cannot race to ASI without incurring significant risk of great power conflict unless either (a) its adversaries are unaware of ASI's importance (contradicting the awareness assumption), (b) development can be kept secret (likely impossible given the scale required), or (c) adversaries believe they can win the race (requiring a close competition in which victory is not assured).

Second, racing heightens the risk of losing control of an ASI system once developed. We do not take a position on the likelihood of loss of control. Instead, we observe that the argument for racing assumes that ASI would wield a decisive advantage over the militaries of global superpowers; accordingly, losing control of such a technology would also present an existential threat to the state that developed it. We also argue that an ASI would only provide a DMA if its capabilities scaled extremely rapidly—precisely the scenario in which loss of control risk is theoretically highest. Finally, a race compounds this risk by creating competitive pressure to develop ASI before adequate control measures are in place.

Third, even a controlled ASI threatens to disrupt the internal power structures of the state that develops it. In the case of a US, a successful ASI project would likely undermine the liberal democracy it purports to defend. An ASI system that would grant a state international advantage would also grant its controllers unprecedented domestic power, creating an extreme concentration of power likely incompatible with democratic checks and balances. Racing exacerbates this risk by requiring development to occur quickly and in secret—and therefore without public input.

These three dangers—great power conflict, loss of control, and power concentration—represent three successive barriers a state would need to overcome to 'win' a race to ASI. Together, they imply that an ASI race poses an existential threat to the national security of the states involved. Assuming states are informed and rational, the strategic situation can be modeled as a trust dilemma: states would prefer mutual restraint to racing and will only race if they believe others will. States can therefore avoid an ASI race by establishing a verification regime—a set of mechanisms that can verify a state's compliance with an international agreement not to pursue an ASI project. An ASI project would be highly distinguishable from civilian AI applications and not integrated with a state's economy—precisely the conditions under which verification regimes have historically succeeded.

The paper concludes that, if an ASI race is motivated, then cooperation to avoid an ASI race is both preferable and strategically sound. The assumptions that make racing seem necessary are the very reasons it is unwise.

Comments1


Sorted by Click to highlight new comments since:

Executive summary: The paper argues that the strategic dynamics and assumptions driving a race to develop Artificial Superintelligence (ASI) ultimately render such efforts catastrophically dangerous and self-defeating, advocating for international cooperation and restraint instead.

Key points:

  1. A race to develop ASI is driven by assumptions that ASI provides a decisive military advantage and that states are aware of its strategic importance, yet these assumptions also highlight the race's inherent dangers.
  2. The pursuit of ASI risks triggering great power conflicts, particularly between the US and China, as states may perceive adversaries' advancements as existential threats, prompting military interventions.
  3. Racing to develop ASI increases the risk of losing control over the technology, especially given the competitive pressures to prioritize speed over safety and the theoretical high risk of rapid capability escalation.
  4. A successful ASI could disrupt internal power structures within the state that develops it, potentially undermining democratic institutions through an extreme concentration of power.
  5. The existential threats posed by an ASI race include great power conflict, loss of control of ASI, and the internal concentration of power, which collectively form successive barriers that a state must overcome to 'win' the race.
  6. The paper recommends establishing an international verification regime to ensure compliance with agreements to refrain from pursuing ASI projects, as a more strategic and safer alternative to racing.

 

 

This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.

Curated and popular this week
 ·  · 1m read
 · 
 ·  · 5m read
 · 
When we built a calculator to help meat-eaters offset the animal welfare impact of their diet through donations (like carbon offsets), we didn't expect it to become one of our most effective tools for engaging new donors. In this post we explain how it works, why it seems particularly promising for increasing support for farmed animal charities, and what you can do to support this work if you think it’s worthwhile. In the comments I’ll also share our answers to some frequently asked questions and concerns some people have when thinking about the idea of an ‘animal welfare offset’. Background FarmKind is a donation platform whose mission is to support the animal movement by raising funds from the general public for some of the most effective charities working to fix factory farming. When we built our platform, we directionally estimated how much a donation to each of our recommended charities helps animals, to show users.  This also made it possible for us to calculate how much someone would need to donate to do as much good for farmed animals as their diet harms them – like carbon offsetting, but for animal welfare. So we built it. What we didn’t expect was how much something we built as a side project would capture peoples’ imaginations!  What it is and what it isn’t What it is:  * An engaging tool for bringing to life the idea that there are still ways to help farmed animals even if you’re unable/unwilling to go vegetarian/vegan. * A way to help people get a rough sense of how much they might want to give to do an amount of good that’s commensurate with the harm to farmed animals caused by their diet What it isn’t:  * A perfectly accurate crystal ball to determine how much a given individual would need to donate to exactly offset their diet. See the caveats here to understand why you shouldn’t take this (or any other charity impact estimate) literally. All models are wrong but some are useful. * A flashy piece of software (yet!). It was built as
Garrison
 ·  · 7m read
 · 
This is the full text of a post from "The Obsolete Newsletter," a Substack that I write about the intersection of capitalism, geopolitics, and artificial intelligence. I’m a freelance journalist and the author of a forthcoming book called Obsolete: Power, Profit, and the Race to build Machine Superintelligence. Consider subscribing to stay up to date with my work. Wow. The Wall Street Journal just reported that, "a consortium of investors led by Elon Musk is offering $97.4 billion to buy the nonprofit that controls OpenAI." Technically, they can't actually do that, so I'm going to assume that Musk is trying to buy all of the nonprofit's assets, which include governing control over OpenAI's for-profit, as well as all the profits above the company's profit caps. OpenAI CEO Sam Altman already tweeted, "no thank you but we will buy twitter for $9.74 billion if you want." (Musk, for his part, replied with just the word: "Swindler.") Even if Altman were willing, it's not clear if this bid could even go through. It can probably best be understood as an attempt to throw a wrench in OpenAI's ongoing plan to restructure fully into a for-profit company. To complete the transition, OpenAI needs to compensate its nonprofit for the fair market value of what it is giving up. In October, The Information reported that OpenAI was planning to give the nonprofit at least 25 percent of the new company, at the time, worth $37.5 billion. But in late January, the Financial Times reported that the nonprofit might only receive around $30 billion, "but a final price is yet to be determined." That's still a lot of money, but many experts I've spoken with think it drastically undervalues what the nonprofit is giving up. Musk has sued to block OpenAI's conversion, arguing that he would be irreparably harmed if it went through. But while Musk's suit seems unlikely to succeed, his latest gambit might significantly drive up the price OpenAI has to pay. (My guess is that Altman will still ma