Comments
We've published an essay series on what we call the intelligence curse. Most content is brand new, and all previous writing has been heavily reworked.
Visit intelligence-curse.ai for the full series.
Below is the introduction and table of contents.
We will soon live in the intelligence age. What you do with that information will determine your place in history.
The imminent arrival of AGI has pushed many to try to seize the levers of power as quickly as possible, leaping towards projects that, if successful, would comprehensively automate all work. There is a trillion-dollar arms race to see who can achieve such a capability first, with trillions more in gains to be won.
Yes, that means you’ll lose your job. But it goes beyond that: this will remove the need for regular people in our economy. Powerful actors—like states and companies—will no longer have an incentive to care about regular people. We call this the intelligence curse.
If we do nothing, the intelligence curse will work like this:
- Powerful AI will push automation through existing organizations, starting from the bottom and moving to the top.
- AI will obsolete even outlier human talent. Social mobility will stop, ending the social dynamism and progress that it drives.
- Non-human factors of production, like capital, resources, and control over AI, will become overwhelmingly more important than humans.
- This will usher in incentives for powerful actors around the world that break the modern social contract.
- This could result in the gradual—or sudden—disempowerment of the vast majority of humanity.
But this prophecy is not yet fulfilled; we reject the view that this path is inevitable. We see a different future on the horizon, but it will require a deliberate and concerted effort to achieve it.
We aim to change the incentives driving the intelligence curse, maintaining human economic relevance and strengthening our democratic institutions to withstand what will likely be the greatest societal disruption in history.
To break the intelligence curse, we should chart a different path on the tech tree, building technology that lets us:
- Avert AI catastrophes by hardening the world against them, both because it is good in itself and because it removes the security threats that drive calls for centralization.
- Diffuse AI, to get it in the hands of regular people. In the short-term, build AI that augments human capabilities. In the long-term, align AI directly to individual users and give everyone control in the AI economy.
- Democratize institutions, making them more anchored to the needs of humans even as they are buffeted by the changing incentive landscape and fast-moving events of the AGI transition.
In this series of essays, we examine the incoming crisis of human irrelevance and provide a map towards a future where people remain the masters of their destiny.
Chapters
1. Introduction
We will soon live in the intelligence age. What you do with that information will determine your place in history.
2. Pyramid Replacement
Increasingly powerful AI will trigger pyramid replacement: a systematic hollowing out of corporate structures that starts with entry-level hiring freezes and moves upward through waves of layoffs.
3. Capital, AGI, and Human Ambition
AI will make non-human factors of production more important than human ones. The result may be a future where today's power structures become permanent and frozen, with no remaining pathways for social mobility or progress.
4. Defining the Intelligence Curse
With AGI, powerful actors will lose their incentive to invest in regular people–just as resource-rich states today neglect their citizens because their wealth comes from natural resources rather than taxing human labor. This is the intelligence curse.
5. Shaping the Social Contract
The intelligence curse will break the core social contract. While this suggests a grim future, understanding how economic incentives reshape societies points to a solution: we can deliberately develop technologies that keep humans relevant.
6. Breaking the Intelligence Curse
Avert AI catastrophes with technology for safety and hardening without requiring centralizing control. Diffuse AI that differentially augments rather than automates humans and decentralizes power. Democratize institutions, bringing them closer to regular people as AI grows more powerful.
7. History is Yours to Write
You have a roadmap to break the intelligence curse. What will you do with it?
I think that there is a radical solution: have the AGI aligned to a certain treaty that requires the AGI, instead of obeying all orders except for the ones determined by the Spec, to harvest at most a certain share of resources and to help humans only in certain ways that amplify humanity and don't cause it to degrade, like teaching humans about the facts that mankind has already discovered or pointing out mistakes in humans' works. Or protecting mankind from some other existential risks that are hard to deal with, like a nuclear war that might be caused by an accident. An even more radical idea close to hopium is that this type of alignment is even easier than the ones that induce the Curse or force people "to work makeshift government jobs" or to collect a generous basic income.