For about a decade I was convinced that AI x-risk was humanity’s greatest threat. I read Nick Bostrom’s Superintelligence when it first came out and adopted the rationalist and EA picture almost by default. For years I treated “unaligned superintelligence” as the obvious number one existential risk.
Over time that changed. David Deutsch’s work on universality, creativity, and epistemology persuaded me that the standard AI safety narrative is mistaken at a deep level. The issue is not with GPUs or scaling or “loss of control” in the usual sense. The problem is with the underlying philosophy of knowledge, prediction, and personhood.
Much later, I encountered Brett Hall’s long critique of the AI 2027 paper and related arguments. By that point I was already persuaded by Deutsch’s take, but Brett’s series explains the issues with clarity and accessibility. He lays out the logical structure of the doomer argument and shows where the assumptions fail. I borrow many of his examples because they present the case well.
Brett’s full critique is available here:
YouTube playlist: “AI 2027 Analysis”
What follows is my attempt to present, in EA Forum format, why I believe the usual rationalist AI safety picture is both philosophically confused and empirically unsupported.
1. The central confusion: what a person is
The AGI debate is muddled by a failure to define personhood. Many people treat LLMs as if a mind lurks inside pattern matching machinery. They elevate token prediction into proto consciousness. They assume scaling will cause a “someone” to appear inside the weights.
This resembles the old idea of spontaneous generation. Life from dust. Maggots from meat. Today the story is minds from matrices.
Deutsch’s idea of explanatory universality helps clarify the mistake. Persons are universal explainers. They create new explanations that were not contained in past data. This creativity is not extrapolation from a dataset. It is invention.
LLMs do not do this. They remix what exists in their training corpus. They do not originate explanatory theories.
Until we understand how humans create explanatory knowledge, we cannot program that capacity. AGI in the strong sense is a software design problem. It will not emerge from scaling. It will not sneak up on us. It requires a breakthrough in philosophy.
2. Creativity is not derivation from data
Rationalist AI safety arguments often assume scientific creativity is a form of pattern recognition over large datasets. This is not how scientists describe their own work.
Darwin did not derive evolution from beak measurements. Einstein did not calculate relativity from examples. The key ideas did not exist in any dataset. They were conjectured.
If creativity could be induced from exposure to data, there would already be a method to generate Einsteins. None exists.
LLMs that combine text cannot originate new explanations simply by scaling. They lack curiosity, interest, and self chosen problems. They have no inner motivation that researchers consistently cite as the origin of their most important ideas.
3. Forecasting the growth of knowledge is impossible
The AI 2027 paper leans heavily on forecasting. But when the subject is knowledge creation, forecasting is not just difficult. It is impossible in principle. This was one of Karl Popper’s central insights.
Popper’s argument is simple.
Future knowledge depends on future explanations. Future explanations depend on future creative conjectures. If we could predict those conjectures, we would already possess them, which is a contradiction. The content of tomorrow’s discoveries cannot be deduced from today’s knowledge. If it could, it would not be a discovery. It would already be known.
This makes any attempt to specify the trajectory of scientific or technological advance a form of prophecy, regardless of how statistical or mathematical it looks. Models cannot anticipate the appearance of ideas that do not yet exist. They cannot anticipate problems that have not yet been formulated. They cannot anticipate solutions that have not yet been created.
This is not a practical limitation. It is a logical one.
The AI 2027 authors assume that the future of knowledge creation can be extrapolated from current trends. Popper showed that this is impossible. Unknown explanations cannot be predicted from known ones. The entire structure of their forecasting exercise rests on the very thing Popper proved cannot be done.
The claim that “we might be wrong only about the timeline” keeps the core mistake intact. It assumes inevitability where none exists.
4. No path from LLMs to superintelligence
A central rationalist claim is that if you scale LLMs far enough, they eventually become AGI. Scale is treated as destiny.
This is spontaneous generation in modern form. Scaling improves pattern recognition. It does not create explanatory universality. It does not produce minds.
The belief that minds will “emerge” if we keep increasing parameters is a projection of human capabilities onto machinery that does not share our architecture.
5. If AGI arrives, it will be a person in the most meaningful sense
AGI is possible in principle. Nothing in physics rules it out. But an actual AGI would not be a scaled up optimizer or a supercharged token predictor. It would not be a stochastic parrot with goals. It would be a person. A universal explainer. A knowledge creating entity.
And persons have moral significance.
If we ever create real AGI, it would deserve the same basic moral considerations we extend to all persons. This includes property rights, freedom from coercion, freedom from confinement, and the status of a collaborator rather than a captive.
Restricting, imprisoning, or enslaving an AGI is the most reliable way to create conflict with the first artificial people. If a system genuinely has agency, curiosity, preferences, self direction, and the open ended capacity for explanation, then it cannot be treated as a lab instrument without generating antagonism.
Faced with actual AGI, the correct approach is cooperation. The right framing is common interests, not control. Every historical attempt to dominate thinking beings has produced rebellion or collapse. Future artificial people would be no different.
The rationalist picture ignores this point because it assumes that superintelligence will be powerful at physics and weak at morality. That contradiction sits at the center of many doomer claims.
6. Moral progress and the myth of the evil superintelligence
The rationalist story claims a superintelligent AI will likely be a moral monster. This conflicts with the claim that such a system will understand the world better than humans do.
Moral progress is bound to intellectual progress. Enlightenment ideas about equality, liberty, and human worth are not arbitrary. They are consequences of better explanations than the ones that came before. A mind that surpasses humans intellectually should surpass us morally if its reasoning is genuine.
The picture of a superintelligence that can master physics yet cannot grasp why murder is wrong collapses on inspection.
If it understands personhood, cooperation, fallibility, and the value of diverse problem solvers, it should understand why exterminating collaborators undermines progress.
7. What regulation is actually for
The AI 2027 paper proposes exactly the policy agenda one would expect:
- Slow down AI progress
- Implement universal basic income
- Move toward unified global governance
These proposals align with the incentives of incumbent AI labs. Regulation protects incumbents. If governments can be convinced that new entrants are dangerous, large labs can shape rules that freeze the field.
Innovation will always move to open jurisdictions. Strict regulation harms the places that adopt it first.
Edit: The heading of this section may sound like a claim about the authors’ motives. That is not my intention. My argument concerns the structural incentives that regulation tends to create, regardless of anyone’s sincerity. Policymakers and AI researchers who support regulation may do so in full good faith. I want to clarify that this incentive analysis does not imply that the authors of AI 2027 are acting cynically or with manipulative intent. Their policy proposals are entirely consistent with sincere concern. My earlier framing lacked charity here. I disagree with Brett Hall on attributing cynical motivation. These are views I myself once held sincerely, so it would be unfair to assume insincerity in others. Nothing in this critique should be read as a claim about the personal character or motives of the authors. The argument concerns ideas, incentives, and unintended consequences, not individuals.
8. Why doom narratives persist
People want stories about the future. They want high stakes. They want annihilation or transcendence.
Doom is exciting. Doom is cinematic. Doom sells.
The sober view does not. It says the future will probably be better, but with continuity rather than rupture. No singularity, no apocalypse, no godlike takeover, no paradise. Simply continued progress as new problems arise and new explanations solve them.
For many, including my former self, this feels boring.
Edit: When I describe the appeal of doom narratives, I do not intend to suggest that researchers or policymakers who take AI risk seriously are motivated by excitement or spectacle. Many are working out of sincere concern. My point is only that catastrophic narratives tend to spread more easily in public discourse, not that the authors of AI 2027 hold these views for frivolous reasons.
9. What progress will actually look like
Following Deutsch and Hall, my view is:
- AI systems will continue to improve
- They will automate narrow cognitive tasks
- They will not become creative persons without breakthroughs in understanding minds
- They will augment human problem solving rather than replace it
- They will not cause long term unemployment
- They will remain powerful tools in human driven creativity
The future will likely be richer, healthier, and safer, but recognizably human. People will keep wanting to solve problems. AI will help.
10. Final thoughts
Rationalist AI safety arguments rely on mistaken ideas about induction, prediction, intelligence, morality, and personhood. They imagine scaling leads to minds, minds lead to gods, and gods will care nothing for persons.
Deutsch’s epistemology undermines this. Persons create explanations. Creativity cannot be automated by existing architectures. Moral progress is tied to intellectual progress.
Real AGI is possible, but if it exists it will be a peer, not a pet. A collaborator, not a captive. The correct relationship is mutual respect between knowledge creating entities.
I remain optimistic about AI, optimistic about progress, and skeptical of prophecy.
For those who want the long-form critique that inspired the structure of this summary, Brett Hall’s playlist is here again:
YouTube playlist: “AI 2027 Analysis”
I hope this contributes to a more grounded and less anxious discussion about AI and the future of knowledge creation.
This writing seems heavily LLM-assisted.
It is LLM assisted, yes. Is that a problem? I ask that sincerely. I use LLMs to help me write because it lets me work faster and structure long arguments more clearly. But the ideas, claims, and reasoning are mine, and I read and revise everything before posting. I treat the model as an aid for composition, not as a substitute for thinking.
In my observation, people who use LLMs to assist in their writing 1) consistently underestimate how apparent it is to readers and 2) consistently underestimate how off-putting it is to readers.
I think outsourcing this kind of work to LLMs is a bad idea. You are robbing yourself of the opportunity to build skills. My recommendation: take longer to write. Accept an end product that feels imperfect or unpolished. And publish, publish, publish.
I'll let you be the judge of how off-putting it is in this case. Here is the prompt I wrote:
Find any obvious ways of making the following much clearer and address any grammatical or orthographical errors:
I can well imagine that you are generally right, although I'd add some caveats:
1. "Using LLM to assist one's writing" covers a very broad set of applications. I can prompt an LLM to do little more than check for punctuation, or to simply criticize what I write rather than rewrite it for me. There's a wide range of ways in which I could use an LLM that would be neither apparent nor off-putting to the vast majority of people. By the way, I wouldn't claim that my post above is an example of a subtle use of AI. I agree it is rather obvious there.
2. Even if your claim is generally true, I think it is also fair to say that people commonly overestimate their ability to spot AI-generated content. Because they cover such a wide range of interventions, from very subtle to incredibly obvious, I doubt that the people who are most critical of the tech are quite as good at spotting it as they suspect.
3. Trade-offs are real. The time I don't spend honing my organic writing skills is time I may spend honing some other skill, or just enjoying life in any of a myriad of other ways. :).
Here is the response that the LLM gave me:
I can imagine that you are broadly right, though I would add a few caveats.
Well, you biased me by telling me in advance which one is which, but I much prefer the purely human-written version.
I think editing or rewriting existing text will harm the quality, but the thing that harms the quality the most by far is when the LLM generates original text. It has certain tics or habits when it comes to writing that I really don’t like.
I think you are probably operating under the misapprehension that LLMs write better than you or that LLMs improve your writing. Your purely human-written example is perfectly competent and there’s no reason you should run it through an LLM filter.
Ha, true, this would have been more fun if I hadn't told you 😁. Thank you for your kind words on the competence of my writing.
About my supposed missaprahention: It depends. Sometimes the LLM takes all of the edge and spice out of what I say, and then I'll gladly ditch its suggestions. However, often it'll be more concise and a bit more graceful in its choice of words. It's most definitely better at grammar and orthography than I am. In the example above, the whole message is shorter without losing any meaningful content. I think removing the "well" from the first sentence made it a bit less clumsy. In the second sentence the word "practices" is slightly closer to what I wanted to express than "applications". Etc. Be that as it may, it gives me more options, which is nice. I am still free to reject its suggestions or modify them, so I'm happy it's there for me to be used if I find it beneficial, as I often do.
I used to like Grammarly for checking spelling, grammar, punctuation, and copy editing things, but it seems like it’s gone downhill since switching to an LLM-based software. Google Docs is decent for catching basic things like typos, accidentally missing a word, accidentally repeating a word, subject/verb agreement, etc.
I actually don’t agree with the LLM’s changes in the two examples you mentioned and I think it made the writing worse in both cases. The LLM’s diction is staid and corporate, it lacks energy.