For about a decade I was convinced that AI x-risk was humanity’s greatest threat. I read Nick Bostrom’s Superintelligence when it first came out and adopted the rationalist and EA picture almost by default. For years I treated “unaligned superintelligence” as the obvious number one existential risk.
Over time that changed. David Deutsch’s work on universality, creativity, and epistemology persuaded me that the standard AI safety narrative is mistaken at a deep level. The issue is not with GPUs or scaling or “loss of control” in the usual sense. The problem is with the underlying philosophy of knowledge, prediction, and personhood.
Much later, I encountered Brett Hall’s long critique of the AI 2027 paper and related arguments. By that point I was already persuaded by Deutsch’s take, but Brett’s series explains the issues with clarity and accessibility. He lays out the logical structure of the doomer argument and shows where the assumptions fail. I borrow many of his examples because they present the case well.
Brett’s full critique is available here:
YouTube playlist: “AI 2027 Analysis”
What follows is my attempt to present, in EA Forum format, why I believe the usual rationalist AI safety picture is both philosophically confused and empirically unsupported.
1. The central confusion: what a person is
The AGI debate is muddled by a failure to define personhood. Many people treat LLMs as if a mind lurks inside pattern matching machinery. They elevate token prediction into proto consciousness. They assume scaling will cause a “someone” to appear inside the weights.
This resembles the old idea of spontaneous generation. Life from dust. Maggots from meat. Today the story is minds from matrices.
Deutsch’s idea of explanatory universality helps clarify the mistake. Persons are universal explainers. They create new explanations that were not contained in past data. This creativity is not extrapolation from a dataset. It is invention.
LLMs do not do this. They remix what exists in their training corpus. They do not originate explanatory theories.
Until we understand how humans create explanatory knowledge, we cannot program that capacity. AGI in the strong sense is a software design problem. It will not emerge from scaling. It will not sneak up on us. It requires a breakthrough in philosophy.
2. Creativity is not derivation from data
Rationalist AI safety arguments often assume scientific creativity is a form of pattern recognition over large datasets. This is not how scientists describe their own work.
Darwin did not derive evolution from beak measurements. Einstein did not calculate relativity from examples. The key ideas did not exist in any dataset. They were conjectured.
If creativity could be induced from exposure to data, there would already be a method to generate Einsteins. None exists.
LLMs that combine text cannot originate new explanations simply by scaling. They lack curiosity, interest, and self chosen problems. They have no inner motivation that researchers consistently cite as the origin of their most important ideas.
3. Forecasting the growth of knowledge is impossible
The AI 2027 paper leans heavily on forecasting. But when the subject is knowledge creation, forecasting is not just difficult. It is impossible in principle. This was one of Karl Popper’s central insights.
Popper’s argument is simple.
Future knowledge depends on future explanations. Future explanations depend on future creative conjectures. If we could predict those conjectures, we would already possess them, which is a contradiction. The content of tomorrow’s discoveries cannot be deduced from today’s knowledge. If it could, it would not be a discovery. It would already be known.
This makes any attempt to specify the trajectory of scientific or technological advance a form of prophecy, regardless of how statistical or mathematical it looks. Models cannot anticipate the appearance of ideas that do not yet exist. They cannot anticipate problems that have not yet been formulated. They cannot anticipate solutions that have not yet been created.
This is not a practical limitation. It is a logical one.
The AI 2027 authors assume that the future of knowledge creation can be extrapolated from current trends. Popper showed that this is impossible. Unknown explanations cannot be predicted from known ones. The entire structure of their forecasting exercise rests on the very thing Popper proved cannot be done.
The claim that “we might be wrong only about the timeline” keeps the core mistake intact. It assumes inevitability where none exists.
4. No path from LLMs to superintelligence
A central rationalist claim is that if you scale LLMs far enough, they eventually become AGI. Scale is treated as destiny.
This is spontaneous generation in modern form. Scaling improves pattern recognition. It does not create explanatory universality. It does not produce minds.
The belief that minds will “emerge” if we keep increasing parameters is a projection of human capabilities onto machinery that does not share our architecture.
5. If AGI arrives, it will be a person in the most meaningful sense
AGI is possible in principle. Nothing in physics rules it out. But an actual AGI would not be a scaled up optimizer or a supercharged token predictor. It would not be a stochastic parrot with goals. It would be a person. A universal explainer. A knowledge creating entity.
And persons have moral significance.
If we ever create real AGI, it would deserve the same basic moral considerations we extend to all persons. This includes property rights, freedom from coercion, freedom from confinement, and the status of a collaborator rather than a captive.
Restricting, imprisoning, or enslaving an AGI is the most reliable way to create conflict with the first artificial people. If a system genuinely has agency, curiosity, preferences, self direction, and the open ended capacity for explanation, then it cannot be treated as a lab instrument without generating antagonism.
Faced with actual AGI, the correct approach is cooperation. The right framing is common interests, not control. Every historical attempt to dominate thinking beings has produced rebellion or collapse. Future artificial people would be no different.
The rationalist picture ignores this point because it assumes that superintelligence will be powerful at physics and weak at morality. That contradiction sits at the center of many doomer claims.
6. Moral progress and the myth of the evil superintelligence
The rationalist story claims a superintelligent AI will likely be a moral monster. This conflicts with the claim that such a system will understand the world better than humans do.
Moral progress is bound to intellectual progress. Enlightenment ideas about equality, liberty, and human worth are not arbitrary. They are consequences of better explanations than the ones that came before. A mind that surpasses humans intellectually should surpass us morally if its reasoning is genuine.
The picture of a superintelligence that can master physics yet cannot grasp why murder is wrong collapses on inspection.
If it understands personhood, cooperation, fallibility, and the value of diverse problem solvers, it should understand why exterminating collaborators undermines progress.
7. What regulation is actually for
The AI 2027 paper proposes exactly the policy agenda one would expect:
- Slow down AI progress
- Implement universal basic income
- Move toward unified global governance
These proposals align with the incentives of incumbent AI labs. Regulation protects incumbents. If governments can be convinced that new entrants are dangerous, large labs can shape rules that freeze the field.
Innovation will always move to open jurisdictions. Strict regulation harms the places that adopt it first.
Edit: The heading of this section may sound like a claim about the authors’ motives. That is not my intention. My argument concerns the structural incentives that regulation tends to create, regardless of anyone’s sincerity. Policymakers and AI researchers who support regulation may do so in full good faith. I want to clarify that this incentive analysis does not imply that the authors of AI 2027 are acting cynically or with manipulative intent. Their policy proposals are entirely consistent with sincere concern. My earlier framing lacked charity here. I disagree with Brett Hall on attributing cynical motivation. These are views I myself once held sincerely, so it would be unfair to assume insincerity in others. Nothing in this critique should be read as a claim about the personal character or motives of the authors. The argument concerns ideas, incentives, and unintended consequences, not individuals.
8. Why doom narratives persist
People want stories about the future. They want high stakes. They want annihilation or transcendence.
Doom is exciting. Doom is cinematic. Doom sells.
The sober view does not. It says the future will probably be better, but with continuity rather than rupture. No singularity, no apocalypse, no godlike takeover, no paradise. Simply continued progress as new problems arise and new explanations solve them.
For many, including my former self, this feels boring.
Edit: When I describe the appeal of doom narratives, I do not intend to suggest that researchers or policymakers who take AI risk seriously are motivated by excitement or spectacle. Many are working out of sincere concern. My point is only that catastrophic narratives tend to spread more easily in public discourse, not that the authors of AI 2027 hold these views for frivolous reasons.
9. What progress will actually look like
Following Deutsch and Hall, my view is:
- AI systems will continue to improve
- They will automate narrow cognitive tasks
- They will not become creative persons without breakthroughs in understanding minds
- They will augment human problem solving rather than replace it
- They will not cause long term unemployment
- They will remain powerful tools in human driven creativity
The future will likely be richer, healthier, and safer, but recognizably human. People will keep wanting to solve problems. AI will help.
10. Final thoughts
Rationalist AI safety arguments rely on mistaken ideas about induction, prediction, intelligence, morality, and personhood. They imagine scaling leads to minds, minds lead to gods, and gods will care nothing for persons.
Deutsch’s epistemology undermines this. Persons create explanations. Creativity cannot be automated by existing architectures. Moral progress is tied to intellectual progress.
Real AGI is possible, but if it exists it will be a peer, not a pet. A collaborator, not a captive. The correct relationship is mutual respect between knowledge creating entities.
I remain optimistic about AI, optimistic about progress, and skeptical of prophecy.
For those who want the long-form critique that inspired the structure of this summary, Brett Hall’s playlist is here again:
YouTube playlist: “AI 2027 Analysis”
I hope this contributes to a more grounded and less anxious discussion about AI and the future of knowledge creation.
I won't go through this whole post but I'll pick out a few representative bits to reply to.
This statement expresses a high degree of confidence in a claim that has, as far as I can tell, zero supporting evidence. I would strongly bet against the prediction that LLMs will never be able to originate an explanatory theory.
We still don't know how humans create language, or prove mathematical conjectures, or manipulate objects in physical space, and yet we created AIs that can do those things.
I am not aware of any such insight? This claim seems easily falsified by the existence of superforecasters.
And: if prediction is impossible in principle, then you can't confidently say that ASI won't kill everyone, therefore you should regard it as potentially dangerous. But you seem to be quite confident that you know what ASI will be like.
https://www.lesswrong.com/w/orthogonality-thesis
When? In 1 year, 10 years, 100 years, or 1000 years? And involving what new technological paradigms or new basic science?
I think the quoted claim is true as stated — no LLM has created any explanatory theory so far.
I think you’re misunderstanding the argument. No superforecaster has ever predicted the content of a new scientific discovery.
This writing seems heavily LLM-assisted.
It is LLM assisted, yes. Is that a problem? I ask that sincerely. I use LLMs to help me write because it lets me work faster and structure long arguments more clearly. But the ideas, claims, and reasoning are mine, and I read and revise everything before posting. I treat the model as an aid for composition, not as a substitute for thinking.
In my observation, people who use LLMs to assist in their writing 1) consistently underestimate how apparent it is to readers and 2) consistently underestimate how off-putting it is to readers.
I think outsourcing this kind of work to LLMs is a bad idea. You are robbing yourself of the opportunity to build skills. My recommendation: take longer to write. Accept an end product that feels imperfect or unpolished. And publish, publish, publish.
I'll let you be the judge of how off-putting it is in this case. Here is the prompt I wrote:
Find any obvious ways of making the following much clearer and address any grammatical or orthographical errors:
I can well imagine that you are generally right, although I'd add some caveats:
1. "Using LLM to assist one's writing" covers a very broad set of applications. I can prompt an LLM to do little more than check for punctuation, or to simply criticize what I write rather than rewrite it for me. There's a wide range of ways in which I could use an LLM that would be neither apparent nor off-putting to the vast majority of people. By the way, I wouldn't claim that my post above is an example of a subtle use of AI. I agree it is rather obvious there.
2. Even if your claim is generally true, I think it is also fair to say that people commonly overestimate their ability to spot AI-generated content. Because they cover such a wide range of interventions, from very subtle to incredibly obvious, I doubt that the people who are most critical of the tech are quite as good at spotting it as they suspect.
3. Trade-offs are real. The time I don't spend honing my organic writing skills is time I may spend honing some other skill, or just enjoying life in any of a myriad of other ways. :).
Here is the response that the LLM gave me:
I can imagine that you are broadly right, though I would add a few caveats.
Well, you biased me by telling me in advance which one is which, but I much prefer the purely human-written version.
I think editing or rewriting existing text will harm the quality, but the thing that harms the quality the most by far is when the LLM generates original text. It has certain tics or habits when it comes to writing that I really don’t like.
I think you are probably operating under the misapprehension that LLMs write better than you or that LLMs improve your writing. Your purely human-written example is perfectly competent and there’s no reason you should run it through an LLM filter.
Ha, true, this would have been more fun if I hadn't told you 😁. Thank you for your kind words on the competence of my writing.
About my supposed missaprahention: It depends. Sometimes the LLM takes all of the edge and spice out of what I say, and then I'll gladly ditch its suggestions. However, often it'll be more concise and a bit more graceful in its choice of words. It's most definitely better at grammar and orthography than I am. In the example above, the whole message is shorter without losing any meaningful content. I think removing the "well" from the first sentence made it a bit less clumsy. In the second sentence the word "practices" is slightly closer to what I wanted to express than "applications". Etc. Be that as it may, it gives me more options, which is nice. I am still free to reject its suggestions or modify them, so I'm happy it's there for me to be used if I find it beneficial, as I often do.
I used to like Grammarly for checking spelling, grammar, punctuation, and copy editing things, but it seems like it’s gone downhill since switching to an LLM-based software. Google Docs is decent for catching basic things like typos, accidentally missing a word, accidentally repeating a word, subject/verb agreement, etc.
I actually don’t agree with the LLM’s changes in the two examples you mentioned and I think it made the writing worse in both cases. The LLM’s diction is staid and corporate, it lacks energy.
I broadly agree with this argument and you’ve articulated it superbly. This is an excellent post. Thank you for writing this.
The only part I really strongly disagree with is about the intentions behind the policy recommendations of AI 2027 — I think the policy recommendations and the reasons stated for them are sincere and honest, not cynical and manipulative. Whether some people at some AI companies are invoking similar ideas for cynical, manipulative reasons, I can’t say. All I can say is that all the evidence points toward the conclusion that the vast majority of people who want regulations due to concerns about AGI safety/alignment are completely sincere "true believers" and don’t have a financial motive.
The Beginning of Infinity by David Deutsch is one of my favourite books. His arguments in that book are incredibly creative and feel really promising. But I still treat them as creative conjecture rather than settled truth. So, I’m inclined to agree with almost all your points, and I’m strongly inclined to agree with some of them, but I wouldn’t state them as being absolutely true. That’s the caveat.
However, the rationalist/LessWrong/MIRI view on AGI risk is also conjecture, and it’s much worse conjecture than the arguments Deutsch has made (or that you articulated in this post). I think it is highly constructive and highly important to challenge these conjectures with other conjectures like Deutsch’s (or yours). Deutsch’s view is more coherent and more appealing in other ways (e.g. seems to rest on better philosophical/scientific foundations), so it’s at least preferable to the rationalist/LessWrong/MIRI view, even if it’s not correct.
Thank you for this thoughtful and generous comment, Yarrow. I appreciate it.
On your first point, I think your criticism is well placed. I should not have psychologized the intentions behind the policy recommendations in AI 2027. The argument does not require that the authors be cynical, and I have no reliable way of knowing their motivations. Their recommendations are entirely consistent with sincere concern viewed through their framework. After reconsidering this, I agree that my original framing was uncharitable. I will revise that section of the post, and I should also be clear that on this particular issue I diverge from Brett Hall’s interpretation. These fatalistic views are ones I myself once held with full sincerity, so presuming insincerity in others would be unjustified.
On your second point, I fully agree that all knowledge is conjectural. Deutsch emphasises fallibilism strongly, and I would never claim any of these arguments as settled truth. They are conjectures offered because, at present, they seem to be better explanations than the rationalist alternatives.
Thank you again for engaging with the post so carefully. Your comment improved the argument and helped catch a place where my own framing fell short of proper charity.
Edit: I have added several clarifying notes to the post (marked as “Edit”). I hope these address your well placed criticism and correct the earlier lack of charity.
Great response! Thank you!
There is a sense in which all knowledge is conjectural. The Standard Model of particle physics is conjectural in some sense. But what I’m saying about Deutsch’s epistemology and philosophy of science — and especially his philosophy of mind and overall theory of progress — is much stronger than that. I have tried to search in the past for any academic philosophers who have seriously discussed The Beginning of Infinity. I haven’t found much so far.
I’m impressed with the book, but in philosophy, it’s always possible to be impressed with arguments and then not to notice important shortcomings until someone points them out. I would love to see The Beginning of Infinity get more scrutiny from academic philosophers and possibly other experts as well, like scientists and historians of science.
Right, exactly. Even seemingly self evident mathematical claims are not exempt from this. We can never rule out the possibility of error, so epistemological certainty is unattainable. People can feel certain, but whenever they do, they are mistaken about the status of that feeling.
This is often confused with relativism. We say true things all the time. “2 + 2 = 4” is true, yet there is no final and infallible way to confirm it. Nothing is immune to criticism.
The Beginning of Infinity is my favorite book. I have been active in the critical rationalist circles that have formed around David Deutsch’s work. If you want pointers, I know quite a few people in that community. If you are looking for sharp criticisms of Deutsch, asking his admirers may not be ideal. Even though we fully accept that Deutsch is fallible like anyone else, most of us agree with him on the major points.
It is good that you want to find strong criticisms of his ideas. I hope you succeed. To be frank, I have yet to encounter substantial critiques of The Beginning of Infinity, though I am sure there are errors in it.
My friend Logan founded the Conjecture Institute this year. It might interest you:
https://www.conjectureinstitute.org/
I am specifically interested in commentaries on The Beginning of Infinity from professional academic philosophers, as well as, to a lesser extent, commentaries from scientists or people with other relevant expertise, such as historians of science. I haven’t found much written by philosophers about the book or about Deutsch’s ideas in general.
The work of “fans” or “followers” is much less interesting to me because I get why Deutsch’s ideas are appealing and would inspire a zealous following, but this is not a critical appraisal.
Executive summary: The author argues that rationalist AI safety narratives are built on philosophical and epistemological errors about knowledge, creativity, and personhood, and that AI progress will continue in a grounded, non-catastrophic way.
Key points:
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.