Why I No Longer Find Rationalist AI Safety Arguments Persuasive

https://www.lesswrong.com/w/orthogonality-thesis

I won't go through this whole post but I'll pick out a few representative bits to reply to.

Deutsch’s idea of explanatory universality helps clarify the mistake. Persons are universal explainers. They create new explanations that were not contained in past data. This creativity is not extrapolation from a dataset. It is invention.

LLMs do not do this. They remix what exists in their training corpus. They do not originate explanatory theories.

This statement expresses a high degree of confidence in a claim that has, as far as I can tell, zero supporting evidence. I would strongly bet against the prediction that LLMs will never be able to originate an explanatory theory.

Until we understand how humans create explanatory knowledge, we cannot program that capacity.

We still don't know how humans create language, or prove mathematical conjectures, or manipulate objects in physical space, and yet we created AIs that can do those things.

The AI 2027 paper leans heavily on forecasting. But when the subject is knowledge creation, forecasting is not just difficult. It is impossible in principle. This was one of Karl Popper’s central insights.

I am not aware of any such insight? This claim seems easily falsified by the existence of superforecasters.

And: if prediction is impossible in principle, then you can't confidently say that ASI won't kill everyone, therefore you should regard it as potentially dangerous. But you seem to be quite confident that you know what ASI will be like.

The rationalist story claims a superintelligent AI will likely be a moral monster. This conflicts with the claim that such a system will understand the world better than humans do.

8mo*

This statement expresses a high degree of confidence in a claim that has, as far as I can tell, zero supporting evidence. I would strongly bet against the prediction that LLMs will never be able to originate an explanatory theory.

When? In 1 year, 10 years, 100 years, or 1000 years? And involving what new technological paradigms or new basic science?

I think the quoted claim is true as stated — no LLM has created any explanatory theory so far.

I am not aware of any such insight? This claim seems easily falsified by the existence of superforecasters.

I think you’re misunderstanding the argument. No superforecaster has ever predicted the content of a new scientific discovery.

This writing seems heavily LLM-assisted.

It is LLM assisted, yes. Is that a problem? I ask that sincerely. I use LLMs to help me write because it lets me work faster and structure long arguments more clearly. But the ideas, claims, and reasoning are mine, and I read and revise everything before posting. I treat the model as an aid for composition, not as a substitute for thinking.

In my observation, people who use LLMs to assist in their writing 1) consistently underestimate how apparent it is to readers and 2) consistently underestimate how off-putting it is to readers.

I think outsourcing this kind of work to LLMs is a bad idea. You are robbing yourself of the opportunity to build skills. My recommendation: take longer to write. Accept an end product that feels imperfect or unpolished. And publish, publish, publish.

I'll let you be the judge of how off-putting it is in this case. Here is the prompt I wrote:

Find any obvious ways of making the following much clearer and address any grammatical or orthographical errors:

I can well imagine that you are generally right, although I'd add some caveats:

1. "Using LLM to assist one's writing" covers a very broad set of applications. I can prompt an LLM to do little more than check for punctuation, or to simply criticize what I write rather than rewrite it for me. There's a wide range of ways in which I could use an LLM that would be neither apparent nor off-putting to the vast majority of people. By the way, I wouldn't claim that my post above is an example of a subtle use of AI. I agree it is rather obvious there.

2. Even if your claim is generally true, I think it is also fair to say that people commonly overestimate their ability to spot AI-generated content. Because they cover such a wide range of interventions, from very subtle to incredibly obvious, I doubt that the people who are most critical of the tech are quite as good at spotting it as they suspect.

3. Trade-offs are real. The time I don't spend honing my organic writing skills is time I may spend honing some other skill, or just enjoying life in any of a myriad of other ways. :).

Here is the response that the LLM gave me:

I can imagine that you are broadly right, though I would add a few caveats.

“Using an LLM to assist one’s writing” covers a wide range of practices. I might ask an LLM to check punctuation or to critique what I wrote instead of rewriting it. Many forms of assistance are subtle enough that most people would neither notice nor be bothered by them. To be clear, my post above is not an example of subtle use. I agree it is quite obvious there.
Even if your claim holds in general, people often overestimate their ability to detect AI generated text. Since the level of intervention can vary from minimal to very heavy, I doubt that the harshest critics are as good at spotting it as they think.
Trade offs matter. Time I do not spend polishing my unaided writing is time I can put into other skills or simply enjoying life in other ways.

Well, you biased me by telling me in advance which one is which, but I much prefer the purely human-written version.

I think editing or rewriting existing text will harm the quality, but the thing that harms the quality the most by far is when the LLM generates original text. It has certain tics or habits when it comes to writing that I really don’t like.

I think you are probably operating under the misapprehension that LLMs write better than you or that LLMs improve your writing. Your purely human-written example is perfectly competent and there’s no reason you should run it through an LLM filter.

7mo

Ha, true, this would have been more fun if I hadn't told you 😁. Thank you for your kind words on the competence of my writing.

About my supposed missaprahention: It depends. Sometimes the LLM takes all of the edge and spice out of what I say, and then I'll gladly ditch its suggestions. However, often it'll be more concise and a bit more graceful in its choice of words. It's most definitely better at grammar and orthography than I am. In the example above, the whole message is shorter without losing any meaningful content. I think removing the "well" from the first sentence made it a bit less clumsy. In the second sentence the word "practices" is slightly closer to what I wanted to express than "applications". Etc. Be that as it may, it gives me more options, which is nice. I am still free to reject its suggestions or modify them, so I'm happy it's there for me to be used if I find it beneficial, as I often do.

7mo

I used to like Grammarly for checking spelling, grammar, punctuation, and copy editing things, but it seems like it’s gone downhill since switching to an LLM-based software. Google Docs is decent for catching basic things like typos, accidentally missing a word, accidentally repeating a word, subject/verb agreement, etc.

I actually don’t agree with the LLM’s changes in the two examples you mentioned and I think it made the writing worse in both cases. The LLM’s diction is staid and corporate, it lacks energy.

I broadly agree with this argument and you’ve articulated it superbly. This is an excellent post. Thank you for writing this.

The only part I really strongly disagree with is about the intentions behind the policy recommendations of AI 2027 — I think the policy recommendations and the reasons stated for them are sincere and honest, not cynical and manipulative. Whether some people at some AI companies are invoking similar ideas for cynical, manipulative reasons, I can’t say. All I can say is that all the evidence points toward the conclusion that the vast majority of people who want regulations due to concerns about AGI safety/alignment are completely sincere "true believers" and don’t have a financial motive.

The Beginning of Infinity by David Deutsch is one of my favourite books. His arguments in that book are incredibly creative and feel really promising. But I still treat them as creative conjecture rather than settled truth. So, I’m inclined to agree with almost all your points, and I’m strongly inclined to agree with some of them, but I wouldn’t state them as being absolutely true. That’s the caveat.

However, the rationalist/LessWrong/MIRI view on AGI risk is also conjecture, and it’s much worse conjecture than the arguments Deutsch has made (or that you articulated in this post). I think it is highly constructive and highly important to challenge these conjectures with other conjectures like Deutsch’s (or yours). Deutsch’s view is more coherent and more appealing in other ways (e.g. seems to rest on better philosophical/scientific foundations), so it’s at least preferable to the rationalist/LessWrong/MIRI view, even if it’s not correct.

8mo*

Thank you for this thoughtful and generous comment, Yarrow. I appreciate it.

On your first point, I think your criticism is well placed. I should not have psychologized the intentions behind the policy recommendations in AI 2027. The argument does not require that the authors be cynical, and I have no reliable way of knowing their motivations. Their recommendations are entirely consistent with sincere concern viewed through their framework. After reconsidering this, I agree that my original framing was uncharitable. I will revise that section of the post, and I should also be clear that on this particular issue I diverge from Brett Hall’s interpretation. These fatalistic views are ones I myself once held with full sincerity, so presuming insincerity in others would be unjustified.

On your second point, I fully agree that all knowledge is conjectural. Deutsch emphasises fallibilism strongly, and I would never claim any of these arguments as settled truth. They are conjectures offered because, at present, they seem to be better explanations than the rationalist alternatives.

Thank you again for engaging with the post so carefully. Your comment improved the argument and helped catch a place where my own framing fell short of proper charity.

Edit: I have added several clarifying notes to the post (marked as “Edit”). I hope these address your well placed criticism and correct the earlier lack of charity.

8mo*

Great response! Thank you!

There is a sense in which all knowledge is conjectural. The Standard Model of particle physics is conjectural in some sense. But what I’m saying about Deutsch’s epistemology and philosophy of science — and especially his philosophy of mind and overall theory of progress — is much stronger than that. I have tried to search in the past for any academic philosophers who have seriously discussed The Beginning of Infinity. I haven’t found much so far.

I’m impressed with the book, but in philosophy, it’s always possible to be impressed with arguments and then not to notice important shortcomings until someone points them out. I would love to see The Beginning of Infinity get more scrutiny from academic philosophers and possibly other experts as well, like scientists and historians of science.

Right, exactly. Even seemingly self evident mathematical claims are not exempt from this. We can never rule out the possibility of error, so epistemological certainty is unattainable. People can feel certain, but whenever they do, they are mistaken about the status of that feeling.

This is often confused with relativism. We say true things all the time. “2 + 2 = 4” is true, yet there is no final and infallible way to confirm it. Nothing is immune to criticism.

The Beginning of Infinity is my favorite book. I have been active in the critical rationalist circles that have formed around David Deutsch’s work. If you want pointers, I know quite a few people in that community. If you are looking for sharp criticisms of Deutsch, asking his admirers may not be ideal. Even though we fully accept that Deutsch is fallible like anyone else, most of us agree with him on the major points.

It is good that you want to find strong criticisms of his ideas. I hope you succeed. To be frank, I have yet to encounter substantial critiques of The Beginning of Infinity, though I am sure there are errors in it.

My friend Logan founded the Conjecture Institute this year. It might interest you:

https://www.conjectureinstitute.org/

Fundamental Philosophical Errors in Taking Children Seriously

I am specifically interested in commentaries on The Beginning of Infinity from professional academic philosophers, as well as, to a lesser extent, commentaries from scientists or people with other relevant expertise, such as historians of science. I haven’t found much written by philosophers about the book or about Deutsch’s ideas in general.

The work of “fans” or “followers” is much less interesting to me because I get why Deutsch’s ideas are appealing and would inspire a zealous following, but this is not a critical appraisal.

Elliot Temple

7mo

-5

Hi, I've been an independent, non-academic philosopher for over 20 years. I was mentored by David Deutsch for a decade and I helped him with The Beginning of Infinity. I'm no longer a fan of Deutsch and I have commentary, much of it critical, on Deutsch's work:

The Historicism of David Deutsch

Analysis of David Deutsch’s The Final Prejudice

David Deutsch Smears Ayn Rand

Deutsch Misquoted Turing

Misquotes by David Deutsch

You can contact me at [email protected] if you want to discuss further.

SummaryBot

Hard-to-reverse decisions destroy option value

Executive summary: The author argues that rationalist AI safety narratives are built on philosophical and epistemological errors about knowledge, creativity, and personhood, and that AI progress will continue in a grounded, non-catastrophic way.

Key points:

The rationalist AI safety view mistakes pattern recognition for personhood, assuming minds can “emerge” from scaling LLMs, which the author compares to spontaneous generation.
Following David Deutsch, the author defines persons as “universal explainers” capable of creative explanation rather than data extrapolation, a process current AI systems cannot perform.
Drawing on Karl Popper, the author argues forecasting the growth of knowledge is impossible in principle because future explanations cannot be derived from existing ones.
Scaling LLMs does not yield AGI, since pattern recognition lacks explanatory creativity; true AGI would require philosophical breakthroughs about mind and knowledge.
A genuine AGI would be a moral person deserving rights and cooperation, not control, since attempts to dominate intelligent beings historically lead to conflict.
The notion of an “evil superintelligence” contradicts itself: a mind superior in understanding should also surpass humans morally if its reasoning is sound.
Proposed AI regulation often benefits incumbent labs and risks stifling innovation by concentrating power and freezing competition.
Doom narratives persist because they are emotionally and narratively compelling, unlike the more likely scenario of steady, human-centered progress.
Future AI will automate narrow tasks, augment human creativity, and improve living standards without replacing humans or creating existential catastrophe.
Rationalist AI safety’s core mistake is philosophical: creativity and moral understanding cannot emerge from scaling pattern recognizers, and real AGI, if achieved, would be a collaborator, not a threat.

This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.

Comments

Curated and popular this week

Stefan_Schubert·9y ago·Curated 16h ago·14m read

This post is co-authored with Ben Garfinkel. It is cross-posted from the CEA blog. A PDF version can be found here. Summary: Some strategic decisions available to the effective altruism m...

If you're agentic, work in biosecurity

sharmaayushmaan🔸·5d ago·7m read

Disclaimer: Although I work on the Groups Team at CEA, I’m writing this in a personal capacity, and this post does not constitute an endorsement by CEA. Agency - the realisation that you really can just do things. TL;DR Biosecurity needs people (of any background) who are agentic and have a high execution velocity and track record....

Introducing Impact List: a ranking of philanthropists by expected lives saved

Elliot Olds·1d ago·6m read

TL;DR: I'm releasing a website that ranks philanthropists according to EA principles and research, and allows users to re-rank the list using their own assumptions. I'd like feedback and help making it better. I'd especially like ideas for how to make the results more trustworthy. Funding may be available. I recently built Impact List (impactlist.xyz), a site which ranks people by their positive impact via donations. The goal is t...

Recent opportunities to take action

Marginal Victories: career advising and opportunities for U.S. democracy preservation & political work

Annika Burman 🔸·3d ago·2m read

I'm stepping down as Hive's Executive Director, and we're hiring my successor

SofiaBalderson, Hive·3d ago·3m read

Starting an EA group @ SUNY Binghamton

micahzarin·2d ago·1m read