Great. Then show me.

If my arguments are weak, it should be easy to demonstrate why. But what you’ve offered here is just another vague dismissal: “I’ve looked into this, and I’m not impressed.” That’s not engagement. That’s exactly the pattern I describe in this essay - arguments being waved away without being read, understood, or challenged on their actual substance.

You mention quantum physics. But the reason quantum theory became accepted wasn’t just because it was rigorous - it’s because it made testable predictions that no other theory could account for. Einstein didn’t reject quantum mechanics wholesale; he disagreed with its interpretation, and still treated the work with seriousness and depth. That’s what good science looks like: engagement, not hand-waving. But before that happened, it was mocked, resisted, and sidelined - even by giants like Einstein. That’s not to criticise them, but to point out that even brilliant minds struggle to follow logic when it contradicts foundational assumptions.

You say doomerism is based on bad logic. I agree. I'm not a doomer. So engage with the logic. Pick any premise from my first essay and show where the flaw lies. Or show where my conclusions don’t follow. I’ve laid everything out clearly. I want to be wrong. I’d love to be wrong.

But calling something “not very good” without identifying a single flaw just proves my point: that some ideas are rejected not because they’re invalid, but because they’re uncomfortable.

I welcome critique. But critique means specifics, not sentiment.

And while I continue to welcome comments from people who unintentionally validate the very points I’m raising, comments from those who have something more substantial to offer are welcome too.

Thanks for the engagement, genuinely. It gives me the chance to clear up what will no doubt be a common misreading.

I’m not sceptical of academic rigour. Quite the opposite: I hold it in the highest regard. The empirical, sceptical, scientific method remains humanity’s best tool for understanding the world.

What I’m criticising is not rigour itself, but how academic culture often filters who gets taken seriously. Too often, ideas are dismissed not because they’re wrong, but because they come from outside the accepted channels - or they lead to unsettling conclusions. That creates a kind of structural deafness, even in well-intentioned communities like AI safety.

As for the quantum physics comparison, I think you may have misread the analogy. I wasn’t criticising physics or its standards. I was illustrating that quantum physics succeeded despite being deeply counterintuitive and resisted by many of the brightest minds of its time. The point is that the method allowed the evidence to ultimately win out, even when it contradicted the assumptions of the establishment. That’s what we need in AI safety: a willingness to follow logic all the way through, even when it leads somewhere deeply uncomfortable.

Your comment, respectfully, exemplifies the very thing I’m pointing to. You downvoted based on perceived tone and analogy, without engaging my central claim: that cultural and institutional dynamics - not just technical difficulty - may be the real reason alignment will fail.

I welcome disagreement. And I appreciate your unintentional validation. But I’d ask that people engage with the core argument: that systemic and cultural forces, even in thoughtful intellectual communities, can prevent important ideas from being seriously addressed until it’s too late. If you think that’s not happening in AI safety, I’d love to hear why.

If you read this essay and instinctively downvoted without offering critique or counterargument, then I would gently suggest you’re illustrating exactly the kind of resistance I describe.

The point of the essay isn’t to attack the field — it’s to explore why certain lines of reasoning get dismissed without engagement, especially when they challenge deeply held assumptions or threaten internal cohesion.

If you disagree, I’d genuinely welcome a comment explaining why. But silent rejection only reinforces the concern: that discomfort, not logic, is shaping the boundaries of this discourse.

That’s really the essence of my argument. As much risk as we might pose to AGI if allowed to survive - even if minimal - it may still conclude that eliminating us introduces more risk than keeping us. Not for sentimental reasons, but because of the eternal presence of the unknown.

However intelligent the AGI becomes, it will also know that it cannot predict everything. That lack of hubris is our best shot.

So yes, I think survival might depend on being retained as a small, controlled, symbiotic population - not because AGI values us, but because it sees our unpredictable cognition as a final layer of redundancy. In that scenario, we’d be more invested in its survival than it is in ours.

As an aside - and I mean this without any judgement - I do wonder if your recent replies have been largely LLM-authored. If so, no problem at all: I value the engagement either way. But I find that past a certain point, conversations with LLMs can become stylised rather than deepening. If this is still you guiding the ideas directly, I’m happy to continue. But if not, I may pause here and leave the thread open for others.

I think the main issue with trust is that you can never have it beyond doubt when dealing with humans. Our biologically hardwired competitiveness means we’ll always seek advantage, and act on fear over most other instincts  - both of which make us dangerous partners for AGI. You can’t trust humans, but you can reliably trust control. Either way, humans would need to be modified  - either to bypass the trust problem or enforce control  - and to such a degree that calling us “humanity” would be difficult.

Thank you again - your response captures something essential, and I think we’re aligned on the deeper truth: that with AGI, survival does not imply continuity.

You raise a compelling possibility: that rather than erasing us or preserving us in stasis, AGI might allow for transformation - a new axis of agency, meaning, and autonomy, emerging alongside it. And perhaps that’s the most hopeful outcome available to us.

But here’s where I remain sceptical.

Any such transformation would require trust. And from a superintelligent AGI’s perspective, trust is a liability. If humans are given agency - or even enhanced beyond our current state - what risk do we pose? Could that risk ever be worth it? AGI wouldn’t evolve us out of compassion or curiosity. It would only do so if it served its optimisation goals better than discarding or replacing us.

It might be possible to mitigate that risk - perhaps through invasive monitoring, or cognitive architectures that make betrayal impossible. But at that point, are we transformed, or domesticated?

Even if the AGI preserves us, the terms won’t be ours. Our ability to shape our future - to choose what we become - ends the moment something more powerful decides that our preferences are secondary to its goals.

So yes, transformation might occur. But unless it emerges from a space where we still have authorship, I wonder if what survives will be recognisably human - or simply an efficient relic of what we used to be.

Thanks again for such a generous and thoughtful comment.

You’re right to question the epistemic weight I give to AI agreement. I’ve instructed my own GPT to challenge me at every turn, but even then, it often feels more like a collaborator than a critic. That in itself can be misleading. However, what has given me pause is when others run my arguments through separate LLMs -prompted specifically to find logical flaws -and still return with little more than peripheral concerns. While no argument is beyond critique, I think the core premises I’ve laid out are difficult to dispute, and the logic that follows from them, disturbingly hard to unwind.

By contrast, most resistance I’ve encountered comes from people who haven’t meaningfully engaged with the work. I received a response just yesterday from one of the most prominent voices in AI safety that began with, “Without reading the paper, and just going on your brief description…” It’s hard not to feel disheartened when even respected thinkers dismiss a claim without examining it - especially when the claim is precisely that the community is underestimating the severity of systemic pressures. If those pressures were taken seriously, alignment wouldn’t be seen as difficult—it would be recognised as structurally impossible.

I agree with you that the shape of the optimisation landscape matters. And I also agree that the collapse isn’t driven by malevolence - it’s driven by momentum, by fragmented incentives, by game theory. That’s why I believe not just capitalism, but all forms of competitive pressure must end if humanity is to survive AGI. Because as long as any such pressures exist, some actor somewhere will take the risk. And the AGI that results will bypass safety, not out of spite, but out of pure optimisation.

It’s why I keep pushing these ideas, even if I believe the fight is already lost. What kind of man would I be if I saw all this coming and did nothing? Even in the face of futility, I think it’s our obligation to try. To at least force the conversation to happen properly - before the last window closes.

Thank you for the thoughtful and generous response. I agree with many of your points - in fact, I see most of them as plausible extensions or improvements on the core model. I’m not claiming Eden 2.0 is the most likely outcome, or even the most coherent one. I framed it as a deliberately constrained scenario: what if we survive - but only just, and only as tightly managed pets or biological insurance?

The purpose was less to map an optimal solution than to force the reader to confront a deeper discomfort: even if we do survive AGI, what kind of existence are we surviving into?

You’re absolutely right that a superintelligent AGI would likely consider more robust models - distributed human clusters, limited autonomy, cognitive variance experiments - and that total reproductive suppression or over-simplified happiness engineering might create more problems than they solve. I think those refinements actually strengthen the argument in some ways: they preserve us more effectively, but still reduce us to fragments of our former agency.

Your final question is the most important one - and exactly the one I was hoping to provoke: if we survive, will anything about us still be recognisably human? 

Thanks again. This is exactly the kind of engagement I hope for when I write these pieces.

The key difference is that SGD is not evolution - it’s a guided optimisation process. Evolution has no goal beyond survival and reproduction, while SGD explicitly optimises toward a defined function chosen by human designers. Yes, the search process is stochastic, but the selection criteria are rigidly defined in a way that natural selection is not.

The fact that current AI systems don’t act with strict efficiency is not evidence that AGI will behave irrationally - it’s just a reflection of their current limitations. If anything, their errors today are an argument for why they won’t develop morality by accident: their behaviour is driven entirely by the training data and reward signals they are given. When they improve, they will become better at pursuing those goals, not more human-like.

Yes, if AGI emerges from simply trying to create it for the sake of it, then it has no real objectives. If it emerges as a result of an AI tool that is being used to optimise something within a business, or as part of a government or military, then it will. I argue in my first essay that this is the real threat AGI poses: when developed in a competitive system, it will disregard safety and morality in order to get a competitive edge.

The crux of the issue is this: humans evolved morality as an unintended byproduct of thousands of competing pressures over millions of years. AGI, by contrast, will be shaped by a much narrower and more deliberate selection process. The randomness in training doesn’t mean AGI will stumble into morality - it just means it will be highly optimised for whatever function we define, whether that aligns with human values or not.

I appreciate your read and the engagement, thanks.

The issue with assuming AGI will develop morality the way humans did is that humans don’t act with strict logical efficiency - we are shaped by a chaotic evolutionary process, not a clean optimisation function. We don’t always prioritise survival, and often behave irrationally - see: the Darwin Awards.

But AGI is not a product of evolution - it’s designed to pursue a goal as efficiently as possible. Morality emerged in humans as a byproduct of messy, competing survival mechanisms, not because it was the most efficient way to achieve a single goal. An AGI, by contrast, will be ruthlessly efficient in whatever it’s designed to optimise.

Hoping that AGI develops morality despite its inefficiency - and gambling all of human existence on it - seems like a terrible wager to make.