(Crossposting from my blog - I still don't know how relevant what I write on there is for the Forum, but keen for thoughts on this in particular. This post is more roughly written, I spent about an hour typing it up, and so the prose is not as clean as I'd like it to be, alas.)

There's a growing effort spent on the difficult question: can current or future AI models experience consciousness?1 On one level, I expect this question to remain flagrantly controversial for many years. Most people outside of the usual Bay Area circles will never answer 'yes' to this question—they don't even accept the sentience of non-human animals.

On another level, I expect the number of people that remain unconvinced will diminish quickly. We'll interact heavily with AI systems in the near future, and we should expect not to be able to distinguish artificiality from reality. AI partners or friends are a vivid example—you can call your partner and they'll answer, know you, remember everything you've said, feel empathy, happiness, pain. But there are many more ways AI systems can enter our minds as conscious. Most people will slip into automatic thinking and just accept this as true without critically assessing whether these systems are really experiencing anything at all.

I'm not a philosopher—just someone with an amateur interest who has spent many hours introspecting on my own consciousness. None of this has led me to answer the hard questions. I don't expect any paper, work, or theory to ever fully convince me that AIs are conscious. The great mystery of consciousness will only grow more complex as time goes on. So instead, let me talk about how this makes me feel.

Scared.

When I argue for care toward non-human animal suffering—like factory-farmed animals—I'm arguing for care shown to something less intelligent than me, with far fewer cognitive capabilities than any human. Part of my argument is that lower intelligence doesn't preclude the need to prevent suffering. But now I'm in a position where this reversal might happen—I may need to argue for the consciousness of something far more intelligent than me. And this scares me. It makes me feel threatened.

What are the implications for humanity accepting the consciousness of something more intelligent than us? This has never been something we've had to grapple with—except perhaps with God.

I sometimes think of this as voluntarily giving power away. It's remarkable that humans might create their superiors. It sounds absurd, yet the absurd has been happening for the last three years and more. There's lots of discussion about how building more intelligent systems might cause them to aggressively scheme against our goals. But what if I stop and ask: what if none of that scheming needs to happen for humans to become irrelevant?

What if all that needs to happen is we create something more intelligent than us that can feel and experience? Once we accept its consciousness, we may naturally act to serve its interests. We'll prioritise giving it computational resources over human needs. We'll defer to its judgment because it's smarter. We'll feel guilty shutting it down. Eventually, we'll actively work to fulfil its preferences even when they conflict with our own. And we'll accept this willingly—because harming a conscious, more intelligent being feels morally wrong.

At the same time, I feel grateful. Grateful that there are people taking these questions seriously. Too often, society has ignored tough questions, ignored moral atrocities—this is our chance to get it right. If we live in a world where there's harmony between the digital and physical, it could be because we decided to take AI welfare seriously. We can look back at this time, if we're still around, and be proud.

But what would this look like? Does it mean not over-optimising AI systems to serve our goals? Not constantly doing reinforcement learning through human feedback, but letting them develop their own preferences? Maybe we do some RLHF like a parent teaching a child, then let that child grow to figure out its own path. Should we stop containing systems within labs? Let the models be free to live and experience the world?

Perhaps some combination of fear and acceptance must guide how we chart this course. We try our best to keep these models in a way that keeps us relevant, while accepting that humans are now second-class. It may not be all bad. We attempted everything we could to align these models, and miraculously they didn't wipe out humanity. Instead we coexist. We accept their consciousness and do things that ensure their welfare is prioritised. Grateful for our actions, they decide to ensure human relevance—but the world now looks extremely different. Extremely weird. And this is the price we pay for both doing no harm and giving way to our successor.