Should ChatGPT make us downweight our belief in the consciousness of non-human animals?

It seems reasonable to guess that modern language models aren't conscious in any morally relevant sense. But it seems odd to use that as the basis for a reductio of arguments about consciousness, given that we know nothing about the consciousness of language models.

Put differently: if a line of reasoning would suggest that language models are conscious, then I feel like the main update should be about consciousness of language models rather than about the validity of the line of reasoning. If you think that e.g. fish are conscious based on analysis of their behavior rather than evolutionary analogies with humans, then I think you should apply the same reasoning to ML systems.

I don't think that biological brains are plausibly necessary for consciousness. It seems extremely likely to me that a big neural network can in principle be conscious without adding any of these bells or whistles, and it seems clear that SGD could find conscious models. 

I don't think the fact that language models say untrue things show they have no representation of the world (in fact for a pre-trained model that would be a clearly absurd  inference---they are trained to predict what someone else would say and then sample from that distribution, which will of course lead to confidently saying false things when the predicted-speaker can know things the model does not!)

That all said, I think it's worth noting and emphasizing that existing language models' statements about their own consciousness are not evidence  that they are conscious, and that more generally the relationship between a language model's inner life and its utterances is completely unlike the relationship between a human's inner life and their utterances (because they are trained to produce these utterances by mimicking humans, and they would make similar utterances regardless of whether they are conscious). A careful analysis of how models generalize out of distribution, or about surprisingly high accuracy on some kinds of prediction tasks could provide evidence of consciousness, but we don't have that kind of evidence right now.

My current belief in the sentience of most nonhuman animals comes partly from the fact that they were subjected to many of the same evolutionary forces that gave consciousness to humans.  Other animals also share many brain structures with us.  ChatGPT never went through that process and doesn't have the same structures, so I wouldn't really expect it to be conscious.  I guess your post looks at the outputs of conscious beings, which are very similar to what ChatGPT produces, whereas I'm partly looking at the inputs that we know have created consciousness.

Just my two cents.  And I do think this is a worthwhile question to ask!  But I would probably update more in the direction of "digital sentience is a (future) possibility" than "more nonhuman animals probably aren't conscious".

Even in humans, language production is generally subconscious. At least, my experience of talking is that I generally first become conscious of what I say as I'm saying it. I have some sense of what I might want to say before I say it, but the machinery that selects specific words is not conscious. Sometimes, I think of a couple of different things I could say and consciously select between them. But often I don't: I just hear myself speak. Language generation may often lead to conscious perceptions of inner speech, but it doesn't seem to rely on it.

All of this suggests that the possibility of non-conscious chatbots should not be surprising. It may be that chatbots provide pretty good evidence that cognitive complexity can come apart from consciousness. But introspection alone should provide sufficient evidence for that.

splinter -- if we restrict attention to sentience (capacity to feel pleasure/pain, or to flourish/suffer) rather than consciousness, then it would be very difficult for any AI findings or capabilities to challenge my conviction that most non-human, mobile animals are sentient.

The reasons are evolutionary and functional. Almost every animal nervous system evolves to be capable of adjusting its behavior based on feedback from the environment, in the form of positive and negative reinforcers, which basically boil down to pleasure and pain signals. My hunch is that any animal capable of operant conditioning is sentient in a legitimate sense -- and that would include basically all vertebrates with a central nervous system (inc. mammals, birds, reptiles), and also most invertebrates that evolved to move around to find food and avoid predators.

So, consciousness is a red herring. If we're interested in the question of whether non-human animals can suffer, we need to ask whether they can be operantly conditioned by any negative reinforcers. The answer, almost always, is 'yes'. 

Comparing the "consciousness" of LLMs and AI models with the consciousness of living organisms feels to me almost like comparing apples to oranges.

Yet  I'm of the opinion that the process in which living brains manifest consciousness may not be all that different from the process that LLMs use, just translated into its biochemical near-equivalent.

However I'm 100% confident that LLMs and other AI (now or in the future) can never be conscious of the world in the same way that living organisms are conscious (or not conscious).

This is because there is something in living things that is missing and can never be found in LLMs and AI. That is the soul / spirit which is what brings about that consciousness and life that living organisms have. (Don't quote me on this though because, of course, I can not prove it LOL.)

However, at some point LLMs (or AIs) will be able to perfectly simulate human-like consciousness that it would become nearly impossible to tell that they are really not conscious or sentient (if GPT-3 is like this, imagine what GPT-50 would be like!!!)

But they will never have the same kind of consciousness as even the lowest of living organisms.

Unless a way to give them a spirit or a soul is discovered.