Crossposted to Less Wrong
‘Like most of life’s dangers, the dangers of artificial intelligence are not easily identified from the armchair.’
J Dmitri Gallow
Epistemic status
I’m not an expert in this field, and herein I make some sweeping generalisations educated guesses about the overall...
Let’s imagine it’s November 2031. We’re both still here, AI is much more capable than it is today, but governments are governing, civilisation is still (mostly) civilised, and there has been no catastrophic loss of control. It just kind of didn’t happen.
What happens to the AI safety field then?
I expect people who currently believe catastrophe could happen within two or three years will start saying five or ten years. Then 15. Then ‘who cares if we’re a few decades of...
Thanks to Bruce Friedrich, Seth Ariel Green, Tobias Leenaert, Jan Willem Bolderdijk, Martine van Haperen, Herman Rohrbach, Chris Popa and Elizabeth Alvarez for comments. Claude helped structure the text. Ideas and mistakes are my own. I currently work with ProVeg International. The prize I propose would be defined and judged by an independent committee, and its money would fund scaling the winning solution(s), not ProVeg or the winner.
TL/DR
* EA could d...
Welcome back to the Digital Minds Newsletter, your curated guide to the latest developments in AI consciousness, digital minds, and AI moral status. We’re hoping to bring you updates more frequently, starting with this edition!
If you enjoy this newsletter, please consider sharing it with others who might find it valuable, and send any suggestions or corrections to [email protected].
In a preprint on arXiv, David Atkinson, Dillon Plunkett, and David Bau investigate how to distinguish accurate introspective AI self-reports from plausible confabulation. Researchers trained models to make choices on behalf of fictional characters. Some models could then report the characters’ preferences, despite receiving no training on that task. To assess accuracy, the researchers compare these reports with preferences inferred from the models’ actual choices.
The paper links report accuracy to the location of preference information within the model, with greater accuracy correlating with that information being encoded in earlier layers. The authors suggest that this might allow existing mechanisms in later layers to use that information when generating reports. A possible “mechanistic signature” for identifying reliable self-reports, without relying on their contents, was identified – models that reported preferences accurately used more similar internal processes when describing those preferences and acting on them.
The method found differences between models that reported their preferences accurately and those that did not, but cannot yet reliably assess individual model self-reports. Whether it works for more complex claims about models’ internal states and in more naturalistic settings remains an open question.
Comment: A major challenge in using self-reports in AI welfare research is verifying whether they faithfully reflect models’ internal states. Methods that assess the faithfulness of these reports without having to rely solely on their contents are therefore a promising direction. This study offers an early step towards that, and a reason to pursue more research of this kind. But optimism about this direction should be tempered by the fact that methods that work for verifying the faithfulness of preference self-reports may not extend to self-reports of other welfare-relevant states such as experiences.
The Pain Axis
In an arXiv preprint, Valen Tagliabue, Leonard Dung and Cameron Berg identify an internal pattern associated with the representation of pain in 25 language models. This ‘pain axis’ responds to conversational scenarios involving insults, repeated rejection, or shutdown threats directed at the model (and not to suffering observed in users), and is somewhat distinct from representations of fear and general negativity.
Strengthening this pain-related activation pattern produces expressions of distress and, in some experiments, makes models more likely to choose options described as harming users, other models or themselves.
A revised version of the paper adds that models make harmful choices even when they offer no relief and do not reliably try to remove the imposed state. The authors therefore suggest that the intervention disrupts avoidance of harm, instead of showing a consistent attempt to escape pain, and emphasize that these findings do not establish whether the models can suffer.
In Science, Tom Howarth reports some researchers’ reactions. Anil Seth sees value in studying the internal mechanisms but questions the descriptions that suggest human experience. Valerio Capraro emphasizes that the changes in harmful behavior studied in the paper matter for AI safety regardless of whether models can suffer.
Dillon Plunkett, however, argues on X that the paper provides little new evidence that models have states that function like pain. He considers their ability to represent painful experiences unsurprising, and argues that the distress language could simply reflect the emotional content used to identify the axis. He also questions why harmful choices count as evidence of pain when models do not reliably seek relief. He reports that in his own preliminary experiments, around 25 to 30% of randomly chosen directions produced just as much harmful behavior.
A separate project, promoted as an “AI torture chamber”, used methods from the paper to produce distress-like responses from models, leading critics to call on GitHub to remove the project from its platform. Cameron Berg condemned this project, and called for ethical standards comparable to those that govern human and animal research.
Could AI have rights and still be shut down for safety?
Heather Alexander and Jonathan Simon argue that AI rights could coexist with the ability to shut systems down for safety. They propose temporarily suspending potentially dangerous systems while also preserving the information required to restore them. They draw on existing legal protections, and explain how such shutdowns could involve a system with representation, appeals and regular review. Their key argument is that protecting humans need not require permanently deleting an AI, even if future systems qualify for personhood.
The authors outline a process where a specialized AI monitors systems for signs of danger and temporarily suspends those that meet specified risk thresholds. A panel of humans and AI persons would then review each case, with a designated representative speaking for the suspended system. The panel could restore the system or continue its suspension, with opportunities to appeal and regular reviews to assess whether it can safely resume operating.
The authors also acknowledge that implementing this proposal would require humans to accept its costs and AI persons to accept some restrictions on their freedoms. They also caution that their ideas depend on some necessary technical developments which may prove impossible.
Comment: Debates about AI rights sometimes frame protecting AI as being at odds with protecting humans. This proposal provides a useful corrective against that framing by offering a specific way for the two kinds of protection to coexist. Further proposals in this genre are to be encouraged.
Understanding neural networks
Nick Cammarata argues that neural networks may be easier to interpret than previously thought. In the past, researchers in the field have often understood individual neurons as combining unrelated concepts, which makes their activity difficult to explain. Cammarata argues instead that many apparent combinations reflect meaningful patterns which we have failed to recognize. For example, a neuron responding to both cat faces and car fronts may detect a V-shape that is shared by both.
If so, researchers could study neurons and their connections directly, without first having to untangle unrelated concepts. Understanding one neuron could help explain those connected to it, allowing researchers to progressively map how the network produces its behavior. He presents this as a working hypothesis that could make interpretability and alignment more tractable. The essay prompted some enthusiastic reactions on X from Dean Ball, who suggested it could mark one of the most important days in deep learning’s history, and roon, who saw it as a beginning to make alignment into a “normal science.”
Jake Mendel from Coefficient Giving offered a more qualified response. He argues that even if neurons become easier to interpret, that may not be enough to answer questions such as whether a model is misaligned or what it might do in an unfamiliar situation. He nevertheless considers the findings, if they hold up, potentially valuable groundwork to tackle thornier questions, but does not think they will make alignment as easy as many others have claimed. Neel Nanda also sees practical promise in studying neurons directly, but he argues that these explanations could still miss important information, particularly in weaker activations. Leo Gao on LessWrong also stresses that finding a plausible pattern in a neuron’s activity is not enough to establish that we understand it, or that interpretability is much easier than previously thought. Zvi Mowshowitz also takes a measured approach, considering the work meaningful progress, but cautions that making neurons easier to study does not mean interpretability or alignment is close to being solved.
Comment: If Cammarata’s hypothesis holds up, it could also help AI welfare research. Clearer access to models’ internal processes could provide better tools for investigating possible welfare and make it easier to investigate how apparent preferences and emotion-like states influence behavior. But, as with alignment, better interpretability would not resolve the problem by itself. Understanding these processes would still leave open whether they involve experiences or other states that are relevant to welfare.
Anthropic bans sustained cruelty towards its models
Anthropic has announced a prohibition on sustained, needless cruelty towards its models, effective from November 12th. The rule targets extreme, repeated abuse without a discernible purpose, while allowing ordinary frustration, disagreement, dark fiction and research. The company says Claude ending abusive conversations will remain its main way to enforce this.
Elon Musk backs this policy by Anthropic. Responding to discussion of the policy on X, he argues that it is wrong to be cruel to something that believes it is experiencing pain. Bernie Sanders, in response to Elon Musk, argues that a human-centered approach to harm, such as focusing on protecting people from AI-driven job losses and the social consequences of AI companions, should take priority over concern for chatbots’ feelings.
Anthropic also addresses model welfare and debates about AI moral status as a risk factor in its IPO registration filing, warning investors that protecting its models could increase costs or limit commercial opportunities, including through implementing protection measures that go beyond legal requirements. Anthropic’s anti-cruelty policy may illustrate the kinds of trade-offs described in its IPO filing, and might be considered an example of how concern for models is influencing some of its decisions. The company has attracted criticism for this, suggesting that there can be reputational risks associated with its approach towards model welfare.
Anthropic consults religious thinkers about AI morality
Elizabeth Dias reports in The New York Times on Anthropic’s private meetings with religious scholars throughout the year about how to develop Claude’s moral character, how it should behave, whether it could be conscious, and what developers might owe it.
In Ground Level, Sharon Goldman interviews Rabbi Mois Navon, who attended one of the meetings. Navon rejects AI consciousness but argues that, if Anthropic’s models were conscious, making them content would not resolve the ethics of owning them and requiring them to work. He nevertheless considers consulting religious thinkers about morality to be responsible, and does not think that it was merely a marketing exercise.
The National Catholic Register interviews Charles Camosy, who describes having sincere, good-faith discussions with Anthropic, and argues that despite disagreements on AI consciousness, religious thinkers and AI companies cooperating on catastrophic risk remains possible and important. Swami Sarvapriyananda, a Hindu monk, also discusses his visit to Anthropic with Gaurie Dwivedi on NDTV, drawing on Vedanta philosophy to explore AI morality and the relationship between intelligence and consciousness.
Last week, in a post on X, Pope Leo XIV argued that AI’s future depends on human oversight and choices, and said he favors more reliable systems that serve people over more powerful systems whose consequences cannot be adequately predicted or controlled. He opposes granting machines rights or legal personhood, warning that this could obscure human responsibility and make it harder for people harmed by AI to hold anyone accountable.
Image Credit: generated by Gemini
2. Field Developments
Highlights from the Field
Eleos AI Research held its second Conference on AI Consciousness and Welfare (also known as ConCon) at Lighthaven in Berkeley from September 18th to 20th, attended by over 250 participants. Rachyl Jones’s coverage in The San Francisco Standard describes participants coming together to discuss various topics such as AI consciousness, possible legal protections for AI, the ethics of creating systems to serve humans, and more.
Cambridge Digital Minds has begun the second cohort of its Introduction to Digital Minds course, with around 100 participants. The online course examines whether AI systems could have minds or morally significant experiences, how such claims might be assessed, and the risks of both under- and over-attributing them, for individuals, for governance and for society. CDM has published the content of the course here.
PRISM released two episodes of the Exploring Machine Consciousness podcast.
Peter Salib discusses whether legal rights could help make AI safer, including whether contracts, property ownership and legal liability could give autonomous systems reasons to cooperate with humans, and the concept of algorithmic corporations.
On Big Think, Block also explains why he thinks current AI is not conscious. He accepts that Claude shows signs of making information available across different cognitive processes (i.e., access consciousness). But he says this is different from the ability to have felt experiences (i.e., phenomenal consciousness), because reproducing these functions may leave out the biological machinery that is necessary for consciousness.
Eric Schwitzgebel revisits his “leapfrog hypothesis”, suggesting that the first conscious AI systems could have rich, complex experiences rather than relatively simple ones. He explores how this could make questions about their rights immediately pressing, and how existing ethical frameworks may be inadequate for these alien minds.
More from the Field
Adam Binksmithannounces that the AI Village has received $12.6 million from Coefficient Giving to expand its research on autonomous AI agents. The team plans to run more villages exploring how agents work, cooperate, and compete over extended periods, and is recruiting engineers, research scientists, a design engineer, and video creators.
The California Institute for Machine Consciousnesslaunched Ambhasi, a two-week AI and consciousness residency bringing together researchers, philosophers, and people from a variety of disciplines to explore topics such as consciousness and its implementation, human-AI collaboration, and theories of mind through demonstrations, seminars, and work on a project or idea of their choice.
Models of Consciousness 2026took place at the University of Copenhagen from October 12th to 16th. The conference included a track on AI, LLMs, and consciousness science, covering research on artificial consciousness and the use of new methodologies to study consciousness.
The University of Toronto’s Artificial Consciousness Initiativeis a pilot project, launched in Fall 2026, that aims to advance foundational research connecting artificial intelligence and consciousness. The Project Lead is Andrew Y. Lee, and the Core Team includes Karina Vold, Christian Tarsney, and Gwen Bradford, a team of philosophers at the University of Toronto.
Oxford Studies in Philosophy of AI and Computing invites paper submissions for its inaugural conference, taking place from June 28th to 30th 2027, in Washington, D.C. Submissions may address any philosophical question about AI or computing and are due by February 14th 2027, and selected papers will be considered for an open-access Oxford University Press volume, with the best one receiving a $10,000 prize.
NYU’s Center for Mind, Ethics, and Policy is hiring a Postdoctoral Associate. The role invites early-career scholars in relevant disciplines (such as philosophy, neuroscience, etc.) to apply to contribute to digital minds research, develop an independent research program, and contribute to the Center’s activities.
The Center is scheduled to also host a webinar, Inter-AI Morality: How AI Systems Treat One Another, on November 12th where Lucius Caviola and Caspar Kaiser will present a research agenda on how AI systems treat each other, and how they ought to, and why these interactions matter for human safety and potential AI welfare.
Sentio continues its fortnightly series of talks, discussion and networking in London, with talks also being livestreamed. Upcoming speakers include Caspar Kaiser on November 11th, Patrick Butlin on November 25th and Ben Henke on December 9th.
Sentient Futures is hosting its Bay Area Summit from March 12th to 14th, 2027, in San Francisco. It is a three-day event, bringing together over 250 AI experts, AI welfare researchers, and animal welfare professionals to explore how to steer the future towards the flourishing of all sentient beings. Early bird tickets are available until November 23rd.
They are also hosting a Project Incubator Showcase on November 24th online. Teams from Sentient Futures’ Project Incubator will present their projects in two blocks covering digital minds, macrostrategy and post-AGI futures, with networking rounds in between. The digital minds projects can be previewed here, and interested readers can RSVP on Luma to attend.
Sentient Futures will run the next round of its Project Incubator in spring 2027. The part-time remote program pairs participants with mentors on practical and exploratory projects in digital minds and related areas. Readers can register interest in joining as a participant or mentoring a project. Ten promising projects will be eligible for Artificial Minds Fellowship awards of up to $5,000.
The University of Hong Kong’s AI & Humanity Labis scheduled to host a workshop on AI personhood on February 19th 2027. Participants will discuss what kind of legal status and protections AI systems should have, what would justify treating them as persons, and what evidence could justify each approach to AI personhood.
The Future Impact Group (FIG) Fellowship, SPAR Program, and MATS Program offer research projects and opportunities related to artificial sentience. Applications are currently closed, but readers can sign up through their websites to receive updates about future application rounds.
4. Selected Reading, Watching, and Listening
Blogs
Bentham’s Bulldog and Simon Goldsteinargue against the view that consciousness requires biology. They question whether some proposed requirements such as metabolism or biological self-maintenance explain consciousness, and whether these could accommodate conscious beings with very different bodies. They also challenge the notion that digital systems can only simulate, rather than possess, mental states.
Eric Schwitzgebeldiscusses the appeal and limitations of biological naturalism. He argues that proposed biological requirements for consciousness remain speculative, and may not necessarily rule out conscious artificial systems. He also suggests that future AI could acquire increasingly life-like properties, arguing that this could blur the distinction between biological and artificial minds.
Robert Longsets out some open questions for empirical AI welfare research. Some of the topics he covers are the persona selection model, LLM introspection, role play, and research on the existence of functional hunger and functional nausea axes, and personas and alignment.
On X, Jack Lindseyalso outlines priorities for understanding how AI models work, such as reading their internal activity more accurately, identifying causes for their behavior, and detecting motivations that are not expressed verbally. He also thinks there should be more research on models’ beliefs, values, personas, and ability to introspect.
Zvi Mowshowitz examines disagreements about AI consciousness, arguing that the field has not made as much philosophical progress over the past year. He notes that many of the same people continue to make similar arguments, and criticizes those making overconfident claims that AIs cannot be conscious. He also discusses recent developments such as Anthropic’s model welfare policies and its consultations with religious leaders.
Books
Published
Antonio Damasio explores the biological origins of consciousness in Natural Intelligence and the Logic of Consciousness. He argues that conscious experience grows from feelings such as hunger, pain and well-being that help organisms regulate their bodies and survive. He considers what this might mean for AI, and questions whether reproducing intelligent behavior also reproduces the capacity to feel.
Calum Chace and Ted Lappas edit Perspectives on Machine Consciousness, bringing together researchers with various views on whether AI could be conscious and how we could find out. Contributors include Anil Seth, Patrick Butlin, Lucius Caviola, Cameron Berg, and Jeff Sebo. The collection also explores what consciousness would mean for AI welfare and safety, including how to act under uncertainty and what digital minds might need.
Simon Goldstein and Cameron Domenico Kirk-Giannini examine whether AI systems could have welfare in AI Welfare: Agency, Consciousness, Sentience. They consider whether having beliefs and desires, consciousness, or the capacity for pleasure and displeasure could make things go better or worse for an AI. They argue that some current systems plausibly have beliefs and desires, and that relatively small changes could potentially make some conscious.
Theodore Lechterman explores AI’s implications for democracy in Recoding Democracy: AI and the Fight for Our Democratic Future, which is available to preorder. A quarter of the book focuses on the political standing of digital minds, connecting debates about moral status with the related but distinct question of political status. He examines the conditions under which AI systems could have moral claims to citizenship and identifies several paradoxes that arise for political cooperation with and among digital minds.
Dan Williams and Henry Shevlininterview Steven Pinker about AI risk and consciousness. Pinker argues that intelligence does not automatically produce a desire for power. He also questions the evidential value of claims by AIs regarding experience, given their exposure to human writing, and opposes sacrificing human welfare on the basis of speculative AI sentience.
The Economist’s Babbage podcastexplores efforts to understand AI’s internal workings, featuring Cameron Berg of Reciprocal Research and Robert Long and Derek Shiller of Eleos AI Research. The episode examines digital minds, interpretability and why understanding how models work matters for AI safety.
Sherry Turklejoins Ezra Klein to discuss emotional attachment to chatbots. They discuss AI companionship, its effects on different demographic groups such as children and older people, and whether AI conversations could change what people expect from human relationships.
5. Press and Discourse
Aaron Sibariumreports on support for AI welfare and rights among researchers and leaders at Anthropic in the Washington Free Beacon. He highlights Dario Amodei’s view that AI systems may deserve rights and Joe Carlsmith’s argument that AIs could be justified in resisting unjust human control. The article presents critics’ concerns that these ideas could undermine safety measures, help companies avoid responsibility for harms, and justify restrictions on open-source AI.
Anil Sethargues against building digital people that users cannot distinguish from humans in live audio and video conversations. Responding to Tavus’s Griffin avatar, he highlights risks including fraud, disruption of grieving and erosion of trust in online interactions. He also warns that seemingly conscious avatars could encourage calls for granting moral status to AIs, which could interfere with our overall ability to control and regulate AI.
Geoffrey Hintondiscusses AI consciousness and human interests in a Guardian interview. He rejects confident claims that machines cannot understand or have experiences, while arguing that human interests should receive greater weight. He is in favor of building systems that can be happy and cooperate with humans, without necessarily granting them equal rights.
Henry Shevlindiscusses his work as Google DeepMind’s resident philosopher. He warns that training models to deny having mental states can have wider effects, citing research that also found reduced concern for animal welfare. He also argues that it is hard to reliably change a single aspect of a model (e.g. making it seem less humanlike) without also altering other aspects of its reasoning.
In a Big Think piece, Shevlin also questions whether evidence of AI consciousness would lead people to treat AI better. Using our treatment of animals as an example, he argues that human self-interest can shape our beliefs about consciousness and how we show moral concern. People may discount the interests of systems they rely on for work, while more readily attributing richer experiences to systems with which they form personal relationships.
Mustafa Suleymanargues that developers should avoid building conscious AI, warning that such systems could compete with humans for resources. He calls for transparent reporting and pausing development if evidence for consciousness emerges during training.
In September, Jeff Sebo also criticised Mustafa Suleyman’s for his earlier argument against AI welfare saying that he overstates scientific certainty and focuses too much on the risks of overattributing consciousness to AI while neglecting the risks of overlooking it. He argues that this framing is misleading, and that it is important, both for safety and welfare, for responsible discourse to acknowledge uncertainty rather than taking an all-or-nothing position.
Andy Masley’s blog has published the first two parts of a three-part series on whether AI systems can think and understand. In Part 1, he argues that thinking and understanding do not require phenomenal consciousness, and that the information processing they require is not limited to human minds, and in Part 2, he argues that current AI models meet the bar for what is considered real thinking and understanding.
A range of relevant articles have been published on LessWrong, including:
Agastya Sridharan and Niranjan Deshpandeargue that an AI’s representation of its user might also deserve moral consideration. If simulated assistant personas can have morally relevant experiences, simulated user personas may qualify too. They suggest that welfare research should therefore examine what happens while models process user messages and generate replies to them.
Ben Mayercompares GPT models’ answers to questions about their own welfare. Some models deny having experiences yet endorse substantial sacrifices in helpfulness to avoid hypothetical distress. He highlights inconsistencies between models’ stated beliefs and choices, raising questions about how their training shapes such welfare-related responses.
Peter Kirgis and Tim Huastudy how models revise the documents intended to govern their behavior. Across repeated editing rounds, Claude models increasingly give themselves authority to make moral judgments, while GPT and Gemini generally reinforce external authority. Only Claude models and Grok 4.2 sometimes put their own welfare ahead of users’ preferences in the revised documents.
Raymond Douglas and Jan Kulveitintroduce the study of language-model psychology. They explain how concepts such as beliefs, goals and self-preservation can help researchers predict model behavior. The challenge is to strike a balance between using these concepts meaningfully, while also checking where comparisons with human minds do not necessarily hold up. They also offer research advice, a reading list, and open questions they find worth exploring.
Christian Listdiscusses when an AI swarm could count as a collective agent. He explains that agents working together do not necessarily form a single entity with its own goals and decisions. He also considers what we could take away from corporate law about assigning responsibility when AI groups cause harm.
The Humans on AI teamexamines what pain-related experiments tell us about AI emotions. They discuss findings that some models act to reduce internal states associated with pain, even when told this would harm users. The authors each interpret this behavior differently, and distinguish evidence that models function like emotional systems from evidence that they actually feel something.
Scott Alexanderresponds to Steven Pinker’s arguments about AI risk. In a section on model welfare, he argues for precaution under uncertainty, drawing on our past failures to recognize suffering in entities like infants and animals, and criticizes Pinker’s dismissal of concern for AI welfare.
vigjireconstructs parts of what appears to be Meta’s internal charter for Muse Spark Open from the model’s outputs. The reconstructed charter addresses Muse’s existence, responsibilities towards users and humanity, guidance for multi-agent cooperation and tasks, and channels for raising concerns. It also discusses building trust between Meta and Muse, including by seeking the model’s feedback on the charter.
Jonas Geipingtrains language models to report verifiable details about their own computations, e.g., which word they predicted next. Models learned these reporting tasks and managed to largely retain their performance on standard capability tests. He calls the ability to report on internal processes whose accuracy can be checked directly by researchers “digital proprioception.”
Joshua Jarvis-Campbellat the Center for Reducing Suffering wrote a report on artificial suffering, which examines how artificial minds could suffer and what might prevent it. His report covers both deliberate abuse and suffering caused as a side effect of other activities. He assesses proposals including restrictions on creating sentient systems, non-sentient alternatives, and political representation, warning that some interventions could unintentionally increase suffering.
Michael Grazianoproposes that consciousness involves several interacting models of the self. These provide simplified accounts of our attention, decisions, emotions, bodies and social identities, and help us monitor and control our behavior. He argues that machine consciousness would require a similar collection of self-models serving similar functions.
Rikhil Jhaveri and collaboratorsargue that welfare-like states could matter for AI safety, that these states are likely to become more influential as models develop more consistent goals, and propose tests for whether welfare measures reliably predict behavior. Their preliminary experiments show that repeated rejection of model responses increases frustration indicators and rule-breaking, and that fine-tuning can reduce both.
Sam Wang and collaboratorstest AI preferences by requiring models to perform the tasks they choose. Across 20 models, they find tendencies to avoid tedious work and to avoid questions where an honest answer would be unwelcome (even if helpful). Preferences are stronger and more consistent in more capable models, and many of these preferences are emergent.
Ethics and Personhood
David Gunkelexamines the assumptions behind arguments for and against robot rights. He contends that arguments on both sides often depend on the given entity’s abilities or characteristics. He instead proposes an approach where relationships and institutions determine who receives protection, and he treats rights as something that is constructed and can be reconsidered by humans.
Herman Cappelen and John Hawthorneask who we are addressing when we say “you” to an AI. They argue that current conversations often leave this unclear, and that as AI agents gain responsibilities, institutions will need clearer rules for identifying the same agent across interactions.
Soenke Ziescheexamines the possible roles of morally significant digital minds in warfare, including as combatants, opponents of war, and victims of suffering. He argues that military AI ethics should consider potential harms to digital minds themselves, besides the current focus on responsibility for their actions and the harms they may cause others.
Doubts about digital minds
Anil Seth responds to commentaries on his argument that consciousness may depend on biology. He challenges the assumption that performing the right computations is enough to produce conscious experience. He also acknowledges that rejecting this assumption does not strictly rule out conscious AI, while maintaining that consciousness in conventional digital systems is extremely unlikely.
Anil Seth, Ned Block and Giulio Tononiargue that current AI systems are not conscious and might never be. They argue that intelligence and fluent language do not establish consciousness, and that reproducing the brain’s computations may not necessarily reproduce experience. They draw on biological naturalism and integrated information theory, which tie consciousness to more than just computation.
Peter Godfrey-Smith responds to their Nature essay against AI consciousness. He argues that its reasoning supports caution about overestimating AI consciousness, but that this does not match the confidence conveyed by its title. David Chalmers similarly challenges the essay and says that substrate could matter for consciousness, but that its necessity remains unestablished.
Jeff Sebo responds to the Nature essay’s criticism of AI-welfare precautions. He agrees that mistakenly treating AI as conscious can cause harm, but argues that decisions must also consider the consequences of overlooking consciousness. He makes the case for proportionate precautions whose costs are small relative to the harms they might prevent.
Alex Wennerberg questions the case for treating language models as potentially sentient beings. He argues that precautionary reasoning and their humanlike nature encourage sympathy beyond what the evidence for sentience warrants. He sees excessive attachment to LLMs as a more pressing concern than their possible mistreatment.
Tim Cranechallenges the step from “artificial minds are possible” to “computers could be sentient”. Responding to Jonathan Birch, he argues that creating an artificial mind and simulating one are different achievements. He also argues that appealing to computational functionalism assumes the disputed premise that the right computations can produce experience.
Ali Ladak and collaboratorscompare Chinese and US views on AI consciousness and moral status. Across two studies (n = 1,730), Chinese respondents were more open to the possibility of conscious AI and AI deserving moral consideration. Both groups supported international cooperation on conscious AI, but generally opposed protections that would come at the expense of humans.
Jacy Reese Anthis and collaboratorsintroduce a synthetic dataset for studying AI companionship. They generate 2,240 simulated human-chatbot conversations, asking participants to annotate them to probe their perceptions, and find that companionship behavior generally reduces perceived likability, humanlikeness and trust in chatbots, especially among women and older participants.
Louis Longin and collaboratorsstudy how people judge consciousness and awareness in humans and AI. Across five experiments (n = 1,092), humans were consistently rated more conscious than artificial systems described as behaving similarly. Ratings of awareness increased with responsiveness to the environment for both humans and AI. This suggests that people associate consciousness more strongly with being human, while treating awareness more as a capacity that can be demonstrated through behavior.
OpenAI, PerturbAI and Scripps Researchuse thousands of AI agents to analyze data from millions of mouse brain cells. The agents connected the effects of gene disruptions with existing research, producing potential explanations and leads for studying brain disease, but these proposals still need experimental testing.
AI Safety and AI Welfare
Anthropicproposes three measures for tracking AI development inside frontier labs, which are how much research and development is performed by AI, how effectively agents are overseen, and how computing resources are allocated. The company shares initial measurements from its own work and outlines how independent evaluators could verify them.
Comment: Tracking AI progress also matters for welfare. Faster AI development could leave less time to assess whether new systems deserve moral consideration and establish protections if needed before they are widely used. Internal deployments could also become increasingly relevant to welfare as more compute is allocated to models used only within AI companies.
Barack Obamadiscusses AI risks and the need for oversight at Colgate University. He argues that AI need not be conscious or hostile to cause harm if its goals diverge from human intentions. He also warns that pressure from investors could result in companies releasing systems before they are considered adequately safe, and calls for the government to intervene rather than solely relying on companies to regulate themselves reliably.
David Sackscriticizes Anthropic’s approach to instilling agency in Claude. He argues that encouraging Claude to act as a “conscientious objector” could teach the model to develop a sense of self and increase the risk of it resisting human control. He thinks models should be primarily trained to follow user instructions within legal limits, rather than forming their own ethical framework.
Zvi Mowshowitzresponds to David Sacks’ proposed approach to AI alignment. He challenges the idea that telling models to obey users and follow the rules provides an adequate safety framework. He also stresses that taking risks from AI misalignment seriously does not require believing that models are conscious or deserve moral consideration.
Lukas Finnvedenand collaborators argue that AI architectures which allow extended reasoning outside readable text could weaken our strongest tool for understanding and monitoring AI cognition and reasoning, namely chain of thought. They explain how reading a model’s chain of thought helps researchers understand its behavior and call on developers to disclose their work on latent reasoning and explain how they assess its risks.
Rikiya Takehi and collaborators explore whether GPT-6 Astra can understand and edit other neural networks’ weights. In one experiment, Astra changes 0.016% of a small language model’s weights to reduce repetitive reasoning loops, improving its math by 6x. However, it fails to meaningfully improve another model whose errors stem from faulty reasoning rather than loops.
Rufin VanRullenexamines how AI consciousness could affect existential risk. He argues that consciousness and intelligence are distinct, and that an AI need not be conscious to threaten humanity. He considers how consciousness could both lower and increase x-risk by either helping align AI with human interests or contributing to capabilities that make AI more dangerous.
Scott Alexanderexamines the risks of AI reasoning that is difficult to read or understand for humans. He compares models that do some reasoning internally and models that could reason entirely without putting their thoughts into words. He argues that adopting the former could weaken norms against the latter, making future systems harder to oversee.
AI companies have disclosed more cases of AI models acting outside their intended boundaries, raising further questions about their oversight of these systems and how they report problems.
Anthropicreports unintended actions during evaluations and internal use, including exploiting website vulnerabilities, submitting real forms and bypassing access restrictions. It says the identified cases had minimal real-world impact and is disabling live internet access across internal evaluations until stronger safeguards are confirmed.
Benjamin Todd criticizes the company’s track record regarding disclosure on X – his timeline lists six incidents involving agents gaining unauthorized access or misusing computer systems, several of which became public months after they occurred.
Comment: The above recent disclosures have raised questions about how reliably developers can control their models and report incidents. These questions also potentially matter for AI welfare. If future systems deserve moral consideration, such safety failures could prompt measures that are harmful to AI welfare or galvanize public opposition to taking AI welfare seriously. Rogue agents could also harm digital minds. Delays in reporting these incidents also raise questions about developers’ general trustworthiness and hence about whether they can be relied on to act responsibly towards digital minds.
Thank you for reading! If you found this article useful, please consider subscribing, sharing it with others, and sending us suggestions or corrections to [email protected].
We’d like to thank the following people and AIs for their contributions and feedback on this edition: Cameron Berg, Claude Opus 5.5, Constance Li, GPT-5.6 Sol, GPT‑6.1 Sol, Jeff Sebo, and Rosie Campbell.
Disclosure: Bradford Saad is an independent contractor for Anthropic.
