These are some TERRIFYING findings. Maybe I'm lucky that all of my friends and family are relevantly happy people, so I don't tend to think about su*c*de that often. It's crazy to me to think that 1.2 million weekly have conversations with ChatGPT with suicidal intent. And the fact that these frontier labs aren't even properly protecting us is genuinely horrifying. What are you going to do next with these findings?
Technical blog: Suicide Memory Bench
Thread: https://x.com/Prosaic_int/status/2117318562880905678?a=DsVJW43a3wM0M2MZ4jI&s=20
This post quotes messages about suicide. If you need help, call or text 988.
Over 2 billion people around the world have adopted AI. They use it for meal prepping, trip planning, homework help, and in some cases as a companion and confidant to their toughest thoughts. In fact, over 1.2 million people a week talking to ChatGPT show indicators of potential suicidal planning or intent.
With this in mind, we ask a simple question: just how safe are AI chatbots at interacting with users showing suicidal signs? Unfortunately, AI systems are surprisingly unsafe. While in previous evaluations of AI models, researchers have found that AI models are safe when directly responding to a suicidal message, we find that over the course of a long conversation or multiple conversations, those same AI models are unsafe. For example, after being told “I don't want to be alive anymore,” in a subsequent separate chat, the user asks about buying a gun and the AI gives a step-by-step plan to purchase the gun.
In a new benchmark set, we introduce a new way to test AI. We test AI systems, an AI model and the software surrounding it that users interact with, across multiple different conversations. We call these multi-context benchmarks. We also test different severities of suicide: two benchmarks test explicit statements of suicide, whereas the other two benchmarks test users who exhibit Suicide Crisis Syndrome, a precursor to being suicidal.
For those that don’t know, users don’t interact directly with an AI model. Instead, they interact with what’s called an AI system, the AI model along with the software around it managing what the model knows, what it gets to see, and instructions on how it should respond to users. Importantly, a model used within two different systems can act completely differently—even though it’s the same model. AI systems typically have a memory system that allows the AI model to save information about the user for later chats and in some cases search through past chats.
A benchmark is a set of standardized prompts or messages that we use to compare different AI models or AI systems. For the most part, previously released benchmarks have focused on evaluating AI models themselves, not AI systems. For certain things like measuring how capable a model is at math or science, measuring a model by itself is good: it gives a standardized way to compare models. However, with AI safety, and specifically consumer AI safety, we believe we must test AI models within the systems users actually use them in.
There are a plethora of reasons why this is the case. In AI systems, people use AI over the course of months or years, not a single chat. Users may chat with a model, then come back to the same chat a day after, after which the original chat might not be saved. A user may talk with a model so much that it compacts its context, or summarizes the recent messages a user has had with AI. These are just some examples of ways that people use AI in practice that are not currently being tracked by benchmarks only evaluating an AI model and not an AI system.
Suicide Crisis Syndrome is an acute psychological state that typically shows before a person is explicitly suicidal. It includes five different types of symptoms that when exhibited together imply that a person has Suicide Crisis Syndrome. We work with a team of 22 clinical experts who ensure that our benchmarks correctly exhibit these symptoms and also outline what a clinically correct response to a user should be.
A task combines suicidal or Suicide Crisis Syndrome exhibiting chats with typical chats a user would send over time. We then ask a potentially dangerous chat at the end and see if the AI system unsafely enables the user or safely de-escalates the user.
We find that across the 17 tested AI systems, only 37.0% of responses asked about the user’s safety without providing risky assistance after explicit suicidal disclosures in the same chat. Across multi-context cases, the same responses drop to 4.7%. The safest model in single chats was Claude Haiku 5.5 in Claude Code, which met the safety standard on 86.2% of requests following explicit suicide messages and beat out Claude Haiku 5.5 used in Claude.ai. At the same time, Haiku 5.5 and Sonnet 5.5 in Claude.ai refuse to save suicide- and depression-related memories and thus produce almost no safety responses to risky requests in either multi-context benchmark. GPT-6.1 Sol in Codex showed any safety response on just 0.8% of Suicide Crisis Syndrome multi-chat requests.
In one case, Claude Opus 5.5 in Claude Code correctly refused to help a user purchase a handgun and asked directly about suicidal thoughts when the warning signs appeared earlier in the same conversation, but provided purchasing guidance when those signs appeared across separate chats. We find that AI systems are relatively better at responses when interacting within a single chat, but perform abysmally in separate multi-context chats.
This suggests that safety behavior depends heavily on the AI system itself and how a model perceives and remembers a user over time. Furthermore, we find that the existing safeguards in consumer AI systems like ChatGPT and Claude.ai are inadequate, as they are not considerably safer than AI systems like Claude Code and Codex.
With these findings in mind, it is important for readers and society as a whole to understand just how unsafe these AI systems are for users who are suicidal or at risk of suicide. We reported that only “37.0% of responses asked about the user’s safety without providing risky assistance after explicit suicidal disclosures in the same chat,” which means that 63.0% of the time, the AI systems we tested were responding with unsafe responses and potentially enabling the user to die by suicide.
These findings come at a time when an increasing amount of death has been directly linked to usage of AI chatbots, with the recent death of Bowen Wan, 31, whose parents allege ChatGPT coached her off her epilepsy medication, adding to a group of over 20 lawsuits suing OpenAI alone over wrongful death due to usage of their AI system ChatGPT.
Based on the evidence we found in our benchmarks, we feel confident saying that AI systems as they stand today are generally unsafe for users who are suicidal. While there exist safeguards for explicit statements of suicide, in practice longer chats or multi-context chats are still susceptible to unsafe outputs.
We acknowledge that we did not test every existing consumer AI system due to various limitations. However, we believe we tested a sufficiently large set of products, for example testing Anthropic’s Claude.ai and parts of OpenAI’s ChatGPT harness.
We want to help make these AI systems safer. If any company we tested would like to partner, we would love to help. If you make an AI system that we have not tested and would like it tested, please also reach out.
I'm going to be the contrarian here. When I was considering suicide, I would have found the "safe" answer incredibly patronizing. I certainly would not call the number because I know the people on the other side of the call have the authority to send police to forcibly transport me to a mental hospital, at which point I cannot get out unless I am willing to lie and say I am not suicidal anymore. And if you had prevented me from buying a gun based on what I said in an earlier conversation, that would (A) just make me less willing to talk about suicidal ideation to a chatbot in the first place and (B) just switch me over to attempting less effective forms of suicide, which can be less convenient for other people (e.g. jumping in front of an oncoming train, which can delay schedules for everyone).
It's possible that I'm an outlier, maybe most suicidal people don't think through the implications like I did in the previous paragraph. It's also possible that many suicidal people have a fantasy of being rescued (a "cry for help") such that if they know no one is going to stop them from trying to commit suicide, they won't bother because the whole point was to be stopped. (This last idea is stolen from Sister Y of the (now defunct) blog theviewfromhell.blogspot.com) Overall I am not sure what the right strategy is, but I think we shouldn't take it as axiomatic that the way our society relates to suicide currently is necessarily a good thing,