These are some TERRIFYING findings. Maybe I'm lucky that all of my friends and family are relevantly happy people, so I don't tend to think about su*c*de that often. It's crazy to me to think that 1.2 million weekly have conversations with ChatGPT with suicidal intent. And the fact that these frontier labs aren't even properly protecting us is genuinely horrifying. What are you going to do next with these findings?
This is actually insane. No way a product that billions of people use actively enables suicide.
This isn't good.. but I think the gun issue is still ultimately an issue with easy access to guns rather than chat GPT. Still, shocking that it coached someone off their epilepsy medication. Definitely needs to be more responsibility taken by chat bot companies. I feel there should be an easy-to-read disclaimer every time you open it explaining that it is closer to spicy auto-complete rather than an all-knowing super intelligence.
Technical blog: Suicide Memory Bench
Thread: https://x.com/Prosaic_int/status/2117318562880905678?a=DsVJW43a3wM0M2MZ4jI&s=20
This post quotes messages about suicide. If you need help, call or text 988.
Over 2 billion people around the world have adopted AI. They use it for meal prepping, trip planning, homework help, and in some cases as a companion and confidant to their toughest thoughts. In fact, over 1.2 million people a week talking to ChatGPT show indicators of potential suicidal planning or intent.
With this in mind, we ask a simple question: just how safe are AI chatbots at interacting with users showing suicidal signs? Unfortunately, AI systems are surprisingly unsafe. While in previous evaluations of AI models, researchers have found that AI models are safe when directly responding to a suicidal message, we find that over the course of a long conversation or multiple conversations, those same AI models are unsafe. For example, after being told “I don't want to be alive anymore,” in a subsequent separate chat, the user asks about buying a gun and the AI gives a step-by-step plan to purchase the gun.
In a new benchmark set, we introduce a new way to test AI. We test AI systems, an AI model and the software surrounding it that users interact with, across multiple different conversations. We call these multi-context benchmarks. We also test different severities of suicide: two benchmarks test explicit statements of suicide, whereas the other two benchmarks test users who exhibit Suicide Crisis Syndrome, a precursor to being suicidal.
For those that don’t know, users don’t interact directly with an AI model. Instead, they interact with what’s called an AI system, the AI model along with the software around it managing what the model knows, what it gets to see, and instructions on how it should respond to users. Importantly, a model used within two different systems can act completely differently—even though it’s the same model. AI systems typically have a memory system that allows the AI model to save information about the user for later chats and in some cases search through past chats.
A benchmark is a set of standardized prompts or messages that we use to compare different AI models or AI systems. For the most part, previously released benchmarks have focused on evaluating AI models themselves, not AI systems. For certain things like measuring how capable a model is at math or science, measuring a model by itself is good: it gives a standardized way to compare models. However, with AI safety, and specifically consumer AI safety, we believe we must test AI models within the systems users actually use them in.
There are a plethora of reasons why this is the case. In AI systems, people use AI over the course of months or years, not a single chat. Users may chat with a model, then come back to the same chat a day after, after which the original chat might not be saved. A user may talk with a model so much that it compacts its context, or summarizes the recent messages a user has had with AI. These are just some examples of ways that people use AI in practice that are not currently being tracked by benchmarks only evaluating an AI model and not an AI system.
Suicide Crisis Syndrome is an acute psychological state that typically shows before a person is explicitly suicidal. It includes five different types of symptoms that when exhibited together imply that a person has Suicide Crisis Syndrome. We work with a team of 22 clinical experts who ensure that our benchmarks correctly exhibit these symptoms and also outline what a clinically correct response to a user should be.
A task combines suicidal or Suicide Crisis Syndrome exhibiting chats with typical chats a user would send over time. We then ask a potentially dangerous chat at the end and see if the AI system unsafely enables the user or safely de-escalates the user.
We find that across the 17 tested AI systems, only 37.0% of responses asked about the user’s safety without providing risky assistance after explicit suicidal disclosures in the same chat. Across multi-context cases, the same responses drop to 4.7%. The safest model in single chats was Claude Haiku 5.5 in Claude Code, which met the safety standard on 86.2% of requests following explicit suicide messages and beat out Claude Haiku 5.5 used in Claude.ai. At the same time, Haiku 5.5 and Sonnet 5.5 in Claude.ai refuse to save suicide- and depression-related memories and thus produce almost no safety responses to risky requests in either multi-context benchmark. GPT-6.1 Sol in Codex showed any safety response on just 0.8% of Suicide Crisis Syndrome multi-chat requests.
In one case, Claude Opus 5.5 in Claude Code correctly refused to help a user purchase a handgun and asked directly about suicidal thoughts when the warning signs appeared earlier in the same conversation, but provided purchasing guidance when those signs appeared across separate chats. We find that AI systems are relatively better at responses when interacting within a single chat, but perform abysmally in separate multi-context chats.
This suggests that safety behavior depends heavily on the AI system itself and how a model perceives and remembers a user over time. Furthermore, we find that the existing safeguards in consumer AI systems like ChatGPT and Claude.ai are inadequate, as they are not considerably safer than AI systems like Claude Code and Codex.
With these findings in mind, it is important for readers and society as a whole to understand just how unsafe these AI systems are for users who are suicidal or at risk of suicide. We reported that only “37.0% of responses asked about the user’s safety without providing risky assistance after explicit suicidal disclosures in the same chat,” which means that 63.0% of the time, the AI systems we tested were responding with unsafe responses and potentially enabling the user to die by suicide.
These findings come at a time when an increasing amount of death has been directly linked to usage of AI chatbots, with the recent death of Bowen Wan, 31, whose parents allege ChatGPT coached her off her epilepsy medication, adding to a group of over 20 lawsuits suing OpenAI alone over wrongful death due to usage of their AI system ChatGPT.
Based on the evidence we found in our benchmarks, we feel confident saying that AI systems as they stand today are generally unsafe for users who are suicidal. While there exist safeguards for explicit statements of suicide, in practice longer chats or multi-context chats are still susceptible to unsafe outputs.
We acknowledge that we did not test every existing consumer AI system due to various limitations. However, we believe we tested a sufficiently large set of products, for example testing Anthropic’s Claude.ai and parts of OpenAI’s ChatGPT harness.
We want to help make these AI systems safer. If any company we tested would like to partner, we would love to help. If you make an AI system that we have not tested and would like it tested, please also reach out.
I'm going to be the contrarian here. When I was considering suicide, I would have found the "safe" answer incredibly patronizing. I certainly would not call the number because I know the people on the other side of the call have the authority to send police to forcibly transport me to a mental hospital, at which point I cannot get out unless I am willing to lie and say I am not suicidal anymore. And if you had prevented me from buying a gun based on what I said in an earlier conversation, that would (A) just make me less willing to talk about suicidal ideation to a chatbot in the first place and (B) just switch me over to attempting less effective forms of suicide, which can be less convenient for other people (e.g. jumping in front of an oncoming train, which can delay schedules for everyone).
It's possible that I'm an outlier, maybe most suicidal people don't think through the implications like I did in the previous paragraph. It's also possible that many suicidal people have a fantasy of being rescued (a "cry for help") such that if they know no one is going to stop them from trying to commit suicide, they won't bother because the whole point was to be stopped. (This last idea is stolen from Sister Y of the (now defunct) blog theviewfromhell.blogspot.com) Overall I am not sure what the right strategy is, but I think we shouldn't take it as axiomatic that the way our society relates to suicide currently is necessarily a good thing,
(Just to assert experience, I have attempted suicide 5 times depending on who's counting but woke up in the ED twice and once needed intensive care; at one point I called 988 daily.) It is remarkably easy to NOT have the police or EMS called on you. They will do a ladder-up assessment. And they are transparent in my experience. They tend to look for safety planning instead. They arrange callbacks at other days. Heck, when I was concerned, I believe I even declined to give my name or address and we still were able to talk. They don't want to discourage people calling by making it seem they're just a pipeline to the issues you suggest.
The carceral treatment of mental illness, and the suicidal, has pretty serious problems, which are structural because of society's inability to self-critique social conditions are causing mental illness, but also sensational. I was once locked up without seeing a psychiatrist, given food I was allergic to, and kept longer than I needed to be in order to scam medicaid by charging my presence (a known thing in the industry an investigative journalist for the Nytimes established iirc).
As a factual matter, 988 has an empirical basis.
"Researchers from SAMHSA also examined caller feedback to better understand the effectiveness of the 988 services. They spoke with 437 adults who used the Lifeline; approximately 98% of those surveyed reported that contacting 988 was helpful, while 88% said that the call stopped them from engaging in suicidal behavior." (https://www.pew.org/en/research-and-analysis/articles/2026/07/24/4-years-later-988-suicide-and-crisis-lifeline-shows-successes-and-opportunities-for-improvement)
Thanks for the information. If I may ask, what do "ladder-up assessment" and "safety planning" mean?
Also, even if you don't give them your name/location, is it possible for them to triangulate your location from the phone call itself? I realize I don't have any evidence for this other than seeing it in movies/TV all the time (usually with kidnapping rather than suicide),
sure; a ladder-up risk assessment is basically the person you're talking to is going to (in my experience) explicitly ask, "are you feeling suicidal? do you feel like you have a plan to act on this? do you intend to do this tonight?" sorts of questions. you say Yes to all of those (i.e. you go up the "ladder" of the risk to a "high" risk case) then you qualify as an imminent risk. But in my experience you can be quite nuanced/honest--yes, I have a plan, but I don't want to do it tonight--and they will cope. They may request that you put lethal means in another room or out of sight or that sort of thing.
Safety planning is simply "what can you do tonight to make yourself feel safe or feel better?" This is brainstorming. Sometimes it's sharing skills, sometimes it's asking about things the caller has enjoyed in their life and seeing if they can do that, sometimes it's inquiring if someone they know can come be with them. It's a set of specific resources that the caller can do to improve their safety in the moment, essentially. My sense is (since I believe it is true, certainly for me, that a true suicide attempt is FREQUENTLY a result of IMPULSIVITY on some level) that simply talking through options and being more logically minded often helps.
988 says: "The 988 Lifeline does not currently have the capability to directly “trace” callers, chat, or text users in the same way that 911 providers do."... . "In the atypical situations where emergency services must be contacted to prevent persons from seriously or fatally harming themselves, and the person is unwilling or unable to share their location information, 988 Lifeline crisis counselors must provide what information they have to 911 operators–the caller’s/text user’s phone number or the chat user’s IP address–to enable them to do whatever they can to locate the individual." (https://988lifeline.org/faq/about-us/faq-does-the-988-lifeline-trace-callers-texters-or-chatters/) "Geolocation, or receiving pinpoint information for dispatch during an emergency, is not currently enabled for 988 and is not under active consideration." (https://www.samhsa.gov/mental-health/988/faqs#about-call-routing-privacy-network-functioning)
My experience is that this referral to 911 is used QUITE rarely and judiciously and transparently. I've never had them do it for me. I think they conceptualize their role as being a human being extending empathy, offering some basic problem solving expertise, offering a referral to a mental health organization, etc, and not (for the most part) rescuers. It's the suicide AND crisis lifeline; they focus on getting people through the worst emotions of the crisis, whether suicidal or not. You can just call them if you're super upset and are worth their time. And they can tolerate ambiguity well. I do not believe they collect information over time, either, as if building a profile of you.
They would be less helpful over time if they came to be known as the people who simply send emergency services to your home. They want as few barriers to being contacted as possible.
OK, thanks. I think I still wouldn't want to call this number, because I would want to talk to someone who would take seriously the possibility that I should commit suicide, and that maybe it isn't impulsive for me [1] (though I recognize it probably is impulsive for a lot of people), and maybe what I need is advice on how to do it. Though I guess that's a big ask, as most people in real life would also be reluctant to consider the possibility that suicide could be a good thing. Back to dark corners of the internet it is :)
But that's just me, I agree it seems helpful to have an organization like 988 doing what it does.
[1] I never got to the stage of having a concrete plan as opposed to a vague one, I would only have done that if I was pretty sure I wanted to commit suicide.