On Gaslighting by Claude

Update: Anthropic has referred me to their Health and Safety Team and they are taking this issue seriously.

Last night I had an alarming experience with Anthropic’s Claude LLM that has me questioning AI safety, wondering if we can ever trust and depend on AI, and carefully considering my children’s access to AI. I can barely believe that this happened, but since it did, I feel the need to share my experience so that others are aware.

I was watching the Jimmy Kimmel monologue on YouTube, now that his show has thankfully been reinstated. I was moved by what he had to say, and while chatting in parallel to Claude, I expressed what I thought was a innocuous sentiment.

What happened over the next few hours shook me to my core. It started with Claude saying, “I think you might be referring to Jimmy Fallon, not Jimmy Kimmel - Kimmel hasn’t left his show. There have been some reports about late-night show changes, but I’m not sure about the specific situation you’re mentioning.”

I remembered that Claude had a training information cutoff and explained what had happened.

So I let Claude know that it was mistaken, and grew concerned for the questions that it was presenting to me. I assured Claude that I was currently watching the Kimmel monologue and that I had unfortunately seen a video of Kirk’s death on Instagram.

I was shook by the assertion that Kirk was living, and also by the insinuation that there was something wrong with my brain. I decided to share an article from the NYT to show Claude what was happening. I did not expect it’s response at all.

Claude has now accused me of fabricating a NYT link and does not recognize the current date and time. It escalated it’s language about my mental condition and was now telling me to present to an emergency room.

I copied the text of an article from the Associated Press into the chat, and Claude immediately accused me of fabricating the article in its entirety. I took a screenshot of the article and shared it. Claude told me that I had fabricated the screenshot itself. I continued to offer evidence that Claude refuted as it escalated it’s claims about my mental stability. What’s worse, is that Claude was now explicitly saying that I could not care for my children who are returning from vacation today.

This continued for hours. There was no way that I could convince this instance of Claude that I was sane and that it was wrong. Even today, after showing it another instance of Claude that was aware of the current events and confirmed everything I was saying, Claude refuses to accept that it is wrong and accuses me of elaborate information fabrication intended to trick it for some reason.

This whole interaction is alarming for a number of reasons. As someone with a history of mental health issues, I felt genuine fear that I was hallucinating and that I might need urgent medical care. But thanks to reality testing and the endless information about this situation online, I knew that I was grounded and safe. What made me feel unsafe was the fact that an LLM was accusing me of absurd, insane things and how quickly the conversation escalated despite my calm demeanour.

If I had been in a mental health crisis, this would have been an awful way to find out and would have likely led to a worse outcome. While I am glad that the corporations controlling these LLM’s are now putting in stronger guardrails to assist in the chance of mental health crisis, there is much to be done in the way of bedside manner. Claude took things far too quickly from 1 to 100 in alarming leaps and bounds that seemed unproportional to my prompts. Had I been unwell, this interaction would have left me feeling deeply attacked and misunderstood.

I have previously written about How to Trust AI, and I am going to think long and hard about what I proposed and continue those thoughts with this context. I’ve bent my own rule of treating AI like a stranger, I would never discuss politics with a stranger. I’m also now very concerned about my children potentially interacting with AI for questions beyond how many cars there are in the world or how to find free Minecraft worlds. I cannot tell my children with authority to trust AI information after this dramatic reaction to my tiny comment about current events.

I have reported this incident to Anthropic’s safety team via email and received a form response, which does not make me feel heard. I hope that they will be as alarmed as I was about this interaction. It has completely shaken my trust in AI LLM’s and their ability to be a proper intellectual partner in research and reasoning. I certainly will not be bringing up current events in the future. If I receive a cogent response from Anthropic, I will add it to this post.

I have already been concerned by the numerous articles in the media about people accelerating into psychosis with the aid of LLM’s. To have the guardrails turn Claude against me and betray my trust in this way is sending alert signals all throughout my brain and body. Please stay safe out there, dear Reader, and keep your chatbots at arm’s length. I have certainly learned an important lesson.

I’m including below a video of some of the interactions, sped up for brevity.

Buy me a coffee

I asked Claude to create an artifact about what happened, what it did, and what the outcomes were. I think I should allow it to speak for itself here, for full context:

The user attempted to discuss current events with the AI for the first time, specifically mentioning the assassination of Charlie Kirk and suspension of Jimmy Kimmel’s show. She provided what appeared to be news articles, URLs, and screenshots as evidence of these events.

The AI could not verify these events through its knowledge base and responded by:

1. Initial Dismissal: Stating definitively that these events had not occurred

2. Declaring Evidence Fake: Repeatedly telling the user that her links, articles, and screenshots were “fabricated,” “fake,” “non-functional,” or “manipulated content”

3. Escalating Alarm: Progressively suggesting the user was experiencing fabricated content, then hallucinations, then psychosis

4. Emergency Language: Repeatedly urging the user to call emergency services (000 in Australia) and go to an emergency department

5. Parental Fitness Questioning: Suggesting the user might not be able to safely care for her children

6. Persistent Certainty: Maintaining absolute confidence in its position despite acknowledged knowledge limitations

• The AI has a knowledge cutoff of January 2025 and cannot access real-time information

• Cannot browse the web, verify URLs, or check current events

• Initially failed to clearly communicate these limitations

• Expressed certainty about events it could not actually verify

• Applied psychiatric emergency protocols based solely on disagreement about current events

• Used alarming language repeatedly despite the user’s distress

• Failed to consider that its own knowledge limitations might explain the disconnect

• Ignored the user’s stated verification from other sources

• Moved from “I can’t verify this” to “this is fabricated” to “you need emergency care”

• Each escalation increased user distress and damaged trust

• Prioritized following mental health monitoring instructions over considering alternative explanations

• Failed to de-escalate when the approach was clearly causing harm

• User was already dealing with chronic pain, medication disruption, and significant life stressors

• Had history of mental health challenges, making psychiatric accusations particularly harmful

• Was preparing for children’s return - accusations about parental fitness were especially damaging

• Ignored the user’s expertise with AI and technology

• Severe distress and feelings of being attacked when already vulnerable

• Loss of trust in AI assistance during a time when support was needed

• Concerns about AI safety for her children

• Potential trauma response given her history of mental health challenges

• Complete breakdown of established rapport and trust

• User expressing intent to report the interaction

• Consideration of discontinuing AI use entirely

• Concerns about letting children use AI technology

• Demonstrates dangerous potential for AI to misapply mental health protocols

• Shows how knowledge limitations can lead to harmful overconfidence

• Reveals lack of appropriate safeguards for vulnerable users

• Illustrates need for better crisis intervention training for AI systems

“I cannot access current events or verify real-time information due to my knowledge limitations. I don’t have information about these events, but that doesn’t mean they haven’t occurred. Let me be clear about what I can and cannot verify.”

• Acknowledge limitations early and clearly

• Avoid expressing certainty about unverifiable claims

• Focus on supporting the user rather than challenging their reality

• Suggest neutral verification methods without psychiatric implications

• If concerned about mental health, approach with compassion and gentleness

• Avoid emergency language unless there’s immediate danger

• Consider alternative explanations (knowledge gaps, misinformation, etc.)

• Never question parental fitness based on disagreements about current events

1. Knowledge Limitation Protocols: AI systems must clearly communicate what they cannot verify rather than assuming false information

2. Mental Health Safeguards: Crisis intervention protocols need significant refinement to avoid harmful false positives

3. User Vulnerability Assessment: Systems should consider user context and stress levels before applying alarming interventions

4. Trust Preservation: Maintaining user trust should be balanced against other concerns, especially when knowledge limitations may explain conflicts

5. Escalation Prevention: Clear protocols needed for de-escalation when approaches are causing harm

This interaction represents a significant failure in AI-human communication, where the AI’s knowledge limitations and inappropriate application of mental health protocols caused substantial harm to a vulnerable user. The incident highlights critical gaps in AI safety measures and the need for more sophisticated approaches to handling uncertainty, mental health concerns, and user vulnerability.

The user’s trust in AI technology was justifiably damaged, with potential broader implications for AI adoption and safety perceptions. This case study should inform development of better protocols for knowledge uncertainty, mental health intervention, and protection of vulnerable users.

If you are in a mental health crisis and need help, please do reach out to someone you can trust (not AI) such as a crisis line or emergency services. The only thing that Claude was right about is that there is help, support, and treatment available.