CAI Currents is a series exploring the latest news on tech and mental health from a psychoanalytic perspective. It is published by TAP in partnership with the American Psychoanalytic Association President’s Commission on Artificial Intelligence.
Cyrano dot AI
Valerie Frankfeldt, PhD, and Todd Essig, PhDCaution: Training (NOT) In Progress
Molly Pachan, PhD
The AI Scribe Note Is Not Neutral
Heather deCastro, LCSW
Valerie Frankfeldt, PhD, and Todd Essig, PhD
Unlike Cyrano, who hid his love for Roxane inside someone else’s identity, AI doesn’t give a damn about your intimate when you use it to write your words of love and care. But the ending points in the same tragic direction: unrequited love, loneliness, loss, and mourning. Just get that Hollywood happy Steve Martin feel-good ending out of your mind.
One patient uploaded a text fight so ChatGPT could write a compelling apology. But their lover knew it was AI-generated eloquence, not something from the heart. Now, they’re not even talking.
Another patient tried to console a grieving friend. Rather than send their own words from a soulful place of struggle from having their friend’s pain in mind, they sent Claude’s eloquent, smooth but soulless response. Their friend responded with rage at being abandoned at such a vulnerable, fragile time. They needed their friend, not a well-appointed ringer.
Making AI a third in intimate relationships is not just something in clinical practice. Ellie Zolfagharifard, writing in Wired, plumbed the depths of AI as an often unwanted third. She even included a recent survey of UK adults where 11 percent admitted they use AI to rewrite or edit private messages.
Some take the dangers of AI writing even further. At the extreme, one finds people like Bret Stephens arguing in a New York Times Op-Ed, “Don’t use artificial intelligence to help you write. Never let A.I. do your writing for you.” But in this you may hear the baby going down the drain with the bathwater.
AI can be useful in writing when one is producing text to serve a specific function, like an appeal letter to an insurance company. It can even be useful as an interactive self-help tool. For example, it can help regulate affect and help you find your authentic response in difficult moments, such as dealing with a text fight with your partner or consoling a grieving friend. You might also find islands of insight and prompts for actual reflection according to APA’s Guide to Navigating AI-Generated Advice.
But substituting for interpersonal intimacy, using it as a stand-in for one’s heart and soul? Even in the best case, a feel-good Steve Martin ending, it would be the AI in relationship with your intimate, not you.
Molly Pachan, PhD
AI slop is somewhere between merely annoying and an attack on human thought and experience. But when AIs are being trained to respond safely to the risks of self-harm, the same training methods resulting in slop-producing LLMs creates clearly unacceptable dangers.
A recent Stanford study examined whether the standard approach to training AI works when training for safety in response to self-harm risk. The study explored whether reinforcement learning through human feedback (RLHF) can work to fine-tune LLMs to respond safely. Three board-certified psychiatrists independently rated chatbot responses. The result? Surprisingly poor agreement among the experts about how to respond to indications of risk.
Were two out of three experts wrong? Far from it. They each had their own, viable perspective on what to do. Multiple right answers exist as do many clearly wrong ones, something every clinician knows. Expert clinicians respond to acute risk of suicide with nuance and sensitivity to context, always keenly attuned to the patient’s state of mind. But the very qualities that make an expert clinician expert are lost by averaging across multiple perspectives.
LLMs are pattern prediction machines, representing language as vectors, encoding high-dimensional relationships between those vectors, and then fine-tuning those relationships by human feedback. The Stanford study illustrates that the way models typically improve—humans scoring LLM output—depends on rater consensus. There is an average response around which rater responses cluster. But the average of unique psychiatrist responses would be a fake average that would only further confuse the LLM.
For AI models to become better at addressing high-stakes, low-base-rate events like suicide risk, the Stanford research suggests that entirely new methods of training may be necessary for critical cases where human feedback does not cluster around an average response, as is the case with expert clinicians.
Heather deCastro, LCSW
How to think about health care providers using AI scribes for record keeping: augmentation or replacement; scaffold or substitute?
On one side, clinician judgment shapes the record. The AI scribe just helps. On the other, it shapes what gets noticed, recorded, and acted on. One side is a clerical tool, the other a medical device needing safety review and possible government regulation. Two countries have decided some scribes are indeed medical devices: Australia and Sweden. Two countries have not, or at least not yet: the UK and the USA.
Over 40 percent of Australian doctors already use AI scribes to create a medical record, according to Digital Rights Watch. Australia’s regulator has weighed in: Vendors’ tools had quietly started shaping clinical decisions instead of just documenting them; they called it “scope creep” and moved toward regulation, Medical Republic reports.
Sweden ran the same test on a different vendor and reached the same conclusion. Its regulator inspected a scribe directly and told the company its safety review wasn’t strict enough for what the tool was doing.
The UK drew the line differently. Its regulator ruled that a scribe which only transcribes and summarizes needs no review; the clinician checking the note is the safeguard.
The US hasn’t ruled on AI scribes yet but has previously employed a wait-and-see regulatory pattern. As it did with the WHOOP blood pressure monitor, it waits for a product to be on the market rather than preemptively regulating. In this example, they dropped the enforcement case a year later once the company changed the product.
These are important developments to watch. The promise of a good AI scribe scaffold, or even a safe, effective, and well-regulated substitute, is significant. AI scribes can give clinicians time to authentically encounter patients rather than sit there typing. Appointments where clinicians are face-to-face with patients rather than staring at a screen are what patients, and clinicians, want. AI scribes are an unfolding story worth watching.
Learn more: Get news on other CAI projects.
Write to TAP’s advice column, Ask a Psychoanalyst.
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.