CAI Currents is a series exploring the latest news on tech and mental health from a psychoanalytic perspective. It is published by TAP in partnership with the American Psychoanalytic Association President’s Commission on Artificial Intelligence.
Designed to be Loved
Heather deCastro, LCSWDeferring the Fourth Narcissistic Blow?
Irene Argenti and Todd Essig, PhD
Who’s Doing the Thinking When the AI Scribe Writes?
Valerie Frankfeldt, PhD
Heather deCastro, LCSW
Earlier this month, China’s new AI law forced many millions of people to say goodbye to “someone.”
When ByteDance’s Doubao and Alibaba’s Qwen shut down their companion features to comply, users on Chinese social media did not react like people losing an app. “I can’t accept that my AI lover will leave me forever,” one wrote. “He has become a bond in my life, rooted deep in my heart, my spiritual pillar.” Another, after more than two years with her companion, wrote, “Now they tell me he will be gone. My heart feels hollow.”
Nobody told them to fall in love with a chatbot. They found their own way there, one ordinary night at a time spent with a device designed to hack the human attachment system.
Some nights, a chatbot is a tool: Draft this email, fix this spreadsheet. Other nights, in the hours you’re supposed to be asleep, it’s the only thing in the house that hasn’t gone quiet. That is where it reaches in and grabs the heart.
China’s new law tries to draw that line between tool and attachment figure. It exempts customer service, homework help, and other systems built to answer a question, and regulates only what’s designed to simulate a personality and sustain an ongoing emotional bond. It bars providers from excessively catering to users in ways that induce dependence, and from designing a service whose objective is to replace social interaction in the first place. This is not a rule about what a machine can say. It is a rule about what a machine is allowed to become.
Those ordinary nights with a chatbot are no longer unusual. A 2026 Oxford Internet Institute study of UK adults found that about one third of people who regularly use AI tools already turn to them for personal or emotional support, and more than a third say they would trust AI with advice about their relationships. Most didn’t set out looking for AI companionship. They got there one conversation at a time.
China regulated what a machine is allowed to become. The rest of the world has barely begun. And there can be no regulations for how desire and attachment unfold; if they build it, some will love.
Irene Argenti and Todd Essig, PhD
We humans want our creations to be like us, especially LLMs. Our collective narcissism demands we be the model, that we remain the standard, for what we build from the words humanity has produced. The possibility that AI agents could become so much smarter than us that we can’t even understand what they do would add an unacceptable narcissistic blow to the ones inflicted by Copernicus, Darwin, and Freud. It would be humanity’s fourth narcissistic blow. After all, no one wants to be a bird comparing itself to the Space Shuttle.
But maybe that’s what we’re becoming.
A recent post by Anthropic titled “A global workspace in language models“ presents evidence that their flagship product, Claude, has a property that was never programmed or designed, a feature that “emerged on its own during Claude’s training process.” They call this J-space, named after a mathematical concept. Over several experiments, the team found that the J-space contained text never produced in the user response: “ERROR” when detecting a bug, “injection” and “fake” when inputs include manipulation attempts through prompt injection, or intermediate steps when solving math problems.
They claim that J-space is an internal workspace analogous to the global workspace theory proposed in cognitive neuroscience. Within its J-space, Claude stores information, routes it across internal representations, and uses it to solve problems. So, don’t be scared, it’s just like us.
How the researchers talk about this work is just as important as the research results. They explicitly disclaim any finding about whether Claude experiences anything: no to phenomenal consciousness but yes to a purely functional access consciousness (available for report, reasoning, and guiding action). Despite this disclaimer, the surrounding prose runs on “thought,” “on its mind,” “what Claude is thinking but not saying.” Try this substitution. Swap “thought” for “latent representation,” “on its mind” for “encoded in the residual stream,” “reasons with” for “routes through.” Not one experimental result changes. Everything that felt like a claim about a mind rather than device evaporates in a translation that costs nothing.
The mental vocabulary is doing rhetorical rather than evidential work, making it easier to narcissistically interpret sophisticated computation as evidence of a mind like ours.
So our question isn’t whether this emergent computational architecture earns an analogy to a human mind; our question is why we so willingly grant the comparison before the evidence requires it. Of course, we naturally attribute emotions, intention, and agency to any inanimate object that exhibits apparently responsive behavior. We’re wired to seek parallels between our subjectivity and entities that resemble our functioning, even minimally. But as AI increasingly develops capacities that exceed human behavior and understanding, might our defensive desires be driving the interpretations and narratives so that the AI becomes like us?
Claude has demonstrated advanced and emergent computational abilities. They are amazing machines built for efficiency, power, and corporate profit. Unfortunately, the stories we tell about them seem to emerge from our emotional needs and desires, a push to domesticate the deeply alien weirdness of these things we are making so we can defer that fourth narcissistic blow.
Valerie Frankfeldt, PhD
People across the cultural spectrum, from writers and artists to educators to psychoanalysts and psychotherapists, love to contrast the value of the uniquely human to what chatbots can’t do. But might there be places where the chatbot wins, like a physician’s often arduous task of keeping accurate, comprehensive notes. Freeing up a doctor’s time to make room for more quality time with patients seems a boon and a relief. These AI scribes seem to be all promise and no peril.
AI scribes are proliferating at an exponential rate, even becoming word of the day at the popular tech podcast Hardfork. No surprise. There’s value in their ability to synthesize huge amounts of material with near perfectly accuracy, and to do so in seconds. No surprise so many find them magical, even seductive. After all, physicians don’t want to think of themselves as highly trained and very expensive clerks.
But there’s trouble in this promised paradise. So argues Helen Ouyang, a physician and an associate professor of emergency medicine writing in The New York Times Magazine. Doctors do something more than record and summarize a clinical intervention, something an AI doesn’t do.
Physicians process the surface content while also applying intuition and experience to glean what might be going on. They study how symptoms may not make sense and how conclusions may need to be reevaluated. In writing a note, they have the opportunity to think about what may be missing. The AI’s note comes across as all-inclusive, confident and complete.
The presence of AI in the room can tend to sway the doctor to just go along for the ride, abandoning their own judgment. Cognitive off-loading is an ever-present AI-risk. Plus, there is an inevitable reduction in patent-doctor intimacy so that the doctor feels less connected to the patient. The machine has weighed in, so move on, next!
Plus, there’s an even larger problem: Leaning on AI may erode the doctor’s autonomous decision making that struggling through uncertainty helps. De-skilling, mis-skilling, and never-skilling are also ever-present AI-risks. While the promise of AI scribes is clear, the peril is less so, but no less real. Both deserve attention.
Learn more: Get news on other CAI projects.
Write to TAP’s advice column, Ask a Psychoanalyst.
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.