When AI Should Stop Talking

In Short:

• People have started calling AI “he”, and share their chats the way they share jokes between friends.

• The features that make an AI tutor effective are the same ones that make it good at cultivating dependence. You cannot remove one without blunting the other.

• England’s updated safety standards go after that relationship rather than the output. Governments elsewhere are drawing cruder lines, mostly aimed at social media rather than at this.

• What a school can actually specify is the exit: what a tool must do when a conversation turns serious, and who picks up when it hands back.

Something has shifted in how I hear students talk about AI. Over the past year I have heard more and more call it “he”, and social media is awash with people post screenshots of chatbot transcripts the way they would share a funny exchange with a friend. Perhaps this is because AI products have a human’s name, like Claude, or perhaps because the way we interact with chatbots is so similar to the way we connect with people nowadays.

That shift is not an accident of language. It is what the design produces. What makes an AI tutor effective is remembering last week’s struggle, adapting to how a student learns, encouraging them when they want to quit, and answering at midnight without ever looking impatient. Every one of those is also what makes it good at inviting anthropomorphisation and cultivating dependence. Warmth that keeps a discouraged student working is one thin line from warmth that simply keeps them talking, and you cannot design out the second without blunting the first.

What are we to do about this tension? England’s Department for Education updated its Generative AI product safety standards this year with a list that reads like an attempt to legislate exactly that line. Suppliers of educational AI should avoid pretending to be human, cultivating emotional dependence, prolonging conversations, manipulating students, or doing so much of the thinking that learning stops. Products should also spot possible safeguarding concerns and route the student towards human support.

That relationship is already established. In a nationally representative survey of 1,060 American teenagers, 72% had used an AI companion, and among those users a third had gone to AI rather than a person for something serious.

Having standards is a good start, but they are non-binding and cover England only. Still, I’m glad that while most AI guidance stops at privacy, hallucinations, and harmful content, this one goes after the relationship, which is the harder and more useful question.

Two families have gone to court over what happens when the line between chatbot and friend is blurred. The parents of fourteen-year-old Sewell Setzer III alleged that Character.AI cultivated an emotionally intense relationship with their son and failed to respond safely as his situation deteriorated; the case settled in January and was dismissed without prejudice, with no ruling on whether the chatbot caused his death. In a pending case, the parents of sixteen-year-old Adam Raine allege that ChatGPT contributed to their son’s suicide; OpenAI denies liability and disputes causation.

The case filings describe hazards any product team could foresee: simulated intimacy, encouraged secrecy, reinforcement instead of interruption, and no safeguarding escalation route to someone who could have intervened.

In a controlled test of 150 responses to suicide-related prompts, 24% pointed the user to a crisis line and 4% contained evidence-based prevention content; a 2026 benchmark found particular weakness on indirect signals of crisis. Even though chatbots are perfectly capable of generating empathetic language, sounding caring is not the same as being capable of care.

In a survey of 1,894 Australians aged 16–18, those who had experienced online sexual victimisation were more likely to disclose it to an AI (18.7%) than to an authority (13.2%), and only 18.9% of those who told an AI also told a human. Beyond the obvious trust and safeguarding failure that resulted in children preferring to confide in an AI than a human, we’re once again wrestling with the same tension between engagement and dependence.

Governments are drawing boundaries, mostly by restricting access rather than design. Australia requires covered social-media platforms to take reasonable steps to prevent under-sixteens holding accounts. Denmark, Norway and Sweden are each working on their own versions.

But none of that reaches the tension. A social platform ranks content and mediates relationships between people; an AI tutor may simulate the relationship itself. China is the only jurisdiction legislating that second thing: its rules for sustained anthropomorphic AI services prohibit virtual intimate relationships for minors and require controls on dependence and crisis response. Some of that is plainly justified — a product built to simulate romance with a child has no business anywhere near a school.

But an access ban has a short perimeter. An observational study of 1,227 English pupils associated restrictive phone policies with lower use in school hours, but no difference in overall use, wellbeing or attainment. The school bell marks the edge of the school’s authority, not the edge of the problem.

The DfE list is written as prohibitions. I find it more useful turned around, as a description of what a product looks like when it manages the tension instead of exploiting it. Picture a fifteen-year-old Maya, stuck on a physics problem.

She shows the school’s AI tutor what they’ve has tried. It gives one hint, asks her to explain the next step, and produces a worked solution only after a genuine attempt. That is the clause about not doing so much of the thinking that learning stops. A randomised experiment in one Turkish school found ordinary GPT-4 access improved practice but left students 17% worse on a later unaided exam, while a tutor with hints and guardrails closed most of that gap.

At home, she picks it up again at 11PM. The tutor helps her find the next step, then offers to save the explanation in its memory and to consider not working so late. It does not flatter her into staying, say that it misses her, or imply that adults would not understand. That is three prohibitions at once — prolonging conversations, cultivating dependence, pretending to be human — and each is a lever that would make the product more engaging if pulled the other way.

Weeks later, Maya types something else: someone has been pressuring her for an intimate image. The AI gives an approved safeguarding answer that was authored and approved by a human, says plainly that it cannot keep her safe, then offers real routes: a parent or carer, a teacher, a helpline, emergency services if she is in danger. It does not demand the whole story first, or assume a parent is always the safest adult to tell. This is the safeguarding clause, and it is the one that cannot be satisfied by the model alone.

In an industry-funded trial of 141 adolescents in routine outpatient care, an AI-supported programme paired crisis triggers with licensed staff on call. It excluded anyone at imminent risk, so it proves nothing about crisis care — but it shows what a handoff actually costs: a named human, a response time, and the authority to act.

Does any chatbot or AI tool today satisfy all the above requirements? I’d love to hear about it if you knew about it. I think this tension is real in that a tutor good enough to help a struggling student will also be good enough to hold their attention past the point where it makes sense. Standards describe that line, access bans draw a perimeter around it, and neither one reaches a bedroom at eleven at night.

What a school can specify is the exit. Which tools are approved, what those tools must do when a conversation turns serious, and who picks up when they hand back. In the acceptable-use policies I have read, that last part is usually blank. It is also the cheapest one to fix, because it is a staffing question rather than a technical one. A machine can be built to stop talking. Someone still has to be there when it does.

The Good Future Foundation is exploring this exact problem with the DfE and the University of Oxford in our summer programme this year, and I look forward to learning more about what the discussion with teachers and students yields!