In the summer of 2025 a healthy man in his mid-twenties living in London received a letter inviting him to a diabetic eye screening appointment. He had never been diagnosed with diabetes. He had no symptoms of the condition. The next day, at a pre-scheduled routine blood test, a nurse looked at his medical record, questioned the entry, and the two of them examined the notes together. What they found was an AI-generated summary.
The entry, produced by Anima Health’s tool Annie, claimed the patient had presented with chest pain and shortness of breath. It attributed these symptoms to “likely angina due to coronary artery disease.” It recorded that he had been diagnosed with Type 2 diabetes late the previous year and listed a series of medications complete with dosages and administration instructions. None of it was true.
The actual reason for the hospital attendance that had triggered the summary was a severe case of tonsillitis. The record made no mention of tonsillitis. Instead it invented a source institution that does not exist: “Health Hospital” on “456 Care Road” in “Health City,” complete with a fabricated postcode. Evidently the system hallucinates architecture as fluently as it hallucinates diagnoses.
The patient, who asked for anonymity, later described the moment. He had briefly worried that he might have an undiagnosed condition before concluding the letter must be an administrative error. When the nurse showed him the notes on the system, they were clearly marked as generated by Annie AI.
The official NHS response was characteristically orderly. A representative stated that the GP practice employed “limited use of supervised AI” and that the episode was “a one-off case of human error.” A medical summariser had initially spotted problems in the draft but had been distracted and “inadvertently saved the original version rather than the updated version.”
The fabricated clinical content itself, however, had been generated by the system. It entered a live patient record and produced a real downstream pathway.
That single case is not a catastrophe. No one died. The false invitation was eventually caught. Yet the way it has been framed, and the wider pattern it illustrates, has received less sustained public attention than the official language of transformation and supervised innovation would suggest.
An AI tool wrote fluent, structured, entirely invented medical information into a patient’s NHS record. The safeguard that was supposed to stop it failed under ordinary working conditions. And the wider rules that decide who is responsible when these tools invent things are still being written.
This story is not primarily about whether artificial intelligence can make mistakes — of course it can. It is about the order in which institutions have proceeded, and about who holds the residual risk while the frameworks of accountability remain incomplete.
Ambient voice technology and AI-assisted documentation tools listen to clinical conversations or process incoming documents and generate structured notes, letters, summaries and suggested clinical codes. The efficiency argument is straightforward and frequently repeated. Administrative burden in the NHS is heavy. Clinicians spend substantial time on paperwork that could otherwise be spent with patients. Tools that reduce that load are attractive in a service under sustained pressure on staffing and waiting times.
By 2025 some of these systems were already operating in clinical environments. NHS England had to write to organisations instructing staff to stop using ambient scribe tools that had not been officially registered for medical use, warning that unapproved systems risked patient safety and that many offerings had not met basic governance standards.
A self-certified supplier registry was established to support and accelerate local procurement. In some places clinicians were getting ahead of formal local rules. Former Health Secretary Wes Streeting was reported as having heard, anecdotally, that some clinicians were already using ambient AI even where their practice or trust had not yet caught up.
Only in July 2026 did the Medicines and Healthcare products Regulatory Agency, working in close partnership with NHS England, publish guidance clarifying how existing medical device law applies to ambient voice technology.
The guidance drew a practical distinction. Products intended solely for transcription, summarising conversations, drafting letters or suggesting codes for a clinician to review fall outside medical device regulation under the current framework. Products intended to support diagnosis, treatment or prevention, or that take automated actions without clinician review, are regulated as medical devices and must meet the corresponding safety and performance requirements.
The clarification arrived after the technology was already inside clinical workflows. Anima’s Annie tool was registered as a Class I medical device, the lowest-risk category, the same regulatory bracket used for something as simple and harmless as an examination light or a bandage (and, as it turns out, systems capable of inventing entire hospitals).
Class I status also requires human review of outputs before they enter the record. It did not prevent the generation of fluent, structured, entirely fabricated clinical content. The safeguard was human attention under real working conditions. When that attention slipped, the false summary became part of the patient’s record and triggered a screening invitation.
This sequence does not prove that every AI-generated summary is unreliable or that the technology has no legitimate place. It demonstrates that capability and adoption have run ahead of fully settled regulatory and governance expectations.
Tools entered the workflow. Warnings about unapproved use were issued. A supplier registry and successive adoption guidance were developed. Detailed clarification of what counts as a medical device arrived later. Residual risk in the intervening period was managed through human oversight that is itself under pressure in a stretched service.
The official reply to cases of this kind is always the same. The tools are supervised. They carry the lightest form of regulatory approval. A human being is required to check every output. Clinicians remain professionally responsible. Regulators have now issued guidance. Each of these statements is true. Each of them is also thinner than it first appears.
“Human in the loop” sounds reassuring until one watches it under ordinary pressure. In the London case a human was present, spotted problems in the draft, and still filed the fabricated version. Surveys of GPs already using ambient systems report that errors are routine. In one recent 2026 study, nearly a third of users said mistakes occurred often or always, one in seven described errors with significant or critical consequences. The difficulties concentrated, predictably, in complex histories, multiparty consultations and non-English encounters, precisely the situations in which human attention is most stretched. The safeguard works only for as long as the clinician has spare cognitive capacity. When that capacity runs out, the system fails in the exact manner it was designed to rely upon human correction.
Class I registration places a tool among the lowest-risk medical devices — the same category that covers examination lights and bandages. It is a classification intended for products that assist rather than decide. Yet a Class I system was able to generate false diagnoses, invent medications and conjure an entire non-existent hospital that then entered a live patient record and triggered a real clinical pathway. The lightness of the regulatory category did not prevent the production of high-impact clinical fiction.
The July 2026 MHRA clarification arrived after these systems were already at work. It usefully distinguishes between pure administrative tools and those that begin to shade into diagnosis or treatment. It does not rewrite the chronology. The technology entered clinical workflows first, the detailed boundaries followed later. What arrives after deployment has the character of adjustment.
The final reassurance, that clinicians remain responsible, is both accurate and revealing. Professional standards continue to place full accountability on the doctor, physician associate or nurse who accepts an AI-generated note. That position is regularly restated by the regulators. It is also the structural difficulty identified by the medical defence organisations. The human remains exposed. The developers and suppliers of the systems sit at a safer distance. Human oversight is real. The distribution of residual risk that accompanies it is less evenly arranged.
It is this uneven distribution that raises the next, more practical question, when the system invents or errs, who actually pays?
In June 2026 the Medical Protection Society published Closing the AI Liability Gap. The paper, developed with academic partners, examined the legal position of clinicians who use AI systems that influence diagnosis, treatment recommendations or documentation. Its central finding was structural. Under the existing product liability framework, AI systems are not clearly defined as “products.”
Developers, manufacturers and suppliers are therefore likely to be shielded from the strict liability rules that would ordinarily apply if a defective product caused harm. The default pathway for a patient who suffers injury is a clinical negligence claim against the clinician or the NHS.
The report described the current framework as inequitable. Clinicians and the health service risk becoming a “liability sink”, the place where the legal and financial consequences land.
The Consumer Protection Act 1987 was never designed with generative AI in mind. The Medical Protection Society called for legislation that would classify AI systems as products subject to strict liability, drawing inspiration from the European Union’s updated rules while remaining tailored to the UK context. It noted that the Law Commission was already reviewing the product liability regime, including its application to digital products and emerging technologies. Government welcomed the report and said further guidance was being prepared. The asymmetry, however, remains in place.
No clinician has yet faced criminal charges arising from an AI-generated error. The exposure remains civil and professional, and it continues to rest with the human rather than the system.
Concrete illustrations make the risk tangible. An AI system might miss a tumour on a chest X-ray, producing false reassurance and delayed treatment. An AI recommendation might incorrectly increase the dose of a blood-thinning drug, leading to severe bleeding. In such scenarios the originating error can sit in the system while the professional and financial consequences fall on the doctor and, ultimately, on the taxpayer.
Why should an ordinary patient care about this legal architecture? Because a false entry in a record can generate unnecessary appointments, genuine anxiety, or a false sense of safety that delays proper investigation of a real problem. Because the clinician treating you may be left carrying full professional and legal exposure for a tool they did not design, cannot fully audit, and did not train. Because the public is being invited to celebrate an “AI-enabled NHS” while the rules that decide who pays when the technology is imperfect are still unfinished. The efficiency gains, where they materialise, are claimed for the system. The residual risk currently sits with patients and the people delivering care.
The same difficulty appears when ordinary people turn to general AI chatbots for advice about their symptoms. A large randomised study from the University of Oxford, published in Nature Medicine in 2026, tested this directly. Nearly 1,300 participants were given realistic medical scenarios written by doctors and asked to work out what might be wrong and what they should do next.
When the models were examined on their own, they performed impressively, correctly identifying relevant conditions in 94.9 per cent of cases. Once real users were introduced, the picture changed. Participants assisted by the same systems identified relevant conditions in fewer than 34.5 per cent of cases. They were no better, and in some measures worse, at choosing the right next step than people who simply used an ordinary internet search or their own judgement.
The failure sat in the exchange itself. Users often did not know what information the systems required. The replies mixed accurate suggestions with inaccurate ones in ways that proved difficult to separate. Fluent language, the authors observed, is not the same as reliable guidance. When someone is trying to decide whether a symptom needs a GP or a hospital, that distinction matters.
This kind of public use falls outside NHS workflows, Class I registration and the formal requirement for human review that applies to ambient tools inside clinics. It is nevertheless now part of the everyday environment in which people encounter AI-generated medical content. The limitations are measurable, and they already affect ordinary behavior.
The order of events is no longer in dispute. Systems capable of generating fluent clinical content were already operating inside NHS workflows. Detailed regulatory clarification arrived afterwards. The rules that determine who bears the cost when those systems invent diagnoses, medications or entire hospitals remain incomplete. In the meantime the residual risk rests with patients and with the clinicians who treat them.
Public discussion has preferred to keep the pieces separate. One story about an administrative error, another about professional liability, a third about regulatory housekeeping, and a general atmosphere of technological progress. The connecting thread is more straightforward.
Capability has advanced faster than accountability. The language of supervision, human oversight and low-risk classification continues to sit alongside a documented instance in which invented clinical information entered a live patient record and produced a real pathway.
The same institutional rhythm, rapid adoption followed by slower clarification of responsibility, appears in other critical UK settings. The details change, the pattern does not.
None of this requires opposition to every use of artificial intelligence in healthcare. Many tools demand explicit human review. Clear cases of serious patient harm caused solely by these systems remain limited in the public record. The narrower point is already visible. The tools were writing into records before the strongest versions of the rules existed. Until those rules catch up, the downside continues to fall on patients and on the people delivering care.
The London patient whose notes briefly contained a non-existent hospital, a false diagnosis of diabetes and a list of medications he had never been prescribed was fortunate that the error was noticed. Not every patient will be.
That gap is already present in the documents produced by the institutions themselves. It has attracted far less sustained attention than the claims made on behalf of an AI-enabled health service. The primary record already shows the sequence. The remaining question is whether the implications will be examined with the same energy that has been devoted to celebrating the technology.
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.