This is a newsletter of Faces of Digital Health - a podcast that explores the diversity of healthcare systems and healthcare innovation worldwide. Interviews with policymakers, entrepreneurs, and clinicians provide the listeners with insights into market specifics, go-to-market strategies, barriers to success, characteristics of different healthcare systems, challenges of healthcare systems, and access to healthcare. Find out more on the website, tune in on Spotify or iTunes.
You’ve probably been in this situation: you talk to AI, it’s confident in what it’s replying to you, and then you say: “Actually, I think you’re wrong.” And the response is: “You’re right. This wasn’t accurate, the right response it ….”
In a healthcare discussion, an example could look like this: you describe your problems and ask for an interpretation. AI gives you a confident response, and then you say: “I forgot to mention, I also have this condition…” and the AI chatbot responds:
”That changes your picture completely ...”
This is called the omission problem.
The initial response wasn’t a hallucination. It was a conclusion made on imperfect information, resulting in a wrong recommendation. I don’t know about you, but when I’ve been in this kind of a situation related to my health, I left the discussion feeling utterly confused, with a sense that AI use… does me more harm than good.
Many developers agree that AI is not yet ripe for diagnosis. And at HIMSS26 Europe, the most commonly raised concern was the challenges posed by the patient use of general purpose models for advice. As Megan Coder, Board Member of The Society for Digital Mental Health elaborated around mental health: General models are optimized for conversational fluency and engagement, not designed for clinical accuracy or patient safety, don’t have domain-specific training on mental health data, lack crisis escalation protocols, are not clinically validated or evidence based.
With this in mind, I decided to re-assess the current state of AI in healthcare software development, how vendors are using AI, what hasn’t changed in terms of data governance needs, and more.
Every new AI model seems to be exponentially more capable than the previous one. At the same time, existing medical software such as EHRs, is transforming slowly, due to legacy architecture and components. These can be hard to re-design or refresh with new technology, as too many people are already using the existing product. The integrations in the background that would need to be re-done are time and resource consuming and risky, especially if there is a chance that an update glitch would cause an outage.
Let’s talk about that background on the data structuring level.
For healthcare data to make sense beyond the free text clinical notes, what clinicians say and record needs to be coded. ICD-10 codes classify conditions for billing and statistics, SNOMED codes are more detailed and capture deep clinical care details. And there’s many other classifications.
In the US market, IMO Health has for 30 years been translating clinical language in structured, contextualized data, and is present in every major US electronic health record. As mentioned by John Laursen, it’s not enough for data to be structured, it needs to have semantic continuity. This refers to "sugar disease" recorded in the 1990s and diabetes recorded today resolving to the same concept. Without having such a longitudinal layer, an AI model reasoning across a thirty-year chart isn’t accurate. It resembles reading a foreign language it assumes it understands.
At IMO, 60 clinical informaticists and physicians take care of accuracy and ensuring the longitudinal continuity. The point is: yes, AI can tell you what an ICD-10 code or a SNOMED code for a particular use-case is, but if you want to make sure the outputs in your software are consistently correct, you should search for and connect the right code from the original source.
AI can make many processes easier. But original sources, and plenty of manual checking, is still required for the development of scalable solutions.
What now separates a good clinical AI system from a bad one happens underneath the model: how is standardisation done, which terminology services are used, how is the context structured.
Robert Tovornik, Innovation Lead at Better, points to another puzzling characteristic of AI solutions. The AI model race is largely settled; in his view frontier models are now performing on a more or less same level.
What the sector hasn’t resolved yet, is validation. Deterministic software can be tested against expected outputs. AI is non-deterministic. Every output, even with the same prompt, will be different. This means that trust becomes experiential - accumulated the way you come to trust a colleague. The more experience you have, the more you can judge what produces quality outputs. This is a challenge for procurement processes and certification regimes, which require demonstrable specifications. Nobody has a good answer for this challenge just yet.
Remember the story from the beginning? Omission has no screenshot. It produces an answer that is fluent, plausible, internally consistent, and wrong in a way that is invisible unless you already knew what was missing. That is the CDS problem of the next few years, and it is a data layer problem before it is a model problem.
AI expert and extrepreneur Elena Ikonomovska, built a company around the challenge of the complete context of patient information. She sees chatbots patients increasingly turn to for answers, as the more dangerous new entrants into medicine, not because they are unhelpful but because they are convincing.
Ikonomovska describes what she calls "faithful hallucinations" — confident, plausible outputs that fabricate detail patients have no expertise to challenge and clinicians have no time to unpick. Generative AI in medicine, she argues, risks widening health inequity, since better-resourced patients will access validated clinical tools while others rely on free chatbots.
What about clinicians, who increasingly use AI for clinical reasoning? Philippe Habets, CEO of EvidenceHunt, Dutch-based provider of a search engine for the latest medical research sees the omission problem in the prompt. Any published scientific paper draws conclusions from results of a carefully selected group of patients. The patient in front of a clinician is n=1, a puzzle that needs to be put in context. If you leave one comorbidity out of the question posed to AI, the correct answer might change. And the model has no way of knowing which condition it wasn’t told about. It cannot flag the absence of information it never saw.
EvidenceHunt’s response is to refuse case-level advice altogether. If a clinician does describe a concrete patient case and asks for a response, the system reformulates theclinical question into a generalisable research question answered with citations.
According to Robert Tovornik, testing ideas, building prototypes and showing the end customers the art of possible before actual development are enhancing collaboration between IT experts and clinicians. Some clinicians even come better prepared in IT discussions, since they use AI to better understand IT. This doesn’t come without potential inaccurate interpretations, but it’s changing the development dynamic.
For anyone evaluating or building clinical decision support, the four conversations suggest a fairly concrete set of questions — most of which are about the data layer, not the model.
What did the system not use, and can it tell you? A system that can say “I had no medication history for this patient” is more useful than one that scores higher on a test set.
Which parts of the pipeline are deterministic, and why those parts? If coding, mapping and terminology resolution sit inside the probabilistic layer, ask what failure looks like and how it would be traced.
How is the record’s quality being maintained, independent of the AI? Problem list hygiene, terminology currency and normalisation are unglamorous and are now the binding constraint on everything built above them.
What happens to the query? Where does a clinician’s question go, what is retained, and does the vendor’s business model depend on that retention?
Tune in to the full discussions!
Faces of Digital Health — website · Spotify · Apple Podcasts · LinkedIn
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.