Adding Support Where It’s Actually Needed

One of the soundtracks to my childhood was the NPR talk radio show Car Talk. My dad would have it on every Sunday while we were driving to the farmer’s market. This always confused me a bit, since my dad was never really into cars. What was so special about this one show?

I think I recently learned the answer. Apparently a lot of physicians loved Car Talk, even if they couldn’t care less about cars. That’s because listening to Tom and Ray think is surprisingly similar to being in a doctor’s brain - after all, they both help people who call in with problems by collecting histories and issuing diagnoses. One episode even directly played on these similarities, and they were actually formally explored in a pretty entertaining JAMA article.

Most of the media that’s been written about this connection has broadly discussed diagnostic reasoning, but I want to spend some time specifically focusing on the art of making diagnoses and treatment plans under uncertainty. Telling someone what’s wrong with their car or body, and how to fix it, even when you don’t have much information.

To me, this has always been both the coolest and scariest part about being a doctor. Great clinicians have a Sherlock Holmes-esque ability to bootstrap answers and plans out of even the most limited sets of data. This looks like a real-life magic trick when you’re seeing it in action, and the high stakes of medical decision making certainly don’t make it easier. Incorrect decisions could harm someone’s health or even be a literal matter of life or death, so it’s fairly anxiety inducing to think about how often doctors have to base their advice on calculated guesses.

Throughout the history of medicine, there’s been a long lasting effort to create clinical decision support (CDS) tools that help ease this burden. Recently, the AI revolution has revitalized that effort. Folks are making intelligent technology that can find relevant research, automatically identify diagnostic cues, and create new evidence on-demand to base your decisions off of. Google is even making an AI doctor that you can use as a copilot to check your work against!

Most of this technology functions in a relatively similar way: type in your question along with the context you have available, rub the genie’s magic lamp (AKA the send button), and receive an answer that ideally helps your decision making process. This is super useful in situations where you have both information and time available, but I’d argue that it generally misses the mark in being helpful for making diagnoses under uncertainty. To explain this, I want to explore 2 common reasons behind why uncertainty even arises.

Physicians love talking about Daniel Kahneman’s Type 1 and Type 2 thinking1, but I think many diagnostic challenges as better framed as manifestations of the exploration vs. exploitation dilemma. This refers to the constant choice between obtaining more information before making a decision (exploration) versus using what you already have available to you (exploitation). Doing both at the same time is generally difficult (although not impossible), so finding an optimal balance between the two is crucial.

This is a fundamental concept in economic decision theory and machine learning. I think it should also become one in medicine. It applies to nearly every task we deal with (medical education is a prime example, planning to write more about this later). Let’s specifically explore its relevance to diagnosis here.

Modern diagnostic challenges offer nearly infinite exploration opportunities; there is always another test you can order. However, many of these tests have low diagnostic yield or give you information that isn’t very actionable. As such, more exploration isn’t always better. Unnecessary testing wastes money, time, and resources that could’ve gone towards someone in greater need, and is generally inconvenient for both patients and doctors. It can even worsen care, by delaying treatment and causing data deluge. This is the usual problem with excessive exploration: it has diminishing returns and often yields insights that weren’t worth the cost it took to get them.

Meanwhile, premature exploitation just results in uninformed decisions. This is intuitive - if you spend no time at all understanding what’s wrong with a patient, there is no way you can make a good diagnosis.

Knowing when to switch between these two thinking frames is already an incredibly difficult task. Add in contextual factors, like the insurance and healthcare system’s incentives or the time-sensitive nature of medical issues, and it’s no wonder that an ideal balance seems impossible. I’d wager that this is the fundamental difficulty in making diagnoses under uncertainty: the challenge of deciding on and obtaining the “just right” amount of history, labs, tests, etc. for each case. An astonishing amount of mistakes and issues in our healthcare system can be abstracted to either exploring too much or exploiting too early.

CDS could be incredibly useful for handling this dilemma, but largely isn’t as of now. This product-market mismatch is my take on why most CDS technology has failed to improve clinical outcomes or save meaningful amounts of time/money. Many tools succumb to becoming excessive exploration, where they don’t actually improve decisions despite seemingly presenting lots of useful information.2 In other words: you’re getting to the same diagnosis, you just feel smarter while you’re doing it. Other tools fail because they don’t add anything to the exploitation process - they intake the same information you have already and output the same thing you were going to do anyways. Just this time, admin has to pay for both you and the A.I.

The most useful CDS would treat uncertainty at its root cause either by making it easier to explore accurate, high-yield info or by making us better at exploiting existing data. This could be by making cheaper, faster, or more information-dense tests, or by squeezing new results out of tests people are already ordering (like this model that helps identify liver disease from heart ultrasounds). By increasing the expected value of more exploration or the insights available from exploitation, such technology would make either option in this dilemma far better.

It would also be incredibly useful to have CDS that directly helps in selecting the balance between these two options. I imagine this to look like a Bayesian projector that continuously shows what the yield of more exploration could be and what the take-away from immediate exploitation should be. This should constantly update when new symptoms, lab results, etc. are noted.

The crucial metric for evaluating CDS is specifically how often does using decision support actually result in doctors changing their decisions. The term I propose for this is decision yield.

Doctors will still always have to personally deal with this dilemma and make the final decision about where to draw the line. I think that’s good! There’s never gonna be a universally correct answer here; the balance found in every single case is different because it is subjective to emotional, economic, and other contextual factors. This is where our humanism becomes a huge strength, and why replacing doctors will AI robots will never be possible (unless they all look like Dr. Strange…then it might be pretty cool).

Perhaps the most common causes of uncertainty are simple geotemporal constraints. For example, the rising popularity of video visits is forcing doctors to make more and more diagnostic and treatment decisions without information from a physical exam. Similarly, in many urgent situations there’s no time to take a detailed history before assessing the patient’s problem and how to fix it.

I recently read about a particularly striking example of this where a patient stopped breathing in the elevator of a community hospital about 5 hours after undergoing surgery to remove their thyroid gland. A code blue was called at the nursing station once the elevator stopped, and the responding hospitalist made a quick judgment to order for transfer to the ICU. By the time the patient got there and was intubated, they’d already suffered anoxic brain damage leading to paralysis.

It turned out that the surgical site was bleeding and compressing their airway. This is apparently not a very uncommon complication post thyroidectomy, as a bunch of surgeons, residents, and even medical students from other countries were saying that one of the first things they learn in surgical rotations is that this can get easily handled by removing the surgical stitches. Unfortunately, the hospitalist was ultimately found guilty for negligence in a lawsuit that finished with a multimillion dollar sentiment.

Although CDS could be life-changing for moments like this, it’s currently useless. Even though the answer we’d need from it is supposedly such an elementary concept, literally no support technology exists that would aid this hospitalist in that split second assessment and plan. This is a humongous hole, because these are some of the most critical moments where making the correct decision really matters.

Even in less high-stakes situations, like simple questions in the operating room, CDS is largely inaccessible. Imagine knowing that your surgeon was planning on stepping out while you’re cut open, pulling up his computer and checking with A.I. whether he should repair or trim your meniscus, and then scrubbing back in. “Yeah on second thought, I think I’ll just keep my torn ACL.”

This is actually a great use case for virtual/augmented reality or ambient technologies in medicine. It would be a game changer to have holographic support available during procedures or smart monitors in the hospital that show live, intelligent insights based on a patient’s history. This doesn’t even need to be innovative in the actual type of decision support: a pair of glasses that simply pulls up existing clinical guidelines in the corner of your eye could massively increase usage of gold standard decision support that we already rely on to improve care and decrease legal liability.

The ideal future of CDS should be one where no preventable errors happen just because of spatial or temporal barriers to simple answers.

Despite his problematic nature and questionable tactics, Dr. House was an expert diagnostician. He somehow perfectly knew when to investigate further (breaking into people’s houses for diagnostic clues) versus when to act on what he had at the moment (cancelling tests his team had already ordered). He was often impossibly good at split second decision making, and largely performed the same despite changes in environment or urgency.

CDS should aim to make everyone as good as him (kinder and more professional of course). Physicians already pride themselves on their abilities to arrive at the correct diagnosis given the least amount of information possible - CDS should do the same. It’s impressive that new technology can rival doctors when you give it all the clues it needs, but it’d be useful to have support that gets you to good decisions even when you don’t have these.