It's the Context, Stupid!

David Wallace-Wells posed a question to Dr. Rachael Bedard on The Opinions and then handed the same question to his readers: what does your health data mean to you?

I have been sitting with that question in exam rooms since 2002. Most days it walks in as a request for a test. Most days my work is to slow the request down long enough to ask what we are hoping to learn.

Their conversation opened with a fight. A San Francisco company announced a consumer whole-body imaging service. Doctors pushed back. The technology world replied with the line that surfaces in every one of these arguments: how can more data ever be bad?

It can be bad. It can be dangerous. Here is why, in plain arithmetic, because this is the part that almost nobody explains and it decides everything downstream.

A test can be excellent and still be wrong most of the time it says yes.

Take a hypothetical. A disease that lives in one person out of every thousand. This test achieves a ninety-nine percent accuracy in identifying the disease, a rate superior to most tests you will ever be offered. Now screen ten thousand people.

Ten of those people have the disease, and an excellent test will find close to all ten. The other nine thousand nine hundred and ninety do not have it. One percent of them, right around a hundred people, will be told that something turned up, anyway.

Count the yeses. About 110 people leave with a positive result. Ten of them are sick. A hundred of them are fine.

So the odds are better than nine in ten that the finding in your paper is nothing. Your paper does not say that. Your paper states that someone discovered something. Your family reads the paper. You cannot unread it.

Those hundred people are the false positives, and they are not a rounding error.

They are the primary products of the enterprise. Each one now enters a corridor with a door at the end: a repeat scan in six months, a needle, an operation, a year of thinking of yourself as a person with a finding. The corridor has its own risks and its own bills, and a healthy person’s curiosity built the corridor.

Think of it as signal and noise. A smoke alarm sensitive enough to catch every fire will also sound every time you brown butter. You will pull the battery, or you will call the fire department once a week. Both responses cost something.

Bedard tells the South Korean thyroid screening story well. Universal ultrasound, a large jump in detected cancers, no change in deaths, and a long trail of biopsies and surgeries behind it. That is the arithmetic above, playing out in a real country, on real necks.

On the first day of medical school, we recite the oath and promise to do no harm. Most of us rolled our eyes. We were young, and we intended to be proactive about everything. Hippocrates was working on a longer timeline than we were.

Which brings me to the word in the title. A number carries no meaning until it has context.

Telling a patient that a scan found atypical cells is close to meaningless, and it is not harmless. Sometimes atypical cells are a warning. Sometimes they are nothing at all. The phrase is identical in both cases.

I encourage plenty of people to check their blood pressure at home. I encourage some to check their glucose. Those numbers mean different things depending on the hour, the meal, the cuff, the week, the argument you just had in the car. You need the skill to record them and a frame to read them. I know someone who mixed up his two home devices and reported a sugar as a blood pressure. He was doing exactly what he had been told to do. Nobody had told him what he was looking at.

Then there is the raw material itself. Where did your data come from? Is it accurate? Is it representative? A great deal of medical information was entered, and still is entered, by someone moving fast through boxes at the end of a long shift. Pull your own records and read them. Count the errors. Then decide how you feel about a machine learning something from that pile.

The clinical side has a scarcity problem too, and the scarce thing is time. Physicians face pressure to see more people in less time, leaving virtually no room for the deliberate task of interpreting a result in the context of a patient’s life. The proposed remedy is more computation. I would take that offer more seriously if the same systems could produce a correct bill.

There is also a motive worth naming out loud. Some of this push exists because there is hardware to sell, software to sell, and a subscription running underneath both.

None of which makes me a skeptic of measurement. Bedard is right that the strongest clinical evidence for wearables looks nothing like the wellness market. The heart monitor gives a patient answers to a question I already had. The device bought on a Tuesday because a podcast host praised it answers nothing, because nobody asked anything.

A test ordered to answer a question is medicine. A test ordered to quiet anxiety manufactures more of it.

This is the part I would put to the engineers, who are better at answering than anyone in history. Most of the diagnostic trouble I have watched unfold came from a well executed answer to a poorly chosen question.

My thanks to David Wallace-Wells and Dr. Rachael Bedard for taking this seriously enough to argue about it in public.

Before your next scan, your next panel, your next device, sit with three questions.

1. What am I trying to find out?

2. What would I do differently for each outcome?

3. Who is going to read this, and how well do they know me?

As always, Jordan

Leave a comment