The Home-Field Algorithm

I spend a fair amount of time worrying about a specific version of this problem. A model that performs beautifully on one organization’s data can wobble the moment it meets a population it wasn’t built on. For a long time I treated that wobble as a training-data problem you solve once, upstream, before a model ever reaches scaled use.

A study out of PLOS Digital Health this March says that assumption doesn’t hold up. Chen, Li, and Lin systematically reviewed 171 studies of medical AI models that had already cleared the bar most of us treat as the finish line — approved for, or actively deployed in, real clinical settings, not research prototypes. Across more than 200,000 real patients, the advantage these “mature” models hold over human clinicians turns out to depend almost entirely on whether the patient in front of the model looks like the patients it was built on.

Across all 171 studies, AI models beat human practitioners on accuracy, 81.7 percent to 77.8 percent. That’s the number every vendor deck leads with, and it’s real. But Chen and colleagues split the studies by geography: evaluations that happened in the same country or city where the model was developed (”in-distribution”) versus everywhere else (”out-of-distribution”). In-distribution, the AI advantage actually widens, to 82.9 percent versus 77.3 percent for clinicians. Out-of-distribution (cross-border or cross-city validation), it evaporates. AI scored 74.1 percent against 76.3 percent for humans, a gap so small the p-value came in at .45. Statistically, no gap at all.

This is particularly relevant because these weren’t lab prototypes with an asterisk next to their claims. Every model in this dataset had already cleared regulatory approval or active clinical deployment, the checkpoint we generally assume catches exactly this kind of problem. It didn’t.

The honest way to describe the headline number, then, isn’t “AI beats doctors.” It’s “AI beats doctors on its home field.” Move the game somewhere else and the scoreboard resets to even.

The geography of that home field is not evenly distributed. Geographic concentration across the 171 studies was extreme (a Dagum-Gini coefficient of 0.97), with 95.1 percent of patient cohorts coming from high-income (62.2 percent) or upper-middle-income (32.9 percent) countries — overwhelmingly China (28.7 percent) and the United States (18.9 percent). Not one study in the dataset involved a low-income country. Racial composition followed the same pattern: White (49.1 percent) and Asian (42.6 percent) patients made up most of the cohort, and nearly two-thirds of the studies showed substantial sex imbalance in who the model was tested on.

None of this started at the deployment stage. A 2024 review in npj Digital Medicine, looking one step upstream at the clinical-trial pipeline itself, found the identical skew: of 159 AI-enabled clinical studies, Yang and colleagues found 74 percent were conducted in high-income countries, with exactly one (in Mozambique) coming from a low-income country. Only 6.3 percent were international multicenter studies, the kind of design that would actually test a model against a population it wasn’t born into.

Put the two studies side by side and you get a full pipeline: the trials that build these models skip the populations that need them most, and the deployment data confirms the skip never gets corrected on the way out the door.

Here’s the part that should bother anyone buying AI for a health system rather than building it. “Regulatory-approved” and “already in clinical use” sound like assurances that the population question has been handled. Chen and colleagues’ data says those labels tell you almost nothing about how the model performs on a patient who doesn’t resemble its training set (the aggregate accuracy number vendors publish blends in-distribution and out-of-distribution performance into one reassuring figure).

I’ve sat through more vendor pitches than I can count where the accuracy slide is a single number with no population breakdown attached. After this study, that single number is the wrong artifact to evaluate a vendor on. The right question is closer to: show me your performance sliced by the population it was validated against, and show me what happens outside it.

That’s a different conversation than the one most procurement processes are built for. A security questionnaire tells you whether a vendor encrypts data at rest. A clinical validity questionnaire, as most of us currently write it, tells you an aggregate accuracy number and calls it done. Neither one asks the question this study says actually predicts whether the model will work for your patients: what population was this built on, and how far is my population from that one.

Practically, this changes what belongs in a vendor security and clinical-validity review before a contract gets signed. Training population characteristics and out-of-distribution performance data should sit next to SOC 2 reports and BAAs as baseline diligence items, not optional appendices a vendor provides if asked twice.

It also reframes what “bias in AI” means for people who spend their days procuring the stuff rather than training it. This isn’t an abstract fairness debate that lives in a data science paper. It’s an 8.8-point drop in the AI model’s own accuracy (from 82.9 to 74.1) the moment your patient population stops matching the one the model grew up on. Ask for that number before you ask for the roadmap.