On Empirical Thinking

Medical School teaches you a lot of things, from the bones in the foot to how the Loop of Henle makes you urinate. A lot of it, you forget relatively quickly after taking the relevant USMLE exam for your level of education. Some things, though, stick with you into your doctoring career, and if you’re lucky, you begin to see the relevance of some things even outside of a hospital. For me, the stickiest of all was statistics, and how to think about a test.

Because no test is perfect.

A clinical test, say a blood test or a CT scan, has a few properties that are important to understand. Sensitivity measures how reliably a test identifies people who actually have a condition. A highly sensitive test catches nearly everyone who is sick, but it may also identify some healthy people as sick. It is concerned with “true positives,” in the language of science. Specificity, on the other hand, deals with true negatives. It’s a measure of how reliably a test excludes people who do not have the condition. A lot of specificity means fewer false alarms, but it may miss some people who are actually sick.

When I first learned them during a Boston winter, these concepts seemed like narrow pieces of medical knowledge I had to memorize. But actually, they profoundly affected how I think about nearly everything. Once the ideas click, you see them hiding inside every societal decision we make. The central idea is that every possible judgment call makes two kinds of mistakes. It can punish, exclude, or alarm someone who doesn’t “deserve it,” a false positive, or it can fail to identify someone who does, a false negative. Because no test is perfect, usually increasing sensitivity results in decreasing specificity. The reciprocal is also true, and no test is perfect. Because of this, what we argue about is which mistakes we are prepared to tolerate.

Consider the death penalty. A highly sensitive death penalty would be designed to catch and execute as many guilty people as possible. Cops would have fewer rules of evidence, prosecutors would face fewer procedural obstacles, and appeals upon a guilty verdict would be fast-tracked. This would result in fewer false negatives: fewer murderers would escape lethal injection. But it would inevitably produce more false positives. Innocent people would invariably, sometimes, be executed.

A highly specific death penalty would err on avoiding those false positives. It would demand stronger evidence with due process, provide defendants with extensive procedural protections including appeal after appeal, and require the government to prove guilt beyond a reasonable doubt. Some murderers would go free, but those who were executed would be more likely to be guilty. The familiar idea, to which I ascribe, is that it is better for guilty people dodge the hangman’s noose than for one innocent person to die is. That is actually an argument for specificity.

The same tradeoff appears in social welfare programs, like disability benefits. A sensitive disability system would be one in which anyone unable to work receives support. The tradeoff is that it must accept that some people who could probably work will also obtain benefits. A false positive looks like someone receiving assistance who does not truly need it. A specific system imposes stricter documentation requirements, more frequent reviews, and a higher burden of proof. This swats away the false negatives (someone seeking to fraudulently obtain benefits, for example), but means some people who truly need assistance cannot obtain it.

Much of American politics can be understood as an argument over these competing errors. Conservatives favor sensitivity in punitive measures like incarceration and would enable it with broader surveillance, aggressive enforcement, and rules to ensure offenders do not slip through the cracks; liberals favor specificity. For them, the reverse is true when it comes to social welfare: conservatives want specificity when distributing social benefits and are intensely concerned that an able-bodied person might receive disability payments, or that an undeserving family might obtain food assistance.

Liberals view the coin as flipped, accepting some fraud or overpayment as the price of ensuring that vulnerable people are not abandoned and apply specificity to crime: accepting that some guilty people will avoid punishment because the alternative is a system that imprisons more innocent people. The difference, most of the time and campaign propaganda aside, is that both sides fear harming people with the state’s power, but they do not agree on which person needs protection. Is it the taxpayer? Or the person too poor to pay taxes? Is it the innocent man charged with murder? Or the family of the murder victim?

Once you recognize this pattern, political arguments become more honest. The dispute over voting rules is not simply between people who favor election integrity and people who oppose it. One side is more alarmed by false positives, invalid votes being accepted; while the other is more alarmed by false negatives, eligible citizens being prevented from voting. Is it worse to grant asylum to someone in danger or occasionally admit someone dangerous? COVID was about whether it is worse to harm someone’s freedom and livelihood or to harm someone’s health? They’re arguing about their understanding of principles, but the real principle argued over is the tolerance for error.

There is another thing I learned in medical school that is relevant: the performance of a test depends partly on the population in which it is used. When something is a rare occurrence, all tests may produce more false positives than true positives. This translates directly into policy beyond healthcare. Because voter fraud is exceedingly rare, a broad system designed to detect it may burden many legitimate voters for every fraudulent vote it prevents. What gets interesting is the debate over whether this, the false detection of “fraudulent voters,” is, in fact, the point.

The best response to any of this, instead of placing moral arguments inside Medicare eligibility, is not just louder noise about where to place the threshold. It is to make the test better. In medicine, a more accurate test can increase sensitivity and specificity at the same time. A CT scan is, in fact, usually better than an X-ray (not always, though). It can identify more problems better, and it can identify the lack of a problem better. Sometimes, an anxious hypochondriac who keeps coming back to the ED over and over again just gets a CT scan so we can put their anxiety over vague, unrelated symptoms to rest. Public policy, in theory, should be able to pursue the same goal. It’s not insane to imagine a world in which people who need VA disability benefits get them, and people who don’t, do not.

Sensitivity and specificity began for me as medical-school vocabulary but became a way of understanding the world. So much so that I’d argue they ought to be taught to everyone in high school or even earlier. Because once you get it, you see it everywhere. You won’t ever unsee it.