85% Confident Is Not an Explanation

“The model is 85% confident” sounds like useful information until you try to debug it.

Confident about what?

Confident that it read the document correctly? Confident that it classified the document correctly? Confident that the extracted field is the right field? Confident that the value is readable, but not necessarily meaningful in the workflow?

Those are different problems. Collapsing them into one number is tidy, but it is not especially useful. It gives you the feeling of instrumentation without giving you the thing instrumentation is supposed to provide: a path to the next question.

A confidence score can be useful. I am not arguing that every numeric confidence value should be thrown into the sea, although I understand the temptation. Scores are good tripwires. They can tell you when something deserves attention. They can drive thresholds. They can help an eval framework decide which outputs need a closer look.

But a tripwire is not a diagnosis.

That distinction matters when you are building AI systems that extract structured data from messy documents. The model can return a value, attach a confidence score, and still leave you with almost no idea what actually went wrong.

Imagine a document extraction test fails on a numeric field.

Expected:

"45.1"

Actual:

"451"

The system reports that confidence fell below the threshold.

Fine. Now what?

That failure might be a prompt problem. It might be an OCR problem. It might be a source document quality problem. It might be an expected-data problem in the test fixture. It might be that the model did exactly what the extracted text told it to do, because the OCR quietly turned a stray mark on the scan into a decimal point or quietly removed one that mattered.

The digits are the same. The value is not.

That is the kind of failure where a single confidence score is especially weak. The interesting part is not “confidence was low.” The interesting part is that the actual and expected values are close in a way that suggests a specific class of problem: possible scan or OCR artifact.

A useful eval harness should help you get there faster.

The manual debugging loop is familiar. You look at the source PDF or image. You compare the OCR output against the image. You inspect the prompt and response to see whether the model made a bad inference or faithfully followed bad input. Eventually you develop a theory of the failure.

Maybe the scan had a speck that looked like punctuation. Maybe the OCR dropped a decimal point. Maybe the model saw conflicting text and picked the wrong one. Maybe the prompt did not tell it how to handle numeric formatting. Maybe the expected fixture is wrong because someone transcribed it by hand six months ago and, naturally, now you are debugging the ghost of that decision.

A confidence score does not tell you any of that.

It just sits there looking scientific.

Software has been through this numbers game before. Lines of code used to get treated like productivity, which mostly rewarded people for producing more code. Then code coverage became the number everyone could point at, which was better, but still not the same thing as correctness. You can have 95% coverage and still have tests that politely verify the wrong behavior.

Confidence scores have the same failure mode. The number is not useless, but it is very easy to mistake it for understanding.

This is where structured uncertainty is more useful than fake precision. Instead of asking the model or test framework to emit only this:

{
  "field_name": "amount",
  "value": "451",
  "confidence": 0.85
}

Ask for something that actually helps an engineer decide what to inspect next:

{
  "field_name": "amount",
  "expected": "45.1",
  "actual": "451",
  "confidence": 0.85,
  "uncertainty_reason": "possible OCR or scan artifact",
  "diagnosis": "digits match, but decimal placement differs",
  "recommended_debug_step": "compare OCR output against source image"
}

The confidence number is still there. It did not vanish. But now it is not being asked to carry the whole explanation on its back like some overworked decimal mule.

The useful part is everything around the number.

field_name matters because document-level confidence is usually too broad. A document can be easy to classify and still contain one field that is ambiguous. Or the document can be messy overall while one extracted value is perfectly clear. If the failure is field-level, the uncertainty should be field-level too.

expected and actual matter because eval failures are comparisons, not vibes. The shape of the mismatch often tells you where to look. A completely different value suggests one kind of problem. Same digits with different punctuation suggests another. Missing value suggests another. Correct value pulled from the wrong field suggests another.

uncertainty_reason matters because “low confidence” is not a reason. It is a symptom. Was the source hard to read? Was the document type ambiguous? Did the model understand the text but hesitate over what it means in the workflow? Those are separate failure modes, and they should not be handled the same way.

recommended_debug_step matters because the point of an eval framework is not just to announce failure. It should shorten the loop between failure and understanding.

This does not mean every user needs to see a giant uncertainty object in the product experience. Most users do not want that, and they are right not to. This is engineering instrumentation. It belongs in evals, test reports, review queues, logs, and debugging workflows.

The regular product can stay clean. The test framework should be nosy.

There are at least three broad uncertainty categories that come up repeatedly in document extraction systems.

The first is source quality. Sometimes the document is obviously bad: blurry scan, cropped page, low contrast, handwriting from someone who apparently wrote with a cooked spaghetti noodle. But the more dangerous cases are not always obvious. The document looks readable until one character changes the meaning of a field. Is that number 45.1, or is the dot a stray mark and the number is 451? A confidence score might tell you something is off. A structured diagnosis can tell you to inspect the OCR against the original image.

The second is document type ambiguity. If a system is not sure what kind of document it is reading, forcing it through a specialized extraction path can produce very confident nonsense. A better failure mode is to degrade gracefully: fall back to generic extraction, capture what can be captured safely, and avoid pretending the system knows more than it does.

The third is semantic ambiguity. Sometimes the text is perfectly readable, but the meaning is not something the model should decide automatically. This is especially true in workflows where language has operational or legal consequences. The system may be able to quote the clause. That does not mean it should determine the effect of the clause without a rule, a reviewer, or some explicit decision boundary.

These categories require different handling. Bad OCR means inspect the image and extracted text. Document ambiguity means review classification or use a generic path. Semantic ambiguity means stop automation before the system makes a decision it has no business making.

One confidence score cannot express all of that. Not in a way that is useful.

This is also why I am more interested in confidence as a debugging trigger than confidence as a product claim. “The model is 85% confident” can sound reassuring in a demo. In an eval report, it should make you ask the next question immediately: why 85, and what should I do with that information?

If the answer is “we do not know,” then the score is mostly decoration.

The annoying thing is that large language models are actually well suited to giving us better information here. They can compare values. They can describe ambiguity. They can point to evidence. They can say, in plain language, “the digits match, but the decimal placement differs, which may indicate an OCR issue.”

That does not make the model automatically right. The explanation still needs to be treated as model output, not divine revelation delivered via JSON. But it is a much better starting point for debugging than a naked float.

The goal is not to make the model sound introspective. The goal is to make failures easier to investigate.

If the extraction failed because the prompt is bad, I want to know that. If it failed because the OCR produced bad text, I want to know that. If it failed because the document does not fit the expected type, I want to know that. If it failed because the system is being asked to make a semantic decision that should not be automated, I definitely want to know that.

All of those can produce low confidence. Only one of them might require changing the prompt.

That is the practical reason structured uncertainty matters. It prevents teams from treating every model failure as the same kind of model failure. Without that distinction, you waste time tuning prompts for OCR problems, adjusting thresholds for classification problems, or adding model complexity where the correct answer is “do not automate this decision.”

A good eval framework should not only tell you that a test failed. It should help you understand what kind of failure you are looking at.

Confidence can be the start of that process. It should not be the end.

These machines speak our language. We should stop asking them to compress uncertainty into a decimal and then acting surprised when the decimal does not explain anything.