You’re Applying the Wrong Fix to AI Mistakes

Domain 4 · Lesson 4.3 · Structured Output and Grounded Verification

tl;dr Most people think a wrong answer from an LLM means they need to improve a weak prompt. The ones who moved past that own real mechanisms and fire them at the wrong failures, which costs more because it feels like they have the right architecture in place. There is an order of operations that fixes both.

There are two ways to be wrong about this, and the second one costs more. The first is believing that a wrong answer means you wrote a bad prompt. Something comes back off, so you go back to the instructions and make them firmer. Be accurate. Do not invent values. Double-check the totals before answering. Most people have never tried anything else, and it often works on easy problems enough to keep the habit alive.

You are probably past that. You use a schema. You may have hooks. You have gotten good enough at examples that your extraction survives the ugly documents. So let me aim this squarely at you, because owning the right mechanisms and pointing them at the wrong failures is a worse position than owning none. It comes with the feeling of having solved the problem, and here is what that looks like on a simple example extracting data from an invoice.

In this example, you run your extractor, and three things come back wrong. 1) The tax amount is plausible, and the invoice has no tax line item anywhere on it. 2) The subtotal and the tax do not sum to the stated total. 3) The vendor address is sitting in the bill-to field, and the bill-to address is sitting in the vendor field.

Now do the sophisticated thing. Write the strictest field definitions you can, mark every one of them required, turn on structured outputs so the model is physically incapable of returning anything malformed, and run it again. All three defects come back untouched.

The fabricated tax is a number, and you asked for a number. The arithmetic is wrong, and arithmetic was never part of what you specified. The two addresses are swapped, and both fields hold a valid address, which is the only question a field definition knows how to ask.

Nothing errored, nothing retried, nothing in your logs is red. You have a clean run and three wrong values, and if you measure reliability by whether the pipeline completed, your numbers just went up.

The boundary of what this guarantee covers is the whole argument. The industry name for it is grammar-constrained decoding, which sounds worse than it is. Claude writes an answer one small piece at a time, and at every step it chooses the next piece from everything it could possibly say. A “grammar” here is just the set of rules describing which answers count as well-formed, and yours comes from the field list you defined. Constrained decoding removes every piece that would break those rules from the list before the model chooses. Say you told it the tax amount must be a number, or empty if there is no tax. This invoice has no tax, so the model would like to write the word “none” instead. It cannot. Writing a word starts with a quotation mark, and a quotation mark is not on that list. There is no error and no retry because there was never an incorrect option to choose. You get a number or an empty value every time. Whether the number is correct is a different question with a different solution.

Now read what those rules describe: which fields exist, what type each holds, which are mandatory. Every one is a statement about the form of the answer, and not one mentions the invoice. So the guarantee you bought is that the output is well-formed, and it was never going to be anything else. That feature is working exactly as specified, within a jurisdiction narrower than most people assume when they switch it on.

I should be honest about my own answer here, because I made the expensive version of this about ten minutes after learning the mechanism. Working through this material, I got the fabricated tax immediately. A required field with no legal way to report absence is an instruction to invent, which is the argument I have been making for two lessons. Then I was asked which lesson the other two defects pointed to, and I said they both pointed to the same one.

They are not. The arithmetic and the swap need different mechanisms, and neither is the mechanism that fixes the fabrication. What I had done was take the tool I most recently learned to trust and aim it at everything in front of me, which is the same move as the person adding “be accurate” to their prompt for the fourth time. Better vocabulary, better tooling, identical error.

That is the takeaway here. Prompt-hardening and schema-hardening are not opposites. They are one instinct at two levels of increasing sophistication, and the instinct is reaching before diagnosing. A mechanism used outside its area does not merely fail to help. It erodes your confidence because you tried the real fix and the numbers did not move, and now you are one step away from concluding the model is unreliable, when what was unreliable was your diagnosis.

The rest of this piece outlines the order of operations for fixing each error. How to tell the failure classes apart on sight, which mechanism owns each one, why the anti-fabrication move is weaker than I told you it was two lessons ago, and what the 2026 stack looks like now that tool_use is no longer the recommended path. It ends in a prompt you paste into Claude Code that reads your own failing extractions, sorts each defect by class, and refuses to sell you a schema change for a defect a schema cannot fix. Your pipeline, your documents, your failures, on your machine.