RAG vs fine-tuning for ICD-10 medical coding

Every team building on LLMs hits the same fork. You have messy real-world text and you need a precise output, and you can either retrieve and rerank against a known answer set (the RAG, or retrieval-augmented generation, playbook) or fine-tune a model to produce the answer directly. The reflex is to fine-tune, because it feels like the serious move. This post is one bounded task, worked every way, to show when RAG beats fine-tuning and when it does not.

The task is automated medical coding: a clinician writes “acid reflux, no esophagitis on scope,” and somebody turns that into K219 so the claim can be billed. There are about 74,000 ICD-10-CM codes. It is high volume, repetitive, bounded, and wrong answers cost money, which is exactly the profile where teams reach for an expensive solution and pick the wrong one. I ran the task through every credible approach, generation, retrieval, reranking, an LLM selector, and five kinds of fine-tuning, on one eval measuring accuracy, latency, and cost. Here is the decision rule and the numbers behind it.

The decision rule, up front

After running this task as generation, retrieval, retrieval plus rerank, retrieval plus an LLM selector, and five flavors of fine-tuning, the rule that fell out is simple:

• Output space is a fixed, enumerable catalog you control (codes, SKUs, tariff lines, ontology entries)? Start with retrieval plus an off-the-shelf reranker. It is the only approach that structurally cannot emit an invalid answer, it needs no labels, and it runs local and cheap. On this task it lands around 88% exact-match with two small models and no training.
• Siblings need real reasoning to disambiguate (left vs right, “with” vs “without,” one digit of specificity)? Put a capable model in the selector seat: hand it the retrieved candidates and let it pick. The surprise was that a bigger off-the-shelf model used zero-shot beat every small model I fine-tuned. On the adversarial set this stops being a luxury: for ruled-out diagnoses and laterality the off-the-shelf reranker fell to 38% and 47%, while the same open 12B in the selector seat hit 100% and 95%. Capacity bought more than tuning did.
• Reach for fine-tuning only after you have proven the off-the-shelf baseline is the bottleneck, you have a large representative labeled set, and the task format is something a pretrained model genuinely does not already do. None of those held here, and every fine-tune I ran failed to clearly beat the model I never trained.

The rest of the post is the evidence for that rule, including the failures along the way.

The data is public, clean, and untainted

ICD-10-CM is published by CMS and is public domain: about 74,000 rows of code, description, 71,704 usable after dropping headers and blanks. No model-generated text in the pipeline, which matters commercially: a dataset distilled from GPT-4 or Claude output carries that model’s usage terms into your product. Government classification data does not.

E119  :: Type 2 diabetes mellitus without complications
I10   :: Essential (primary) hypertension
N390  :: Urinary tract infection, site not specified
K219  :: Gastro-esophageal reflux disease without esophagitis

That catalog is the entire knowledge base for retrieval and the training target for generation.

How I built the eval

I authored 90 cases by hand as clinician-style paraphrases mapped to a gold code, spread across 18 ICD chapters. “55-year-old with type 2 diabetes, well controlled, no complications noted” maps to E119. The official descriptions never appear, so a model cannot win by string-matching the code book, and I validated every gold code against the catalog (which caught two of my own slips: a laterality flip and a “with/without” flip). To keep the comparison fair I ran every model blind, giving it only the query and the candidate list, never the gold.

This is a proof-of-concept set, not a clinical benchmark. Ninety hand-built cases are enough to see the structural results and the cost ratios, not to certify a coder for billing. For that you want a human-coded gold set, ideally on real multi-problem notes. To pressure-test the easy single-concept setup before drawing conclusions, I also built a deliberately adversarial 103-case set (negation, laterality, ruled-out diagnoses, long-tail), reported in full below.

Generation is the wrong shape (and this part is robust)

Frame coding as text generation: prompt a small model with the description, train it to emit the code, mask the prompt so the loss covers only the code tokens. distilgpt2 SFT scored 0% exact-match, and the failure mode is the real finding: it emitted a code-shaped token for all 90 queries, but 85 of those codes do not exist in ICD-10-CM. It invents E133411 for type 2 diabetes where the answer is E119.

This is not a small-model artifact, and it is the one place I will lean on absolute numbers, because the gap is enormous (85 of 90, not 3 of 90). Generating over a 74k open vocabulary is unbounded, and the code digits carry no matchable signal. Anything that generates a code can write one that does not exist. Anything that selects from a catalog cannot. That structural fact is the spine of the whole decision.

Retrieval, then rerank

The answer already exists in 71,704 descriptions, so look it up. Embed every description once, embed the query, take nearest neighbors by cosine similarity. EmbeddingGemma on Apple MPS:

index = model.encode_document(descs, normalize_embeddings=True)  # 71,704 x d, once
qv = model.encode_query([query], normalize_embeddings=True)[0]
top5 = (index @ qv).argsort()[::-1][:5]

The bi-encoder scores 81.1% exact, 95.6% category, 57 ms per query, no training, and it cannot return a code outside the catalog. Its top-1 is 81%, but its top-5 holds the right code 97.8% of the time and top-50 holds it 100%. The answer is almost always there, just not first, because the bi-encoder embeds query and code separately and blurs the distinctions that decide the code.

So rerank the top 5 with a cross-encoder that reads query and candidate description together (bge-reranker-v2-m3, 280M, also no training):

cand = retrieve(query, k=5)                       # 5 real codes, 97.8% recall
scores = cross_encoder.predict([(query, c.desc) for c in cand])
best = cand[scores.argmax()]                      # ~40 ms, never leaves the catalog

That lifts exact-match from 81.1% to 87.8%. Note the calibration honestly: 87.8 minus 81.1 is about six cases out of 90, right at the edge of this eval’s resolution. It is directionally clear and mechanistically sensible (joint scoring catches siblings), but if it had come out 85% I would not be surprised. Reranking only the top 5 mattered: at top-20 the same reranker scored worse, because more candidates give it more chances to demote the right one.

The frontier, and what retrieval does to it

I had Claude Opus 4.8 code all 90 cases blind, with no access to the answer key. Generating freely it scored 84.4% exact, 96.7% category, with 7 of 90 codes invalid ICD-10-CM strings. So at n=90 the off-the-shelf local reranker (87.8%) and free-generation Opus (84.4%) are a statistical tie on accuracy, three cases apart. The honest claim is not “300M model beats the frontier.” It is: the local reranker is in the same accuracy band, costs orders of magnitude less, and structurally cannot produce the invalid codes the frontier model does.

Now constrain the frontier model instead of replacing it. Retrieve the top 20 real candidates and ask Opus to pick from that list:

cand = retrieve(query, k=20)                       # 20 real codes, 98.9% recall
chosen = claude_pick(query, cand)                  # selects from the list, cannot invent

The same model that scored 84.4% generating freely scores 93.3% exact, 98.9% category as a selector, with every answer a valid code. That jump (roughly eight cases) is large enough to clear the noise, and it is the most reliable finding in the post.

Can fine-tuning beat off-the-shelf? Five attempts, tested

The obvious next move is to fine-tune a small local model to match the selector and drop the API entirely. I tested that thoroughly, five different ways, and every one lost to the off-the-shelf cross-encoder I never trained. How they lost is the useful part: it tells you when fine-tuning earns its keep and when it just burns budget.

• Naive cross-encoder on catalog hard negatives, official description as the query: 65.6%. The loss collapsed in one epoch because it learned to match the code book’s wording, which real clinical text does not use. Clearly worse, and the gap (over 20 points) is real.
• Distilled cross-encoder, using Opus to write realistic paraphrase queries (580, then a scaled 2,306-example set, with LoRA to limit forgetting): 82.2% / 82.2% / 81.1%. All below 87.8%, but every one of these is inside the noise band of the off-the-shelf number. The honest statement is “no improvement,” not “strictly worse.”
• Generative selector on a real GPU (Modal): Qwen-0.5B SFT on the full catalog went from a base 16.7% to 74.4%, and Gemma-4 E2B (about 5B params, H100) reached 85.6% with full SFT. Real lifts, the laptop 0% was compute starvation not a dead end, but a 5B generative selector still did not clear a 0.3B off-the-shelf reranker.
• DPO on the exact failure mode: 15,000 preference triples preferring the gold code over its confusable sibling. It scored 84.4%, identical to the SFT baseline it started from. It learned the training siblings perfectly (99.4%) and did not transfer.

The turn: distrust the benchmark, and disclose how

When a “solved” problem plateaus across that many attempts, suspect the labels before the model. The best model’s “errors” were often defensible: J00 (common cold) against my J069, F411 (generalized anxiety disorder) for a query that literally said “generalized.” So I rebuilt the gold as multi-valid: any clinically defensible code counts.

Two design choices keep the cleaned gold trustworthy. The adjudication was done by Opus 4.8 in one clean blind pass over the candidate pools (it sees only the query and the pool of real codes, never which system picked what), not by human medical coders:

1. The judge never grades itself. I report the systems Opus did not produce: Haiku 4.5 as the reference selector, the open 12B and 32B models, and the retrievers. For those, the judge is a stronger, different model than the system under test, which is the standard LLM-as-judge setup. The one cell I throw out is Opus grading Opus, circular by construction.
2. The judge is deliberately strict. It accepted a second code on only 7 of 90 queries (average 1.08 codes per query), well below the looser first pass I ran. A symptom code does not count when a diagnosis is documented, and an explicit “unspecified” in the query rules out the more specific siblings. Strictness is the honest direction when the worry is inflating scores. The full judgment and scorer are committed (icd_judge_opus.py, accepted_opus.json).

One caveat survives both choices: an LLM calling another model’s answer “defensible” is a soft signal, not a billing department signing off. With the Opus-judged-Opus row set aside:

System (judge: Opus 4.8)	strict (my single gold)	any-valid (Opus, strict)
Haiku 4.5 selector (reference)	92.2%	96.7%
gemma-4-12B selector (open, self-hosted)	93.3%	98.9%
Qwen2.5-32B selector (open, self-hosted)	94.4%	96.7%
cross-encoder (off-the-shelf)	87.8%	90.0%
bi-encoder retrieval	81.1%	85.6%
Opus 4.8 selector (circular, excluded)	93.3%	98.9%

The real takeaway is not any single number. It is that my single-gold exact-match had four to five points of self-inflicted label noise, my gold systematically defaulted to “unspecified” codes even when the query stated specificity, and several “fine-tuning failures” were partly an artifact of grading against loose labels. The benchmark was a meaningful part of the ceiling.

The lever was capacity, not fine-tuning

So the question changed from “how do I fine-tune a small model up to the frontier” to “the selector task is easy for any capable model, what is the smallest capable model I can run locally with no API and no training?” Haiku already solved it zero-shot. So I ran a bigger open model zero-shot as the selector, fully self-hosted:

Local open model, NO fine-tune, NO API	strict	any-valid (Opus 4.8, strict)
gemma-4-12B-it (zero-shot)	93.3%	98.9%
Qwen2.5-32B-Instruct (zero-shot)	94.4%	96.7%

Note the open 12B lands at 98.9%, not 100%, under the strict clean judge: one genuine miss, which is far more credible than the round number a looser gold gave. The defensible claim is narrow and still useful: a 12B open-weight model, handed retrieved candidates, lands in the same band as the frontier API I had treated as the ceiling, on a GPU I control, with no training. At n=90 I cannot distinguish gemma-4-12B, Qwen-32B, Opus, and Haiku as selectors; they are all within two or three cases. Every small model I fine-tuned, I can distinguish, and they are worse. The lever was capacity used zero-shot.

Beyond the 90: an adversarial eval that actually hurts

The 90-case set is clean. The honest worry is that everything above scores well because the queries are easy. So I built a second eval designed to break things: 103 cases as minimal pairs, where only the discriminating cue changes, across negation (“with esophagitis” vs “no esophagitis”), ruled-out diagnoses (“chest pain, MI ruled out”), laterality (right / left / bilateral), acute vs chronic, specificity, and long-tail codes. Minimal pairs are the point: if a system returns the same code for both members, it is blind to the cue, and I can measure that directly. Every gold code is validated against the catalog, and retrieval recall@20 is 97.1%, so the selector almost always has the right code available to pick.

This is where the easy-eval shine comes off. The off-the-shelf reranker drops from 87.8% on the clean set to 75.7% here, and the damage is concentrated in exactly the cues a clean eval hides:

Pipeline / selector (zero-shot)	overall	laterality	ruled-out	negation	long-tail	acuity	specificity
cross-encoder rerank (off-the-shelf, local)	75.7%	47%	38%	77%	80%	100%	100%
Claude Haiku 4.5 (frontier)	89.3%	74%	100%	81%	95%	100%	100%
Qwen2.5-32B (open, self-hosted)	91.3%	84%	88%	85%	95%	100%	100%
Claude Sonnet 4.6 (frontier)	95.1%	84%	88%	100%	95%	100%	100%
gemma-4-12B (open, self-hosted)	96.1%	95%	100%	92%	95%	100%	100%
GPT-5 (frontier)	96.1%	89%	100%	96%	95%	100%	100%
Gemini 3.5 Flash (frontier)	98.1%	100%	100%	96%	95%	100%	100%

The bi-encoder alone trails at 71.8% overall (laterality 53%, ruled-out 12%). Per-family n is small: laterality 19, ruled-out 8, negation 26, long-tail 20, acuity 12, specificity 18.

Three findings, and they sharpen the decision rule rather than soften it. First, the reranker is genuinely good at some cues and genuinely bad at others: it nails acute-versus-chronic and specificity (it reads “with” versus “without” fine, zero cue-blind pairs), but a cross-encoder is a similarity model, so it cannot reason that “MI ruled out” means do not code the MI, and it blurs left versus right because both descriptions are near-identical. Second, every capable selector fixes exactly those two weak spots: the worst selector in the table still takes ruled-out diagnoses from 38% to 100%, and laterality from 47% into the 74 to 100% range, because reasoning is what those cases need. When disambiguation needs reasoning, the selector is not a small upgrade over rerank, it is the difference between 38% and 100% on the cases that matter.

Third, and this is the part I did not expect: across six independent selectors spanning open weights and three frontier vendors, the open self-hosted gemma-4-12B (96.1%) beats Claude Haiku 4.5 (89.3%) and Sonnet 4.6 (95.1%), ties GPT-5 (96.1%), and trails only Gemini 3.5 Flash (98.1%). The cheap, smallest frontier model (Haiku) is actually the weakest selector here, still a huge gain over rerank but below a 12B you can run yourself. Once the candidate list is good, the selector task saturates and vendor and open-versus-closed barely separate. That is the whole argument for self-hosting the picker.

A couple of things to keep in mind: per-family counts are small (ruled-out is 8 cases, laterality 19), so read these as directional. Because I wrote the eval, I ran every selector blind, handing each model only the query and the candidate list and never my gold. I also tried Opus 4.8 as a selector and it scored 100%, but since I wrote the answers that is an upper bound, not an independent number, so I keep it out of the table.

Cost and latency

This is where the decision actually gets made, and where the numbers are robust because they are ratios, not 90-case accuracies. Estimates, single machine, not a rigorous benchmark:

Approach	Exact	Invalid codes	Latency	$ / 1,000 codes
Bi-encoder retrieval (local)	81.1%	0	~57 ms	~$0.0017
Retrieval + cross-encoder rerank (local)	87.8%	0	~96 ms	~$0.003
Retrieval + open 12B selector (self-hosted)	93.3%	0	GPU-bound	GPU cost only
Retrieval + Claude Opus 4.8 selector (API)	93.3%	0	network	~$1
Claude Opus 4.8, free generation (API)	84.4%	7 / 90	network	~$1.20

Frontier list prices for reference: Haiku 4.5 $0.24/1k, GPT-4o$0.57/1k, Opus 4.8 $1.20/1k. The local reranker is two to three orders of magnitude cheaper per code than any API path, with no network and no key. Free-generation Opus is the worst cell in the table: most expensive, not more accurate than the local reranker at this n, and the only option that invents codes.

When to fine-tune anyway

To not be glib about it, fine-tuning is the right call when the conditions this task failed to meet are actually met:

• You have proven off-the-shelf is the bottleneck. Here it never was: a downloaded reranker and a downloaded 12B model already topped everything I trained.
• You have a large, representative, well-labeled dataset, not a few hundred synthetic paraphrases of the catalog. Real distillation needs scale and a matched case mix.
• The task format is genuinely novel to the base model: an output structure it does not know, domain jargon it truly has not seen, a constraint that cannot be expressed as “pick from this list.”
• You need to compress a known-good large-model behavior into a smaller, faster, cheaper, or air-gapped model for production scale, and off-the-shelf small models underperform. Fine-tuning for cost and latency, after establishing the ceiling, is legitimate. Fine-tuning to discover the ceiling is the mistake I made.

For a bounded lookup with strong off-the-shelf retrieval, none of these hold, and retrieval plus rerank plus an off-the-shelf selector is the answer.

Three gotchas

What I would ship

Retrieve candidates, then select from them. Never generate a code from a blank prompt, and do not fine-tune a small model to discover whether off-the-shelf is good enough, run the off-the-shelf baseline first. The reranker alone is fine if your cases are mostly acuity and specificity, but the adversarial set is clear that once laterality and ruled-out diagnoses are in play you need a reasoning selector, and a self-hosted open 12B does that zero-shot for the price of the GPU. Use a frontier API only if you would rather not run the GPU. Either way, route the close calls (top two candidate scores near each other) to a human.

The next phase is the harder one: multi-problem notes, where several findings (some negated or historical) map to a set of codes, and a concept-extraction step has to decide which findings fire a code at all. That is where end-to-end error actually lives, and it is the one part of this problem I have not measured yet. The single-concept coding step, the part everyone assumes needs a frontier model, is a solved lookup.