I gave it 100 ugly invoices. Three ways my eval was wrong.
Zero false auto-approves and 92% accuracy. The useful part was discovering my measurement was broken three separate times, twice while every single test stayed green.
Shrikar Archak
Long-form writing on AI-native development, fine-tuning, agents, evals, and Claude Code.
Latest posts
I built an AI employee that runs accounts payable
An agent that reads invoices, matches them against purchase orders, and decides. Exactly one step uses a model. Everything that makes it employable is deterministic code, and a test enforces that.
Why document collection is the perfect agentic AI workflow (the chase loop)
Mortgage files, insurance claims, and tax prep reduce to the same chase loop. I built an agentic AI workflow to run it: the agent owns the loop, a human in the loop owns the gates, and autonomy is earned gate by gate.
The agent reliability ladder: what hardening buys depends on the model
I built a tool-calling support agent from scratch plus a trajectory eval, then stacked five small reliability fixes and tested them on three models. What each fix buys depends entirely on how good the model already is.
RAG vs fine-tuning for ICD-10 medical coding
A tested answer to RAG vs fine-tuning on ICD-10 medical coding: retrieval, an off-the-shelf reranker, and a zero-shot LLM selector beat every fine-tuned model on accuracy and cost. Real numbers and the failures.
Post-training a model from scratch: SFT, DPO, and RLHF on your laptop
SFT, DPO, RLHF, PPO, GRPO get talked about more than they get run. Here is a tiny working pipeline that does all three on an 82M model on CPU, with real numbers and a reward-hacking gotcha.
Programmatic SEO without the slop: build the gate, not the generator
Generating 10,000 pages with an LLM is trivial and a trap. The hard part is not shipping slop. Here is a measurable quality gate, five signals, that rejects generic AI content before it publishes.
From ML engineer to AI-native: reskilling toward an edge
The undifferentiated middle of ML engineering is getting squeezed. Your data rigor still wins, the experimentation loop gets a force multiplier, and here is a concrete 90-day plan to move.
Claude Code GitHub Actions: an AI agent in your repo, not your editor
A practical guide to Claude Code GitHub Actions: wire it up in two minutes, get issue-to-PR automation and AI code review on every pull request, plus the architecture, config, and gotchas.
Semantic IDs for vulnerable code: finding 100× more cross-project clones than VUDDY
Learned RQ-VAE Semantic IDs for C/C++ vulnerability clones. Borrowing the TIGER substrate from recsys: on a 5000-function CVE registry SecSid finds 112 cross-project clones; VUDDY finds 1.
Claude Code skills vs subagents vs hooks vs plugins vs MCP: a mental model that finally sticks
Claude Code's five primitives — skills, subagents, hooks, plugins, MCP servers — classified on three axes that matter: lifetime, trigger, ownership. Decision rule, misclassifications, repo cheat sheet.
Prompting, context engineering, RAG, fine-tuning, CPT — when to use which
A decision guide for the five layers of LLM adaptation in 2026, the diagnostic question that actually picks the right one, and the production mistakes that send teams down the wrong path.
Beating Claude Sonnet at document extraction with an 8B model
Fine-tuning Llama 3.1 8B on 1,465 Bills of Lading to beat Claude Sonnet 4.5 on field accuracy — 45× faster and 18× cheaper — plus the brittleness trap that almost wrecked it.
Llama 3.1 8B QLoRA on Bill-of-Lading extraction — full report
Reproducible fine-tuning report: QLoRA r=16 on 1,465 BOL examples, ablations against Claude Sonnet 4.5 and Gemini 3.1 Flash Lite, adversarial test results on 920 layout-diverse examples, and the brittleness recovery.
Beating Claude Sonnet at document extraction with a fine-tuned 8B
Fine-tuned Llama 3.1 8B for 18-field Bill of Lading extraction, hitting 99.6% field accuracy and 89 ms latency — 45× faster than Claude Sonnet 4.5, 18× cheaper, 7 percentage points more accurate on the same test set.
The brittleness trap I almost shipped
A fine-tuned model that scored 100% on the in-distribution test set collapsed to 39.5% schema compliance on layout-diverse inputs. Why your standard test is lying to you, and the methodology that catches it.