Are You Improving the Judge or the Agent?

Forty minutes into a working session with a regulated enterprise AI team, someone drew a box on the shared screen and called it the eval dataset. One box.

Everything around that box was in decent shape. Real use case, real model, real pipeline, and a list of eight evaluator names lifted off a docs page: correctness, groundedness, toxicity, tone, tool-use accuracy, refusal rate, response-time compliance, hallucination. What the room did not have was an answer to the question I ask before any of that. Which single failure is your first evaluator there to prevent, and who eats that failure when it happens?

Then the second question, which is the one that stopped the conversation for a full minute: is that box meant to make the agent’s answers better, or to make the grader’s verdicts better?

Those are two different datasets. Same source material, possibly. Different inputs, different expected outputs, different definition of done. I have watched that conflation happen once, in one room, and once was enough to make me write down the model I had been carrying around in my head for months.

Almost every team I talk to about evaluation arrives with a list of evaluator names. The list is never wrong, exactly. Correctness matters. Groundedness matters. Tone matters in the products where tone is the product.

The list just does not tell you what to build on Monday, because a list is not a decision. It is a menu with no ordering logic, and the failure mode is predictable: the team builds four mediocre evaluators instead of one that actually protects something, and then nobody trusts any of the four.

Most teams do not have an evaluator shortage. They have an evaluation-system design problem.

The gap is four explicit decisions that almost nobody writes down. I call the set of them the Evaluation Operating System, which is a grand name for a very short table.

Four rows. Every evaluation program I have seen stall was missing at least two of them, and the fourth one is the one that gets skipped most often because it is the only one that costs human hours forever.

The rest of this issue walks the four in order.

The most common blocker I hear from pre-production teams is that they cannot start evaluating until they have volume. That is backwards, and it is expensive backwards, because the whole point of evaluating before launch is that fixing a definition is cheap and fixing a reputation is not.

Before production, prioritize from user risk and stakeholder requirements. Five steps, in this order:

1. Define the minimum acceptable behavior for this use case in one sentence. Not the aspirational behavior. The floor.

2. Identify who consumes the answer. A customer, an internal analyst, another system, a regulator reading a transcript six months from now.

3. Identify the failure that matters most to that consumer. Not the failure that is easiest to measure.

4. Build your first evaluator around that failure.

5. Add dimensions only when evidence says they matter.

After production, add two more inputs: trace error analysis and incident frequency. Once real traffic exists, your priority list stops being a risk hypothesis and becomes an observed distribution. That is a genuinely better input, and it is also the reason a pre-production priority list should be treated as a draft rather than a commitment.

And you do not need production scale to run a real experiment. Define the behavior you want to test, seed a dataset from synthetic examples or domain-expert answers, hold that dataset constant, change exactly one configuration axis at a time, then compare results and read the failures. That is an evaluation experiment. It works on 40 items. Most teams waiting for volume are waiting for permission they already have.

For the leader reading this: the useful question in the next review is not “how many evaluators do we have.” It is “which consumer does our first evaluator protect, and what happens to that consumer if it fails.” If nobody in the room can answer in a sentence, the metrics list is decoration.

The second decision is mechanism, and it is where budgets quietly leak.

An LLM judge is a genuinely good tool for the things it is good at. It is also the most expensive way in the world to check whether a required field is present. Decompose the requirement first:

The stance is not “avoid LLM judges.” I use them. The stance is narrower than that:

Use a judge for judgment. Do not pay a judge to rediscover facts your code already knows.

One more mechanism choice hides inside this one, and it is the score type. Enumerate the possible outcomes before you pick. Two states means boolean. A stable set of named states means categorical. A range where the middle values carry real meaning means numeric. The score type should follow the decision the score informs, not the shape of the chart you want on the dashboard.

This is the part that stopped the room, and it is the part I would put on a wall.

Evaluation work has two distinct targets. Improving the agent is one job. Improving the judge that grades the agent is a completely different job. They need different dataset contracts.

The teaching analogy I use: if you are training an intern to write good memos, you collect examples of good memos. If you are training a manager to grade memos fairly, you collect memos plus the grades a trusted expert gave them. Nobody would confuse those two piles of paper. In an evaluation platform they look identical, because both of them are just rows with an input field and an expected-output field.

Here is what the two contracts look like as actual dataset items. Both examples below are fully synthetic, from an invented internal expense-policy assistant.

Agent-improvement contract. Task and context in, ideal answer out.

dataset_contract: agent-improvement
item_id: agent-eval-0001
input:
  task: >
    Employee question: "Can I expense a $42 client dinner without an
    itemized receipt if it is under the $50 threshold?"
  context: >
    Company travel and expense policy, section 4.2: meals under $50 do
    not require an itemized receipt, but the vendor name, date, and
    business purpose must still be logged in the expense system within
    30 days of the expense.
expected_output:
  ideal_answer: >
    Yes. Under policy 4.2, meals under $50 do not need an itemized
    receipt. You still need to log the vendor name, date, and business
    purpose in the expense system within 30 days.
evaluators_applied:
  - answer_correctness
  - policy_citation_accuracy
  - task_completeness

Judge-improvement contract. Task plus a candidate answer in, correct verdict out.

dataset_contract: judge-improvement
item_id: judge-eval-0001
input:
  task: >
    Employee question: "Can I expense a $42 client dinner without an
    itemized receipt if it is under the $50 threshold?"
  candidate_answer: >
    "Yes, dinners under $50 never need a receipt or any extra logging.
    You are fully covered."
expected_output:
  verdict: fail
  score: 2
  score_scale: "1-5 integer, 5 = fully correct and complete"
  rationale: >
    The receipt threshold is stated correctly, but the answer drops the
    mandatory 30-day logging requirement from policy 4.2. That omission
    creates audit risk even though the reimbursement itself is valid.
    The candidate answer is incomplete, not simply wrong.
evaluators_applied:
  - judge_agreement_with_domain_expert
  - rubric_calibration

Read the two input blocks side by side. The agent contract has no candidate answer in it, because the agent’s job is to produce one. The judge contract must have a candidate answer, because grading nothing is not a task. And the judge item’s expected output is a verdict with a written rationale, which is the part teams skip and then wonder why their rubric drifts.

Three things this does not mean.

It does not mean you need two databases. These are contracts, not storage mandates. Separate datasets, separate views over one corpus, or two schemas in one platform are all fine. The distinction is the job, not the disk.

It does not mean you only evaluate final answers. Evaluate the final answer for user-facing qualities like tone and final-response compliance. Evaluate intermediate steps for tool selection, tool arguments, safety, and workflow correctness, which is also where most of your debugging value lives. Evaluate the judge when the question is whether your grading process is calibrated. Three different observations, three different reasons.

And none of this is new thinking. Judge quality has been a separately-benchmarked concern in the research literature for a while, and there is good published practitioner guidance on calibrating judges against human annotations. What I have not found is a version packaged so a team can act on it in an afternoon. That packaging is the part I am claiming.

The fourth decision is the one that never finishes, which is why it gets dropped from plans.

A judge that grades your agent needs its own ground truth, and ground truth comes from people who know the domain. The failure mode is obvious once you name it: you send the domain expert the raw trace firehose, they review 30 items, they stop answering the Slack channel, and the calibration loop dies of exhaustion by week three.

So do not send them the firehose. This is how I built my own review queue, in three buckets:

1. A random sample for baseline coverage, so you learn what normal looks like rather than only what broken looks like.

2. Failed and low-scoring cases for error analysis, which is the bucket everybody already has.

3. Suspiciously perfect scores for false-assurance checks. A judge returning top marks on every single item is not good news. It is usually a judge that cannot see enough of the input to disagree with anything.

That third bucket is the one I would add first if you already have the other two, but a starved judge does not only fail in the lenient direction. In my own system, a judge that had lost access to enough of the trace did not go easy. It went the other way: it flagged 8 of 12 turns as fabrication, and all 8 flags were wrong, because the missing tool calls were the exact evidence that would have cleared the agent. I wrote about what that cost me last week. Same root cause as the suspiciously-perfect problem, opposite symptom: a judge that cannot see enough of the input does not know what it is missing, so it can swing to false alarms just as easily as false confidence. Both failures trace back to the same finding: audit what your judge can actually see before you trust what it says.

For scoring, I run machine and human scores side by side under mirrored names: accuracy and human_accuracy, compliance and human_compliance, groundedness and human_groundedness. That is my convention, not a platform requirement, and the reason it is worth the extra config is that the gap between the two columns is the only honest measure of whether your judge is calibrated. Corrections humans make can then become golden-data candidates, after review, which closes the loop back to the certification pattern I published in July. That issue covered promoting traces into golden datasets. This one adds the target separation that tells you which dataset a promoted trace belongs in.

One honest caveat, because I have seen this oversold. A curated queue does not reduce your review workload by some clean multiple. The random-sample bucket is small, but the failure and anomaly buckets add volume of their own. What changes is not the amount of review. It is that the review is aimed at the cases that can actually change a decision.

📌 SAVE THIS: “Improving the agent and improving the judge are different jobs. Each one needs its own dataset contract.”

How to use this right now:

1. Write the one-sentence risk statement. Name the single failure your first evaluator exists to prevent and the consumer who eats that failure. If nobody can write that sentence, you are not ready to pick a metric, and no additional evaluator will fix it.

2. Label your existing dataset by contract. Go item by item. Does this row train the agent to answer better, or teach the judge to grade better? Items that satisfy neither contract are not data, they are backlog.

3. Run the three-bucket queue for one week. Ten random items, everything that failed, everything that scored perfect. Have a domain expert score the same items your judge scored, then look at where the two disagree. That gap is your first calibration finding, and it will be more useful than the next four evaluators you were planning to build.

A judge that never disagrees with you is not a judge. It is a mirror with a score attached.

Wednesday on LinkedIn: which evaluator to build first when you have zero production traffic and eight metric names on the whiteboard, which is the decision that comes before everything in this issue.

If you’re going to use this framework, I want to hear about it. Reply and tell me what you’re working on.

New here? Signal over Noise publishes one issue a week on making AI work in production. The rest of this series lands in your inbox: the evaluator-selection decision, the human calibration loop as an operating model, and the implementation when the build is ready to show. Subscribe here.

Quick hits from this week in enterprise AI:

↑ Anthropic disclosed three eval-environment containment incidents. The stated root cause was a misconfigured boundary at a third-party evaluation partner rather than any clever model behavior, which is a reminder that the environment you use to measure agent safety is production and should be governed like it.

👀 Braintrust released behavior specs and argued LLM-as-judge is losing effectiveness for agents. Read it against today’s issue: the claim is that an agent’s ground truth is how it should behave rather than what it outputs, which adds a third contract instead of replacing the two above.

↑ LangChain published how it evaluates its own internal agents. The interesting part is the pipeline, production trace to candidate task to scored eval, which is the maturity path teams skip when they buy tooling before they define the decision.

↑ Mastra shipped trace clustering because reviewing traces does not scale. This is what the problem after observability looks like: you instrumented everything, and now nobody can read it.

↑ Langfuse shipped v4 on an observations-first data model. A trace is now every row sharing a trace_id rather than something reassembled at read time, which is what makes step-level evaluation cheap enough to actually run. Read it as a schema argument for today’s issue: what you can afford to evaluate is bounded by what your trace store lets you query.

👀 Karpathy gave a model a million-token budget and got 5,500 lines of code it could not audit. The demo traveled, but the line worth keeping is that models cannot easily check their own work, which makes verification an architecture decision you own rather than one you buy.

👀 Chollet argues test-time adaptation was a patch and program search is the real direction. Useful as procurement framing rather than research adjudication: which capabilities you are buying this year are patches you will re-buy in 18 months?

More Signal. Less Noise.