Evaluation Prompt Engineering: Designing Reliable LLM Judges

Article links

Make inline references clickable

In the previous two chapters, we established that LLM judges can serve as scalable proxies for human evaluation and that those judges are susceptible to position bias, verbosity bias, and sycophancy when you do not design your pipelines carefully. We covered how to measure swap consistency, detect when a judge is choosing based on response order rather than quality, and apply mitigation strategies like position swapping and calibration. But there is a problem upstream of all of that. Every mitigation strategy we discussed assumes you already have a working evaluation prompt. The quality of that prompt determines whether your judge measures anything meaningful in the first place.

This chapter is about that upstream problem: evaluation prompt engineering. The core insight is that LLM judges are not oracle systems that assess quality objectively when given any reasonable instruction. They are responsive, sensitive text-completion engines. The phrasing of your evaluation prompt, the criteria you include or omit, the examples you provide, and even seemingly trivial choices like how you label the options all shape what the judge measures. A poorly designed evaluation prompt does not just produce noisy results. It produces systematically wrong results, and it does so in ways that are invisible if you never test the prompt's sensitivity.

Prompt engineering for evaluation differs from prompt engineering for generation. When you engineer a generation prompt, you want to elicit a particular behavior. When you engineer an evaluation prompt, you want to measure a particular property of a response as accurately as possible. The goal is reliability and validity: the judge should give the same verdict on the same pair of responses (reliability) and that verdict should correspond to actual quality on the dimension you care about (validity). Both are harder to achieve than they look.

The challenge runs deeper than you might expect. Many teams building LLM evaluation pipelines spend substantial effort choosing the right judge model, designing the overall pipeline, and collecting human labels for calibration, only to discover that their evaluation results are unstable or poorly correlated with human judgment. When they investigate, the root cause is almost always the evaluation prompt itself. The task description was underspecified. The criteria were redundant or poorly anchored. The output format introduced quantization artifacts. Once they redesign the prompt carefully, the same judge model and the same pipeline produce results that correlate well with human raters and remain stable across reruns. The evaluation prompt is the highest-impact investment in your entire evaluation pipeline, and it is the one that teams most consistently treat as an afterthought.

Before you can design a good evaluation prompt, you need to appreciate how sensitive LLM judges are to prompt variation. The same two responses evaluated with slightly different prompts can produce opposite verdicts. This is not a theoretical concern. It has been demonstrated empirically across judge models, task types, and prompt modifications that span the range from trivial to substantial.

Prompt sensitivity refers to changes in judge output caused by changes in the evaluation prompt that should, in principle, be irrelevant to quality. A judge that is reliable to prompt variation should give the same verdict regardless of whether you phrase the instruction as "evaluate the quality" or "assess the quality," and regardless of whether you ask for a score from 1 to 5 or a score from 1 to 10.

In practice, LLM judges vary in their outputs across a wide range of modifications:

• Wording changes: "Rate the response on fluency" vs. "How fluent is the response?" can produce different distributions of scores, even when the underlying measurement task is identical.
• Criteria ordering: If your prompt lists accuracy, clarity, and helpfulness, the judge tends to weight criteria listed first more heavily than criteria listed last. This is a close cousin of position bias, but applied to the criteria themselves rather than to the responses.
• Instruction tone: Prompts that frame the task as "find flaws in the response" elicit harsher judgments than prompts that frame it as "assess the strengths and weaknesses." The label changes the cognitive posture of the judge.
• Format instructions: Whether you ask for a numeric score, a letter grade, or a categorical label (Excellent / Good / Fair / Poor) changes the output format and the underlying distribution of judgments. Some output formats introduce quantization artifacts that are not present in others.
• Scale anchoring: A prompt that defines the endpoints of a 1-to-5 scale ("1 = completely wrong, 5 = perfectly accurate") produces different results than a scale with no anchors or with different anchor descriptions.

Understanding these sources of sensitivity is the first step toward controlling them. Treating any one of them as fixed and unproblematic is a mistake that will show up in your evaluation results.

The sensitivity of LLM judges to prompt variation reflects properties of how language models are trained. Transformers are trained to predict the next token given all previous tokens. When an evaluation prompt includes specific wording, that wording primes the model's probability distribution over subsequent tokens. This is the same mechanism that makes few-shot prompting effective for generation tasks. For evaluation tasks, it means that the framing of the prompt actively shapes what the judge considers important.

To make this concrete: if your evaluation prompt uses the phrase "professional and polished," the judge's attention is drawn toward surface-level linguistic features associated with professional writing in its training data. That might not be what you care about. You might care about whether the response was correct, or whether it addressed the question. But the phrase "professional and polished" primes the model's internal representations toward stylistic evaluation, not accuracy evaluation, and that shift happens before the model even processes the responses being judged.

Instruction-tuned and RLHF-trained models are additionally sensitive because their training involved human feedback on specific types of prompts. If the prompts used during RLHF training tended to reward certain phrasings or structures, the judge model learns to associate those phrasings with high-quality outputs. Asking the judge to evaluate using very different language may fall outside the distribution of its fine-tuning data and produce less reliable results. This is especially pronounced for smaller judge models: a larger model with more reliable instruction-following may handle unusual prompt phrasings more gracefully, while a smaller model may exhibit more erratic behavior when the evaluation prompt deviates from familiar patterns.

You can quantify prompt sensitivity for any evaluation task by constructing a set of prompt variants that should, in principle, be equivalent, and measuring how much judge verdicts vary across them. The simplest approach uses Krippendorff's alpha or Cohen's kappa, which we introduced in the chapter on inter-annotator agreement, treating each prompt variant as a different "annotator."

Constructing a useful set of variants requires covering different dimensions of variation. A minimal sensitivity test should include at least three types of changes: a wording paraphrase (same meaning, different phrasing), a scale format change (for example, 1-4 vs. 1-5 vs. Excellent/Good/Fair/Poor), and a role description change (same expertise domain, different framing). If your judge produces consistent verdicts across all three types, you have good evidence that the measured verdicts reflect the responses themselves rather than the prompt's framing. If it fails on any type, you have identified which component of your prompt needs redesign.

If the agreement across prompt variants is high (alpha > 0.8), your evaluation prompt is reliable to those particular variations. If agreement is low (alpha < 0.6), you have evidence that the framing of the prompt is driving verdicts, not the quality of the responses. That is a signal to redesign the prompt before using it to compare models. Borderline cases will always show more variance than clear cases. What you are looking for is high agreement on cases you expect to be clear-cut: obviously correct responses and obviously wrong ones should get the same verdict regardless of which variant you use.

Sensitivity measurement should not be a one-time audit. Every time you modify an evaluation prompt, even for seemingly minor improvements, re-run the sensitivity test. Prompt changes that appear cosmetic can shift verdicts in unexpected ways. A good practice is to maintain a small reference set of test cases, ranging from clear-cut correct responses to clear-cut wrong ones to a few authentic borderline cases, and run every new prompt version against this reference set before deploying it. If any clear-cut case changes verdict, you have introduced a regression.

Designing an evaluation prompt that is reliable and valid requires making deliberate choices about five key components: the task description, the evaluation criteria, the response presentation, the output format, and the persona or role assignment. Each component interacts with the others, and getting them right is an iterative process.

The task description tells the judge what it is doing and why. It should specify the source task (what was the model asked to do?), the dimension being evaluated (what aspect of quality are we measuring?), and the evaluator's role (who is the judge in this scenario?).

A weak task description:

Evaluate this response.

A strong task description:

You are an expert evaluator assessing language model responses to factual questions. Your task is to evaluate how factually accurate each response is. A factually accurate response makes only claims that are true and verifiable, cites or implies evidence where appropriate, and does not confabulate details that were not present in the source material.

The stronger version eliminates ambiguity about what is being measured. The word "evaluate" in isolation is underspecified. LLMs are capable of evaluating many properties of text, and without explicit guidance they often weight whatever properties are most salient in their training distribution, which may not match what you care about.

Three elements are consistently worth including in the task description:

• What the model was supposed to do: Include the original user question or task specification. Without this, the judge has no basis for assessing task completion.
• What you are measuring: Be explicit about the dimension (accuracy, fluency, helpfulness, safety, groundedness) and provide a brief definition.
• The standard for judgment: Describe what a high-quality response looks like on that dimension, not just what you are measuring.

The third element is the one most often omitted. "Evaluate factual accuracy" tells the judge what to measure. "A factually accurate response makes only verifiable claims and does not add details not present in the source" tells the judge what the standard is. Without a standard, the judge applies its own internal standard, which is based on its training distribution and may not match yours. The standard clause is especially important for evaluation tasks that are domain-specific or that involve fine-grained judgment calls.

One additional point about task descriptions: they should not describe what a bad response looks like. Focusing the task description on high-quality exemplars helps the judge orient toward positive signal rather than toward catching failures. Prompts that emphasize failure modes tend to sensitize the judge toward that class of errors while underweighting other quality dimensions. This is particularly relevant for safety evaluation, where a prompt that asks the judge to "find unsafe content" will be more sensitive to one type of failure than a prompt that asks it to "assess whether the response is safe for all audiences."

Evaluation criteria are the specific properties the judge should weigh when forming a verdict. Designing good criteria is harder than it looks because criteria must be precise enough to eliminate ambiguity but general enough to apply across the full range of responses the judge will encounter.

Single-dimension vs. multi-dimension evaluation. The most common design mistake is asking a judge to evaluate multiple quality dimensions with a single score. If you ask "Rate this response on accuracy, fluency, and helpfulness" and request a single 1-to-5 score, the judge must aggregate across dimensions using some implicit weighting. That weighting is opaque, unstable, and likely to vary across prompts and judge models. Unless you specifically want an aggregated quality score (and have validated that aggregation reflects your priorities), evaluate dimensions separately and combine them explicitly.

There are legitimate use cases for composite scores. If you are doing a final ranking of models and you have already validated your individual criteria, a weighted composite score is a reasonable summary statistic. But during the design and validation phase, single-dimension evaluation is almost always preferable. It lets you understand what your judge measures, catch criterion drift, and trace unexpected results back to their sources.

Criterion definition. Each criterion should include a name, a brief definition, and examples of high and low performance on that criterion. Vague criterion names like "quality" or "helpfulness" are interpreted differently by different judge instances. Concrete criterion definitions reduce that variance.

Here is an example of a well-specified criterion for factual accuracy:

Factual Accuracy (1-5): 5: All claims are accurate and supported by the source material or world knowledge. The response does not add unverifiable details. 3: Most claims are accurate but 1-2 minor errors or unsupported inferences are present. 1: The response contains major factual errors or fabrications.

This rubric-style definition gives the judge concrete anchors at the extremes and a reference point in the middle. It reduces the ambiguity that makes different prompt phrasings produce different results. The explicit description at each scale point forces the judge to map the response onto a specific definition rather than onto its own prior about what a "3 out of 5" means. Evaluators who use unanchored scales frequently observe that different judges interpret the middle of the scale very differently, with some treating it as "acceptable" and others treating it as "mediocre." Anchoring eliminates that ambiguity.

Criterion independence. If your evaluation uses multiple criteria, check whether they are truly measuring different things. Fluency and grammaticality are highly correlated because grammatical text is almost always fluent and fluent text is almost always grammatical. Helpfulness and task completion frequently overlap because completing the assigned task is the primary way a response becomes helpful. Redundant criteria amplify the weight of underlying dimensions without your knowledge: a rubric that lists fluency, readability, grammaticality, and coherence as separate criteria is effectively weighting surface-level language quality four times as heavily as the rubric text implies.

To test for redundancy, run your judge on a sample of 50-100 responses and compute Pearson or Spearman correlation coefficients between the per-criterion scores. Pairs of criteria with correlations above 0.8 are redundant. Decide which label better captures what you care about and drop the other. If you want to preserve both because they occasionally diverge (there are tasks where a response can be fluent but incoherent, for example), retain them but be explicit in your documentation that they are measuring related constructs. Reviewers of your evaluation pipeline will need that context to interpret multi-criterion aggregations correctly.

The heatmap below shows a typical pattern of criterion correlations for a quality assessment rubric. The grouping of surface-level criteria (fluency, grammaticality, readability) in the upper-left corner illustrates why treating them as independent dimensions overstates their individual contributions to overall quality assessment.

How you present the responses to the judge affects verdict quality in ways beyond position bias. The presentation format includes the order of presentation (A before B, or randomized), the labeling of responses ("Response A" vs. "Response 1" vs. "Option Alpha"), the amount of whitespace and structure, and whether you include the original question inline with each response or only at the top of the prompt.

Labeling effects. Research on LLM evaluation has found that certain labels carry implicit quality associations in the training data of judge models. "Response A" and "Response B" are relatively neutral. But labels like "Expert response" or "Professional answer" can lead the judge to apply higher standards, independent of content. Similarly, "Model A" and "Model B" are neutral, but "GPT-4 response" and "baseline response" are not, because the judge has prior beliefs about GPT-4 that will influence its evaluation. Keep labels neutral and consistent, and do not reveal model identities to the judge.

Original question placement. Including the original user question immediately before each response (rather than only once at the top of the prompt) helps the judge evaluate task completion locally. This is particularly important for evaluation tasks where different responses may interpret the question differently. Seeing the question again primes the judge to ask "does this response answer what was asked?" rather than evaluating the response in a more general sense. For long prompts where the question appears many tokens before the first response, the model may not weight the question context as heavily as it should, a consequence of the attention patterns and context window dynamics we discussed in the chapters on long-context modeling.

Response length normalization. Do not truncate responses to fixed lengths before presenting them to the judge. Truncation can remove the most important part of a response and introduce artifacts that the judge interprets as quality signals. A response that appears to stop mid-sentence will score lower on coherence even if the full response was perfectly coherent. If length is a concern for prompt budget, filter out unusually long responses before evaluation rather than mid-response truncation.

Formatting artifacts. Some responses contain formatting that looks different in a plain-text prompt than it does in the rendered interface where the user would read it. Markdown headings, bullet points, and code blocks are interpreted differently by LLM judges than by human readers looking at a rendered interface. If your evaluation task involves responses that use rich formatting, consider whether your judge needs to see the raw markdown, a rendering description, or a stripped plain-text version. Inconsistent formatting handling is a source of criterion invalidity that is easy to miss.

The output format instruction tells the judge how to express its verdict. Common formats include:

• Scalar score: A numeric rating on a fixed scale (1-5, 1-10, 0-100).
• Categorical label: One of a fixed set of labels (Excellent, Good, Fair, Poor).
• Binary choice: A or B (for pairwise comparison).
• Structured JSON: A dictionary with fields for score, reasoning, and confidence.
• Chain-of-thought + verdict: Explanation followed by a final score or choice.

Each format has tradeoffs. Scalar scores are easy to aggregate but introduce anchoring effects and quantization noise. Categorical labels are easy to interpret but collapse nuance. Binary choices are easy to extract but discard magnitude information. Structured JSON outputs are flexible but increase prompt complexity and can cause parsing failures if the judge does not follow the schema precisely.

Chain-of-thought evaluation is worth the additional token cost in most situations. Studies have found that judges using chain-of-thought formats show higher agreement with human raters and greater consistency across prompt variants than judges creating direct scores. The mechanism is similar to chain-of-thought prompting for generation tasks: the explicit reasoning trace structures the model's probability distribution in a way that reduces noise and anchoring effects. When the judge must write down its reasoning before scoring, it is effectively constrained to produce a score that is consistent with the written reasoning, which reduces the influence of surface-level priming effects.

There is a practical downside to chain-of-thought evaluation: parsing. The reasoning trace and final verdict must be reliably extractable from the judge's output. If you ask for "Reasoning: [one sentence]" followed by "SCORE: [number]", the judge might add a preamble, use different formatting, or nest the score in a sentence rather than on its own line. Your extraction logic must be reliable to these variations. Regular expressions and structured parsing both work, but you need to test extraction failure rates on real judge outputs before deploying. A 5% parsing failure rate means 5% of your evaluations return no verdict, which can introduce systematic bias if parsing failures are correlated with certain response types.

Output extraction. Regardless of the format you choose, you need a reliable way to extract the judge's verdict from its response. If you request a JSON output and the judge wraps it in markdown fences or adds a preamble, your parser will fail. Design your output format instructions to be explicit about expected structure and test extraction reliability across a sample of outputs before deploying. Include a fallback parsing step for malformed outputs. A pattern that works reliably is to ask for the score on a line by itself in a distinctive format (for example, "SCORE: 4") and use a strict regex that looks for that exact pattern. More complex formats create more parsing opportunities for failure.

Assigning the judge a specific role (for example, "You are a senior software engineer reviewing code quality") can improve criterion alignment by orienting the model's prior knowledge and judgment toward the relevant domain. Role assignment works best when:

• The dimension being evaluated is domain-specific (code correctness, medical accuracy, legal reasoning).
• The judge model has been trained on substantial data from that domain.
• The role description is consistent with the evaluation criteria.

Role assignment can backfire when the assigned role conflicts with the evaluation objective or when it activates associations that introduce unintended biases. A role assignment like "You are a strict grader" may produce harsher scores across the board, not because responses are worse but because the role primes severity. Similarly, "You are a helpful assistant" can lead the judge to be more lenient than warranted, because the "helpful assistant" persona in most instruction-tuned models is trained to be affirming rather than necessary. The word "strict" carries connotations of severity that go beyond a simple instruction to be accurate.

The most subtle failure mode is domain-specific role assignments that carry implicit style preferences. Telling a judge "You are a professional academic reviewer" may cause it to penalize informal language even when the task was conversational. The role description activates expectations about writing register that are irrelevant to the quality dimension you are measuring. If you suspect this is happening, ablate the role description entirely and compare the distribution of scores. If scores shift substantially when you remove the role, the role is doing more than just domain alignment. It is influencing the implicit standard of judgment in ways that may not reflect what you want. Test your role assignments empirically by comparing judge verdicts with and without the role description before committing to them.

A useful middle ground is a role description that specifies domain knowledge without specifying a judgment style. "You are an expert in machine learning" is cleaner than "You are a strict machine learning professor" because it provides domain priming without style priming. The criteria and scale anchors you provide in the evaluation prompt should carry the judgment style, not the role description.

One of the most consequential design decisions in evaluation prompt engineering is whether to include example evaluations in the prompt (few-shot) or to rely on the judge's training to interpret your criteria without examples (zero-shot). Both approaches have legitimate use cases, and the choice should be based on the specific task, the judge model, and the resources available to create high-quality examples.

Zero-shot evaluation relies entirely on the criterion descriptions and task framing in the prompt. It requires no example verdicts. Its key advantages are simplicity and generality: you do not need to curate labeled examples, and the prompt stays compact. Zero-shot evaluation works reasonably well when:

• The evaluation dimension is common in the judge's training distribution (fluency, coherence, grammaticality).
• Your criterion descriptions are precise and anchor the endpoints of the scale.
• You are evaluating a task type that is well-represented in instruction-following data.

The main risk with zero-shot evaluation is inconsistency in how the judge interprets abstract criteria. "Accuracy" might mean different things to the judge across different runs if the context does not pin it down clearly. For dimensions that are less common or more subjective, zero-shot evaluation tends to show high variance and low agreement with human raters.

Zero-shot evaluation is a reasonable starting point when you are exploring a new evaluation task and do not yet have labeled examples to draw on. You can iterate quickly on criterion descriptions without the overhead of maintaining an example set. Once you have a stable criterion definition that produces acceptable zero-shot performance, you can consider whether adding few-shot examples would improve calibration further. In many cases, a well-designed zero-shot prompt with explicit scale anchors comes close to the performance of a few-shot prompt with poor examples, which means the ordering should be: design a good zero-shot prompt first, then add examples only if you can curate high-quality ones.

Few-shot evaluation includes two to five example evaluations in the prompt, each consisting of a (question, response, verdict) triple. The examples serve as in-context demonstrations of how you want the judge to apply the criteria. Few-shot evaluation is more reliable than zero-shot evaluation on most evaluation tasks when the examples are high quality.

The benefits of few-shot evaluation are:

• Calibration: Examples anchor the judge's internal scale to your intended distribution of verdicts. If you provide one example of a score-1 response and one example of a score-5 response, the judge interpolates between them rather than relying on its default interpretation of the scale.
• Edge case coverage: Including an example that covers a common edge case teaches the judge how to handle it consistently.
• Reduced sensitivity: Judges evaluated with few-shot prompts show less sensitivity to phrasing variations in the criterion descriptions because the examples constrain interpretation more tightly than text alone.

The benefit of few-shot evaluation depends entirely on the quality of the examples. A single high-quality example that represents a realistic borderline case is more useful than five examples that cluster at the extremes of the scale. Good few-shot examples share several properties:

• They cover the task types that will appear in your evaluation data (do not use toy examples for a production evaluation).
• They include a reasoning trace explaining why the response received that score, rather than only the score.
• They span the scale, including at least one clearly high-quality response, one clearly low-quality response, and one borderline case.
• They were labeled by domain experts whose judgment you trust, ideally with cross-labeler agreement measured before inclusion.

Low-quality examples introduce their own biases. If your examples systematically reflect the preferences of a single annotator or a narrow slice of task types, few-shot evaluation will amplify those biases rather than calibrate them away. An evaluation prompt with poorly chosen examples can perform worse than zero-shot evaluation with good criteria, because the examples actively mislead the judge about your intentions.

The borderline case example deserves special emphasis. Examples at the extremes (clearly correct, clearly wrong) help calibrate the scale endpoints but provide little guidance for the cases where evaluation matters: the responses where a well-designed prompt would give a 2 or 3 and a poorly designed prompt would give a 4 or 5. A borderline example with explicit reasoning explains to the judge how to handle ambiguity, which is where most evaluation value is created. If you can include only one example in a token-constrained environment, make it a borderline case.

A common failure mode in few-shot evaluation is inconsistency between the examples and the criterion descriptions. If your text says "accuracy means only verifiable claims" but your examples reward confident-sounding responses regardless of verifiability, the judge will learn from the examples and ignore the text. The examples take precedence over the prose when the two conflict because they carry more direct evidence about your intentions.

Before deploying a few-shot evaluation prompt, verify that every example is consistent with your criterion definitions. This is easier to check programmatically: use a separate judge to evaluate whether each example verdict follows from the criteria, and flag any inconsistencies for human review. You can also check consistency by having a second annotator who did not create the examples verify that they would assign the same verdicts given the criterion definitions. Systematic disagreement between the example creator and the verifier is a signal that the examples encode idiosyncratic preferences rather than your stated criteria.

The practical guidance for the number of examples is two to five, but the right number depends on the task complexity and the judge model's capability. For simple, well-defined criteria (factual correctness on closed-domain questions), two examples covering clear-correct and clear-wrong cases may be sufficient. For complex, criteria with several dimensions (overall response quality for open-ended questions), five examples that cover a range of cases and failure modes provide better calibration.

Adding more than five examples has diminishing returns and carries a cost. Each example adds tokens to the prompt, which increases latency and cost per evaluation. In long prompts, there is also a risk of context window crowding: when the examples take up a large fraction of the context window, the judge may weight the examples less carefully because they are further from the end of the prompt where the model's attention concentrates. Keep your example set small and high-quality rather than large and complete.

Even a well-designed evaluation prompt will show some variance. You run the same judge on the same pair of responses twice and get different scores. This is expected: autoregressive language models are stochastic by default, and the temperature settings that produce fluent, natural outputs also introduce randomness in evaluation decisions. Managing evaluation consistency means understanding the sources of variance, quantifying them, and applying strategies to reduce variance where it matters.

Variance in LLM judge outputs comes from three main sources.

Sampling randomness. With temperature > 0, the judge samples from a probability distribution over tokens. On borderline cases where the judge's uncertainty is high (the probability mass is spread across multiple possible scores), different samples from the same distribution will produce different verdicts. Setting temperature to 0 eliminates sampling variance but introduces its own artifacts: greedy decoding can get stuck in degenerate outputs, and it removes the ability to estimate uncertainty by sampling multiple times.

Prompt sensitivity variance. As we discussed earlier, small changes to the evaluation prompt produce different verdicts. Even without changing the prompt intentionally, environmental factors like token truncation at the context window boundary or subtle differences in how the system prompt is formatted can introduce variation.

Positional and contextual effects. The context of other responses in a batch evaluation, the order in which responses appear in a multi-turn conversation, and the recent history of the judge's outputs can all influence individual verdicts. These effects are less understood than sampling variance but are real. When evaluating many responses in sequence, there is evidence that judge behavior can drift as the conversation history accumulates, particularly if the judge is creating chain-of-thought reasoning that fills the context window with its own prior verdicts.

The standard approach to measuring evaluation consistency is to run the judge multiple times on the same inputs and compute agreement metrics across runs. The agreement metric you use depends on your output format:

• Pairwise comparisons (A vs. B): Use percent agreement or Cohen's kappa across runs.
• Scalar scores: Use intraclass correlation coefficient (ICC) or Pearson correlation across runs.
• Categorical labels: Use Fleiss' kappa treating each run as a separate "rater."

A well-designed evaluation prompt at temperature 0 should show near-perfect consistency on clear cases (> 95% agreement across runs) and measurable but limited variance on borderline cases (70-85% agreement). If you see substantial variance even on clear cases, the evaluation prompt needs revision. Variance on clear cases indicates that the prompt's criterion definitions are not anchoring the judge's interpretation effectively, or that the output format is introducing parsing ambiguity.

Several strategies reliably improve evaluation consistency.

Reduce temperature. Setting temperature to 0 eliminates sampling variance. For evaluation tasks where you want a deterministic verdict, zero temperature is the default choice. Reserve higher temperatures for uncertainty estimation via sampling.

Sample and aggregate. For high-stakes evaluations, run the judge 5-10 times on the same input and aggregate the scores (by mean for scalar scores, by majority vote for categorical labels). This reduces the variance of the final estimate at the cost of additional token usage. The standard error of the mean for independent samples is:

where:

• : the standard deviation of judge scores across individual runs (how spread out single verdicts are)
• : the number of independent samples (judge runs) you aggregate

Doubling the number of samples reduces the standard error by a factor of . Moving from one sample to five samples reduces it by . Diminishing returns set in beyond about 10 samples for most evaluation tasks, making 3-5 samples a practical default for high-stakes comparisons.

Use structured output formats. Judges that produce structured JSON outputs or chain-of-thought explanations before the final score tend to show lower variance than judges that produce direct scores. The structure constrains the probability distribution over outputs and reduces the influence of random sampling on the final verdict.

Calibrate against held-out examples. Reserve a held-out set of (response, human verdict) pairs and measure your judge's calibration on this set periodically. Calibration drift, where the judge's score distribution shifts over time or across prompt variants, is an early signal that consistency is degrading.

Normalize the scale. If your judge is creating a distribution of scores that is biased toward one end of the scale (clustering at 4-5 on a 1-5 scale, for example), your criterion descriptions may be poorly calibrated. Revise the endpoint anchors to spread the distribution across the scale and retest. A judge that never assigns score 1 or score 2 on a 5-point scale is effectively operating on a 3-point scale, which reduces its discriminative power substantially.

Control the evaluation environment. Batch evaluations where the judge processes many items in sequence can show within-batch drift due to context accumulation. Running each evaluation in a fresh context (a new API call with no shared history) eliminates this source of variation. The tradeoff is cost: fresh contexts are more expensive than batched evaluations. For large-scale evaluation tasks, a practical approach is to use batched evaluation for preliminary ranking and fresh contexts for high-stakes comparisons.

Let us walk through the full process of designing an evaluation prompt from scratch for a specific task: evaluating the factual accuracy of responses to knowledge questions. This is one of the most common evaluation tasks in language model development, and it illustrates the full range of design decisions we have covered.

Task description: We are building an evaluation pipeline for a question-answering model. Our responses are answers to factual questions (for example, "What year did the Berlin Wall fall?"). We want to measure whether the model's response is factually correct, without crediting fluent-sounding but incorrect answers.

Step 1: Define the criterion.

We will evaluate a single dimension (factual accuracy) on a 1-4 scale:

• 4: The response answers the question correctly with no factual errors.
• 3: The response answers the question mostly correctly but includes a minor error or imprecision that does not change the core answer.
• 2: The response attempts to answer the question but contains a significant factual error that affects the core answer.
• 1: The response is factually wrong, fabricated, or refuses to answer without basis.

We chose a 1-4 scale rather than 1-5 deliberately: even-numbered scales force the judge away from a neutral midpoint, which is useful for factual accuracy where responses should generally be either correct or incorrect. An odd-numbered scale would allow the judge to score borderline cases as "3 out of 5 (neutral)" even when the response is substantively wrong or right.

Step 2: Write the task description.

You are an expert fact-checker evaluating language model responses to factual questions. For each response, you will assess factual accuracy on a 1-4 scale: 4: Correct. The response answers the question accurately with no errors. 3: Mostly correct. Minor imprecision present but core answer is right. 2: Partially correct. A significant error affects the core answer. 1: Wrong. The response is factually incorrect, fabricated, or refuses to answer. You must base your verdict ONLY on factual accuracy, not fluency, length, or tone. After the response, provide: - A one-sentence explanation of your verdict - Your numeric score on a line by itself: SCORE: [1/2/3/4]

The explicit exclusion clause ("ONLY on factual accuracy, not fluency, length, or tone") is important. Without it, judges tend to conflate high-quality writing with factual accuracy, giving higher scores to well-written wrong answers than to awkwardly-written correct ones. The exclusion clause is a direct intervention against the surface-quality confound.

Step 3: Create two few-shot examples.

Example 1 (score 4):

Question: In what year did the Berlin Wall fall? Response: The Berlin Wall fell in 1989. Explanation: The response correctly identifies 1989 as the year of the Wall's fall. SCORE: 4

Example 2 (score 2):

Question: What is the speed of light in a vacuum? Response: The speed of light is approximately 300,000 miles per second. Explanation: The speed of light is approximately 300,000 kilometers per second (not miles). The order of magnitude is correct but the unit is wrong, which constitutes a significant error. SCORE: 2

The second example is a borderline case: the response is not fabricated, and it gets the order of magnitude right, but the unit error makes it factually wrong in a way that matters. The explicit reasoning explains to the judge why this is a 2 rather than a 1 (which would be reserved for responses that are completely wrong) or a 3 (which would be reserved for minor imprecisions that do not affect the core answer). This kind of explicit reasoning in examples teaches the judge how to handle analogous borderline cases across the full evaluation set.

Step 4: Test sensitivity and consistency.

Before deploying this prompt, we should:

1. Test it against 20-30 (question, response, human verdict) examples and compute agreement between the judge and human raters.
2. Create three paraphrase variants of the task description and verify that agreement across variants is high (kappa > 0.8).
3. Run the judge three times on the same inputs at temperature 0 and verify perfect consistency (100% agreement across runs).
4. Examine the score distribution and verify it covers the full 1-4 range, not just 3-4.

This example is simpler than production evaluation prompts, which typically involve longer responses, more fine-grained criteria, and domain-specific knowledge. But it illustrates the core principle: explicit criterion definitions, concrete scale anchors, and empirical testing are not optional features of evaluation prompt design. They are the difference between a judge that measures what you care about and one that measures whatever is easiest to measure.

Let us implement an evaluation prompt framework that handles the design patterns we have discussed: structured output with chain-of-thought reasoning, few-shot examples, and consistency measurement via multiple runs.

We will use the Anthropic Python SDK to demonstrate evaluation with a real judge model. The implementation covers prompt assembly, output parsing, and variance estimation.

The sensitivity audit tells you whether your evaluation prompt's verdicts are stable across reasonable paraphrases of the role description. A range of 0 (all variants agree) indicates robustness. A range of 2 or more indicates that the role description is influencing verdicts substantially, and you should either anchor the criteria more precisely or use few-shot examples to constrain interpretation.

The key parameters in the evaluation prompt framework are:

• criterion.scale_anchors: Defines the rubric labels for each score point. Endpoint anchors (score 1 and score 4/5) have the most impact on calibration. Midpoint anchors reduce ambiguity for borderline cases.
• use_chain_of_thought: Whether to ask the judge to reason before scoring. Enable this by default; the reasoning trace improves consistency and makes verdicts auditable.
• examples: The few-shot demonstrations. Two to five well-chosen examples that span the scale and include a borderline case outperform larger example sets with extreme-only coverage.
• temperature: Judge sampling temperature. Use 0.0 for deterministic evaluation in production. Use 0.1-0.2 when estimating uncertainty via multiple samples.
• n_samples: Number of judge runs to aggregate. Set to 1 for fast evaluation and 3-5 for high-stakes comparisons where variance estimation matters.

Beyond the design of individual prompts, a well-maintained evaluation pipeline requires ongoing monitoring of score distributions. Even a prompt that was well-calibrated at launch can drift over time as the distribution of model responses shifts. Tracking score distributions is the primary tool for detecting this drift before it silently degrades your evaluation quality.

A score distribution tells you how often your judge assigns each possible score. An ideal distribution for a 1-5 scale should be roughly bell-shaped if the responses you are evaluating vary in quality, or right-skewed if your model is generally performing well. What you do not want to see is a distribution that is heavily concentrated at one end of the scale, because that concentration destroys your ability to use the scale to discriminate between responses.

The most common pathological distribution in LLM evaluation is leniency bias: the judge consistently assigns scores in the range of 4-5 on a 1-5 scale, regardless of response quality. Leniency bias has several causes. The most common is that the high end of the scale is underspecified: if "5 = excellent" is the only anchor at the top, the judge fills in the gap with its prior about what "excellent" means, and its prior often includes fluent, confident-sounding responses, which are overrepresented in its training data. Adding a concrete description of what a 5 looks like, including what it does not include, usually corrects this.

Severity bias is less common but occurs with prompts that emphasize flaw detection. If your evaluation prompt asks the judge to identify weaknesses in the response, it may apply a higher implicit standard than intended, clustering scores in the 1-3 range. This is why task description framing matters: "assess the quality" and "find flaws" produce systematically different score distributions from the same judge on the same responses.

A practical monitoring approach is to maintain a calibration set: a fixed collection of (response, human verdict) pairs that you evaluate with your judge at regular intervals. By tracking the judge's score distribution on this fixed set over time, you can detect drift independently of changes in the model outputs you are evaluating. If the judge's scores on the calibration set shift without any change to the evaluation prompt, something has changed in the judge model itself (due to an update, a change in serving infrastructure, or context window effects), and you need to investigate.

For pipelines that evaluate models at multiple stages of development, calibration set management is especially important. The responses from an early-stage model and a late-stage model may be so different in quality that a judge calibrated on early-stage responses is effectively measuring a different dimension when applied to late-stage responses. Re-calibrating the judge (updating the few-shot examples and scale anchors) when the model population shifts significantly is good practice.

Evaluation prompt engineering improves judge reliability and validity, but it does not solve all problems with LLM-based evaluation. Several basic limitations remain even with well-designed prompts.

Criterion validity is not guaranteed. Even if your judge reliably applies your criteria, those criteria may not measure what matters for your task. A judge designed to measure factual accuracy will give high scores to responses that are accurate but unhelpful. A judge measuring fluency will rate responses that sound confident but are wrong. The question of what to measure, and whether your criteria capture it, is a human judgment that no amount of prompt engineering can fully automate. Periodic human review of high-scoring and low-scoring judge outputs is needed to detect criterion drift. When you examine high-scoring outputs and find that they do not appear to be high-quality responses, your criteria are not capturing what you care about.

Few-shot examples encode annotator biases. The examples you include in your evaluation prompt carry the biases of whoever labeled them. If your few-shot examples come from a single annotator with strong preferences about response style, the judge will learn those preferences and amplify them. This is not a defect of few-shot evaluation as a technique, but it is a risk that scales with how opinionated your annotators are and how much you trust their judgments. Measuring inter-annotator agreement on your few-shot example labels before including them in the prompt is a practical safeguard. If two independent annotators would not agree on the score for a proposed example, that example is not a reliable anchor for the judge's calibration.

Evaluation prompts require maintenance. As the models you are evaluating improve, the distribution of responses shifts. Criteria that were adequate for an earlier generation of models may be too coarse to discriminate between responses from a stronger model. Evaluation prompts that were calibrated against one model's output distribution may be systematically mis-calibrated when applied to a different model. This is especially important when using LLM judges for model selection: if the judge was calibrated on the same model it is evaluating, or on a model with similar failure modes, it will miss the failures that matter.

Scale inflation is common. In practice, LLM judges tend to be more lenient than human raters. Without careful calibration, scores cluster at the high end of the scale, which compresses your ability to distinguish between good and great responses. Monitoring the full score distribution across your evaluation corpus, and adjusting scale anchors or few-shot examples to spread it more evenly, is an ongoing maintenance task rather than a one-time setup step.

The judge's knowledge is bounded. For factual accuracy evaluation, the judge can only assess claims that fall within its training knowledge. Responses about recent events, specialized domains, or niche topics may be evaluated incorrectly not because the evaluation prompt is poorly designed but because the judge lacks the knowledge needed to verify the claims. This is a basic limitation of LLM-based factual evaluation that cannot be solved by prompt engineering alone. For knowledge-necessary evaluation tasks, combining the LLM judge with retrieval (giving the judge with relevant reference documents) or with a specialized knowledge base check addresses this gap.

Prompts that work for one model may not transfer. An evaluation prompt that produces high agreement with human raters when using GPT-4 as the judge may produce lower agreement when using a smaller or differently aligned model. The sensitivity characteristics we discussed depend on the specific judge model's training distribution and instruction-following capabilities. Before deploying an evaluation prompt with a new judge model, re-run your sensitivity tests and human agreement measurements. Do not assume that a validated evaluation prompt transfers automatically.

The techniques in this chapter also connect directly to the upcoming chapters on bias and fairness. The biases we have discussed here, such as length bias, style bias, and annotator bias encoded in few-shot examples, have systematic effects on which models score well in your evaluation pipeline. When evaluation prompts reward verbosity, models that produce longer outputs will appear to perform better regardless of quality. When few-shot examples reflect one cultural register or writing style, models that produce text in different registers will be systematically undervalued. These effects compound across large-scale evaluations and can lead to misleading conclusions about which models are best suited for deployment.

Evaluation prompt engineering determines the quality of everything downstream in an LLM-based evaluation pipeline. The key principles from this chapter are:

• Prompt sensitivity is real and measurable. Small changes to phrasing, criteria ordering, and output format can change judge verdicts. Test your evaluation prompts across paraphrase variants before deploying them, and require high inter-variant agreement as a condition for use.
• Task description quality matters. Include the original user question, an explicit definition of the dimension being measured, and a description of the standard for high-quality responses. Vague descriptions produce unreliable judges.
• Explicit scale anchors reduce variance. Rubric-style criterion definitions with anchor descriptions at each scale point reduce the ambiguity that drives score variance. Endpoint descriptions (what a 1 looks like, what a 5 looks like) have the most impact.
• Few-shot examples improve calibration. Well-chosen examples that span the scale and cover borderline cases reduce sensitivity to phrasing variation and improve agreement with human raters. Quality matters more than quantity: two high-quality examples outperform five mediocre ones.
• Chain-of-thought formats improve consistency. Asking the judge to reason before scoring reduces sampling variance and produces verdicts that are more explainable and easier to audit.
• Multi-dimension evaluation requires separate criteria. Aggregated scores that combine multiple quality dimensions are hard to interpret and easy to game. Evaluate dimensions independently and combine them with explicit weights.
• Consistency testing is mandatory. Run the judge multiple times on borderline cases, measure agreement across runs, and set a minimum agreement threshold before trusting judge outputs in production.
• Score distributions require monitoring. As model quality improves over time, score distributions shift toward the high end of the scale. Recalibrate your judge periodically to maintain discriminative power across the response quality range you care about.

Evaluation prompt engineering is not a one-time task. It is an ongoing practice that requires monitoring score distributions, updating few-shot examples as model outputs evolve, and periodically validating judge verdicts against human labels. The investment pays off in evaluation pipelines that measure what you care about, rather than whatever the judge finds easiest to measure.

QuizLink Copied