LLM Preference Evaluation: Pairwise Comparisons and Elo

Article links

Make inline references clickable

When we evaluate language models, we face a basic problem: human judgments are noisy. Ask ten people to rate an essay on a scale of 1 to 5, and you will get ten different answers influenced by personal calibration, mood, and shifting standards. But ask those same people which of two essays is better, and agreement skyrockets. Humans are remarkably consistent at making relative judgments even when absolute ratings vary wildly.

This phenomenon stems from basic principles of human cognition. Psychologists have long observed that humans are highly sensitive to relative differences but struggle with absolute calibration. Just as we can effortlessly determine which of two weights is heavier while struggling to estimate either weight in kilograms, evaluators can distinguish quality differences in language models with far greater consistency than they can assign absolute scores. This principle, related to the concept of Just Noticeable Differences in psychophysics, means that pairwise comparisons tap into a more reliable aspect of human judgment than cardinal rating scales.

Gustav Fechner's 19th-century work on psychophysics established that perception of differences follows a logarithmic relationship with the absolute magnitude of the stimulus. What matters to the human perceptual system is not the absolute level but the ratio between stimuli. This basic property of biological sensing applies directly to the evaluation of text quality: a person assessing whether Response A is better than Response B is engaging a perceptual comparison process that is far more reliable than the process of mapping quality onto an abstract numerical scale. The comparative judgment extracts a signal that the brain is wired to produce reliably; the absolute judgment requires the brain to do something far less natural.

This observation motivates modern preference evaluation. Instead of asking annotators to assign scores, we ask them to compare outputs directly. This approach underpins the reward modeling pipeline we explored in Part XXXVII: Alignment and RLHF, where the Bradley-Terry model converts pairwise preferences into scalar rewards. But preference evaluation extends far beyond RLHF training data collection. It gives the statistical framework for benchmarking systems like Chatbot Arena, validating model improvements, and determining whether a new training recipe produces better outputs.

The challenge, of course, is that collecting pairwise comparisons is expensive. With models, a full tournament requires comparison pairs, and each pair typically needs multiple independent judgments to reach reliable estimates. For ten models, that is 45 pairs. For twenty models, 190 pairs. This combinatorial explosion forces practitioners to think carefully about experimental design, statistical efficiency, and the tradeoffs between coverage and cost. The methods in this chapter address exactly those challenges: how to design a comparison study that extracts maximum information from a constrained annotation budget, how to aggregate noisy judgments into stable global rankings, and how to quantify uncertainty in the resulting estimates.

In this chapter, we examine how to design rigorous comparison studies, aggregate pairwise results into global rankings using Elo rating systems, and determine whether observed differences reflect real model improvements or merely statistical noise. We will connect the preference models from RLHF to the practical work of running large-scale evaluation campaigns.

Absolute rating scales suffer from several systematic biases. Annotators interpret "good" differently depending on their expertise, cultural background, and the distribution of examples they have seen recently. A response that earns a 4 from a generous annotator might receive a 2 from a strict one, even if both agree on the relative ordering of responses. This annotator calibration drift makes it difficult to compare ratings collected at different times or by different people.

The problem becomes particularly acute when evaluation campaigns span multiple days or involve distributed annotation teams. An annotator who begins a session after reviewing particularly poor model outputs may calibrate their scale differently than one who begins after reviewing strong baselines. Even the same annotator may drift over time as they acclimate to model capabilities or fatigue sets in. These shifts create systematic noise that can obscure real model differences or create spurious apparent improvements.

Consider a concrete scenario: you are running a two-week evaluation to compare a new model against a baseline. In the first week, annotators review mostly weak baseline outputs and anchor their "5" rating at a relatively low quality threshold. In the second week, after seeing better model outputs, they recalibrate upward. The new model, evaluated in week two, receives lower ratings not because it is worse but because the scale has shifted. This temporal confounding is invisible in the raw rating data and can easily be mistaken for evidence that the new model underperforms.

Pairwise comparisons eliminate calibration issues by forcing a relative judgment. When an annotator selects Response A over Response B, they communicate real information regardless of whether they would rate both as "excellent" or both as "terrible." This invariance to additive bias makes comparisons the gold standard for subjective evaluation. The mathematical intuition is straightforward: if an annotator consistently applies a personal offset to all their ratings, this constant cancels out when computing differences between models. While absolute ratings suffer from , pairwise comparisons yield , where the calibration constant has vanished.

The advantages of pairwise comparisons extend beyond just eliminating the constant offset. They also reduce the cognitive load on annotators. Rather than mentally constructing a mapping from abstract quality to a numerical scale, the annotator simply identifies which of two concrete outputs better satisfies a given criterion. This simpler task produces more consistent judgments, higher inter-annotator agreement, and lower dropout rates in annotation campaigns. Research in social choice theory confirms that binary choices tend to produce more reliable and reproducible results than rating scales, particularly when the evaluated concepts are complex or multi-dimensional.

However, comparisons introduce their own challenges. They are more expensive than absolute ratings (requiring two outputs per judgment), create potential for position bias (preferring the first option shown), and raise statistical questions about transitivity. If Model A beats Model B, and Model B beats Model C, we expect Model A to beat Model C, but real human preferences often violate this expectation due to context-dependent strengths. For instance, Model A might excel at creative writing but struggle with factual accuracy, while Model B balances both moderately, and Model C prioritizes factuality above all. In such cases, A defeats B on creative tasks, B defeats C on general quality, yet C defeats A on factual queries, creating a preference cycle that confounds scalar rankings.

This problem of intransitivity is not a pathological edge case; it is a near-universal feature of LLM evaluation. The reason is that "quality" is not one-dimensional. A response can be more helpful but less harmless, more concise but less thorough, more confident but less accurate. When the prompt distribution covers multiple task types, different models will show different profiles of relative strengths, and the aggregate ranking will reflect an average over these profiles. How informative that average is depends entirely on whether the question you are asking is "which model performs best on average" or "which model is best for my specific use case."

Designing a preference evaluation requires careful attention to experimental protocol. The simplest form presents two anonymized outputs side-by-side and asks the annotator to choose the better one, with optional buttons for "tie" or "uncertain." But even this simple design embeds dozens of choices that affect the validity and statistical power of your results. The best evaluation pipelines treat comparison design as a scientific experiment, applying the same rigor to randomization, blinding, and quality control that clinical trials apply to drug testing.

To minimize bias, you must randomize the presentation order. If Model A always appears on the left, you will conflate model quality with position preference. A proper A/B test randomly assigns positions per comparison. This keeps any position bias affects both models equally across the dataset. This randomization serves a important statistical purpose: it turns position effects from confounding variables into noise that averages to zero across the experiment. Without randomization, you cannot distinguish whether a 55% win rate for Model A shows real superiority or merely a 5% advantage from consistently appearing in the preferred position.

Equally important is anonymization of model identities. If annotators know they are comparing GPT-4 against an open-source model, their expectations and prior beliefs will contaminate their judgments. This is not mere speculation; experiments in wine tasting, music listening, and product evaluation consistently show that labels change perceived quality even when the underlying product is identical. In language model evaluation, the contamination risk is even higher because the community has strong prior beliefs about which providers produce better models. Blinded evaluation with randomly assigned model identifiers ensures that judgments reflect the actual response quality rather than brand reputation.

When comparing multiple models, use blocking to ensure balanced comparisons. If you have four models (A, B, C, D), you want roughly equal numbers of A-vs-B, A-vs-C, A-vs-D, B-vs-C, B-vs-D, and C-vs-D matchups. This prevents one model from being evaluated primarily against weak competitors, which would inflate its win rate. Blocking also ensures that the comparison graph remains connected, which is needed for global ranking estimation. In statistical terms, balanced blocking minimizes the variance of rating estimates by so that each model faces a representative sample of the competitor population rather than a biased subset.

The statistical validity of a comparison study depends on how many comparisons you collect and which prompts you use. A study that exclusively uses straightforward factual questions will give a misleadingly optimistic picture of models that excel at retrieval but struggle with reasoning. Conversely, a study dominated by creative writing prompts will favor expressive models over precise ones.

Stratified prompt sampling ensures that your evaluation covers the task distribution you care about. If your application involves 40% factual questions, 30% summarization, 20% creative writing, and 10% code generation, your prompt set should approximate this distribution. When you cannot specify the distribution in advance (because you are building a general assistant), you can sample prompts from real user conversations. This keeps the evaluation shows actual usage patterns rather than an idealized benchmark.

Prompt quality matters as much as prompt diversity. Ambiguous or poorly specified prompts produce noisy comparisons because different annotators interpret the task differently. A prompt like "write something about climate change" leaves so much to interpretation that two high-quality responses might be judged differently based solely on which annotator valued conciseness over comprehensiveness. Well-specified prompts with clear criteria reduce this variance and make comparisons more informative.

Even with perfect experimental design, annotation quality determines the floor on what your evaluation can reach. Annotators need explicit guidelines that specify what dimensions of quality to evaluate (helpfulness, accuracy, safety, fluency), how to handle edge cases (what to do when one response is better on some dimensions but worse on others), and how to calibrate their judgments (give examples of responses at different quality levels).

Calibration examples are particularly useful. Before beginning their session, annotators review a set of example pairs where the "correct" answer has been established by expert consensus. This anchors their judgment to the intended quality scale and reduces drift between annotators. Some evaluation platforms use periodic calibration checks throughout the annotation session, flagging annotators whose accuracy on calibration items degrades and triggering retraining or review.

Inter-annotator agreement (IAA) gives a quantitative measure of annotation consistency. For binary win/loss judgments, Cohen's kappa is the standard metric:

where:

• : Cohen's kappa, measuring agreement corrected for chance
• : the observed agreement rate (fraction of comparisons where annotators agree)
• : the expected agreement rate under random labeling (typically 0.5 for binary choices)

Values of are generally considered acceptable for preference evaluation. If is lower, you should investigate whether guidelines are unclear, whether the prompts are inherently ambiguous, or whether some annotators are giving low-quality work. Multiple redundant annotations per comparison allow you to detect and filter unreliable annotators before they contaminate your dataset.

Not all comparisons yield clear winners. Annotators should have the option to declare a tie when responses are indistinguishable in quality. In Elo systems, ties typically award half a point to each competitor, but you can also model them explicitly using the Bradley-Terry model with ties, which extends the binary preference probability to a three-outcome case.

The probability that model ties with model can be modeled as:

where:

• : the probability that model ties with model
• : the relative strength parameter for model against model , based on their rating difference
• : the ratings of model and model , respectively
• : the draw margin parameter controlling how likely ties are
• : the inverse relative strength parameter for model against model

The draw margin parameter controls the zone of ambiguity where models are considered evenly matched. When rating differences are small relative to , ties become probable; when differences are large, ties become vanishingly unlikely. This formulation recognizes that ties are most likely when competitors are closely matched, whereas large skill gaps should produce decisive outcomes. However, most practical implementations treat ties as half-wins for simplicity, accepting a slight bias in exchange for computational tractability.

The decision of whether to allow ties also affects annotation behavior. Systems that force a winner on every comparison eliminate the cognitive option of "these are equivalent," which can reduce deliberation and push annotators toward gut reactions. Systems that allow ties risk annotators using the "tie" option as a way to avoid difficult decisions, inflating the tie rate beyond its true value. Monitoring the tie rate across annotators and flagging unusually high tie rates can help identify this pattern.

The Elo rating system, developed by Arpad Elo for chess rankings in the 1960s, gives an elegant solution for aggregating pairwise comparisons into a global skill rating. Each model maintains a rating , and the difference in ratings predicts the expected outcome of a match. The system works because it converts the sparse, local information of individual match results into a global, transitive ranking that approximates the underlying skill distribution.

Elo's key insight was that skill differences should predict win probabilities, and that those predictions should be updated in proportion to how surprised we are by the actual outcome. If a model rated 1800 beats one rated 1200, we learn almost nothing new because that outcome was nearly certain. But if the 1200-rated model wins, we sharply revise our estimates of both models' true strengths. This asymmetric updating makes Elo a form of online Bayesian inference, continually refining our beliefs about the true ordering of competitors as new evidence arrives.

The Elo system has been adapted beyond chess to many competitive domains: video games (where TrueSkill and Glicko extend Elo's framework), academic competitions, and more recently to the evaluation of machine learning systems. The natural alignment between Elo's assumptions (players have latent skill levels that determine match outcomes probabilistically) and the structure of LLM evaluation (models have latent quality levels that determine preference outcomes) makes Elo a natural choice for preference leaderboards.

Given two models with ratings and , the expected score for Model A is:

where:

• : the expected score for Model A (probability of winning plus half the probability of drawing)
• : the current Elo rating of Model A
• : the current Elo rating of Model B
• : the scale factor determining the rating-to-probability mapping (a 200-point difference yields approximately 76% expected win rate)

The expected score is the probability that Model A wins plus half the probability of a draw. The divisor of 400 creates a convenient scale where differences of 200 points correspond to roughly 3:1 odds, and differences of 400 points correspond to 10:1 odds. This logistic curve ensures that expected scores remain bounded between 0 and 1 while mapping linear rating differences to nonlinear probability changes. The specific value of 400 is conventional, chosen to match the historical chess rating scale, but the system works with any scale factor that matches the observed variance in your comparison data.

The formula is equivalent to a logistic function applied to the rating difference. As the difference grows, approaches 1 (near-certain win). As it shrinks toward zero, approaches 0.5 (coin flip). This sigmoid shape captures the intuition that skill differences become increasingly decisive as they grow larger, but that even large advantages do not guarantee victory against random variation in individual comparisons.

Rating UpdatesLink Copied

The Elo update rule approximates maximum likelihood estimation for the Bradley-Terry model we discussed in Bradley-Terry Model. In the Bradley-Terry framework, the probability that model beats model is:

where:

• : the probability that model defeats model in a pairwise comparison
• : the latent strength parameter for model (higher values indicate stronger models)
• : the latent strength parameter for model
• : the exponential of the strength parameter, so positive values and amplifying differences

If we set , this becomes exactly the Elo expected score formula. The Elo update is in effect an online stochastic gradient descent step on the Bradley-Terry log-likelihood, where is the learning rate. Each update moves the rating in the direction that would increase the likelihood of the observed outcome, with the step size controlled by .

This connection means that Elo ratings, when stabilized, estimate the underlying preference parameters that would be obtained by fitting a Bradley-Terry model to the entire comparison matrix. The online updates give a computationally efficient approximation to the batch optimization problem, making Elo ideal for live leaderboard systems where new comparisons arrive continuously and ratings must update in real time.

The Bradley-Terry model rests on a core assumption: each model has a single latent strength that governs all comparisons, regardless of the specific prompt or task. This assumption is clearly an idealization. In practice, GPT-4 might outperform Claude on mathematical reasoning while Claude outperforms GPT-4 on creative writing. The Bradley-Terry model averages across these task-specific advantages and disadvantages, creating a scalar that captures performance over the empirical distribution of prompts in the evaluation set. Whether this scalar is real depends entirely on whether that prompt distribution matches the distribution you care about in deployment.

When you have collected thousands of pairwise comparisons across dozens of models, you need reliable methods to aggregate these into a final ranking. The challenge is that online Elo updates, while computationally convenient, are sensitive to the order in which matches are processed. A model that faces its strongest competitors early in the sequence will receive lower ratings than one that faces them later, even if both eventually participate in the same number of matches. Batch methods that process all comparisons simultaneously avoid this path dependence.

Rather than iteratively updating Elo ratings, you can fit the Bradley-Terry model directly using maximum likelihood estimation. Given a comparison matrix where counts wins of model over model , the log-likelihood is:

where:

• : the log-likelihood function of the Bradley-Terry model parameters
• : the vector of strength parameters for all models
• : the number of times model defeated model (element of comparison matrix )
• : the number of times model defeated model
• : the natural logarithm
• The summation iterates over all unique pairs of models where is less than

You maximize this subject to the constraint to fix the scale. This constraint is necessary because adding a constant to all values leaves the probabilities unchanged; we need an anchor to make the parameters identifiable. This approach is more statistically efficient than online Elo updates, especially when comparison graphs are sparse (some models never play each other directly). Maximum likelihood estimation uses all available information simultaneously rather than processing matches sequentially, yielding estimates with lower variance when the model assumptions hold.

The MLE approach also produces natural measures of uncertainty through the Fisher information matrix. The curvature of the log-likelihood at its maximum tells us how precisely the parameters have been estimated: sharp curvature implies high precision, while flat curvature indicates that the data are consistent with a wide range of parameter values. This uncertainty quantification is directly interpretable as confidence intervals around each model's rating. This gives the statistical infrastructure needed to determine which differences in rankings are reliable and which are within the margin of error.

A practical challenge arises when the comparison graph is disconnected. If Group A models only ever compared against Group A, and Group B models only against Group B, you cannot determine whether Group A or Group B is stronger. Mathematically, the likelihood becomes flat in directions that shift mass between disconnected components, making the parameters unidentifiable. This requires anchor models: reference systems that bridge different communities of models by playing against all of them.

In practice, strong baseline models like GPT-4 or Claude act as universal anchors. This gives the connective tissue that allows Elo ratings to be comparable across the entire ecosystem. Without such anchors, you might find that Model X has a higher rating than Model Y, but this could merely reflect that X faced weaker competition within its isolated subgroup. The inclusion of widely-evaluated anchor models ensures that the comparison graph remains connected. This allows the ranking algorithm to establish a global scale rather than merely local orderings.

Graph connectivity affects both the technical analysis and measurement validity. Imagine comparing two national chess leagues without any players who competed in both. Any ranking that spans both leagues is purely speculative. The same logic applies to LLM evaluation. If your comparison graph has disconnected subgraphs, any ranking that places models from different subgraphs relative to each other is statistically unjustified. This is why evaluation systems like LMSYS Chatbot Arena are careful to include widely-compared models that serve as reference points across the entire comparison graph.

Regularization gives an alternative approach to the disconnected graph problem. Instead of requiring connectivity, you can add a prior over the parameters that shrinks them toward zero. This prevents any model's rating from diverging to infinity when it has a perfect record (which would happen with the unregularized MLE when a model wins every comparison), and it implicitly connects all models through the shared prior. The regularized objective is:

where controls the strength of the regularization. Larger produces stronger shrinkage toward equal ratings, while smaller allows ratings to spread more freely. The choice of can be made via cross-validation or by treating it as a hyperparameter that shows prior beliefs about how much the models differ in quality.

Random pairing is statistically inefficient. If Model A has a 1500 rating and Model B has 1000, the outcome is nearly deterministic. This gives little information. The variance of the rating update is proportional to , which approaches zero when one model is heavily favored. Active sampling strategies pair models with similar ratings or high uncertainty, maximizing the information gained per comparison.

Thompson sampling or Bayesian optimization approaches can select matchups that most reduce uncertainty in the global ranking, cutting the number of required comparisons by 50% or more compared to random pairing. These methods maintain a posterior distribution over possible rating configurations and select the matchup with the highest expected information gain. For example, if Model C and Model D have similar ratings but high uncertainty, a match between them gives more information about their relative ordering than a match between a certain champion and a certain loser. This adaptive approach is particularly useful when evaluation budgets are limited, letting precise ranking with fewer expensive human judgments.

The information-theoretic framing of active sampling reveals a deep connection to the design of scientific experiments more broadly. In both settings, the goal is to select observations that maximally reduce uncertainty about quantities of interest. For Elo-based evaluation, the quantity of interest is the vector of ratings , and the optimal sampling strategy is the one that minimizes the posterior entropy after observing the new comparison result. Greedy approaches that approximate this criterion by selecting the matchup with the highest expected prediction error are easy to implement and perform well in practice.

Determining that Model A has a higher Elo rating than Model B is insufficient; we need to know whether this difference is statistically significant or within the margin of error.

Intuition about statistical significance in Elo contexts differs from classical hypothesis testing in one important way: the test is not about whether a single comparison's result was due to chance, but about whether the aggregate pattern of many comparisons gives reliable evidence of a true rating difference. With enough comparisons, even tiny true differences become statistically distinguishable. With too few comparisons, even large true differences may not reach significance. Understanding the relationship between effect size, sample size, and power is needed for designing evaluation studies that can detect the differences you care about.

Elo ratings are estimates with associated uncertainty. The asymptotic variance of the Bradley-Terry parameter estimates can be derived from the Fisher information matrix. For model , the variance is approximately:

where:

• : the asymptotic variance of the maximum likelihood estimate for model 's strength parameter
• : the estimated strength parameter for model
• : the number of comparisons between model and model
• The summation aggregates over all opponents that model has faced
• The term is the variance contribution from each matchup, peaking when models are evenly matched

The Fisher information measures the curvature of the log-likelihood function at its maximum; sharper curvature implies more information about the parameter and thus lower variance. Notice that the variance contribution from each opponent is maximized when , confirming the intuition that matches between evenly matched competitors give the most information. Conversely, matches against much stronger or much weaker opponents contribute little to the precision of the estimate.

In practice, bootstrap resampling gives a more reliable variance estimate, especially when the Bradley-Terry assumptions are violated (e.g., due to intransitive preferences or context effects). The bootstrap makes no parametric assumptions about the distribution of preferences, instead using the empirical distribution of observed outcomes to estimate sampling variability.

To compute a 95% confidence interval for the rating difference between models A and B:

1. Bootstrap the comparison data 1000 times, resampling with replacement
2. Fit Bradley-Terry parameters for each bootstrap sample
3. Calculate the empirical 2.5th and 97.5th percentiles of

If the interval excludes zero, the difference is significant at .

This non-parametric approach accounts for both sampling variability and potential model misspecification. By refitting the model on resampled data, we capture the uncertainty arising from having finite comparison samples. Wide confidence intervals indicate that additional comparisons are needed to distinguish the models, while narrow intervals suggest high confidence in the observed ordering.

The practical implications of confidence intervals often surprise practitioners accustomed to reporting point estimates. In evaluation campaigns with limited budgets, it is common to find that all models' confidence intervals overlap substantially, meaning that no pairwise differences are statistically significant even if the point estimates suggest a clear ranking. This is not a failure of the evaluation; it is honest communication of the uncertainty inherent in limited data. The appropriate response is either to collect more comparisons or to acknowledge that the models are effectively tied at the current sample size.

A rough rule of thumb for power planning: detecting a 100-point Elo difference (approximately 64% win rate for the stronger model) at 80% power with requires roughly 150-200 comparisons between the two models. Detecting a 50-point difference (approximately 57% win rate) requires 600-800 comparisons. These numbers scale with the number of models in the evaluation because each model must participate in multiple pairs. Planning your sample size before data collection is needed for avoiding underpowered studies.

In live evaluation systems, you want to detect when a new model is definitively better than the current champion without running infinite comparisons. Sequential probability ratio tests (SPRT) give stopping rules that maintain false positive rates while minimizing the number of required judgments.

The SPRT continues collecting comparisons until the likelihood ratio exceeds a threshold determined by desired (false positive) and (false negative) rates:

where:

• : the likelihood ratio comparing the probability of observed data under alternative vs. null hypotheses
• : the probability of observing the comparison results assuming Model A is truly better (alternative hypothesis)
• : the probability of observing the comparison results assuming no difference between models (null hypothesis)
• : the Type I error rate (false positive probability)
• : the Type II error rate (false negative probability)
• : the decision threshold determining when sufficient evidence has been collected

Unlike fixed-horizon tests, which require specifying the sample size in advance, SPRT adapts to the evidence. If Model A is sharply superior, the test terminates early with a decisive verdict. If the models are nearly identical, the test may continue longer to accumulate sufficient evidence to distinguish them. This approach can reduce evaluation costs by 30-50% compared to fixed-horizon testing while maintaining the same statistical power.

The SPRT framework requires specifying the alternative hypothesis in concrete terms: for example, "Model A has a win rate of at least " rather than the vague claim that Model A is better than Model B. This concreteness is useful, because it forces you to explain the minimum effect size you care about before seeing the data. An evaluation designed to detect a 60% win rate will have different stopping rules than one designed to detect a 70% win rate, and the choice should reflect the practical significance threshold for your deployment decision.

Let us walk through a concrete example. Suppose we are evaluating three models: Alpha, Beta, and Gamma. We collect the following comparison results over 300 total matchups:

• Alpha vs Beta: Alpha wins 45, Beta wins 30, 5 ties
• Alpha vs Gamma: Alpha wins 35, Gamma wins 50, 5 ties
• Beta vs Gamma: Beta wins 40, Gamma wins 35, 5 ties

First, we initialize all ratings at 1500. Using :

For Alpha vs Beta:

Continuing this process iteratively until convergence yields stable ratings. The model with the strongest performance (Alpha against Beta, but weak against Gamma) creates a cyclic preference: Alpha > Beta > Gamma > Alpha, known as intransitivity. Real LLM evaluations often exhibit such cycles when models have different strengths (e.g., one is more helpful but less harmless).

This cyclic preference highlights a basic limitation of scalar rankings. In our example, Alpha might excel at creative writing but struggle with factual accuracy, while Beta balances both moderately, and Gamma prioritizes factual precision above all else. Depending on the prompt distribution, any of these models could appear superior. Intransitivity suggests that the very notion of a single "best" model may be illusory when quality is multi-dimensional, and that Elo ratings should be interpreted as capturing average performance over a specific task distribution rather than universal superiority.

How should we interpret this result in practice? Rather than concluding that the ranking is meaningless, we should recognize it as useful within its scope: the rank captures average performance over the prompt set used in the evaluation. If that prompt set is the distribution of prompts our users will send, then the ranking is maximally relevant. If the prompt set is unrepresentative, the ranking is unrepresentative too. The solution is not to abandon the ranking but to ensure that the evaluation prompt distribution is carefully designed to match the deployment distribution.

The intransitivity also motivates stratified reporting. In addition to a global Elo ranking, reporting per-domain win rates (creative writing, factual Q&A, code generation, summarization) gives a richer picture of each model's strengths and helps users select the right model for their specific application. A model with a lower overall Elo might be the clear choice for a specific domain where it has a large win-rate advantage.

Code ImplementationLink Copied

Understanding the theory of preference evaluation is one thing; seeing how it operates at scale is another. Two systems have shaped how the field thinks about preference-based LLM evaluation: LMSYS Chatbot Arena and the RLHF training pipeline. Each uses pairwise comparisons for a different purpose, and each has made different design choices that reflect the tradeoffs inherent in large-scale deployment.

Chatbot Arena, launched by the LMSYS research team in 2023, operates as a crowd-sourced evaluation platform where real users interact with two anonymized language models simultaneously and vote on which response they prefer. As of 2025, it has collected tens of millions of pairwise votes across hundreds of models, which makes it the largest open-ended LLM evaluation dataset in existence.

Several design choices make Chatbot Arena particularly useful. First, users submit their own prompts rather than being given preset questions. This ensures that the prompt distribution shows actual user interests rather than a curated benchmark, making the resulting rankings directly needed to real-world deployment. Second, all comparisons are blind: model identities are hidden until after the vote, preventing brand bias from contaminating judgments. Third, the system uses models in production, letting comparisons that would be prohibitively expensive under controlled laboratory conditions.

The platform has revealed patterns that controlled evaluations often miss. In early evaluations, GPT-4 dominated the rankings, but as the model population expanded to include instruction-tuned variants and specialized models, the top of the leaderboard became more competitive and the rankings more volatile. This volatility shows real capability improvements, but it also shows the sensitivity of Elo ratings to the evolving population of competitors. A model that faces stronger opponents as the field advances will see its rating decline even if its absolute capabilities remain constant, a phenomenon known as rating deflation.

To address rating deflation, Chatbot Arena periodically reruns historical comparisons using current models, effectively anchoring the rating scale to a fixed reference point. This longitudinal anchoring is important for tracking whether the field is making real progress or merely experiencing rating inflation from an expanding pool of competitors.

The preference evaluation framework described in this chapter is also the foundation of the RLHF training pipeline we explored in Part XXXVII. There, the goal is not to rank finished models but to collect training signal for a reward model that will guide the policy optimization. The annotation pipeline is similar in structure: human annotators compare pairs of responses to the same prompt and indicate which response better satisfies the desired criteria. But the statistical requirements differ in important ways.

For a ranking leaderboard, you need enough comparisons to produce stable, statistically significant ratings for each model. For RLHF reward modeling, you need a large, diverse set of comparisons that covers the space of responses the policy might generate. Coverage matters more than per-model depth. If the reward model never sees comparisons involving certain types of responses (very long responses, responses with numerical calculations, responses in rare languages), it will extrapolate poorly on those types and the policy will find ways to exploit those gaps during training.

The quality of RLHF preference data also determines the quality of the reward model, which in turn determines the effectiveness of the alignment process. Preferences that are inconsistent (the same annotator giving different answers to functionally identical comparisons), biased (preferring verbose responses regardless of content), or misaligned with the intended objectives (rewarding confident-sounding responses regardless of accuracy) will produce reward models that incentivize exactly the wrong behaviors. This is why the annotation guidelines, calibration procedures, and quality control measures described earlier in this chapter are not just academic concerns; they directly determine whether the resulting AI system is safe and helpful.

One limitation of crowd-sourced evaluation like Chatbot Arena is that the prompt distribution is uncontrolled. Users submit whatever they feel like, which means the distribution shifts over time as user demographics change, as the platform gains or loses popularity in specific communities, and as the models' capabilities improve and alter the types of prompts users find interesting to submit.

Arena-Hard addresses this by curating a fixed set of challenging prompts from the Chatbot Arena data, specifically selecting prompts that produce high variance in model performance. By using a fixed prompt set, comparisons across time become more directly comparable, and the resulting ranking is more interpretable. The tradeoff is that a fixed set becomes stale as models improve: prompts that were challenging in 2023 may be trivially easy for models trained in 2025. Periodic refreshes of the benchmark set are necessary to maintain its discriminative power.

This tension between replicability (requiring a fixed prompt set) and relevance (requiring a dynamic prompt set that shows current capabilities) is one of the enduring challenges of LLM evaluation. There is no perfect solution; different evaluation contexts call for different tradeoffs, and the best practice is to be explicit about which tradeoff you are making and why.

The preference evaluation system involves several parameters whose settings materially affect the quality of results:

• k_factor: Controls the sensitivity of Elo rating updates. Higher values (e.g., 32) allow ratings to respond quickly to new information but introduce volatility; lower values (e.g., 16) stabilize ratings but slow adaptation to real improvements. Selection should consider the expected rate of model improvement and the cost of rating volatility versus the risk of missing real changes in quality. In practice, dynamic K-factors that decrease as models accumulate more matches offer the best of both worlds.

• initial_rating: The starting Elo rating for all models (typically 1500), serving as the reference point for relative skill comparisons. While the absolute value is arbitrary, consistency within an evaluation ecosystem is needed for real comparisons. If new models always enter at 1500 regardless of their true strength, initial ratings create temporary distortions that take many matches to correct.

• scale: Conversion factor between Bradley-Terry beta parameters and Elo ratings ( ), mapping log-odds differences to the standard Elo scale. This parameter determines how much probability changes per Elo point and should be calibrated to match the empirical variance observed in your specific domain. A domain where human annotators have high variability will benefit from a larger scale factor that spreads ratings more widely.

• n_bootstrap: Number of resampling iterations for bootstrap confidence intervals (commonly 1000), balancing computational cost with statistical precision. Higher values give smoother estimates of the sampling distribution but increase computation time linearly. For preliminary results, 500 bootstraps suffices; for publication-quality confidence intervals, 2000 or more is preferable.

• alpha: Significance level for confidence intervals (default 0.05 for 95% intervals), determining the width of uncertainty bounds around rating estimates. Lower alpha values give more conservative bounds appropriate for high-stakes deployment decisions where false positives (incorrectly declaring a new model better) carry significant costs.

Preference evaluation systems have enabled the current generation of language model benchmarks, but they carry important limitations. Understanding these limitations is not an invitation to abandon the approach; it is a prerequisite for applying it wisely and interpreting results with appropriate skepticism.

The Elo system assumes transitivity (if A > B and B > C, then A > C), yet human preferences over LLM outputs frequently violate this assumption. A model optimized for helpfulness may defeat one optimized for safety in head-to-head comparison, which defeats a code-specialized model, which in turn defeats the helpfulness model when coding tasks are evaluated. These preference cycles mean Elo ratings compress multi-dimensional quality into a single scalar, potentially misleading users about which model is "best" for their specific use case. The appropriate response is not to remove the scalar ranking (which gives useful coarse-grained signal) but to supplement it with stratified per-domain rankings that reveal where each model excels.

Elo ratings are relative, not absolute. A model with a 1600 rating today might be stronger or weaker than a model with a 1600 rating from last year, depending on the evolving population of competitors. This non-stationarity creates challenges for longitudinal tracking of progress. Without maintaining fixed anchor models that persist across evaluation periods, you cannot determine whether the field is improving collectively or if rating inflation is occurring. Chatbot Arena has addressed this by treating GPT-4-Turbo as a long-term anchor, but this creates a different problem: if the anchor itself improves over time through version updates, the anchor's stability is compromised. True longitudinal tracking requires either frozen model checkpoints or explicit rating normalization procedures.

The annotation cost problem is more severe than it first appears. Detecting a 20-point Elo difference (roughly 53% win rate) requires hundreds of comparisons per model, creating significant annotation costs. The relationship between effect size and required samples follows standard power analysis: smaller differences require quadratically more data to detect reliably. For organizations running continuous evaluation pipelines, these costs can become prohibitive when many models are evaluated across many tasks. This is precisely what motivates LLM-as-Judge approaches, which replace expensive human annotations with automated model evaluations. We will explore those methods in the next chapter.

Gaming and overfitting to evaluation metrics are risks that grow as preference leaderboards become high-stakes. When models are explicitly trained to win comparisons rather than to be helpful, preference evaluation loses its validity as a signal. There is already evidence that some models are fine-tuned on examples from high-performing responses in Chatbot Arena, effectively using the evaluation as training signal and creating a form of contamination. Detecting and preventing such gaming requires regularly rotating the prompt set, monitoring for unusual win-rate patterns, and auditing model training procedures. This is an active and unsolved problem in evaluation methodology.

Preference evaluation also conflates multiple dimensions of quality into a single signal. When an annotator selects Response A over Response B, we cannot determine whether they did so because A was more accurate, more helpful, more engaging, or more concise. This aggregation is sometimes a feature (it shows an overall user judgment) and sometimes a bug (it makes it impossible to diagnose why one model outperforms another). The practical implication is that preference evaluation should be combined with dimension-specific automatic metrics and expert audits to give a complete picture of model capabilities.

Despite these challenges, preference-based evaluation has become the standard for RLHF data collection and leaderboards like LMSYS Chatbot Arena. This gives a more reliable signal than automated metrics for open-ended generation quality. The method successfully aggregates noisy individual judgments into stable population-level estimates, even if the scalar ranking obscures important nuances in model capabilities. Its stability across annotator calibration differences, its natural alignment with how humans experience quality, and its strong theoretical foundations in the Bradley-Terry model make it the gold standard against which other evaluation methods are benchmarked.

Preference evaluation turns the noisy problem of absolute quality assessment into the more tractable task of relative comparison. By using the Elo rating system and its connection to the Bradley-Terry model, we can aggregate thousands of pairwise judgments into reliable global rankings. Key takeaways include:

• Pairwise comparisons eliminate annotator calibration bias but require careful randomization to control for position effects. The invariance to additive shifts makes them fundamentally more reliable than absolute ratings for subjective quality assessment.

• Elo ratings give an online update rule that converges to maximum likelihood estimates of the Bradley-Terry model parameters, with the conversion formula

  where:

  • : the Elo rating on the standard scale (centered at 1500)
  • : the Bradley-Terry strength parameter
  • : the scale factor determining the rating unit size
  • : the natural logarithm of 10, converting from log-odds to base-10 exponential scale.

• Statistical significance requires bootstrap confidence intervals or sequential testing procedures; rating differences smaller than 30 Elo points typically require hundreds of comparisons to validate.

• Active sampling strategies that match models of similar strength can reduce evaluation costs by half compared to random pairing, prioritizing informative matchups over predictable blowouts.

• Intransitive preferences and multi-dimensional quality remain basic challenges, suggesting that scalar Elo rankings should be supplemented with task-specific evaluations and that preference cycles should be monitored as indicators of diverse model specializations.

• Real-world systems like LMSYS Chatbot Arena show that crowd-sourced preference evaluation can scale to millions of comparisons, but they require careful design choices around anchor models, rating deflation, and prompt distribution control to remain valid over time.

These methods form the foundation for both human evaluation campaigns and the emerging paradigm of LLM-as-Judge, which we explore next.

QuizLink Copied