Fairness Metrics: Demographic Parity, Equalized Odds

Article links

Make inline references clickable

When a hiring algorithm rejects qualified candidates from certain demographic groups at a higher rate, is that unfair? What if a medical risk model under-predicts illness severity for one population, even when its overall accuracy looks fine? These questions cannot be answered by accuracy alone. They require a vocabulary of fairness: a set of mathematical definitions that let you precisely specify what it means for a model to treat different groups equitably.

The field of algorithmic fairness emerged as a research discipline in the mid-2010s, driven by a wave of high-profile cases where automated decision systems were found to systematically disadvantage certain populations. Audits of recidivism risk tools in the criminal justice system revealed that Black defendants were assigned higher risk scores at rates inconsistent with their actual recidivism outcomes relative to white defendants. Facial recognition systems showed substantially higher error rates for darker-skinned women compared to lighter-skinned men. Hiring algorithms trained on historical data learned to penalize features correlated with being a woman, not because the features were causally related to job performance, but because the training data reflected past discriminatory hiring practices. These cases drew attention to the gap between a model's aggregate accuracy and its equitable treatment of individuals.

Researchers responded by developing formal definitions of fairness that could be measured, compared, and optimized, much like accuracy or precision. The resulting literature is rich and sometimes confusing: there are dozens of distinct fairness metrics with different names in different papers, many of which measure related but not identical things. This chapter cuts through that complexity by organizing the major definitions into coherent families and explaining the precise relationships between them.

Fairness metrics formalize intuitions about equitable treatment into measurable quantities. They give you a way to ask: does my model perform equally well across demographic groups? Does it fail in systematically different ways for different populations? Are individuals with similar characteristics treated similarly? The answers to these questions depend heavily on which fairness definition you adopt, because the definitions are not equivalent, and in most real-world settings they cannot all be satisfied at once.

This chapter covers the major families of fairness metrics: group fairness criteria like demographic parity and equalized odds, individual fairness and counterfactual approaches, calibration as a fairness requirement, and the fundamental mathematical tensions between them. By the end, you will understand how to calculate these metrics and why choosing between them requires thinking carefully about what kind of fairness matters most in your specific context.

Building on the bias measurement techniques from the previous chapter and the mitigation strategies explored in Bias Mitigation, fairness metrics are the formal evaluation layer that lets you verify whether your interventions achieve fairness, and according to which definition. Where bias measurement asks "does my model exhibit disparities?", fairness metrics provide the precise mathematical language to say "which kind of disparities, how large, and what would it mean to eliminate them."

Group fairness, also called statistical fairness or demographic fairness, asks whether a model produces similar outcomes across different demographic groups. The groups are typically defined by a sensitive attribute, such as race, gender, age, or disability status. Group fairness is the most commonly used family of fairness criteria in practice because it maps naturally onto legal and regulatory frameworks, is straightforward to measure from outcome data, and does not require specifying a similarity function between individuals.

Let's establish notation. We have:

• : input features (excluding the sensitive attribute, or possibly including it)
• : sensitive attribute (e.g., for group 0, for group 1)
• : true label (the ground truth outcome)
• : predicted label (the model's output)
• : predicted probability score, where for some threshold

Group fairness metrics compare properties of the joint distributions across values of . The key question is whether the distribution of outcomes, conditional on group membership, looks the same for all groups.

Demographic parity, also called statistical parity or independence, requires that the model's positive prediction rate is the same across all groups. Formally:

where:

• is the fraction of individuals in group 0 who receive a positive prediction
• is the same quantity for group 1

The intuition is direct: if 30% of group 0 applicants are approved for a loan, then 30% of group 1 applicants should also be approved, regardless of other differences between the groups. The model's decisions, aggregated at the group level, should look the same for both groups. The criterion is also called "independence" because it requires that be statistically independent of : knowing a person's group should give you no information about whether they receive a positive prediction.

In practice, you often measure the demographic parity difference (DPD) and the demographic parity ratio (DPR):

where:

• DPD of 0 and DPR of 1 both indicate perfect demographic parity
• The US Equal Employment Opportunity Commission's "four-fifths rule" considers selection rates below 0.8 (DPR < 0.8) as evidence of adverse impact in employment contexts
• Negative DPD values mean group 1 is selected at a lower rate than group 0

Demographic parity has an important weakness: it makes no reference to the true labels . A model that randomly approves applicants at equal rates satisfies demographic parity, but it is also useless. More subtly, in domains where the base rate of the positive outcome differs across groups, for example if actual default rates differ between demographic groups due to historical economic inequality, enforcing demographic parity forces the model to approve some higher-risk applicants from one group and reject lower-risk applicants from another. Whether this is appropriate is a policy question, not a statistical one.

Demographic parity is most suitable when you believe the differences in observed base rates are themselves artifacts of historical discrimination or measurement bias, rather than reflecting real differences in the underlying construct being predicted. In hiring, if equally qualified candidates from different groups have unequal representation due to past exclusion, demographic parity can serve as a corrective mechanism. If the base rate difference reflects real differences in the outcome variable, demographic parity will intentionally create misclassifications.

Equalized odds, introduced by Hardt, Price, and Srebro (2016), adds the true labels back into the picture. It requires that both the true positive rate (TPR) and the false positive rate (FPR) are equal across groups:

where:

• The condition gives the true positive rate: among truly positive individuals, both groups have the same chance of receiving a positive prediction
• The condition gives the false positive rate: among truly negative individuals, both groups have the same chance of receiving a false positive

This is a stronger requirement than demographic parity. It says the model must make the same kinds of errors at the same rates for both groups. If a model is better at detecting true positives for group 0 than group 1, it violates equalized odds even if its overall accuracy is identical across groups. The key insight is that equalized odds conditions on the true label, which means it only makes sense if the true label is reliable and unbiased. If is itself a product of biased measurement or past discrimination, conditioning on it can reproduce that bias.

Equalized odds is also called "separation" in the fairness literature, because it requires that the joint distribution of be the same conditional on . In other words, once you know whether someone truly belongs to the positive class, their group membership should have no bearing on their predicted outcome.

The equalized odds difference measures the maximum disparity across the two conditions. Taking the maximum ensures that a violation of either the TPR or FPR equality is captured:

where:

• The first term inside is the absolute FPR difference: how much more often does one group receive a false positive prediction?
• The second term is the absolute TPR difference: how much more often are one group's truly positive individuals correctly identified?
• A value of 0 indicates perfect equalized odds

Equalized odds protects both types of individuals simultaneously. Among those who are truly positive (qualified candidates, patients who need treatment, customers who will repay a loan), both groups are equally likely to be recognized. Among those who are truly negative, both groups are equally likely to be falsely flagged. This symmetric protection is what makes equalized odds particularly useful in contexts where both kinds of errors carry significant consequences.

Equal opportunity is a relaxation of equalized odds that focuses only on the true positive rate:

where:

• Only qualified or positive-labeled individuals are compared across groups
• The false positive rate is allowed to differ between groups

The motivation is asymmetric: in high-stakes domains like hiring, criminal justice, or medical diagnosis, missing a truly qualified individual (a false negative) is often the most consequential error. Equal opportunity ensures that among truly qualified applicants, both groups have the same chance of being selected. It does not require that both groups receive the same number of false positives, which can matter in contexts where the "false positive" outcome is not especially harmful to the individual being misclassified.

Consider college admissions. Among equally qualified students from different demographic groups, equal opportunity requires that each group has the same admission rate. It does not require that the overall admission rate be equal across groups, nor that the false positive rate (admitting students who will struggle academically) be equal. The focus is entirely on giving qualified individuals from all groups equal access to the positive outcome.

When deciding between equalized odds and equal opportunity, consider whether false positives matter differently than false negatives in your context. If both error types carry significant harm to the affected individuals, equalized odds is more protective. If false negatives are primarily harmful (denying opportunity) while false positives are relatively benign, equal opportunity is sufficient.

Predictive parity, also called calibration within groups or conditional use accuracy equality, requires that the precision is equal across groups:

where:

• is the fraction of individuals predicted positive in group who are positive (the precision, or positive predictive value, for group )

This is the flip side of equal opportunity. Instead of asking "given you are truly positive, what is your chance of being predicted positive?", predictive parity asks "given the model predicts you positive, what is the probability you are positive?" A model satisfies predictive parity if its positive predictions are equally reliable for both groups. A score of should mean the same thing regardless of which group you belong to.

Predictive parity has an important connection to calibration. A well-calibrated model produces probability scores such that for all score values . If a model is calibrated within groups, then any threshold applied to both groups will produce equal precision, giving predictive parity.

The COMPAS recidivism risk tool controversy is a useful case study here. The tool's developer, Northpointe, showed that among defendants assigned a "high risk" score, the recidivism rate was similar for Black and white defendants, showing calibration within groups. ProPublica showed that the tool assigned high-risk scores to Black defendants who did not reoffend twice as often as to white defendants who did not reoffend, showing a violation of equalized odds: specifically, a higher false positive rate for Black defendants. Both findings were mathematically correct. They reflect different fairness definitions applied to the same model, and their apparent contradiction is not accidental. As we will see, it is mathematically unavoidable.

The symmetric counterpart is negative predictive parity, which requires that the negative predictive value is equal across groups: negative predictive parity requires that the negative predictive value is equal across groups:

This asks: given the model predicts someone negative (rejects, denies, flags as low risk), what is the probability they are negative? If the model is better at correctly identifying true negatives for one group, its negative predictions are more trustworthy for that group. In criminal justice, this means: if the model says "low risk," is it equally likely to be correct for Black and white defendants? Negative predictive parity is rarely discussed compared to the four main criteria above, but in high-stakes contexts, asymmetric treatment of negative predictions can cause harm.

Let's build intuition with a concrete example. Imagine a simplified hiring classifier that predicts whether a candidate will be hired ( ) for two demographic groups. We will compute each fairness metric by hand to make the relationships concrete.

Suppose we have the following data, recorded as a confusion matrix split by group:

From this table we can compute all the rates:

• Group 0 (100 total, 50 truly positive): positive prediction rate = (40+20)/100 = 60%, TPR = 40/50 = 80%, FPR = 20/50 = 40%, Precision = 40/60 = 67%
• Group 1 (100 total, 40 truly positive): positive prediction rate = (15+5)/100 = 20%, TPR = 15/40 = 37.5%, FPR = 5/60 = 8.3%, Precision = 15/20 = 75%

Computing the fairness metrics:

• DPD = 0.20 - 0.60 = -0.40. Group 1 is selected at a rate 40 percentage points lower than group 0.
• DPR = 0.20 / 0.60 = 0.33. Well below the EEOC four-fifths threshold of 0.80; this would constitute adverse impact.
• Equal opportunity difference = |0.375 - 0.80| = 0.425. Among truly qualified candidates, group 1 is identified at a rate 42.5 percentage points lower. Qualified group 1 candidates are being passed over at a dramatically higher rate.
• Equalized odds difference = max(|0.083 - 0.40|, |0.375 - 0.80|) = max(0.317, 0.425) = 0.425. The larger disparity drives the equalized odds metric, coming from the TPR gap.
• Predictive parity difference = |0.75 - 0.67| = 0.08. Precision is close but not identical; a positive prediction for group 1 is slightly more reliable.

This example illustrates a pattern common in practice: the model simultaneously fails to find true positives in group 1 (low TPR) while producing too many false positives for group 0 (high FPR). These failures have different downstream harms. Group 1 candidates miss real opportunities they have earned. Group 0 candidates who do not deserve the positive outcome receive it, which may dilute program quality or create unfairness toward group 0 members who are displaced by less-qualified peers.

Notice that this model comes relatively close to predictive parity (0.08 difference in precision) even though it badly violates demographic parity and equalized odds. This is not a coincidence, and the mathematical reason for it will become clear when we examine the impossibility theorem.

Let's implement these metrics in code. We will use scikit-learn for generating classifier outputs and compute fairness metrics directly, then visualize how they interact.

First, let's install the required libraries and generate a synthetic dataset.

Threshold SensitivityLink Copied

All the group fairness criteria discussed so far share a common structure: they compare aggregate statistics across two or more demographic groups. This is useful and legally motivated, but it has an important limitation. A model can satisfy demographic parity by making random positive decisions at equal rates, even if it treats very similar individuals from the same group wildly differently. Conversely, a model that always assigns the same score to everyone in a group satisfies group fairness while being completely ignorant of individual circumstances.

Individual fairness asks a different question: are similar individuals treated similarly, regardless of their group membership?

The formal definition, due to Dwork et al. (2012), says the model should be Lipschitz continuous with respect to a meaningful input similarity. Two applicants who are nearly identical in their qualifications should receive nearly identical predictions, regardless of which demographic group they belong to. The Lipschitz condition bounds how much the output can change as a function of how similar two individuals are: if and are very close in the similarity metric (small ), then their predictions must also be close (small ). The constant controls the maximum allowable sensitivity.

The practical challenge with individual fairness is defining the similarity metric . For the definition to be meaningful, the similarity must capture task-relevant similarity, not just numerical distance in feature space. Two candidates with the same GPA from universities of very different selectivity are not equally qualified, but a naive Euclidean distance would treat them as similar. Constructing the right similarity metric often requires domain expertise and can encode its own contested value judgments. What counts as "equally qualified" in hiring? What makes two medical patients "similar" for the purpose of treatment decisions? These are policy questions that cannot be answered by the mathematics alone.

There is also a practical approximation challenge. Verifying individual fairness exactly requires checking every pair of individuals in the dataset, which scales quadratically. In practice, auditors often sample pairs and test consistency, or use Lipschitz-based certificates for specific model classes.

Individual fairness and group fairness are not equivalent and neither implies the other:

• A model can satisfy individual fairness but violate demographic parity, if the demographic groups differ in the feature distribution. If group 0 applicants have systematically higher qualifications, a fair-to-individuals model will approve more of them, creating a demographic parity violation.
• A model can satisfy demographic parity while violating individual fairness, if it randomly selects individuals to reach the target proportions without regard to individual qualifications.

Let's implement a simple individual fairness check based on consistency scores. The idea is to identify pairs of individuals who are similar in the input space but belong to different groups, and measure how different their predictions are.

Counterfactual fairness asks: would my decision change if this person belonged to a different demographic group, holding everything else constant?

This definition, from Kusner et al. (2017), uses causal reasoning. The notation means "imagine a world where person 's sensitive attribute is changed to , but everything causally unrelated to remains the same." If the prediction does not change under this counterfactual intervention, the classifier is counterfactually fair.

The difficulty is that computing counterfactuals requires a causal model of how the features are generated. If race affects which schools are available, which GPA scale is used, and which neighborhoods a person lives in, then a true counterfactual requires specifying all of these downstream effects. Changing only the sensitive attribute while holding everything else fixed is only coherent if you have a complete causal model that separates what changes from what stays the same when you change .

Naive approaches that simply remove the sensitive attribute from the feature set fail because other features may encode the same information through causal pathways. In our hiring example, removing the "group" column does not prevent the model from learning to use reported skill scores as a proxy, because reported skill scores were generated with a group-specific offset. Removing a proxy leaves the information in the data; it just becomes harder to trace. This is the proxy problem that causal fairness approaches are designed to address.

Counterfactual fairness is appealing because it aligns with legal intuitions: a person's protected status should not determine their outcome, and the way to check this is to ask what would have happened if they belonged to a different group. But it is computationally demanding and requires causal assumptions that are often difficult to validate from observational data alone. In practice, researchers use structural causal models (SCMs) to operationalize counterfactual fairness, identifying which features are direct descendants of in the causal graph and which are not. Features that are not causally influenced by can be used safely; features that flow through causal pathways from require special treatment.

Let's demonstrate a simplified counterfactual fairness check. We will test whether flipping the group indicator (while holding the other features constant at their observed values) changes the model's prediction.

As covered in the Calibration chapter from the Evaluation Fundamentals part of this book, a calibrated model produces risk scores that match empirical frequencies. Calibration within groups extends this requirement: among all individuals from group who receive score , the actual positive rate should be approximately .

where:

• is the predicted probability
• is a specific probability value
• The condition holds separately for each group

Calibration within groups is a meaningful fairness requirement because it ensures that the score means the same thing for everyone. A predicted probability of 0.7 should mean "70% chance of the positive outcome" for group 0 and group 1 alike. If the model is better calibrated for one group, then the same numerical score carries different information depending on group membership, which can lead to systematically different downstream consequences when decision-makers interpret scores as risk estimates.

The relationship between calibration and predictive parity is important to understand. Predictive parity at a fixed threshold requires equal precision: . If the model is calibrated within groups (meaning for all ), then applying the same threshold to both groups will produce the same precision for both groups, achieving predictive parity. Calibration within groups is therefore a sufficient condition for predictive parity. It is a stronger condition because it requires the score-outcome relationship to match at all score levels, not just at the binary threshold.

Let's visualize calibration curves separately for each group to check whether the model is calibrated within groups.

When calibration curves for both groups track the diagonal closely and track each other, the model is calibrated within groups: the predicted probability means the same thing regardless of group membership. When one group's curve is above the diagonal, the model is under-confident for that group (actual outcomes occur more frequently than predicted). When a curve is below the diagonal, the model is over-confident.

The COMPAS recidivism risk tool controversy is a useful case study here. The tool's developer, Northpointe, showed that among defendants assigned a "high risk" score, the recidivism rate was similar for Black and white defendants, showing calibration within groups. ProPublica showed that the tool assigned high-risk scores to Black defendants who did not reoffend twice as often as white defendants who did not reoffend, showing a violation of equalized odds: specifically, a higher false positive rate for Black defendants. Both findings were mathematically correct. They reflect different fairness definitions applied to the same model, and their apparent contradiction is not accidental. As the impossibility theorem explains, this is exactly what must happen when base rates differ between groups.

One of the most important theoretical results in algorithmic fairness is that most group fairness criteria are mutually incompatible in the general case. You cannot satisfy demographic parity, equalized odds, and predictive parity simultaneously unless one of two special conditions holds: either the base rates are equal across groups, or the classifier is perfectly accurate.

The Chouldechova impossibility theorem (2017) shows that if the base rates of the positive outcome differ between groups ( ), then no classifier can simultaneously satisfy all three of:

1. Predictive parity: equal precision across groups,
2. Equal opportunity: equal TPR across groups,
3. Equalized FPR: equal false positive rates across groups,

unless the classifier achieves perfect accuracy.

The proof follows from the relationship between precision, TPR, FPR, and base rate. Precision asks: "given a positive prediction, what fraction are truly positive?" We can expand this using the law of total probability:

where:

• is the base rate (prevalence of the positive outcome in the group)
• : the probability of a positive prediction among truly positive individuals
• : the probability of a positive prediction among truly negative individuals
• : the fraction of truly negative individuals

The crucial implication: if TPR and FPR are both forced to be equal across groups (equalized odds), then the only thing that can vary in the precision formula is the base rate . When base rates differ between groups, equal TPR and FPR mathematically force unequal precision. There is no way around this: the formula is an identity, not an approximation.

Working through the logic in the other direction: if we demand equal precision (predictive parity), and we know the base rates differ, then the TPR and FPR cannot both be equal. The formula dictates that at least one of TPR or FPR must differ. A model trying to maintain equal precision across groups with different base rates must compensate for the base rate difference by adjusting error rates.

This is not a failure of any particular algorithm. It is a mathematical constraint on the joint distribution of outcomes, predictions, and groups. When base rates differ between groups, there is a fundamental tension between giving everyone equal access (demographic parity), finding everyone's true positives equally well (equal opportunity), and making equally reliable positive predictions (predictive parity).

The Kleinberg et al. (2016) impossibility result shows a similar tension between calibration within groups and equalized odds: a calibrated classifier cannot achieve equal TPR and equal FPR across groups with different base rates, except in degenerate cases. These are independent proofs of overlapping impossibilities, together mapping out the full set of what cannot be achieved simultaneously.

Let's visualize this impossibility directly.

The Impossibility TriangleLink Copied

Knowing that fairness metrics conflict, you have to make a deliberate choice. The impossibility result does not tell you which metric to prioritize. That is a values question that depends on your specific context, who bears the costs of different errors, and what legal and ethical obligations apply. The following considerations structure that decision.

False positives and false negatives carry different costs in different contexts, and in different directions for the individuals affected versus the institutions making decisions.

In a hiring context:

• A false positive means hiring someone who does not perform well. Cost to the institution: wasted resources, opportunity cost, possibly a legal claim if the hire is let go.
• A false negative means rejecting someone who would have performed well. Cost to the individual: a lost opportunity, potential discrimination claim.

If false negatives cause more harm (you are denying opportunity to qualified people), equal opportunity (equalizing TPR) is the more relevant criterion. If false positives cause more harm (you are providing a service or resource to people unlikely to benefit, at cost to those who would), equalizing FPR or achieving predictive parity matters more.

In criminal justice, the asymmetry runs in the other direction. A false positive in a recidivism risk tool means detaining or monitoring someone who would not have reoffended. The cost falls entirely on the individual being incorrectly classified as high-risk. A false negative means under-supervising someone who does reoffend. The costs there fall on the community and on the victim. Neither direction is obviously worse, which is why criminal justice applications generate the most contentious fairness debates.

Equalized odds requires both TPR and FPR to be equal, which protects both types of positive-class individuals (through equal TPR) and both types of negative-class individuals (through equal FPR). Predictive parity protects the institution making decisions: it ensures that positive predictions are equally reliable as guides to action, so the institution doesn't bear higher error costs when acting on predictions for one group versus another.

When the costs of errors fall primarily on the individuals being classified, as in parole decisions, medical diagnosis, or college admissions, individual-centric criteria like equalized odds or equal opportunity are more ethically appropriate. When the institution bears primary costs, for example a bank losing money on defaulted loans, predictive parity arguments carry more weight.

A useful test is to ask: if you were in the group with lower predicted probability, which fairness criterion would you want? Most people answer equalized odds or equal opportunity, because they want their actual qualifications to be recognized at the same rate as everyone else's. This perspective helps clarify the distributional ethics of the choice.

The impossibility theorem only bites when base rates differ between groups. But before accepting observed base rate differences as ground truth, it is worth asking why they differ.

In many real-world settings, observed base rate differences are themselves artifacts of biased data collection. Criminal recidivism data is shaped by policing patterns: if one community is over-policed, its members are more likely to have prior arrest records, not necessarily because they commit more crimes. Using such data as ground truth and then demanding predictive parity creates a circular argument: it encodes bias in the labels, calls the resulting model "calibrated," and declares the model fair.

Medical diagnosis datasets often have lower true positive rates for groups that were historically excluded from clinical trials or have lower access to healthcare. When such groups receive later diagnoses, it is not because they have lower rates of disease; it is because the diagnostic pathway is harder for them to complete. Training on such data and conditioning on as if it were unbiased will propagate the healthcare access inequity.

If you suspect the ground truth labels are themselves biased, group fairness metrics that condition on (like equalized odds) can propagate rather than correct that bias. In these cases, demographic parity (which ignores ) or causal approaches that try to reconstruct an unbiased outcome may be more appropriate, even though they are harder to justify mathematically in isolation.

In the United States, the Equal Credit Opportunity Act, Fair Housing Act, and Equal Employment Opportunity Act create legal fairness obligations that roughly align with demographic parity through the "four-fifths rule" for adverse impact. If the positive selection rate for any group is less than 80% of the rate for the highest-selected group, that constitutes prima facie evidence of adverse impact. The European Union's AI Act creates additional requirements for high-risk AI systems, requiring fundamental rights impact assessments and ongoing monitoring for bias.

These legal standards do not map cleanly onto any single mathematical definition of fairness. Legal compliance is a floor, not a ceiling. You may satisfy the four-fifths rule while still having large TPR disparities that cause real harm to individuals from under-selected groups. The law has not caught up with the full richness of the fairness metric literature, which means practitioners bear responsibility for thinking beyond legal compliance.

Most fairness definitions consider a single sensitive attribute at a time: race, or gender, or age, without their combination. This creates a significant blind spot. A person's experience of discrimination is shaped by individual identities and by the intersection of multiple identities simultaneously. Legal scholar Kimberlé Crenshaw coined the term "intersectionality" in 1989 to describe how Black women faced discrimination that was neither fully captured by analyses of race alone nor gender alone. Their specific combination of identities created unique vulnerabilities that single-axis analyses missed entirely.

The same problem arises in algorithmic fairness. A model may be fair for women overall and fair for Black individuals overall, but systematically unfair specifically for Black women as a subgroup. If the subgroup is small enough that it does not move aggregate statistics significantly, a standard group fairness audit will not catch the disparity. Worse, mitigation interventions designed to fix the racial disparity or the gender disparity independently may leave the intersection unchanged or even worsen it.

Intersectional fairness requires disaggregating metrics by combinations of sensitive attributes. Instead of checking and separately, you check against all other combinations. This approach is significantly more data-intensive because intersectional subgroups can be small, leading to high variance in metric estimates. With binary sensitive attributes, there are subgroups to check, which creates both a statistical power problem and a multiple comparisons problem.

Several formal frameworks address intersectional fairness, including minimax fairness (optimizing for the worst-off subgroup rather than average group parity) and multi-calibration (requiring calibration within all subgroups simultaneously, including intersectional ones). These approaches acknowledge that "fair on average" can coexist with systematic unfairness for specific populations, and they shift the burden of proof toward the model developer to demonstrate fairness across all relevant subgroup combinations.

When selecting a fairness criterion for a specific application, work through these questions systematically:

• What are the positive and negative outcomes? Define clearly what means and who is helped or harmed by each type of error.
• What does ground truth represent? Is it a reliable, unbiased measurement, or is it itself a product of historical bias?
• Who bears the cost of errors? The individuals being classified, or the institution making decisions?
• What are the legal requirements? What adverse impact rules or fairness mandates apply?
• What is the base rate situation? If base rates differ significantly across groups, which impossibility trade-offs are you willing to accept?
• Are there intersectional subgroups at risk? Are there small demographic intersections that might be systematically disadvantaged without appearing in aggregate statistics?

No framework answers these questions automatically. The choice of fairness metric is ultimately a values question wrapped in mathematics.

The fairness metrics discussed so far have been framed around binary classification, which covers many high-stakes applications like hiring, lending, and medical diagnosis. But language models raise additional fairness challenges that these metrics do not directly address.

When a language model generates text, it does not produce binary predictions. It assigns probabilities to tokens, sentences, and continuations. Measuring fairness in generative settings requires adapting the binary classification framework or developing new approaches. Several directions have emerged:

Counterfactual input perturbation. One approach is to measure how model outputs change when a sensitive attribute in the input is swapped. Does a language model give different career advice when the user's described gender changes? Do summarization systems preserve positive framing about one group and negative framing about another? These tests operationalize a form of counterfactual fairness adapted to text generation.

Stereotype and association probing. Another approach measures whether the model associates certain demographic groups with certain roles, occupations, or characteristics at different rates. Word association tasks and template-filling probes like "The [doctor/nurse] was [adjective]" measure these associations directly. These tests are related to but distinct from classification fairness metrics; they measure representation harms rather than classification outcome disparities.

Toxicity and sentiment disparities. Metrics like the probability of generating toxic continuations, or sentiment differences in auto-completed text, vary across demographic groups mentioned in the prompt. These disparities can be quantified with group-level comparisons analogous to demographic parity, adapted to probability distributions over text rather than binary outcomes.

Embedding-level fairness. Representation bias in word and sentence embeddings can be measured using geometry. The WEAT (Word Embedding Association Test) and related methods measure the cosine similarity between demographic group terms and attribute terms, testing whether the embedding space encodes stereotype associations. These are related to but different from outcome-based fairness metrics; they measure what the model has learned to represent, rather than what decisions it makes.

These generative and representation-based fairness concerns are covered in more detail in the chapter on Representation Harms. The key point is that the fairness metric framework developed here, grounded in classification outcomes and group statistics, is a necessary starting point but must be extended for the full range of language model applications.

Fairness metrics are necessary but not sufficient for building fair AI systems. Understanding their limitations prevents the false comfort of "my model passes the fairness audit."

Goodhart's Law applies to fairness. When demographic parity becomes a target metric, optimizing it can achieve the metric without achieving the underlying goal. A model that randomly upsamples predictions for a disadvantaged group can satisfy demographic parity without being fair in any individual sense. The phenomenon is well-documented: once a measure becomes a target, it ceases to be a good measure. In high-stakes settings, fairness audits should include qualitative review and stakeholder input alongside quantitative metric checks.

Metrics aggregate over individuals. Equalized odds says "on average, TPR is equal across groups." But individual members of each group may experience very different outcomes. A fair metric can mask extreme unfairness for specific subpopulations, especially when those subgroups are small or intersectional. A model that is perfectly fair for men and for white individuals overall can still systematically disadvantage white women or men of color in ways that aggregate statistics will not reveal unless you specifically look for them.

Ground truth is often itself biased. In criminal justice, healthcare, and hiring, the labels used for training are produced by human systems with documented biases. Fairness metrics that condition on , including equalized odds and equal opportunity, will propagate those biases if is not itself fair. Choosing metrics that are robust to label bias, or investing in better ground truth measurement, is often more impactful than optimizing existing metrics on biased labels.

Fairness metrics do not capture harm. The harm caused by a false positive in a parole decision, keeping someone incarcerated who would not reoffend, is qualitatively different from the harm of a false positive in a product recommendation, showing an ad someone does not want. Fairness metrics are dimensionless numbers; they do not encode the stakes of the decisions they evaluate. A fairness analysis is only as meaningful as the human judgment about what the errors cost for the people affected.

Sample size and statistical power matter. Fairness metrics estimated on small samples are noisy. A demographic parity difference of 0.05 computed on 50 individuals from each group carries wide confidence intervals and should not be treated as precise. In practice, fairness audits should report confidence intervals or conduct hypothesis tests, not just point estimates. When subgroups are small, as in intersectional analysis, the statistical power to detect real disparities may be low, which means failing to find a disparity is not the same as confirming fairness.

Despite these limitations, fairness metrics have had real impact. They created a common language for discussing algorithmic discrimination, enabling audits of deployed systems that previously operated without accountability. They influenced policy frameworks in the United States, European Union, and beyond, creating legal teeth for fairness obligations that previously had no measurement standard. They demonstrated to the industry that accuracy and fairness are not the same objective, a realization that prompted widespread adoption of fairness-aware modeling practices. The field has moved from intuitive notions of "this feels unfair" to quantitative, falsifiable claims, which is a prerequisite for systematic improvement even when the measurements are imperfect.

Fairness metrics translate ethical intuitions about equitable treatment into measurable mathematical quantities. The key concepts covered in this chapter are:

• Demographic parity: positive prediction rates are equal across groups, regardless of true outcomes. Simple to measure and does not require conditioning on labels; ignores whether the labels themselves are biased.
• Equalized odds: both TPR and FPR are equal across groups. Provides symmetric protection for both positive and negative class individuals; requires reliable ground truth labels.
• Equal opportunity: only TPR is equal across groups. A targeted protection for the positive class, appropriate when false negatives are the primary harm.
• Predictive parity: precision is equal across groups. Ensures the model's positive predictions are equally reliable; favors the institution making decisions over the individuals being classified.
• Individual fairness: similar individuals are treated similarly according to a task-specific similarity metric. Requires defining what "similar" means, which is a domain-specific problem.
• Counterfactual fairness: predictions do not change when a person's sensitive attribute is counterfactually changed while holding causally unrelated factors constant. Requires a causal model of feature generation.
• Calibration within groups: risk scores mean the same thing for all demographic groups. A sufficient condition for predictive parity and a foundation for trustworthy probabilistic predictions.

The most important theoretical result is the impossibility theorem: when base rates differ across groups, no imperfect classifier can simultaneously satisfy equalized odds and predictive parity. This forces a deliberate choice about which criterion matters most for your application.

Choosing among fairness metrics requires thinking carefully about: the costs of different error types and who bears them, whether ground truth labels are reliable and unbiased, intersectional subgroups that may be hidden in aggregate statistics, and the legal and regulatory context.

In the next chapter, we explore representation harms, a different dimension of fairness that concerns classification decisions and the ways language models encode and reproduce stereotypes and erasure in their outputs.

QuizLink Copied