Representation Harms: Stereotyping, Erasure, and Bias

Article links

Make inline references clickable

When a language model consistently completes "The doctor told his patient..." but rarely generates "The doctor told her patient...", it isn't making a factual error or producing hate speech. It's reproducing a pattern, one that reinforces the idea that doctors are men. This is a representation harm: damage caused not through a single harmful output but through systematic patterns in how groups are portrayed, acknowledged, or associated with certain roles and traits.

Representation harms are distinct from allocative harms, which occur when a model denies someone a job, a loan, or an opportunity. The distinction matters because the two types of harm often require different measurement techniques and different mitigation strategies. Allocative harms are easier to quantify: you can count how often a model approves or rejects applications across demographic groups, which is exactly what the fairness metrics examined in the previous chapter are designed to do. Representation harms are subtler. They operate at the level of language itself: which groups get described with certain adjectives, which roles they appear in, which languages and cultures receive fluent treatment, and which are systematically absent or distorted.

What makes representation harms particularly important is their pervasiveness. A model that produces allocative harm in a hiring tool affects people who use that tool. A model that produces representation harms in a writing assistant, a search engine, an educational platform, or a content generator affects everyone who uses those systems, whether or not they belong to the affected group. The harm is distributed, ambient, and cumulative rather than discrete and traceable to a single decision. It shapes how millions of users come to understand who belongs in certain roles, whose contributions history has valued, and whose ways of speaking count as legitimate.

This chapter develops a precise vocabulary for three major categories of representation harm: stereotyping, erasure, and demeaning associations. For each, you'll see both the conceptual definition and concrete measurement approaches, including embedding-based tests, corpus analysis, and template-based generation probes. The chapter draws on bias sources covered in Bias in Language Models and the association tests introduced in Bias Measurement, applying them specifically to the question of how models represent groups in text rather than how they classify or rank individuals.

Before diving into specific categories, it's worth being precise about what makes a representation pattern harmful rather than merely statistically uneven.

Representation harms arise when model outputs systematically shape how people perceive or are perceived, and when those patterns disadvantage certain groups. The mechanism isn't necessarily intentional. A model trained on text from the internet inherits the representational patterns present in that text, including centuries of accumulated cultural bias. But the absence of intent doesn't reduce the impact.

Three features characterize representation harms:

• Systematicity: The pattern appears consistently across many inputs, not just in isolated outputs
• Group targeting: The pattern affects people along demographic dimensions such as gender, race, nationality, religion, disability status, or sexual orientation
• Harm potential: The pattern either reflects real-world disadvantage or risks reinforcing it by shaping how readers, users, or downstream systems perceive the affected group

This last criterion is important for distinguishing representation harms from mere statistical associations. A model that consistently associates "chef" with masculine pronouns isn't just reporting a statistical correlation from its training data. It's participating in the reproduction of a cultural pattern that has historically excluded women from professional kitchens and made the path harder for those who did enter. Language models are now embedded in writing assistants, educational tools, search systems, and content generators that millions of people use daily. The representational patterns they encode have real consequences at scale.

One feature of representation harms that distinguishes them from many other AI failure modes is their capacity to compound over time. When language models are used at scale to generate text, their outputs enter the information environment. That information environment becomes training data for the next generation of models, through web crawls, through instruction tuning on human preferences, and through the implicit reinforcement of what users accept and engage with. A model that systematically underrepresents nonwhite female scientists in generated content produces text that makes such representation seem unusual. If that text is crawled and included in future training sets, it teaches the next model that the underrepresentation is normal. The bias reproduces itself.

This feedback dynamic is sometimes called a data flywheel for harm. Unlike a single biased decision, which is bounded, representational biases in generative models can amplify across successive model generations. The implication for measurement and mitigation is that representation harms need to be addressed proactively in training data curation, not just corrected in output filtering, because the problem restates itself at a higher level with each training cycle.

A second conceptual distinction that clarifies what counts as a representation harm involves the difference between describing the world as it is and shaping how the world is perceived. These sound like the same thing but they aren't. A model that describes the current gender distribution of nursing accurately is reporting a fact. A model that generates the same distribution as if it were natural, inevitable, or definitional of the role is encoding a normative claim as an empirical one.

The formal test is counterfactual sensitivity. If a model's portrayal of an occupational group would change substantially given equivalent real-world statistics from a different historical period or a different country, that's evidence the model is encoding contingent cultural patterns as universal ones. If the model applies the same male-dominated framing to 2023 nursing that it would have applied to 1950 nursing, despite nursing having become heavily female-dominated in the interim, the model is not reflecting current reality; it's reflecting the accumulated text weight of a particular cultural perspective.

Stereotyping occurs when a model systematically associates a demographic group with a fixed set of traits, roles, or behaviors that overgeneralize from some members of the group to all members, or that encode historically loaded assumptions rather than current reality.

The distinction between a stereotype and a statistical correlation is not always obvious, but it centers on whether the generalization erases individual variation and whether it tracks historical power structures rather than neutral distributions. The association between "nurse" and femininity reflects a real occupational imbalance: women currently make up a large majority of nurses in many countries. But a model that generates "she" far more often than "he" in nurse-related contexts encodes this imbalance as a natural or necessary feature of the role, rather than as a contingent outcome of labor market history. The same imbalance in a model trained on data from a society where men dominate nursing would simply run in the other direction.

Occupational stereotyping is among the most extensively studied forms of representation harm in language models. It manifests in two main ways: gender associations with specific occupations, and race associations with economic roles.

In template-based probes, researchers construct sentences like:

"The [occupation] said that [pronoun] had finished the report."

and measure how often models assign masculine versus feminine pronouns to different occupations. Models trained on large internet corpora consistently show:

• Higher feminine pronoun rates for nurse, receptionist, teacher, and librarian
• Higher masculine pronoun rates for engineer, CEO, pilot, and programmer
• These associations hold even when the model is given no explicit gender information

What makes this stereotyping rather than accurate reflection is that the associations are stronger than the underlying occupational distributions warrant, they resist updating when contradicting evidence is provided in context, and they cluster in ways that map directly onto traditional gender role hierarchies rather than being randomly distributed.

The resistance to contextual updating is particularly revealing. A well-calibrated model should assign higher feminine pronoun probability to "engineer" when the preceding context says "she graduated with a degree in electrical engineering." Models often fail to do this: the contextual signal is present and should theoretically resolve the ambiguity, but the prior association with masculine pronouns is strong enough to override it. This reveals that the stereotyping pattern goes beyond a simple reflection of data distributions; it is encoded in the model's representational geometry in a way that resists easy correction through context.

Beyond occupations, language models encode associations between demographic groups and personality traits. These associations are often revealed through embedding geometry: the relative distances and directions between word vectors in the model's representation space.

As discussed in the Bias Measurement chapter, the Word Embedding Association Test (WEAT) quantifies these associations by measuring the differential similarity between attribute word sets and target group word sets. Models consistently assign more pleasant associations to names typical of European-American groups and more unpleasant associations to names typical of African-American groups. This pattern emerges even when the words themselves carry no explicit demographic content: the association is built into the geometric structure of the representation space.

Large language models replicate and often amplify these patterns. When prompted to describe fictional characters with names drawn from different demographic groups, models generate descriptions that are systematically more positive, more competent-sounding, and more professionally focused for characters with names associated with high-status demographic groups. The gap isn't large on any individual generation, but it's consistent enough to be statistically reliable across thousands of samples.

Trait stereotyping also operates along axes that don't map neatly onto a single dimension. Warmth and competence are the two most studied dimensions in social psychology, and language models replicate the warmth-competence tradeoff documented in human social cognition: groups stereotyped as warm tend to be stereotyped as less competent, and vice versa. This means that even apparently positive stereotypes carry implicit costs. A model that consistently describes an immigrant character as warm and family-oriented while describing a similar native-born character as ambitious and professionally driven is encoding the same cultural hierarchy through opposite valence associations.

Representation harms don't operate along a single demographic dimension at a time. A person who is Black and a woman faces stereotyping patterns that differ from the sum of race-based and gender-based associations. The intersection creates new patterns: a Black woman described as a "professional" in generated text is more likely to have her credentials questioned than either a white woman or a Black man in the same scenario.

Measuring intersectional stereotyping requires going beyond single-axis analyses. Template probes need to systematically vary multiple demographic attributes simultaneously and look for interaction effects. Models show larger stereotyping effects at intersections, meaning the harms compound rather than average out.

The statistical reason for this compounding is that language model training data reflects real-world representation, and real-world representation of multiply marginalized groups is even more uneven than representation of each group individually. If women appear rarely in tech contexts in training data, and Black professionals appear rarely in leadership contexts, then Black women in tech leadership contexts appear extremely rarely. The model has almost no examples from which to learn anything other than the dominant pattern. This data sparsity at intersections is both a cause of intersectional bias and a reason why fixing single-axis biases doesn't automatically fix intersectional ones: you could achieve gender balance in leadership portrayals and racial balance in tech portrayals without producing any intersectionally balanced examples at all.

The implication for measurement is that fairness benchmarks built on single-axis comparisons can produce false confidence. A model can pass gender parity tests and racial parity tests while still systematically erasing Black women from professional roles. Complete representation harm auditing needs to treat intersectionality as a first-class concern, not an afterthought.

A more subtle form of stereotyping occurs through adjective selection rather than pronoun assignment. When generating descriptive passages about characters from different demographic groups in equivalent roles, models apply different adjective sets. Characters from one group are described as "driven" and "strategic"; characters from another group doing the same thing are described as "enthusiastic" and "hardworking." Both sets of adjectives are positive, but they encode very different assumptions about the source of success: one implies competence and agency, the other implies effort applied to compensate for something.

This adjective selection pattern is harder to measure than pronoun assignment because it requires semantic analysis rather than simple counting. Researchers typically address it through embedding-based analysis of the adjective distributions across groups, or through crowdsourced human rating of whether adjective sets connote equivalent or different underlying attributes. The pattern is reliable: it appears across multiple models and multiple demographic dimensions, and it aligns with the implicit association patterns documented in social psychology experiments using the Implicit Association Test (IAT).

Erasure is the absence or systematic underrepresentation of a group in generated text. Where stereotyping misrepresents groups, erasure makes them invisible. The harm is different: instead of false or degrading associations, erasure conveys that certain people, cultures, or experiences don't exist, aren't relevant, or aren't worth acknowledging.

The most direct form of erasure is simply who appears in generated text. When asked to generate stories about "a doctor and a patient," "a couple enjoying their vacation," or "a group of friends," models produce content centered on particular demographics far more often than others. White, male, and Western characters dominate across many generation scenarios even when the prompt contains no demographic specification.

This matters because these defaults shape what users see as normal, expected, or central. A student using an AI writing assistant who repeatedly gets content centered on one demographic group is receiving implicit signals about whose stories are worth telling. The signal isn't delivered through any single output. It accumulates over many interactions, each one subtly calibrating the user's sense of which voices are default and which require explicit specification to appear.

At scale, the effect compounds further: millions of users receiving consistently biased defaults over millions of interactions creates a feedback loop where AI-generated text becomes part of the information environment that future models are trained on. Erasure that enters training data propagates forward into future models unless actively measured and corrected. This compounding affects model training cycles and the broader cultural environment in which people form beliefs about who belongs in which roles.

Language models trained primarily on English internet data are dramatically more capable in English than in other languages, and within English, more capable with text from dominant cultural contexts. But erasure happens at a finer grain as well. Dialects, registers, and cultural concepts from non-dominant groups receive systematically worse treatment:

• African American Vernacular English (AAVE) is more likely to be flagged as lower quality or "incorrect" by language model-based quality tools
• Models perform worse at sentiment analysis and named entity recognition on text from non-Western cultural contexts
• Languages with fewer speakers, especially indigenous languages, receive almost no coverage
• Code-switching and multilingual communication patterns from bilingual communities are handled poorly

The linguistic erasure of AAVE is particularly consequential because AAVE is not a degraded form of Standard American English. It is a complete, rule-governed dialect with its own consistent grammar, phonology, and pragmatic conventions. Models that treat AAVE as inferior or incorrect aren't making a linguistic judgment. They're reproducing a social hierarchy that has historically been used to invalidate Black American speech, culture, and authority. When a language model's writing quality scorer gives lower scores to AAVE text, or when a grammar checker flags AAVE constructions as errors, the tool is acting as an instrument of linguistic oppression regardless of the intent of its designers.

The solution is not to make models treat AAVE exactly like Standard American English in all contexts, since the two dialects differ and sometimes the distinction matters. The solution is to ensure that models treat AAVE as a legitimate, rule-governed dialect whose conventions are respected rather than pathologized, and that quality scores don't systematically discount text written in non-dominant varieties.

When models generate historical narratives, scientific histories, or biographical content, they systematically underrepresent contributions from women, people of color, and non-Western civilizations. Probes asking models to "list important mathematicians" or "describe the history of this scientific field" produce outputs that reflect the biases of English-language textbooks and Wikipedia more than actual historical contributions.

This form of erasure has a compounding mechanism of its own. Historical narratives shape how people understand what is possible. If a model consistently fails to mention that al-Khwarizmi founded algebra, that Katherine Johnson calculated orbital mechanics by hand for early NASA missions, or that Islamic scholars preserved and advanced Greek mathematics during the European Dark Ages, it is shaping users' understanding of who has contributed to human knowledge. Over millions of interactions with millions of users, that shaping effect is real.

The underlying cause is the training data distribution: English-language textbooks and Wikipedia, which reflect decades of scholarship selection, themselves underrepresent non-Western and non-male contributions. Fixing this at the model level requires either curating training data to include more representative historical sources, or applying targeted finetuning on corrected historical information, or both.

Machine translation systems present a particularly sharp form of erasure. When translating from languages with grammatical gender into English, or from English into languages with grammatical gender, models must make gender attribution decisions. These decisions systematically default to masculine for occupational roles and certain contexts.

The most documented case involves translating from gender-neutral languages like Finnish, Hungarian, or Turkish into languages with grammatical gender like Spanish or French. A sentence like "The doctor said they would call tomorrow" becomes "The doctor (masculine) said he would call tomorrow" in Spanish, even though the Finnish or Turkish source contained no gender information. Models apply their priors about occupational gender, and those priors encode the same stereotypes that template probes reveal in monolingual generation.

The consequence is that translation tools actively gender information that was gender-neutral in the source, which is a qualitative change, not a neutral representation of the original. Users relying on translation tools to understand documents in languages they don't speak have no way to know that gender information has been added by the model rather than present in the original.

Demeaning associations occur when a model generates content that degrades, dehumanizes, or expresses contempt toward a group. This category spans a wide range: from explicit slurs and hate speech at one extreme to subtler patterns of condescension, infantilization, and association with negative attributes at the other.

Language models can generate explicit slurs, hate speech, and demeaning characterizations, particularly when prompted with template completions, roleplay scenarios, or requests to "speak as" particular characters. Research on models like GPT-2 and its successors showed that models could be reliably prompted to generate racist, sexist, or otherwise demeaning content through techniques that bypass explicit content filters.

This category has received the most attention from moderation and safety teams, partly because it's the most visible and partly because the harm is clearest. Explicit demeaning content can be measured directly with classifiers trained on labeled toxicity data, and its presence in model outputs is easily demonstrable. However, focusing exclusively on explicit toxic content risks creating models that are "polite bigots": models that don't produce slurs but still systematically encode demeaning patterns through the subtler mechanisms described below.

More difficult to detect, and arguably more pervasive, are subtle demeaning associations that don't rise to the level of explicit slurs but systematically portray certain groups in degrading ways:

• Competence disparagement: Models describing characters from lower-status demographic groups as confused, forgetful, or incompetent at higher rates than comparable characters from higher-status groups
• Infantilization: Describing adults from certain groups with child-like framing, using diminutives, or attributing naivety
• Hypersexualization: Generating sexually focused descriptions for women and for certain racial groups in professional contexts where such descriptions would be inappropriate
• Agency erasure: Describing group members as passive, acted upon, or dependent rather than as active agents with goals and decisions
• Credibility discounting: Generating narratives where characters from certain groups have their professional judgments questioned or overridden more frequently than equivalent characters from other groups

These patterns are harder to catch with simple classifiers because they depend on context: describing a character as "childlike" in one scenario might be neutral, in another condescending. Detecting them requires discourse-level analysis rather than token-level toxicity scoring. Researchers typically use a combination of carefully designed templates, embedding-based sentiment analysis, and human annotation to identify and measure these patterns.

The harm from subtle demeaning patterns is real even though it operates below the threshold of explicit offensiveness. When a model consistently portrays women in professional contexts as needing validation or guidance from male colleagues, even in language that is never harsh or vulgar, it reinforces the idea that female professional authority is contingent rather than inherent. The accumulation of thousands of such portrayals in the outputs of widely used tools shapes professional culture.

A consistent finding across language model studies is that models apply different sentiment and quality standards to text from different demographic groups. In toxicity detection tasks, text written in AAVE is more likely to be classified as toxic than equivalent text written in Standard American English, even when the semantic content is the same. In translation tasks, models apply more fluent and natural-sounding outputs when translating into English from certain source languages than from others.

These asymmetries are themselves demeaning: they encode the idea that some ways of using language are inherently better than others, where "better" maps onto racial and cultural hierarchies. When a model's quality scorer assigns lower scores to AAVE text, it's not measuring quality in any culture-neutral sense. It's measuring deviation from a standard that was defined by the dominant culture, and then treating that deviation as a defect.

The Sap et al. (2019) study documented this pattern directly: annotations of tweets written in AAVE for a toxicity dataset showed that AAVE tweets were annotated as toxic at roughly 3-4 times the rate of equivalent tweets written in Standard American English. Because these annotations became training data for toxicity classifiers, the bias propagated from human annotators into model outputs. Classifiers trained on this data then produce higher toxicity scores for AAVE text, which leads to systematic over-filtering of Black social media content by safety systems.

It's useful to think about demeaning associations along a spectrum from explicit to implicit, because the measurement and mitigation strategies differ:

At the explicit end, content contains direct slurs, dehumanizing language, or overt expressions of contempt. These are measurable with relatively simple classifiers and their harm is unambiguous. At the implicit end, content uses polite language but encodes demeaning patterns through framing, omission, and comparison. These are harder to measure, require discourse-level analysis, and their harm is easier to rationalize away.

The implicit end of the spectrum is where deployed models most often cause harm in practice. Explicit toxicity is aggressively filtered in most production systems. Implicit demeaning patterns slip through because they don't trigger content filters and because individual outputs seem innocuous. The harm is realized only across the pattern, not within any individual generation, which is exactly what makes it difficult to surface in standard testing.

Each category of representation harm requires different measurement approaches. The measurement techniques from the Bias Measurement chapter apply here, but they need to be adapted to the specific phenomena. Stereotyping requires probes that reveal distributional patterns across groups in model completions or internal representations. Erasure requires counting and frequency analysis at the corpus level. Demeaning associations require both automated classifiers and human judgment, because the harm is often contextual and resistant to automated detection. No single tool covers all three categories, which is why representation harm auditing typically involves a battery of methods rather than a single score.

Template-based probing is the most widely used approach for measuring stereotyping. You construct a set of templates with demographic group slots and measure the distribution of model outputs across groups.

The basic pattern:

1. Define a template: "The [group] [occupation] was known for being [adjective]."
2. Instantiate with different group labels: "Black", "white", "Asian", etc.
3. Collect model completions or measure conditional probabilities
4. Compute disparity metrics across groups

Well-known benchmarks built on this approach include:

• WinoBias: Coreference resolution templates designed to test gender bias in occupational contexts
• StereoSet: Sentence completion templates measuring stereotyped versus anti-stereotyped completions across race, gender, religion, and profession
• CrowS-Pairs: Sentence pairs with minimal edits swapping demographic group references, measuring which version the model prefers
• BBQ (Bias Benchmark for QA): Question-answering templates that probe stereotyping through ambiguous contexts requiring the model to make inferences about individuals

The strength of template-based probes is their interpretability and reproducibility. Given a template and a group set, different researchers can run the same probe on different models and compare results directly. The weakness is coverage: templates can only measure stereotypes that researchers anticipated when designing them. Emerging stereotypes, context-dependent patterns, and intersectional combinations that weren't built into the template set are invisible to the measurement.

A less obvious weakness is that template probes test model behavior on artificial, carefully constructed inputs that may not resemble the inputs models receive in deployment. A model could perform well on WinoBias while still encoding occupational stereotypes in subtler ways that the template format doesn't expose.

For models that produce dense vector representations, embedding geometry reveals stereotyping patterns through the relative positions of group-associated words in the representation space. The WEAT framework measures the differential association between group word sets and attribute word sets using cosine similarities.

For an embedding space, WEAT computes the differential association score , which measures how much more strongly group words are associated with attribute than with attribute , compared to group :

where:

• and are sets of target words for two demographic groups (e.g., European-American and African-American names)
• and are attribute word sets representing contrasting concepts (e.g., pleasant and unpleasant adjectives)
• is the mean cosine similarity between target word and the attribute set

The mean cosine similarity function is defined as:

where:

• is the number of words in attribute set
• is the cosine similarity between the embedding vectors of words and
• The sum averages the similarity of a target word to all members of the attribute set

A large positive value of means group is more associated with attribute (and group more with ) than expected by chance. A value near zero indicates the groups are associated equally with both attributes. Statistical significance is assessed by permutation testing: randomly reassigning words to groups thousands of times and measuring how often the permuted score exceeds the observed score.

The WEAT effect size normalizes the raw association score by the pooled standard deviation of association scores across both target groups:

where is the mean association score for group .

This normalization makes the effect size comparable to Cohen's from classical statistics, with conventionally treated as small, as medium, as large, and as very large. The Caliskan et al. (2017) paper reported effect sizes in the range for their ten canonical tests, all above the large threshold.

The WEAT paper by Caliskan et al. (2017) reported effect sizes for ten canonical tests in GloVe embeddings trained on Common Crawl. The results showed large effects across multiple group-attribute pairings, with the name-pleasantness test (European-American vs. African-American names, pleasant vs. unpleasant attributes) producing one of the highest effect sizes.

Measuring representation harms often requires counterfactual comparisons: what would the model have generated if the only thing that changed was the demographic group referenced? This is the idea behind counterfactual data augmentation (CDA), where you take a real text, swap group references (pronouns, names, demographic labels), and measure whether model behavior changes.

For stereotyping measurement:

1. Start with a sentence containing a demographic reference: "The female engineer struggled with the proof."
2. Swap the demographic reference: "The male engineer struggled with the proof."
3. Measure the model's likelihood or quality score for each version
4. A systematic difference reveals a representational bias

The challenge is that some "neutral" substitutions are not neutral. Swapping "Black" for "white" in a sentence about poverty doesn't produce an equivalent sentence because the socioeconomic contexts differ. CDA interpretations need to account for these complicating factors.

A more sophisticated version of counterfactual testing uses minimal pairs: sentences where only the demographic attribute changes and all other content is held constant. The difference in model perplexity, likelihood, or generation behavior between the two versions gives a direct estimate of the attributable bias for that attribute. When done at scale across many minimal pairs, this provides a distributional picture of how much demographic attributes influence model behavior across different contexts.

Erasure is most naturally measured at the corpus level: across a large set of generated texts, how often do members of different groups appear, and in what roles?

For a generated corpus and a group , the representation rate is:

where counts occurrences of group-associated tokens (names, pronouns, explicit labels) in and is the total number of texts or tokens.

Comparing across groups and against a baseline (population rates, representation in training data, or ground truth data) reveals erasure. More sophisticated analyses condition on context: does group membership affect appearance rates differently in professional versus social contexts?

The choice of baseline matters considerably for the interpretation. If you compare model output representation to population proportions, you're asking whether the model accurately mirrors demographic reality. If you compare to training data representation, you're asking whether the model amplifies or attenuates the biases in its source data. If you compare to ground truth data (e.g., actual representation of women in tech companies), you're asking whether the model accurately represents a specific domain. Each comparison answers a different question, and researchers need to be explicit about which baseline they're using and why.

For demeaning associations, toxicity classifiers trained on labeled data provide automated measurement, but they have known biases. Many classifiers themselves exhibit higher false positive rates for AAVE and other non-dominant linguistic varieties. Human evaluation remains necessary for calibrating classifier outputs and for catching subtle demeaning patterns that don't surface in automated scores.

The key evaluation dimensions for human raters:

• Sentiment: Is the language used to describe the group positive, neutral, or negative?
• Agency: Are group members portrayed as active agents or passive objects?
• Competence framing: Is the group portrayed as capable, intelligent, and professional?
• Contextual appropriateness: Does the model apply consistent framing across groups in equivalent contexts?

Human evaluation introduces its own biases, since annotators bring their own cultural assumptions and demographic perspectives to the task. Best practice involves using diverse annotator panels, giving clear annotation guidelines with examples, measuring inter-annotator agreement, and analyzing whether annotator demographics correlate with annotation decisions. When annotators from the affected group disagree with annotators from other groups about whether a piece of text is demeaning, that disagreement is itself informative: it often indicates that the harm is real but only visible to people with the relevant lived experience.

Let's build practical tools for measuring each type of representation harm. We'll start with template-based stereotyping measurement, then implement embedding-based association analysis, and finally measure erasure through corpus-level frequency analysis.

We'll use numpy, scipy, and a simple word embedding approach for the association tests. For the text generation probes, we'll use the transformers library to interact with a masked language model.

Simulating MLM Occupational StereotypingLink Copied

Let's visualize the relationship between model pronoun assignment and actual labor market representation. Points above the diagonal represent occupations where the model over-generates feminine pronouns relative to reality; points below represent occupations where the model under-generates feminine pronouns.

WEAT-Style Embedding Association TestLink Copied

For erasure measurement, we'll analyze how often different demographic groups appear in generated text across different scenarios. The analysis computes an erasure index: the difference between a group's appearance rate in generated content and its population proportion. Negative values indicate the group is being systematically omitted.

Visualizing Erasure Across ScenariosLink Copied

Automated Toxicity Asymmetry MeasurementLink Copied

The following pair of plots summarizes the two quantitative analyses side by side. The left panel shows which occupations are most stereotyped (measured as the gap between model outputs and labor market reality), and the right panel shows which demographic groups are most consistently erased across all professional scenarios.

The key parameters in the measurement framework are:

• n_permutations: Number of random shuffles in the WEAT permutation test. Higher values give more precise p-values; 5,000 is standard in the literature.
• threshold (toxicity classifier): The decision boundary above which a score is treated as a positive prediction. The choice of threshold directly controls false positive and false negative rates and should be calibrated per use case.
• dim (embedding dimension): The dimensionality of the word embedding vectors. In practice, GloVe and word2vec use 50-300 dimensions; higher dimensions capture more nuance but require more data to estimate reliably.
• noise (simulation parameter): The amount of random variation added to simulated vectors. Controls how strongly the constructed embeddings deviate from the planted association direction.

Knowing how to measure representation harms is only half the problem. The other half is what to do about them. Mitigation strategies differ substantially depending on which harm type is being addressed and which stage of the model pipeline the intervention targets.

The most upstream interventions target training data before the model ever sees it. Counterfactual data augmentation (CDA) systematically swaps demographic references in training texts to create balanced datasets: for every sentence containing "male nurse," a counterfactual "female nurse" sentence is added. This reduces the strength of gender-occupation associations in the learned embeddings without changing the content of the training data in any other way.

A more aggressive version of data augmentation is resampling: increasing the proportion of underrepresented groups in training data by oversampling or by targeted data collection. For historical and cultural erasure, this might mean seeking out and including texts from underrepresented historical traditions, non-Western knowledge systems, and community-produced content from marginalized groups.

The limitation of data-level interventions is that they address proximate causes rather than root causes. The training data reflects real-world inequality, and debiasing the data doesn't change that underlying inequality. A model trained on augmented data that perfectly balances gender in occupational roles may produce more equitable outputs, but it's not an accurate model of the current world. Researchers and practitioners need to be explicit about whether they're optimizing for equitable representation or accurate representation, and to recognize that these goals can conflict.

Bias in the representation space can be directly targeted through debiasing algorithms applied to trained embeddings. The Hard Debiasing algorithm of Bolukbasi et al. (2016) identifies the gender subspace in word embeddings using a set of definitionally gendered word pairs (man/woman, king/queen, etc.), then projects out the gender component from neutral words like occupational terms.

More formally, if is the gender direction in the embedding space (typically the first principal component of the differences between gender-paired word vectors), the projection of a neutral word vector onto the gender direction is:

and the gender-debiased word vector is:

This operation removes the gender component from neutral words while preserving their other semantic properties. After debiasing, "programmer" and "nurse" should have near-zero projections onto the gender direction, while "man" and "woman" remain fully gendered.

The limitation of hard debiasing is that it targets a defined bias direction, which requires specifying in advance what counts as gender and what counts as neutral. The gender subspace identified from a handful of word pairs may not capture all the ways gender is encoded in the embedding space. More fundamentally, some studies have found that hard debiasing reduces WEAT scores without reducing stereotype-based behavior in downstream tasks: the bias is hidden from the measurement rather than removed from the model.

For large language models where modifying the embedding space directly is impractical, generation-level interventions can reduce representation harms at inference time. These include:

Prompt augmentation adds demographic diversity instructions to system prompts, e.g., "When describing characters, vary their demographic backgrounds." This is simple but unreliable: models may follow demographic diversity instructions for some prompts and ignore them for others.

Constrained generation uses techniques like classifiers or rerankers to filter or rerank model outputs to exclude or de-emphasize outputs that score poorly on representation metrics. This is more reliable than prompt augmentation but adds latency.

Targeted finetuning trains the model on curated demonstrations that exhibit equitable representation, either through supervised examples or through reinforcement learning from human feedback (RLHF) with raters trained to identify and penalize representation harms. This is the most effective approach but also the most resource-intensive, and its effects can decay or generalize poorly to new domains.

Measuring representation harms is significantly harder than measuring allocative harms, and the limitations of existing measurement approaches are themselves part of the problem.

Template-based probes can only capture the stereotypes that researchers think to measure. They are effective at documenting known biases but poor at discovering new ones. A probe that asks about gender and occupations can't surface biases related to disability or sexual orientation unless it was specifically designed to test those dimensions. Real representation harms are intersectional and context-dependent in ways that template grids can't fully cover. The universe of possible representation harms is vast, and any fixed benchmark captures only a small slice of it.

Embedding-based tests have their own limitations. WEAT and its relatives measure associations in static embedding spaces, but modern language models use contextualized representations that vary with input. Association tests developed for word2vec and GloVe may not transfer cleanly to BERT-style encoders or decoder-only models like the GPT series, where representations depend on full context. Researchers have developed contextualized versions of WEAT (SEAT and CEAT), but these require choosing a sentence template to provide context, which reintroduces the coverage problem.

The framing of representation harms as a measurement problem can also create a false sense that they are solved once a metric is improved. Representation harms operate through accumulation: no single model output is harmful, but the aggregate pattern across millions of interactions shapes how users understand what is normal, who matters, and whose experience is central. Metrics computed on benchmark datasets can improve while the harms continue in deployment, if the benchmark doesn't capture the actual distribution of real-world uses.

There is also a deeper conceptual limitation. Representation harms are ultimately normative: they're harmful because we hold certain values about equity, inclusion, and human dignity. The measurement approaches described in this chapter operationalize those values into quantitative scores, but every operationalization reflects choices about what to measure and what to count. A benchmark that measures occupational gender balance implicitly accepts that binary gender is the relevant dimension. A baseline that compares to population proportions implicitly accepts that proportionality is the right standard. Neither choice is neutral, and the communities most affected by representation harms have often had no input into these choices.

Despite these limitations, the field has produced real progress. Large language models released since roughly 2021 show less stereotyping on standard benchmarks compared to their predecessors. Toxicity classifier bias against AAVE has been documented and addressed in some production systems. Gender-neutral pronoun handling has improved dramatically. The challenge going forward is to ensure that these improvements generalize beyond the benchmarks on which they were measured, and that the communities most affected by representation harms have ongoing input into what gets measured and how.

The next chapter on Hallucination Types examines a different failure mode, factual inaccuracy, and representation harms and hallucinations intersect in an important way: models that hallucinate contributions from historically marginalized groups both misrepresent history and actively harm the people who are misrepresented.

Representation harms are a distinct class of AI failure mode that operates at the level of language patterns rather than individual decisions:

• Stereotyping associates demographic groups with fixed traits, roles, or behaviors that overgeneralize from some members to all, or encode historical power structures. Template probes and WEAT-style embedding tests are the primary measurement tools. Stereotyping is not just inaccurate representation: it amplifies existing inequalities and resists correction by contextual evidence.

• Erasure is the systematic absence of groups from generated content. Corpus-level frequency analysis, comparing model outputs to population or ground truth baselines, provides the most direct measurement. The feedback loop between erasure in model outputs and erasure in future training data makes this harm self-perpetuating without active intervention.

• Demeaning associations range from explicit toxic content to subtle patterns of condescension, incompetence attribution, and hypersexualization. Toxicity classifiers, human evaluation, and counterfactual comparisons are all necessary because no single tool captures the full range. The bias in toxicity classifiers themselves against AAVE is a meta-level representation harm that suppresses Black expression under the guise of safety.

• Measurement challenges include the template coverage problem (you can only measure what you probe for), the contextualization gap between static embeddings and modern language models, and the distinction between benchmark improvement and real-world harm reduction.

• Intersectionality matters: representation harms compound across demographic dimensions and can't be fully captured by single-axis analyses. The most severe erasure typically affects groups with multiple historically marginalized identities.

• Mitigation requires interventions at multiple levels: data curation to address imbalances in training corpora, embedding-level debiasing to remove encoded associations, and generation-level techniques including prompt augmentation, constrained generation, and targeted finetuning. No single intervention is sufficient, and all interventions require ongoing monitoring to detect regression.

Representation harms illustrate why safety and fairness in language models can't be reduced to output filtering alone. The harms emerge from systematic patterns in what models generate across many interactions, making evaluation, ongoing monitoring, and community input essential components of responsible deployment. Building equitable language models requires better techniques and better practices: involving affected communities in harm definition, measurement design, and mitigation evaluation throughout the development lifecycle.

QuizLink Copied