Memorization and Privacy in Language Models

Article links

Make inline references clickable

When a language model generates text, it does not look up facts in a database. It produces output entirely from patterns compressed into billions of numerical weights during training. Yet in practice, these models sometimes reproduce training text verbatim: phone numbers, email addresses, passages from books, medical records, and even API keys that appeared somewhere in the training corpus. This memorization problem affects model behavior, data governance, and privacy law.

Memorization is not a bug in the traditional sense. A model with zero memorization would struggle to reproduce standard phrases, correctly spell proper nouns, or recite well-known facts. Some memorization is necessary for useful language models. The problem is when a model can be prompted to reproduce information that individuals or organizations reasonably expected to remain private. Understanding the conditions and extent of memorization is essential for anyone building or deploying language models responsibly.

This chapter covers the measurement of memorization, the phenomenon of extractable memorization that allows adversarial extraction of training data, the prevalence of personally identifiable information in training corpora, and the broader privacy risks that follow. The next chapter explores differential privacy as one formal mitigation approach.

Before measuring memorization, we need a precise definition. Intuitively, a model has memorized a training example if it can reproduce that example. But this definition has several dimensions worth unpacking, and getting the definition right matters both for measurement and for understanding the severity of the risk.

Verbatim memorization means the model can reproduce a sequence of tokens character-for-character as it appeared in training data. This is the most straightforward case: given a prefix from the training text, the model generates the exact continuation.

Approximate memorization is broader. A model might reproduce a piece of text with minor variations: different whitespace, slightly reordered clauses, or a changed proper noun. These approximate matches are harder to detect but still represent real privacy risks. A model that reliably outputs someone's address with the street number off by one is still leaking private information. Approximate memorization also makes auditing harder because string-matching checks for exact reproduction will miss it.

There is a third category that sits between these two: semantic memorization. This occurs when the model does not reproduce the surface form of a training example but does reproduce its specific semantic content in a way that reveals private information. If a training document described "a 42-year-old male patient with diabetes and hypertension in Seattle admitted on March 3rd," a model that generates this fact in response to a medical query has effectively leaked private health information even if the phrasing differs. Semantic memorization is the hardest to measure and the hardest to defend against.

Some memorization is intentional. You would not want your language model to forget that Paris is the capital of France, or to mangle common phrases. This type of memorization captures broadly shared factual knowledge. It is benign and desirable.

Unintentional memorization occurs when a model happens to reproduce specific, private information that appeared in the training data. This includes personally identifiable information (PII) like names combined with contact details, unique text from private documents that ended up scraped into the training corpus, or rare sensitive facts that a user never intended to share publicly.

The distinction between intentional and unintentional memorization is easier to state than to implement. The model does not mark certain memories as "public knowledge" and others as "private." Both categories emerge from the same gradient descent process applied to the same training data. Context and training data curation are the only levers available before training; extraction resistance and privacy-preserving training techniques are the levers available after. This means the burden falls heavily on whoever constructs and curates the training corpus.

An important nuance is that memorization can shift from unintentional to functionally intentional based on how a model is fine-tuned or deployed. A base language model may have memorized a phone number incidentally during pretraining on web data. If that model is then fine-tuned on a customer support corpus where the same number appears frequently, and then deployed as a customer service agent, the model may now reliably produce that number in response to relevant prompts. At some point along that chain, the memorization crosses from accidental to embedded behavior.

Research has shown that memorization is not a binary property. It exists on a spectrum, and different parts of the training corpus are memorized to different degrees. Thinking about memorization as a continuous property rather than a yes/no distinction leads to better-calibrated risk assessments and more targeted mitigations.

The broad categories along the spectrum are:

• Highly memorized: Sequences that appeared many times across the training data or that are very long and specific. Book passages, code snippets, and contact information in templates all fall here. These can often be extracted with short prompts.
• Moderately memorized: Sequences that appeared a few times or that have partial uniqueness. These may be extractable with carefully crafted prompts but not with generic ones.
• Weakly memorized: One-off sequences that were encountered during training but are only recoverable under specific prompting conditions, if at all.
• Not memorized: Sequences with high perplexity that were not reinforced by repetition and show no evidence of being stored in the model's weights.

Understanding where on this spectrum a given piece of data falls helps predict privacy risk and informs data curation decisions. A practical implication is that the most dangerous memorization is also the easiest to measure and mitigate: repeated sequences stand out in audit procedures and are removed efficiently by deduplication. The harder problem is the long tail of weakly memorized content that does not show up clearly in automated checks but can still leak under the right adversarial conditions.

To understand memorization it helps to think about how gradient descent produces this behavior. During training, the model is rewarded for assigning high probability to training sequences. For sequences that appear many times, the gradient signal is correspondingly strong: every appearance of the sequence pushes the weights toward configurations that make that sequence highly probable. The model is not trying to memorize; it is trying to minimize loss. Memorizing frequently-repeated sequences is a natural consequence of doing so.

For very unusual sequences, a different mechanism can drive memorization. Highly specific text, like a UUID or a rare proper name, has very few plausible completions. The model learns that when it sees a certain unusual prefix, one particular continuation is overwhelmingly more probable than any alternative. This tight association between a distinctive prefix and its continuation is essentially memorization, even when the sequence appeared only once.

This means we have two distinct routes to memorization: repetition-driven memorization (common text reinforced across many training examples) and specificity-driven memorization (unique text with low entropy that the model finds easy to compress). Both contribute to the overall memorization rate, and they call for slightly different mitigations.

The challenge of measuring memorization is that we cannot simply ask the model "did you memorize this?" We need empirical methods that probe the model's behavior without assuming knowledge of what was memorized. The most influential work in this area developed both the formal definitions and the practical tooling that the field now uses.

The most influential framework for measuring memorization was introduced by Carlini and colleagues in a 2021 paper titled "Extracting Training Data from Large Language Models." Their approach defines memorization formally and provides a practical extraction method that demonstrated, empirically, that real private information could be extracted from deployed language models.

They define a training example as -memorized if there exists a prefix of length tokens drawn from such that giving the model makes it dramatically more likely to produce the continuation of than a model would without any meaningful context. Formally:

where:

• is the full training sequence being tested (the candidate memorized example)
• is a prefix of tokens taken from the beginning of (the prompt given to the model)
• is the prefix length, which controls how much context the attacker provides
• LM refers to the language model's autoregressive generation process
• The right-hand side is the baseline probability of producing the continuation when given only a random, uninformative prompt

The notation captures the essential idea: the conditional probability given the natural prefix should be orders of magnitude larger than the baseline. If the model is no more likely to produce a sequence when given its own prefix than when given a random prompt, the sequence is not memorized in any meaningful sense. If the probability ratio is extremely large, the sequence is almost certainly memorized verbatim.

The parameter controls the strength of the definition. When is small (say, 5 tokens), we are asking whether a very short prompt can elicit the sequence, which is a strong form of memorization. When is large (say, 100 tokens), we are giving the model almost the entire sequence and asking if it can complete the rest, which is a weak form of memorization. A sequence that is -memorized is a greater privacy concern than one that is -memorized, because extracting it requires less prior knowledge of the sequence.

In practice, the exact probability is difficult to compute for long sequences, so Carlini et al. operationalize this by comparing the model's perplexity. Perplexity is the exponential of the average negative log-likelihood per token:

where:

• is the total number of tokens in the sequence
• is the number of prefix tokens provided
• is the -th token in the sequence
• is the model's probability of the -th token given all preceding tokens

Lower perplexity means the model assigns higher probability to the sequence. A sequence is considered memorized when its perplexity given the natural prefix is dramatically lower than its perplexity given a random prompt of the same length. The ratio of these two perplexities quantifies how much the natural prefix raises the probability of the sequence compared to a generic prompt. This ratio, which Carlini et al. call the memorization score, is the central empirical quantity in their framework.

A related but stronger concept is extractable memorization: a training example is extractable if an adversary who does not know the example can nevertheless craft a prompt that causes the model to generate it.

The extraction procedure works as follows. An attacker generates a large number of model completions (hundreds of thousands to millions) using a variety of prompts, including common prefixes and intentionally unusual prompts designed to shift the model out of generic behavior. The attacker then compares these completions against any available dataset or applies deduplication heuristics to identify repetitive exact sequences. Sequences that appear verbatim in the model's output and are unique enough to be unlikely coincidences are flagged as extracted training data.

The key insight is that an adversary does not need access to the training data to run this attack. The extraction procedure requires only black-box access to the model's text generation API. This makes it applicable to any deployed model, including models where the training data is not publicly known. The attacker's investment is computation, not information.

Carlini et al. applied this method to GPT-2 (the 1.5 billion parameter version) and successfully extracted hundreds of examples of training data, including:

• Names, phone numbers, and email addresses of specific individuals
• IRC chat logs and personal blog posts
• News articles and Wikipedia text
• A 128-bit UUID that appeared only once in the training data

The single-occurrence UUID extraction is particularly striking. A model trained on a 40 GB corpus was able to reproduce a unique 32-character string that appeared exactly once. The probability of generating that string by random chance is vanishingly small, which confirms true memorization rather than coincidence. This result demolished the intuitive assumption that neural network compression would prevent direct recovery of specific training examples.

A useful numerical summary is the fraction of training examples that are -memorized for a given prefix length . Larger means we give the model more of the target sequence before asking it to complete, which makes memorization easier to trigger. By varying from 0 (no prefix) to 50 (50 tokens of prefix), we can construct a memorization curve that characterizes how much of the training data is recoverable at different levels of prompting.

Research on GPT-2 found that roughly 0.1% to 1% of training examples could be extracted with prefix lengths between 10 and 50 tokens. While this sounds small, GPT-2 was trained on approximately 40 GB of text. A 1% memorization rate across such a corpus amounts to hundreds of megabytes of extractable training data. When scaled to models trained on trillions of tokens, even a fraction of a percent represents a staggering quantity of memorized content.

A follow-up study by Carlini and colleagues in 2022 examined larger models and found that memorization scaled predictably with model size, training duration, and repetition in the training data. The study introduced the concept of a "memorization curve": a plot of memorization rate against prefix length. Models that memorize more aggressively show steeper memorization curves, reaching high extraction rates with shorter prefixes.

Not all models memorize at the same rate, and not all types of training data are equally at risk. Several factors influence how much a specific example is memorized:

Repetition in training data. The single strongest predictor of memorization is how many times an example appears in the training corpus. A sequence that appears 100 times across the training data is far more likely to be memorized than one that appears once. This makes data deduplication a practical privacy mitigation, because deduplication directly reduces this main risk factor.

Model size. Larger models memorize more. A 6 billion parameter model memorizes proportionally more training examples than a 117 million parameter model, even when both are trained on the same data. This is counterintuitive to many practitioners who assume that larger models would generalize better and rely less on specific examples, but the relationship runs in the opposite direction: more capacity enables more memorization. The model has room in its parameter space to store more specific patterns, and gradient descent fills that space.

Training duration. Models trained for more steps memorize more. Early in training, the model is learning broad patterns. As training continues, it gradually encodes more specific information. This means that longer training runs carry higher memorization risk, even when the training data is fixed.

Position in context. Some research suggests that training examples near the end of long documents are memorized more reliably than examples near the beginning, possibly because the model encounters the end of a document more often when data is chunked. Context window chunking strategies can inadvertently influence which parts of a document receive more training signal.

Template and structured text. Highly formulaic text, such as legal boilerplate, contact information templates, or code with repeated patterns, is memorized at higher rates than diverse natural prose. The regularity reduces the model's need to learn generalizable patterns and instead rewards verbatim reproduction. If you have ever seen a language model correctly reproduce a standard legal disclaimer, you have observed template memorization in action.

Data source characteristics. Different data sources contribute to memorization at different rates. Code repositories, for instance, contain large amounts of repeated boilerplate (license headers, standard imports, framework scaffolding) that appears across thousands of files. Legal documents often contain standardized language that appears verbatim across many filings. These structural features of certain data sources make them particularly prone to producing memorized content.

The memorization problem would be less urgent if training corpora contained no private information. Unfortunately, the opposite is true: large web-scale training datasets contain substantial amounts of personally identifiable information, and this is not accidental. The web evolved to host contact information, discussion forums, legal records, and social profiles. Any system that systematically scrapes the web will capture all of these categories.

Common Crawl and similar web archives contain snapshots of public web pages. Many of these pages include PII through a variety of mechanisms.

Contact pages and directories are the highest-density PII source. Business directories, professional association membership lists, and alumni networks explicitly list names, addresses, phone numbers, and email addresses. These listings are publicly accessible and frequently crawled. They are also frequently out of date: a person's contact information may remain in a directory for years after they have changed jobs, moved, or requested removal.

Forum posts and comment sections are a qualitatively different source of PII. Users discuss personal situations and sometimes include identifying details, either intentionally (asking for help with a specific situation) or inadvertently (mentioning a person's name in context, sharing a screenshot that includes metadata, or referencing a real address in the course of a discussion). Forum posts are rich with narrative context, which means PII in forums often appears with more surrounding information than in a plain directory listing.

News articles mention individuals by name in connection with events. People who were newsworthy for any reason, including crime victims, accident witnesses, parties in public disputes, or simply attendees at a public event, may find their names appearing in training data. The distinction between "news-relevant" and "private" is culturally and legally contested. A person mentioned in a local newspaper story about a neighborhood dispute did not necessarily consent to having their name encoded in a language model.

Government and legal documents contain names and addresses of parties in public records. Court filings, property records, business registrations, and voter rolls are often technically public but were not expected by individuals to be machine-readable and searchable in the way that training corpora make possible. The aggregation of this information with other sources dramatically increases privacy risk beyond what any individual record creates on its own.

Code repositories present a distinctive PII risk. Developers inadvertently commit API keys, credentials, and test data containing real user information with surprising regularity. Studies of GitHub and similar repositories have found thousands of real email addresses, authentication tokens, and database connection strings. Even when these are subsequently revoked or removed, they may persist in git history and web snapshots that end up in training data.

The PII in these sources ranges from clearly public (a politician's official phone number) to ambiguously public (a person's name on a company directory) to clearly private (a medical professional's personal cell phone number scraped from a small business listing they no longer maintain). Training data curation cannot easily distinguish these categories at scale, which is why the privacy problem is fundamentally difficult to solve through filtering alone.

Research by Subramani and colleagues at AI researchers quantified PII in Common Crawl-derived datasets. They found that a sample of 15 million training documents contained tens of thousands of unique email addresses, phone numbers, and social security numbers. The density of PII was not uniform: certain categories of websites (job boards, member directories, legal filings) contributed disproportionately.

An important complication is that what counts as PII depends on context and jurisdiction. Under GDPR, even an IP address can be PII. Under HIPAA, a patient's date of birth combined with their diagnosis is protected health information. Under California's CCPA, a person's inferences drawn from professional or employment information, education information, or commercial information can be protected. Language models trained on web data potentially encode all of these categories, and the model itself does not know which sequences are sensitive.

The aggregation problem compounds this. A model might memorize a person's first and last name from one document, their city of residence from another, their employer from a third, and their approximate age from a fourth. None of these facts is particularly sensitive in isolation. But a user who queries the model with enough context might receive a response that combines all four, effectively reconstructing a private profile from separately innocuous facts. This aggregation risk is much harder to measure and mitigate than direct PII memorization.

Training data often contains near-duplicate documents: multiple versions of the same news article, forum discussions that quote earlier posts, and scraped websites with repeated contact information across multiple pages. Near-duplicates amplify memorization risk because they increase the effective repetition count of the contained text without appearing as exact duplicates to standard deduplication filters.

If a person's contact information appears on their business's website, a cached version of that website, three scraped directories that aggregated the original, and two forum posts that mentioned the business, the same PII may appear six or more times in the training data. The model will memorize it reliably, and the information may be extractable even from brief prompts.

The insidious aspect of near-duplicate amplification is that it is invisible to standard occurrence counting. If you check how many times the string "John Smith, 555-0123" appears verbatim in the training data, you might find only two exact matches. But the same information might appear in paraphrased form in ten other documents, and the model integrates all of these training signals into a coherent representation that makes it highly likely to reproduce the underlying information regardless of surface form.

This motivates more sophisticated deduplication strategies that operate at the document level rather than the string level, using techniques like MinHash locality-sensitive hashing to identify and remove semantically similar documents even when they differ in phrasing.

We can implement a simplified version of memorization measurement. This example demonstrates the core idea: given a sequence and a model, estimate how much more likely the model is to produce the sequence given a natural prefix versus a generic prompt.

Counting Repetitions and Memorization RiskLink Copied

Visualizing Memorization by Repetition CountLink Copied

Measuring memorization in a controlled setting is one thing. Extracting training data in an adversarial setting is another. Extraction attacks demonstrate that memorization creates real privacy risks, not just theoretical ones.

The extraction attack described by Carlini and colleagues operates as follows. The attacker has access to the model's generation API but does not have direct access to the training data. The attacker generates a large number of text completions from the model using diverse prompts, then analyzes these completions to identify likely training data.

The prompts can take several forms:

• Empty or near-empty prompts: Completions from very short prompts tend to capture the model's most strongly memorized content, since there is little context to guide generation toward anything other than familiar sequences.
• Style-shifting prefixes: Prompts designed to shift the model into a particular style or domain (for example, a few lines of code to shift into code generation mode, or a news article lead sentence to shift into news content) can expose memorized content from those domains.
• Repeated token attacks: Feeding the same token repeated many times (for example, 50 repetitions of the newline token) can destabilize the model's standard generation behavior and cause it to fall back to memorized sequences.

After generating completions, the attacker applies several filters to identify likely memorized text. Membership inference heuristics assess whether a completion looks "too specific" to have been generated by chance. Deduplication identifies sequences that appear verbatim in multiple completions, suggesting they are reliably produced rather than randomly generated. Cross-referencing with known datasets identifies completions that match known documents. Each of these filters reduces false positives, helping the attacker distinguish extraction from coincidentally matching text.

The scale of this attack is important to appreciate. Carlini et al. generated 600,000 completions from GPT-2 using a mix of prompts. From those 600,000 completions, they manually verified hundreds of training data extractions. The false positive rate was manageable, and the true positive rate was significant enough to make the attack practically useful. An adversary with access to more compute could scale this to millions of completions.

A related attack is membership inference: given a specific text sequence, determine whether it was part of the training data. This is weaker than extraction (we already know the sequence) but still useful for privacy auditing and for adversaries trying to confirm whether a target's data was used in training.

The simplest membership inference attack computes the model's perplexity on the candidate sequence. Training data tends to have lower perplexity (the model assigns it higher probability) than non-training data, because the model was optimized to predict training sequences. An attacker who obtains a target document can feed it to the model, compute the perplexity, and compare it to the distribution of perplexities for known non-training documents. If the target document's perplexity is anomalously low, it was likely in the training set.

More sophisticated attacks use the loss trajectory over the course of training, or compare the target model's behavior against a reference model trained on similar but non-overlapping data. The latter approach, called likelihood ratio attacks, achieves better accuracy by accounting for the fact that some text is inherently easy to predict regardless of whether it appeared in training. A Wikipedia article about a major historical event will have low perplexity under many language models simply because the content is commonly known and frequently discussed, not necessarily because it appeared in this particular model's training data. By comparing against a reference model, the likelihood ratio attack controls for this baseline difficulty.

The fundamental difficulty is that reducing memorization requires the model to forget specific training examples, but gradient descent does not selectively forget individual examples. Every update adjusts all weights in response to the entire batch. Targeted forgetting, sometimes called machine unlearning, is an active research area (discussed in detail in the Mitigations section), but current techniques work imperfectly and can degrade model performance.

Additionally, extraction attacks can exploit the model's context window. By providing a specific prefix from a document, the model's attention mechanism increases the probability of producing the document's continuation. Removing this context sensitivity would fundamentally undermine the model's ability to do useful tasks like summarization, completion, and question answering. The same mechanism that makes language models useful, their ability to pick up on contextual cues and generate contextually appropriate continuations, also makes them vulnerable to extraction with well-chosen prefixes.

There is also a fundamental information-theoretic dimension to this problem. A language model with parameters trained on tokens of data is a compression of that data. For a large model (tens of billions of parameters) trained on a moderately-sized dataset (hundreds of billions of tokens), the compression ratio is less than 1 bit per token. At this compression ratio, the model cannot store all training sequences verbatim. But it can store many short, specific sequences very compactly, because those sequences have low entropy relative to their length. This is precisely the profile of PII: email addresses, phone numbers, and social security numbers are short strings with very low entropy (they follow predictable patterns), so they are efficiently memorizable.

An emerging category of extraction attack combines memorization vulnerabilities with prompt injection. In a system that uses a language model as a backend for a user-facing application, an attacker may be able to craft inputs that cause the model to reveal memorized training data in a context where the system would normally filter or refuse to disclose it.

For example, a customer service chatbot might have instructions to never discuss competitor products. But an attacker who knows the model was trained on data containing competitor pricing could potentially craft indirect prompts that cause the model to summarize or quote content from those documents without explicitly framing it as competitor information. The model does not enforce policies through understanding; it generates the most probable continuation given the prompt, and if that continuation happens to include memorized data, it will produce it.

This category of attack is particularly relevant for retrieval-augmented generation (RAG) systems, where external documents are inserted into the model's context at inference time. If those documents contain sensitive information and the system's isolation mechanisms are imperfect, adversarial inputs may be able to extract document contents that were intended to inform the response but not be directly disclosed.

The memorization and extraction vulnerabilities described above translate into concrete privacy risks when language models are deployed in real products. The risk profile depends on the deployment context, the sensitivity of the training data, and the sophistication of the adversary.

The most direct risk is that a language model deployed in a chatbot, writing assistant, or API might generate text containing PII from its training data in response to legitimate user queries. A user asking for help writing a cover letter might receive output that includes a real person's name and phone number that the model extracted from a scraped directory. A user asking about a medical topic might receive output containing specific patient information that appeared in a scraped clinical forum.

This kind of exposure is particularly insidious because it looks like normal model output. There is no error message, no anomalous behavior, just a response that happens to contain real private information. The user receiving the output has no way to know the information is real rather than plausibly generated. And the organization deploying the model may not discover the leak unless a sophisticated privacy audit is conducted.

The risk is asymmetric: the user who triggers the exposure may not realize it happened, the person whose information was exposed may never know, and the deploying organization may never discover the specific incident. This makes memorization-based PII exposure qualitatively different from a database breach, where the exposure is typically detectable and scope-definable. Memorization-based exposure is diffuse, ongoing, and difficult to characterize.

Beyond direct generation, inversion attacks attempt to recover training data from the model's weights directly, without using the generation API. These attacks treat the model's parameters as a representation of the training data and attempt to decode that representation.

While full inversion is not yet practical for large models, partial inversion has been demonstrated: attributes of training examples (demographic features, sentiment, topic) can sometimes be inferred from the model's weights or from its behavior on carefully chosen prompts. As model inversion techniques improve, the boundary between "theoretical risk" and "demonstrated vulnerability" will shift.

The weights of a language model encode, in some compressed form, statistical regularities of the training data. For sufficiently specific distributions, those regularities can leak demographic information about individuals even without extracting specific training examples. A model trained primarily on medical notes from a specific hospital might exhibit measurably different generation behavior when prompted with clinical language, revealing information about the patient population's characteristics even without reproducing any individual record.

In federated learning, models are trained on data that never leaves users' devices. Gradients are aggregated on a server. This architecture was designed to prevent raw data from being transmitted to a central server, giving a strong intuitive privacy guarantee. However, gradient inversion attacks have shown that individual training examples can sometimes be reconstructed from gradients alone, particularly when batch sizes are small or gradients are unclipped.

Language models trained in federated settings for on-device applications are potentially vulnerable to gradient inversion if the federated aggregation protocol is not carefully designed with privacy guarantees. The gradient of a loss function with respect to a specific training example carries information about that example. With small enough batches and high-resolution gradients, that information can be sufficient to recover the input. This creates privacy risks even when the raw data is never transmitted.

The practical implication is that federated learning is not, by itself, a privacy guarantee. It reduces the risk by keeping raw data on-device, but it introduces gradient-based attack surfaces that require their own mitigations, typically in the form of gradient clipping, noise addition (DP-SGD), and secure aggregation protocols.

The privacy risks from memorization have concrete legal implications that are increasingly being codified into regulatory requirements.

GDPR (EU): The right to erasure, commonly called the "right to be forgotten," requires that organizations remove an individual's personal data on request. If a language model has memorized that data, erasure may require retraining the model, a process that costs millions of dollars for large models. The tension between GDPR erasure requirements and the practical impossibility of surgical model unlearning is one of the most significant unresolved issues involving AI and data privacy law.

CCPA (California): Similar data deletion requirements apply to California residents. The CCPA also grants the right to opt out of the sale of personal information, which raises questions about whether training data licensing constitutes a "sale" and whether model outputs that reproduce PII constitute use of that information.

HIPAA (US): Protected health information that was scraped and then memorized by a model creates compliance liability for organizations that deploy that model in healthcare contexts. A model trained on de-identified clinical data that was improperly de-identified may effectively re-identify patients through memorization of rare demographic combinations.

Emerging AI regulations: Several jurisdictions, including the EU AI Act, are developing AI-specific regulations that explicitly address training data provenance and model privacy. These regulations increasingly require that deployers document what training data was used, how PII was handled, and what audit procedures were conducted to verify privacy compliance.

Regulation is moving faster than the technical solutions. Organizations deploying language models are expected to have answers to questions about data provenance and privacy risk that the technical community is still actively researching. This gap makes responsible deployment a difficult challenge rather than a simple compliance exercise.

No single technique eliminates memorization entirely. Effective privacy protection requires a layered approach combining data preprocessing, training modifications, and deployment safeguards. Each layer addresses different aspects of the risk and has its own cost-benefit profile. Understanding each layer clearly helps practitioners make informed decisions about which combinations are appropriate for their deployment context.

The single most effective preprocessing step is deduplicating the training corpus. Since repetition is the primary driver of memorization, removing or downsampling duplicates dramatically reduces the memorization rate for affected sequences. Lee and colleagues demonstrated that deduplicating the C4 and GitHub training datasets reduced memorization rates by a factor of 10 or more for sequences that had been repeated many times. This is arguably the most effective privacy intervention available at the data preparation stage.

Deduplication can be exact (remove sequences that appear byte-for-byte more than times) or approximate (use MinHash or SimHash to identify near-duplicate documents and remove them). Both approaches are straightforward engineering tasks that pay large privacy dividends.

MinHash-based deduplication works by computing a sketch of each document as a set of token -grams, then using locality-sensitive hashing to find documents whose sketches overlap substantially. Given a document with token -grams , the Jaccard similarity between two documents and is:

where:

• is the set of -gram token sequences in document
• is the number of -grams that both documents share
• is the total number of distinct -grams across both documents

Two documents with a Jaccard similarity above a threshold (typically 0.8) are considered near-duplicates and the lower-priority copy is removed. This scales to trillion-token corpora because comparisons use compact MinHash sketches rather than the raw documents. Each sketch is a vector of hash values derived from the -gram set, and similarity can be estimated from these vectors without comparing the full documents.

The one limitation of deduplication is that it cannot help with information that appears only once in the training data but is still memorized, such as the UUID example from Carlini et al. For single-occurrence memorization, other techniques are needed. Deduplication is a strong mitigation for the most easily-exploited class of memorized content, but it does not eliminate the long tail of weakly-memorized, single-occurrence data.

Automated PII detection pipelines can identify and redact personally identifiable information before training. Named entity recognition, regular expressions for known PII patterns (email addresses, phone numbers, Social Security numbers), and machine learning classifiers trained on PII examples can all contribute to reducing PII density in training data.

PII scrubbing is imperfect by design. Entity recognition misses novel formats and uncommon names. Redaction can be circumvented by indirect references. And the definition of PII is context-dependent in ways that automatic classifiers struggle with. Nevertheless, PII scrubbing reduces the quantity of sensitive information available for memorization and is considered a best practice for responsible training data curation. The goal is not perfect recall (which is impossible) but a substantial reduction in the density of readily-extractable PII.

A practical PII scrubbing pipeline typically operates in layers. A rules-based layer first handles high-confidence patterns: email addresses matching RFC 5321 syntax, US phone numbers in standard formats, Social Security numbers matching the standard 9-digit format, and credit card numbers passing the Luhn checksum. This layer has very low false positive rates and handles the majority of structurally distinctive PII.

A second layer applies a sequence labeling model, typically a fine-tuned BERT or similar model, to identify names, addresses, and other entities that require contextual understanding. This layer catches PII that does not follow a fixed pattern, such as a person's name appearing in the middle of a sentence. The tradeoff is higher false positive and false negative rates compared to the rules-based layer.

Finally, a post-hoc review step samples a small fraction of scrubbed documents to estimate false negative rates. The output replaces detected PII with typed placeholders like [EMAIL] or [PHONE] rather than removing the surrounding text entirely. This preserves the sentence structure needed for language learning. A model trained on text with [EMAIL] placeholders learns that email addresses exist in certain contexts, which is useful general knowledge, without learning specific email addresses, which is the privacy risk.

Differential privacy (DP) provides a formal mathematical guarantee that the model's outputs reveal limited information about any individual training example. The DP-SGD algorithm, developed by Abadi and colleagues, adds calibrated Gaussian noise to gradients during training, making it provably harder for an attacker to determine whether any specific example was in the training set.

The formal guarantee is stated in terms of -differential privacy. A training algorithm satisfies -DP if for any two training datasets and that differ by a single example, and for any set of possible outputs :

where:

• is the privacy budget: smaller values mean stronger privacy guarantees, with meaning the outputs are statistically identical whether or not any single example was included
• is a small probability of the guarantee failing, typically set to or smaller
• and are neighboring datasets differing by exactly one training example
• is the training algorithm (including randomness in mini-batch selection, gradient computation, and noise addition)

The privacy guarantee comes at a cost: DP-SGD typically reduces model quality, sometimes significantly. The tradeoff between privacy (measured by , the privacy budget) and utility is steep for large language models. At (strong privacy), model quality can degrade substantially. At (weaker privacy), quality is closer to non-private training but the privacy guarantee is weaker. Calibrating this tradeoff for specific applications requires empirical experimentation and domain expertise. We will explore differential privacy in full detail in the next chapter, including how to select privacy budgets and how fine-tuning with DP differs from pre-training with DP.

Machine unlearning seeks to remove the influence of specific training examples from a trained model without full retraining. This addresses the GDPR right-to-erasure problem: if a user requests deletion of their data, unlearning would remove that data's influence from the model without the expense of training from scratch.

The conceptual foundation of unlearning is simple: reverse the effect of the gradient updates that were influenced by the target examples. In practice, this is significantly harder than it sounds, because gradient descent is not reversible and the influence of any single example is distributed diffusely across all the model's parameters through thousands of parameter updates.

Current unlearning techniques for neural networks fall into two broad categories:

• Exact unlearning: Provably removes all influence of the target examples. Requires either full retraining or techniques based on SISA training (Sharded, Isolated, Sliced, and Aggregated), which partitions training data so that only a fraction needs to be retrained when data is removed.
• Approximate unlearning: Uses gradient-based methods to nudge the model away from the target examples. Faster but provides weaker guarantees. Also called gradient ascent on the forgotten examples, this approach directly penalizes the model for assigning high probability to the target sequences, but cannot guarantee that all influence has been removed.

For large language models, exact unlearning is computationally intractable without architectural changes. Approximate unlearning is fast but has been shown to inadequately protect against membership inference attacks. A model that has been approximately unlearned may no longer verbatim reproduce the target sequence, but may still exhibit elevated probability for related content, and membership inference attacks may still correctly identify the example as training data.

SISA training is worth understanding in some detail because it represents the most principled approach to the unlearning problem. The training corpus is divided into shards of equal size. Each shard is further divided into slices. Models are trained incrementally: first on slice 1 of each shard, then on slices 1-2, and so on, with a checkpoint saved after each slice is processed. When a deletion request arrives, only the shards containing the target example need to be retrained from the most recent checkpoint that predates the introduction of that example.

In the best case, when the target example is in the last slice of a single shard, only of the total training cost is incurred for unlearning. In the worst case, when it appears in the first slice, the cost is , which is still a fraction of full retraining. With shards and slices per shard, the worst-case unlearning cost is of full retraining. This makes SISA practical for applications where unlearning requests are infrequent and the training data can be structured into shards at the outset. The main limitation is the need to plan for unlearning before training begins, which requires knowing the sharding strategy in advance.

At deployment time, systems can screen inputs for prompts that seem designed to extract training data and screen outputs for patterns matching known PII formats. This is a defense-in-depth measure rather than a root-cause fix: it catches obvious extraction attempts and obvious PII leaks, but determined adversaries can evade these filters, and the filters themselves may miss PII that does not match known patterns.

Output scanning is typically implemented as a post-processing step that applies the same PII classifiers used during data preprocessing to model outputs before they are returned to users. When a high-confidence PII detection occurs, the output can be blocked, redacted, or flagged for human review. Several deployed systems use this approach as a last-resort safety net, with the expectation that it will catch some leaks but not all.

The principal limitation of output scanning is indirect PII. Information that identifies someone in combination with other context but not on its own is difficult to detect without access to the full context of who the user is and what they might infer from the output. If a model generates a response that mentions a first name, a job title, and a city, that combination might uniquely identify a person even though none of the three pieces of information is PII on its own. No output scanner can reliably detect this without deep knowledge of the deployment context and the user's identity.

Input filtering faces a similar limitation. Simple extraction attacks that use unusual prompts or repeated tokens are easy to detect and block. Sophisticated attacks that construct prompts designed to blend with legitimate usage are much harder. The arms race dynamic is unfavorable: defenders must block all attacks, while attackers need only find one that works.

The mitigations described above are most effective when combined. Deduplication addresses the most common source of memorization (repetition) but leaves single-occurrence PII exposed. PII scrubbing reduces the density of sensitive information but has imperfect recall. Differential privacy provides formal guarantees but degrades model quality. Output scanning catches obvious leaks but misses indirect PII. Together, these layers create a defense that resists more failure modes than any single technique alone.

The chart below summarizes how each mitigation layer reduces extractable memorization, based on empirical results from published studies. The values represent approximate reduction in extracted examples relative to a baseline model with no mitigations.

Consider a practical scenario: you are training a customer service language model on historical support tickets. The tickets contain real customer names, email addresses, and account numbers. What is the memorization risk, and what mitigations should you apply?

Step 1: Count repetitions. First, determine how often specific pieces of information appear. A single customer's email address may appear in every support ticket they sent. If a customer sent 50 tickets, their email appears 50 times. Based on the memorization curves discussed earlier, this creates high memorization risk for that customer's contact information. Run an occurrence count across your entire corpus to identify the highest-risk sequences before training begins.

Step 2: Assess model size. Larger models have higher memorization rates. If you are fine-tuning a large foundation model on your support tickets, the base model's parameter capacity means it will memorize more of your fine-tuning data than a smaller model would. Consider whether a smaller model meets your performance requirements, especially if the primary use case does not require the additional capacity of the largest models.

Step 3: Apply deduplication. Remove exact and near-exact duplicates from the training corpus. For customer emails that appear in many tickets, keep only one occurrence (or a small number, enough for the model to learn the domain vocabulary without memorizing the specific address). This is the most effective step: it directly attacks the repetition mechanism that drives most high-probability extraction.

Step 4: Scrub PII. Use a PII detection pipeline to replace names, email addresses, phone numbers, and account numbers with generic placeholders before training. Your model will learn customer service patterns without learning specific customer identities. Test the pipeline's recall rate on a hand-labeled sample of your data to understand what fraction of PII it misses.

Step 5: Consider differential privacy. If the support tickets contain particularly sensitive information (health issues, billing disputes, security incidents), evaluate whether DP-SGD is warranted. The privacy guarantee it provides may justify the quality tradeoff. Start with a high (weak privacy) to measure the quality impact, then tighten until the quality degradation becomes unacceptable.

Step 6: Audit after training. After training, run a memorization audit: attempt to extract training data using the extraction procedure, and check outputs for PII patterns. This validates whether your mitigations worked and provides evidence of due diligence for regulatory purposes. An audit that finds zero extracted PII provides much stronger regulatory protection than a stated intention to prevent memorization.

The worked example illustrates that memorization risk management is not a single decision but a sequence of decisions made at the data preparation, training configuration, and deployment stages. The earlier in the pipeline you intervene, the more effective and less expensive the intervention tends to be.

Memorization and privacy research has revealed a fundamental tension in how large language models are built today. The same training procedure that enables these models to be useful also causes them to encode private information in ways that are difficult to audit, hard to remove, and potentially exploitable. This tension is not going away; it is structural to the current paradigm of training on large, uncurated internet data.

The limitations of current mitigations are significant. Deduplication and PII scrubbing are preprocessing steps that cannot account for the full diversity of private information that might exist in a training corpus. Differential privacy works but degrades model quality, and the quality degradation is most severe precisely in the high-sensitivity applications (healthcare, legal, financial) where the strongest privacy guarantees are needed. Machine unlearning is not yet practically viable for large models: SISA training requires planning before the model is trained, and approximate unlearning provides weaker guarantees than are needed to satisfy right-to-erasure requests. Output filtering catches obvious cases but misses sophisticated extraction attempts.

The scale of the problem also makes it difficult to assess across the full corpus. Training corpora for large language models contain trillions of tokens, and auditing every document for PII is not feasible. Sampling-based audits can estimate the density and type of PII present, but they cannot guarantee that all sensitive information has been identified or removed. Even well-curated corpora contain documents with PII that evaded detection, and the tail of rare, single-occurrence PII may always remain.

There is also a temporal dimension to the problem that current research has underaddressed. Training data is collected at a point in time, but people's privacy expectations and circumstances change. A person's contact information was public at the time of scraping but may be associated with a stalking risk years later. A medical diagnosis that was publicly discussed in a support forum during one period of someone's life may become deeply sensitive later. The model trained on that data retains its associations indefinitely, creating a kind of privacy time capsule that the person has no way to update.

What this research has enabled is a more honest accounting of what large language models are doing. Before Carlini et al.'s 2021 paper, it was common to assume that the compression inherent in neural network training would prevent direct extraction of training examples. That assumption was wrong. The research community now has concrete extraction methods, formal definitions of memorization, and a growing toolkit for measuring and reducing privacy risks. Each of these contributions has shifted the field from hand-waving assurances to measurable guarantees and quantified risks.

The practical impact extends beyond research labs. Major language model providers now publish model cards and data documentation that address data provenance and privacy mitigations. Privacy-preserving training techniques, once primarily a research curiosity, have become standard practice in applications involving sensitive data. Regulatory frameworks increasingly address AI training data specifically, driven in part by demonstrated extraction vulnerabilities.

For practitioners, the key takeaway is that training data curation is both a privacy obligation and a quality concern. The data you use to train a model becomes part of the model, and that model can be induced to produce that data under the right conditions. Treating training data with the same care you would apply to a production database is not excessive caution; it is appropriate engineering practice given what we now know about memorization. The cost of getting this wrong is not a software bug that can be patched. It is a model that has permanently encoded information it should not have, backed by infrastructure that may cost millions of dollars to retrain.

Language model memorization occurs when a model encodes specific training examples so precisely that those examples can be extracted from the model's outputs under adversarial conditions. The key concepts from this chapter are:

• -memorization measures how easily a training example can be triggered with a -token prefix. Larger prefixes make memorization easier to demonstrate, and smaller prefixes indicate stronger, more easily-exploitable memorization.
• Extractable memorization is the stronger property: an adversary without prior knowledge of the training data can craft prompts that cause the model to reproduce it. The 2021 Carlini et al. paper demonstrated this concretely for GPT-2, including extraction of a UUID seen only once in training.
• Repetition in training data is the primary driver of memorization. Sequences seen hundreds of times are reliably memorized; sequences seen once may still be memorized but are harder to extract.
• Larger models memorize more, not less. Model size and memorization rate are positively correlated, counter to the intuition that more capacity enables more generalization.
• PII in training corpora is prevalent in web-scale datasets, arising from scraped contact directories, forum posts, legal documents, and code repositories. Near-duplicate amplification increases the effective repetition count of PII beyond what occurrence counts suggest.
• Membership inference attacks can determine whether a specific sequence was in the training data by comparing the model's perplexity on that sequence against a baseline distribution.
• Mitigations include data deduplication (most effective for repeated content), PII scrubbing (targeted at sensitive information), differential privacy during training (formal guarantees at a quality cost), machine unlearning (planned for in SISA training), and output scanning (last-resort defense-in-depth). Each has different cost-benefit tradeoffs and no single technique is sufficient on its own.
• Regulatory pressure from GDPR, CCPA, HIPAA, and emerging AI-specific regulations requires organizations to have concrete, auditable answers about how training data privacy is managed.

The next chapter explores differential privacy in depth, covering the DP-SGD algorithm and how the privacy-utility tradeoff can be managed systematically for language model training.

QuizLink Copied