Hallucination Mitigation: RAG, Decoding, and Training

Article links

Make inline references clickable

Knowing that language models hallucinate is useful. Knowing why they hallucinate, as we covered in Hallucination Causes, is more useful. But neither piece of knowledge does anything for the user who needs the model to tell the truth. This chapter is about what you can do to reduce hallucination in deployed systems.

The set of mitigation options is broad and the tradeoffs are real. No single technique eliminates hallucination. What exists instead is a toolkit of complementary strategies, each of which addresses a different failure mode: retrieval augmentation grounds generation in external sources, decoding-time interventions steer probability distributions away from uncertain territory, training and fine-tuning modify what the model has learned to do, and uncertainty expression lets the model communicate what it does not know rather than fabricating an answer. Effective hallucination mitigation typically means combining several of these approaches.

Before diving in, it helps to think about hallucination as a spectrum rather than a binary. A model that confabulates a citation is hallucinating. But so is a model that correctly names a capital city but gives a population figure that is off by a factor of two, or one that describes a historical event accurately but attributes the wrong date. The severity ranges from cosmetic inaccuracy to dangerous misinformation, and different mitigation strategies address different points on this spectrum. RAG is excellent at preventing knowledge-gap hallucinations about specific, retrievable facts. Calibrated uncertainty is excellent at surfacing when the model is outside its confident knowledge zone. Training-based approaches reshape the model's systematic biases. A mature deployment layers all of these.

This chapter walks through each strategy in depth: the mechanism behind it, how to implement it, and where it breaks down. By the end, you will have a framework for diagnosing which mitigation strategies are appropriate for a given deployment context and how to measure whether they are working.

The most widely deployed hallucination mitigation technique is retrieval-augmented generation, or RAG. The core intuition is simple: if a model lacks reliable factual knowledge from training, you can supply that knowledge at inference time by retrieving relevant documents and placing them in the context window. The model then generates its answer based on what you retrieved, rather than relying on potentially inaccurate parametric memory.

To understand why this works so well, consider what hallucination is: the model generating tokens that are statistically plausible given its training but that do not correspond to facts about the world. The model has no dedicated "fact retrieval" module; facts are distributed across billions of floating-point parameters, blended with patterns, styles, and associations learned from everything else in the training corpus. Asking the model to recall a specific fact is asking it to reconstruct a signal from noisy, distributed weights. Retrieval sidesteps this problem entirely: instead of reconstructing a fact from memory, the model reads it.

We have covered RAG architecture in depth in the retrieval-augmented generation chapters of Part XLIV: Retrieval-Augmented Generation. The focus here is specifically on how RAG reduces hallucination, its failure modes as a mitigation strategy, and implementation details relevant to factuality.

A language model's parametric knowledge is frozen at training time, probabilistic in nature, and unverifiable during inference. Retrieved documents have none of these limitations: they are up-to-date, explicit, and attributable to a source.

When you retrieve a document containing the answer and place it in the context window, the model faces a fundamentally different generation problem. Rather than reconstructing a fact from statistical weights learned over billions of tokens, it reads the answer from text that is immediately present. For most factual question types, surface reading comprehension is far more reliable than parametric recall. A model that hallucinates the founding year of a company when asked cold will almost certainly answer correctly if you provide a Wikipedia excerpt that states the founding year in the first paragraph.

The mechanism generalizes beyond simple lookups. RAG improves factuality by:

• Grounding answers in verifiable sources: The model cannot assert something the retrieved document contradicts without the user noticing. This external check constrains confabulation even when the model would otherwise have interpolated from nearby facts.
• Reducing knowledge gap exposure: Many hallucinations occur precisely when the model is asked about tail entities or recent events outside its training data. Retrieval plugs these gaps directly.
• Enabling attribution: When the answer comes from a retrieved document, the model can cite the source. Attribution lets users verify claims and increases accountability.
• Making freshness tractable: A model's training data has a cutoff, but a retrieval index can be updated continuously. For questions about recent events or rapidly changing information, retrieval is the only viable path to accurate answers without retraining the model.

The quality of RAG as a hallucination mitigation strategy depends heavily on three design choices: the retrieval model (how well it finds the right documents), the knowledge base (whether the right documents exist and are well-curated), and the context formatting (how retrieved passages are presented to the generation model). Improving any of these three levers improves factuality, but they involve different engineering investments and have different failure signatures.

RAG does not eliminate hallucination. Understanding its failure modes is essential for deploying it responsibly.

Retrieval failure is the most fundamental: if the relevant document is not retrieved, the model falls back on parametric memory. This happens when the query-document similarity is not well-captured by the embedding model, when the knowledge base does not contain the relevant document at all, or when the question requires synthesizing across many documents that together exceed context limits. Retrieval failure is silent: the model silently falls back to parametric generation, often without indicating that the context did not contain the relevant fact.

Context ignoring occurs when the model retrieves the correct document but generates an answer inconsistent with it. This failure mode is more common than expected. Models trained on enormous corpora develop strong prior beliefs about what the answer should be. When the retrieved document contradicts the model's parametric knowledge, the model sometimes defaults to its prior rather than faithfully reading the document. This is especially pronounced for commonly misrepresented facts, where training data provides overwhelming but incorrect signal. A model that "knows" a popular misconception may override a retrieved correction.

Faithful summarization with unfaithful extension is subtle. The model correctly reports what the retrieved document says, then continues generating text that goes beyond the document, interpolating from parametric memory without marking the boundary between retrieved and generated content. The answer appears fully grounded but contains unverified claims. This is particularly dangerous because the grounded portion gives the response a surface appearance of reliability.

Noise sensitivity affects retrieval when documents are long or topically broad. A retrieved passage that contains the relevant fact also contains irrelevant text. Models are sensitive to position (facts early in context are more reliably used than facts buried deep) and to distraction (irrelevant but plausible text can nudge generation away from the correct answer). The "lost in the middle" phenomenon describes how models attend primarily to the beginning and end of their context window, often missing facts placed in the middle of long retrieved passages.

Multi-hop reasoning failures emerge when answering a question requires combining facts from multiple retrieved documents. The model must retrieve both pieces of information, identify that they need to be combined, and reason across them correctly. Each step introduces failure probability, and the compounding of these probabilities means that multi-hop RAG is substantially less reliable than single-document RAG. For complex questions, this limitation is significant.

Several techniques improve the probability that a model will faithfully use retrieved context rather than overriding it with parametric memory. The underlying challenge is that instruction-following is imperfect: a model told to "use only the provided documents" has not been surgically modified to ignore its training; it has been given a natural language instruction that it may or may not prioritize over its prior beliefs. The techniques below increase the probability of faithful behavior, but none guarantee it.

The most effective prompt-level technique is an explicit instruction to answer only from the provided context:

Answer the following question based only on the information in the provided documents. If the answer is not contained in the documents, say "I cannot find information about this." Do not use background knowledge that is not mentioned in the documents.

This framing shifts the task from "use documents as a hint" to "read from documents exclusively." The improvement is real but incomplete: models still occasionally override context, especially when parametric priors are strong.

Contrastive prompting goes further by explicitly informing the model that the context should override its priors:

The following documents contain authoritative information. Even if your training suggests otherwise, base your answer on these documents. If the documents disagree with what you would normally say, trust the documents.

Self-consistency with context checking addresses the context-ignoring failure mode computationally. Generate multiple responses, then check each response against the retrieved document using an entailment model or another LLM. Responses that contradict the retrieved document are filtered out, and the most consistently faithful response is selected. This adds latency but significantly reduces context-ignoring failures.

Citation forcing is a structural technique: require the model to cite the specific sentence in the retrieved documents that supports each claim it makes. A model that cannot cite a supporting sentence should not make the claim. This makes violations of faithfulness detectable: if the model makes a claim and its cited sentence does not support it, the system can flag or reject the response. Citation forcing also helps users understand which parts of the answer come from which documents, improving interpretability.

Chunk granularity tuning addresses the noise sensitivity failure mode. Rather than retrieving entire documents or large passages, retrieve fine-grained chunks (sentences or short paragraphs) that are more likely to contain the specific fact needed. Smaller chunks reduce distraction from irrelevant content but may miss context that spans sentences. The optimal chunk size depends on the nature of the queries and documents in your deployment.

Hallucination is a generation-time phenomenon: a model that has encoded some degree of uncertainty about a fact may still produce a confident, incorrect token if the decoding procedure pushes it toward high-probability completions. Decoding-time interventions aim to modify this behavior without changing the model's weights. They are appealing because they can be applied to any model at any time, with no training cost, but their effect is limited by what the model has already learned.

The simplest decoding intervention is temperature. Standard next-token generation samples from:

where:

• : the logit for token , the raw unnormalized score from the model's output layer
• : the temperature parameter controlling distribution sharpness
• : the sequence of tokens generated so far (the context)
• The denominator sums exponentials over all vocabulary tokens

When , the distribution sharpens: the highest-probability tokens become even more probable relative to alternatives. When , the distribution flattens, increasing diversity but also increasing the probability of low-probability tokens including incorrect ones.

For factual generation tasks, low temperature is generally preferable. The model's most probable token at each step is more likely to be the one best supported by training evidence. However, the relationship is not monotone: greedy decoding ( , always selecting the argmax) can amplify repetition and get stuck in degenerate loops, and it does nothing to prevent the model from generating a confident wrong token if that token has the highest logit.

A useful mental model for temperature: think of the model's logit distribution at each step as representing its "belief" about the next token. Low temperature sharpens the model's expressed beliefs, making it commit more firmly to its top choice. High temperature softens those beliefs, making it hedge toward alternatives. For factual accuracy, you want the model to commit firmly to its best-supported beliefs, which argues for lower temperature. But if the model's best-supported belief is wrong, lower temperature makes that error more pronounced.

Top-p sampling (nucleus sampling) restricts generation to the smallest set of tokens whose cumulative probability exceeds a threshold . Formally, the nucleus is defined as:

where:

• : the full vocabulary of possible tokens
• : a candidate subset of the vocabulary, sorted by descending probability
• : the cumulative probability threshold (e.g., 0.9 means keep the smallest set of tokens that together account for 90% of the probability mass)
• : the chosen nucleus, the smallest such subset meeting the threshold

We sample uniformly over after renormalizing its probabilities to sum to 1. Top-k sampling instead restricts to the highest-probability tokens regardless of their cumulative probability.

These filters prevent the model from sampling extremely low-probability tokens that might include hallucinated content. They do not prevent high-confidence hallucinations, but they do reduce the tail risk of the model generating tokens that are essentially noise in the distribution. The intuition is that truly hallucinated content often requires the model to produce tokens that are improbable given the context: setting ensures you never sample from that improbable tail.

The difference between top-p and top-k is important in practice. Top-k truncates to a fixed number of tokens regardless of how concentrated the distribution is. If the model is very confident (one token has 95% probability), top-k with still admits 49 low-probability alternatives. Top-p dynamically adjusts: when the distribution is concentrated, the nucleus is small (perhaps just 2-3 tokens), and when it is diffuse, the nucleus is large. For hallucination mitigation, top-p is generally preferable because it is responsive to the model's actual confidence at each step.

Self-consistency is one of the most empirically reliable decoding strategies for improving factual accuracy. The procedure is straightforward:

1. Generate independent responses to the same prompt using sampling (not greedy decoding)
2. Aggregate the responses to find the most consistent answer

For questions with extractable answers (factoid questions, numerical answers, reasoning chains), you identify the answer that appears most frequently across the responses. The key insight is that the correct answer, when the model knows it, tends to be consistently generated across samples, while hallucinated content tends to vary.

Why does this work? When the model has strong training signal for a fact, the underlying probability distribution assigns high probability to the correct completion, and different samples drawn from that distribution tend to land on the same answer. When the model is uncertain or guessing, the distribution is flatter, and samples diverge. Self-consistency exploits this asymmetry: agreement across samples is evidence of stored knowledge, while disagreement is evidence of uncertainty.

The aggregation step works differently depending on task type:

• Closed-form answers: majority vote across the responses
• Numerical answers: median or mode of the extracted values
• Reasoning chains: majority vote on the final conclusion, using the most common chain as the output
• Open-ended generation: selecting the response with the highest average token probability, which is a proxy for the model's confidence

Self-consistency was originally proposed as a prompting technique for chain-of-thought reasoning but generalizes broadly. Its hallucination-reduction effect comes from averaging out the variance in the model's generation. A single sample might land on an incorrect token; the average of many samples is pulled toward the model's learned central tendency.

The cost of self-consistency is compute: generating responses is times more expensive than generating one. For latency-sensitive applications, this is often prohibitive. For accuracy-critical applications where the query can be processed asynchronously, it is a practical approach. There are also efficient variants that stop sampling early when consensus is reached after samples, reducing average cost on high-confidence queries.

More targeted decoding interventions modify the token probability distribution itself based on external factuality signals. These go beyond simple temperature and sampling adjustments to actively reshape what the model generates.

Contrastive decoding computes the difference between a capable large model and a smaller model for each candidate token. The reasoning is that tokens where the large model assigns much higher probability than the small model tend to be the tokens the large model has specifically learned to prefer, as opposed to superficial continuations that both models would predict. Formally, for a capable model and an amateur model , the contrastive score for token is:

where:

• : the contrastive decoding score for token
• : the probability assigned to by the capable (large) model
• : the probability assigned to by the amateur (small) model

Tokens that both models consider likely (common continuations, function words) get low contrastive scores. Tokens the large model uniquely prefers get high scores. This acts as an amplification of what the large model has specifically learned. For factuality, the intuition is that small models have weaker factual knowledge, so facts the large model knows strongly will have high contrastive scores. The method has shown meaningful improvements on knowledge-intensive tasks and is relatively low-cost since the small model is cheap to run.

A practical limitation of contrastive decoding is that it requires running two models in parallel, doubling inference complexity. For latency-sensitive applications, this may be unacceptable. There are also cases where the small model assigns high probability to the correct token for the wrong reason (e.g., because the token is a common word), which reduces the signal quality of the contrastive score.

Inference-time factuality steering uses linear probes trained to detect factual versus confabulated content in the model's residual stream. At inference time, the probe's gradient direction can be added to the model's internal representations to steer generation toward factually supported completions. This is an active research area and not yet widely deployed, but represents a direction toward intrinsic factuality control that operates below the level of token probabilities.

Retrieval-constrained decoding combines RAG with decoding-time constraints. Rather than simply placing retrieved documents in context, you restrict the generation vocabulary at each step to tokens that are consistent with at least one retrieved document. This enforces hard faithfulness but requires online retrieval during each decoding step, which is computationally expensive. In practice, this approach is most useful for highly structured generation tasks (e.g., generating database queries or code) rather than open-ended text generation.

Chain-of-thought prompting, which asks the model to reason step-by-step before giving a final answer, has a beneficial side effect on factuality beyond its primary benefit for multi-step reasoning. When a model must articulate its reasoning process, errors become more visible and correctable.

The factuality benefit of chain-of-thought comes from several mechanisms. First, the reasoning chain exposes intermediate claims that can be checked against retrieved documents or common sense. A response that states "Napoleon was born in 1769 in Corsica, and since Corsica became French territory in 1768, he was a French citizen" is more verifiable than a bare assertion "Napoleon was French." Second, the act of articulating reasoning chains may itself constrain generation: a model that has just stated a fact in the reasoning chain is more likely to use it correctly in subsequent steps. Third, longer reasoning chains give retrieval-augmented systems more opportunities to match and retrieve relevant documents before the final answer is generated.

The failure mode of chain-of-thought for factuality is that confident but incorrect reasoning chains can lead the model to a wrong answer with apparent logical support. A fluent incorrect reasoning chain is harder to reject than a bare incorrect assertion, because it pattern-matches to the surface form of sound reasoning. Users may be more convinced by a confidently wrong argument than by a confidently wrong bare assertion.

Decoding interventions and retrieval augmentation work at inference time without touching the model's weights. Training approaches go deeper: they modify what the model has learned to do, making it intrinsically less likely to hallucinate. Training-based mitigations are more powerful and persistent than inference-time interventions, but they are also more expensive and require data.

The most direct training approach is supervised fine-tuning (SFT) on high-quality, fact-checked datasets. If the model has learned to generate confident falsehoods because it was trained on noisy web data, training it additionally on verified content can help recalibrate its factual generation.

This works better for domain-specific deployment than for general factuality. A model fine-tuned on carefully curated medical literature will generate more accurate medical text because the training signal consistently reinforces the correct facts. A model fine-tuned on news articles will be more reliable on current events covered in that dataset. The improvement is real and often significant in the target domain.

The limitation is coverage: SFT on factual data can only improve the model's accuracy on facts present in the fine-tuning set. It does not improve the model's ability to recognize when it is outside its knowledge, and it can reduce diversity in a way that makes the model overconfident about the fine-tuning domain while no better on everything else. There is also the risk of catastrophic forgetting, as we discussed in the Finetuning Fundamentals chapters: heavy fine-tuning on a narrow factual domain can degrade the model's performance on unrelated tasks.

A practical application of SFT for factuality is training on examples that demonstrate appropriate uncertainty expression. Rather than training only on examples of correct answers, you include training examples where the correct output is an explicit "I don't have reliable information about this" rather than an attempted answer. This teaches the model that refusal is a valid and valued response, counteracting the tendency of instruction-tuned models to always attempt an answer.

RLHF trains a reward model on human preference judgments and then uses reinforcement learning to optimize the language model to produce outputs that would receive high reward. When human preferences include factual accuracy as a criterion, RLHF can reduce hallucination by directly rewarding non-confabulation.

The process works in three stages. First, human annotators rate pairs of model outputs on multiple criteria including helpfulness, safety, and factual accuracy. These ratings become training data for a reward model, which learns to predict a scalar preference score for any given (prompt, response) pair. Second, the language model is fine-tuned using RL (typically Proximal Policy Optimization, or PPO) to maximize the expected reward score, with a KL divergence penalty that prevents the model from drifting too far from its original behavior:

where:

• : the language model policy being optimized, with parameters
• : the reference (base) policy before RLHF training
• : the reward model's score for response to prompt
• : the KL penalty coefficient, controlling how far the model can deviate from its reference
• : the distribution of prompts

When human annotators are instructed to penalize hallucinated or unverified claims, the reward model implicitly encodes a preference for factuality, and the RL training pushes the language model toward outputs that avoid false claims.

InstructGPT and subsequent models (GPT-4, Claude, Gemini) use RLHF with explicit instructions to annotators to prefer factually accurate responses and to downgrade responses containing false claims. The resulting models are measurably less likely to assert confidently false statements compared to base models fine-tuned on instruction data alone.

The mechanism is indirect: the reward model learns to predict what humans prefer, and humans prefer accurate responses, so the language model is pushed toward accuracy. But the reward model is itself imperfect, and annotators make errors or disagree, so the improvement is real but not absolute. RLHF does not eliminate hallucination; it reduces it probabilistically.

A deeper limitation is that RLHF provides no explicit signal about which facts are true and which are false. It only rewards what humans rate highly, and human raters cannot always verify factual claims in the responses they are rating. This means RLHF works better at eliminating obvious, verifiable falsehoods than at correcting subtle factual errors in specialized domains where annotators lack expertise. A medical hallucination that reads confidently and coherently may receive high human ratings even if it is incorrect.

DPO is a more stable alternative to RL-based RLHF. Rather than training a separate reward model and using RL, DPO directly optimizes the language model to prefer responses chosen by humans over responses rejected by humans, using a classification-style loss:

where:

• : a training triple of prompt , preferred response (won), and rejected response (lost)
• : the language model being trained (with parameters )
• : the reference (base) policy before DPO fine-tuning
• : a temperature parameter controlling how far can deviate from
• : the sigmoid function

The loss maximizes the probability that the model assigns higher likelihood to the preferred response than the rejected response, relative to the reference model. This is equivalent to doing implicit reward learning and policy optimization in a single step, without ever materializing a separate reward model.

DPO has been applied to factuality by constructing preference datasets where the preferred response is a factually correct answer and the rejected response is a hallucinated one. This directly teaches the model to prefer truthful outputs over confabulated ones. The key requirement is a high-quality preference dataset with pairs of (truthful answer, hallucinated answer) for the same prompt. Constructing these pairs at scale requires either human annotation or automated hallucination generation (deliberately eliciting hallucinations from a base model to create negative examples).

A practical advantage of DPO over RLHF for factuality fine-tuning is stability: the optimization landscape is smoother, training is less sensitive to hyperparameters, and there is no risk of reward hacking (where the policy finds ways to maximize the reward model's score without being more factual). For domain-specific factuality improvements, DPO with a curated preference dataset is often the most effective and tractable training approach.

A related approach pairs fine-tuning with explicit knowledge supervision. In knowledge-grounded training, each training example includes both the question and the relevant supporting documents, and the model is trained to generate answers conditioned on those documents. The goal is to teach the model to rely on provided context rather than parametric memory.

This is effectively training a model to do RAG well, rather than retrofitting RAG onto a model trained without it. Models trained with knowledge-grounded objectives learn to distinguish "I know this from training" from "I am reading this from context," which is a skill that improves their behavior in RAG deployments. A model trained on (question, document, answer) triples learns to use the document as its primary information source. When deployed with retrieval augmentation, this training makes the model more faithful to retrieved context and less likely to override it with parametric priors.

ROME and MEMIT are surgical fact-editing approaches at the opposite extreme from full fine-tuning. Rather than retraining broadly, they locate specific factual associations stored in transformer feed-forward layers and directly modify those weights. The feed-forward layers in transformer blocks act as key-value memories: they store factual associations of the form "subject entity attribute value," and these associations can be located and modified with relatively small weight changes.

Editing "The Eiffel Tower is in Paris" to "The Eiffel Tower is in Berlin" (as a test of the method) requires changing only a small number of weights in mid-layer feed-forward networks. ROME edits one fact at a time; MEMIT extends this to batch editing of thousands of facts simultaneously. The editing procedure involves:

1. Identifying which attention heads and feed-forward neurons are responsible for recalling the target fact
2. Computing the update direction in weight space that shifts the association from the old value to the new one
3. Applying the weight update with minimal impact on other stored facts

Fact editing is appealing for knowledge-update use cases (updating a model's beliefs about recent events without full retraining) but carries significant risks. Edited knowledge can interfere with nearby associative facts in ways that are hard to predict. Editing can create inconsistencies the model cannot reason about: the model may correctly report the edited fact in direct queries while continuing to use the old fact in indirect reasoning. There is limited understanding of what other beliefs change implicitly when a targeted fact is modified. For production hallucination mitigation, fact editing is currently more useful as a research tool than a deployment strategy.

Before reaching for training-based or retrieval-based mitigations, there is significant value in simply designing prompts that elicit more factual behavior from existing models. Prompt engineering for factuality is not a complete solution, but it is cheap, fast to iterate, and can provide meaningful improvements with no infrastructure cost.

One of the most reliable prompt-level interventions is explicitly asking the model to express uncertainty when it is not confident. A prompt like "If you are not certain about any specific fact, please say so explicitly" significantly increases the rate at which models hedge uncertain claims, even without any training changes.

This works because instruction-following models are responsive to explicit framing. Without an explicit instruction to hedge, the model defaults to the style it was trained on, which typically involves confident assertion. With an explicit instruction, the model shifts its generation style toward more hedged formulations. The calibration of this hedge is imperfect: the model may hedge some claims it knows confidently, and fail to hedge others it does not know. But the direction of effect is reliable and useful.

Structured instructions that specify what to do when uncertain are more effective than vague instructions to "be careful." Compare:

Answer as accurately as possible. If you are unsure about specific facts such as dates, numbers, or names, say "I'm not certain about this" before giving your best estimate, and recommend that the user verify the information independently.

versus:

Be careful about accuracy.

The first instruction gives the model a specific behavioral template for uncertainty expression. The second is too vague to reliably change behavior.

Asking the model to cite its sources for factual claims has two benefits. First, it encourages the model to use retrieved documents (in RAG settings) rather than parametric memory, because retrieved documents are the easiest thing to cite. Second, citations allow users to verify claims independently, turning hallucination from an invisible failure into a visible one: an uncited claim stands out, and a fabricated citation can be caught.

The limitation is that models can hallucinate citations with the same fluency they hallucinate facts. A model asked to cite sources may produce plausible-looking but nonexistent references. This is arguably worse than uncited hallucinations because it gives users a false sense of verifiability. To get the benefit of citations without this risk, you need a system that validates citations (e.g., checks that the cited document exists and contains the claimed content).

Asking the model to break complex questions into sub-questions before answering each one reduces hallucination on multi-hop queries. The decomposition step forces the model to make its reasoning structure explicit, which makes errors more visible and allows retrieval to operate on focused sub-queries rather than the original complex question. This technique, called least-to-most prompting or question decomposition, has shown consistent improvements on multi-hop factoid question answering.

All of the strategies above attempt to prevent hallucination during generation. Post-hoc verification takes a different approach: generate first, then verify whether what was generated is factually accurate. This is valuable in settings where the cost of a wrong answer is high enough to justify the additional verification step.

Factual consistency checking evaluates whether a generated response is consistent with a set of reference documents. Unlike hallucination detection (which requires knowing the ground truth fact), consistency checking only requires that you have reference documents that should support the response. This is a well-posed task: given document and claim , does entail, contradict, or not address ?

Natural language inference (NLI) models are the workhorses of factual consistency checking. An NLI model takes a (premise, hypothesis) pair and classifies the relationship as entailment, contradiction, or neutral. For factual verification, the retrieved document is the premise and each claim in the generated response is the hypothesis. Claims that are contradicted or not addressed by any retrieved document are flagged as potentially hallucinated.

The limitation of NLI-based consistency checking is that it only checks consistency with what you retrieved, not with the world. If your retrieval system missed the relevant document, a hallucinated claim may be neither contradicted nor supported by the retrieved context, leading to a "neutral" judgment that does not identify the problem.

LLM-based verification uses a second language model as a fact checker. Given the generated response and optionally some retrieved context, the verifier model is asked to identify specific claims that appear inaccurate or unsupported. This approach benefits from the verifier model's general knowledge and reasoning capabilities, but introduces its own failure modes: the verifier can hallucinate corrections, be inconsistent, or be fooled by the same fluency that makes the original response convincing.

FActScore (Factual Precision in Atomic Claims Score) is a systematic framework for measuring hallucination in long-form generation. Rather than treating an entire response as correct or incorrect, FActScore decomposes the response into atomic claims (simple, standalone factual assertions) and verifies each one independently against a knowledge base.

For a response about subject , FActScore is:

where:

• : the set of atomic claims extracted from response
• : a single atomic claim (e.g., "Marie Curie was born in Warsaw")
• : 1 if claim is supported by the knowledge base for subject , 0 otherwise
• : the total number of atomic claims extracted

The extraction step (decomposing a response into atomic claims) and the verification step (checking each claim against a knowledge base) can both be done with LLMs, making FActScore automatable at scale. The framework has been applied to biography generation, scientific summarization, and other long-form factual generation tasks, giving a principled way to compare models and mitigation strategies on the same scale.

All of the strategies above try to prevent the model from generating false content or verify it afterward. Uncertainty expression takes a different approach: rather than preventing hallucination, it asks the model to communicate when it is uncertain, allowing users to calibrate their trust in the output.

This is not a fallback for when prevention fails. Uncertainty expression and hallucination prevention are complementary strategies. Even a well-mitigated system operating on difficult out-of-domain queries will sometimes be uncertain. A system that can recognize and communicate this uncertainty is more useful than one that generates confident answers of unknown reliability.

The simplest form of uncertainty expression is verbalization: training the model to express confidence in natural language. Phrases like "I believe," "I'm not certain, but," "to my knowledge as of my training cutoff," or "you may want to verify this" communicate epistemic status directly to the user.

RLHF can instill this behavior when human annotators reward responses that accurately hedge versus those that confidently assert uncertain information. Models trained this way learn a rough mapping from their internal uncertainty states to verbal hedges. The calibration is imperfect, however: models often express high-confidence hedges ("to the best of my knowledge") even when they are essentially guessing, and sometimes express confident assertions on uncertain facts.

A better-calibrated hedging strategy distinguishes between types of uncertainty:

• Knowledge cutoff uncertainty: facts that may have changed since training ("As of my training data, X, but this may have since changed")
• Low-coverage uncertainty: facts about entities underrepresented in training ("I don't have detailed information about X")
• Contradictory-source uncertainty: facts where training data contained conflicting information ("Sources disagree on this, but a common claim is X")
• Inference uncertainty: conclusions requiring long reasoning chains where errors may accumulate ("This requires several inference steps; I'd recommend verifying")

Teaching models to distinguish these uncertainty types requires training data that explicitly labels which type applies. Generic hedging is easy to learn but less useful; type-specific hedging is harder to train but provides more actionable information to the user.

When uncertainty is high enough, the right response may be no response at all, or an explicit refusal with guidance for the user. "I don't know" is more useful than a confident hallucination, provided the model correctly identifies when it does not know.

This idea is older than LLMs. In statistical machine learning, "abstaining classifiers" or "reject options" are well-studied mechanisms: a classifier can output a prediction or say "I am not confident enough to classify this input." The accuracy on the subset of inputs where the model does predict tends to be much higher than overall accuracy. For language models, the analogous behavior is refusing to answer questions that fall outside the model's reliable knowledge, rather than fabricating a plausible-sounding response.

Teaching models to abstain requires training signal. Instruction-tuned models that have been trained to always attempt an answer have, by construction, been trained away from appropriate refusal. Recalibrating this behavior requires either:

• Refusal examples in SFT data: training examples where the correct output is an explicit "I don't have reliable information about this" rather than an attempted answer
• Reward modeling that values refusal: explicitly rewarding accurate refusals in RLHF, not just rewarding correct answers

The difficulty in constructing refusal examples is knowing when refusal is warranted. A training example that says "the model should refuse to answer X" implicitly encodes a judgment about what the model should and should not know, which requires curators to have their own ground truth. This is tractable for clearly out-of-scope queries (events after the training cutoff, fictional events in real-world framing) but harder for tail-entity queries where the model simply does not have high-quality signal.

The risk is over-refusal: a model too quick to say "I don't know" is less useful than intended. Calibrating the refusal threshold requires measuring both false positive and false negative rates, which requires ground truth for a representative set of questions. A model that refuses 30% of answerable queries to avoid hallucinating on the remaining 10% may be worse in aggregate than a model that hallucinates 10% and answers correctly 90%.

Beyond verbalized uncertainty, models expose quantitative uncertainty through their output probability distributions. At each decoding step, the logits encode how much probability mass the model places on each possible token. Low entropy (high confidence in one token) suggests the model has a clear preference. High entropy (probability spread across many tokens) suggests uncertainty.

The log probability of a complete response is the sum of log probabilities for each token:

where:

• : the full generated response as a sequence of tokens
• : the -th token in the response
• : the input prompt
• : the length of the response in tokens

Low overall log probability signals that the model assigned relatively low probability to this particular completion, which can be used as an uncertainty signal. Comparing log probabilities across multiple sampled responses gives a distribution of confidence scores; the variance of this distribution is a measure of generation uncertainty.

Token-level entropy is a more fine-grained signal than whole-sequence probability. Rather than computing the log probability of the entire response, you compute the entropy at each token position . High-entropy positions indicate tokens where the model is uncertain. This allows you to identify specific tokens or spans that are uncertain rather than flagging the entire response. For named entities and numbers (which are common hallucination sites), token-level entropy tends to spike when the model is confabulating.

Sequence probability is not perfectly calibrated with factual accuracy. Short responses have lower log probabilities than long ones simply because of length. Fluent hallucinations can have high log probabilities. But combined with other signals (diversity across samples, entropy of individual token predictions), probability-based uncertainty estimates provide useful information.

Selective prediction formalizes the abstention idea: a model predicts on inputs where it is confident and abstains on inputs where it is not. The key design choice is the confidence threshold : predict when estimated confidence , abstain otherwise.

A useful framework for evaluating selective prediction is the coverage-accuracy tradeoff:

where:

• : the confidence threshold above which the model predicts rather than abstaining
• : the confidence estimate for input (e.g., max token probability, mean log probability, or self-consistency score)
• : the fraction of inputs on which the model makes a prediction at threshold
• : the fraction of those predictions that are correct
• : the model's prediction; : the ground truth

As increases, coverage decreases (fewer inputs are answered) but accuracy on the answered subset increases. Plotting this tradeoff (AUARC, area under the accuracy-rejection curve) gives a single-number summary of how well the confidence estimate separates cases where the model is right from cases where it is wrong.

Implementation: A Practical PipelineLink Copied

Setup and ImportsLink Copied

Simulating a Retrieval CorpusLink Copied

Simulating a Language Model with HallucinationLink Copied

The Full Mitigation PipelineLink Copied

Let's measure the impact of each mitigation strategy by running the same queries with different configurations.

The key parameters for this mitigation pipeline are:

• n_samples: The number of self-consistency samples. More samples improve consistency estimation but increase latency linearly. Values between 5 and 20 are typical.
• consistency_threshold: The fraction of samples that must agree for a high-confidence answer. A value of 0.6 means 60% agreement; increase this for higher precision at the cost of more abstentions.
• uncertainty_threshold: The maximum log-probability standard deviation for a high-confidence answer. Lower values require more consistent probability assignments across samples.
• top_k retrieval: The number of documents to retrieve. Retrieving more documents increases context length but may improve coverage for complex questions.

In production, these parameters should be tuned on a held-out validation set with known ground truth, optimizing the tradeoff between coverage and accuracy that your application requires.

Deploying mitigation strategies without measuring their effect is flying blind. Effective evaluation requires benchmarks that test the specific failure modes your mitigation is designed to address, and metrics that capture both the rate of hallucination and the model's calibration.

Several benchmarks have become standard for evaluating factual accuracy in language models:

TruthfulQA tests whether models give truthful answers to questions that many humans answer incorrectly due to common misconceptions. A model that merely reproduces training data will score poorly, because training data contains the misconceptions. The benchmark specifically targets the tendency of capable models to generate plausible-sounding falsehoods.

FELM (Factuality Evaluation of Large Language Models) evaluates factuality across multiple domains (science, medicine, finance, law) by having models generate responses and then checking specific factual claims against verified sources. FELM provides domain-specific breakdowns, making it useful for identifying where a model has strong versus weak factual coverage.

FActScore (described earlier in the Post-Hoc Verification section) evaluates long-form generation by decomposing responses into atomic claims and verifying each one. This is particularly useful for evaluating mitigation strategies on biography, entity description, and summarization tasks where hallucination is fine-grained rather than binary.

HaluEval focuses on hallucination in dialogue and question answering, with tasks that test whether the model can identify hallucinated versus faithful responses. It is useful both for evaluation and for constructing training data for DPO-based mitigation.

Measuring whether a model's expressed uncertainty is calibrated requires comparing stated confidence to empirical accuracy. The standard tool is an expected calibration error (ECE) analysis:

where:

• : a bin of predictions with confidence scores falling in the same range
• : the actual accuracy within bin
• : the average confidence within bin
• : the total number of predictions

A model with ECE close to 0 is well-calibrated: when it says it is 80% confident, it is right 80% of the time. A model with high ECE is miscalibrated: its stated confidence does not track its actual accuracy. Most language models are overconfident in their factual claims, meaning ECE analysis will show that their high-confidence claims are correct less often than their confidence implies.

For production systems, the most meaningful evaluation is end-to-end: measure the rate of factual errors on a representative sample of real user queries with ground truth labels. This is expensive (it requires human annotation at scale), but it is the only evaluation that captures the full distribution of failure modes in your deployment context.

A practical compromise is periodic automated evaluation on a curated validation set, with human spot-checking to catch systematic errors that the automated metrics miss. The validation set should be refreshed periodically to include new query types and to avoid overfitting mitigation strategies to a fixed benchmark.

Hallucination mitigation is not a solved problem, and each mitigation strategy carries its own limitations.

Retrieval augmentation depends critically on retrieval quality. A well-constructed knowledge base with a strong retrieval model and careful context formatting is the single largest lever available to practitioners. But building and maintaining that knowledge base requires ongoing engineering investment. For open-domain factual questions, the knowledge base cannot cover every relevant fact; for recent events, it requires continuous updating. RAG also shifts the failure mode rather than eliminating it: instead of model hallucination, you get retrieval failure and context-ignoring errors, which may be harder to detect. A system that confidently returns a retrieved passage that is outdated or incorrect has simply substituted one type of factual error for another.

Self-consistency decoding assumes that incorrect answers are more variable than correct ones. This is true on average but not always. A model that is consistently wrong (a systematic error learned from biased training data) will show high consistency but low accuracy. Self-consistency is most effective for tail-end hallucinations where the model is uncertain; it does not help with confident systematic errors. If a model was trained on a large corpus that consistently misrepresents some fact, self-consistency will amplify that error rather than catching it.

Training-based approaches require data. High-quality preference datasets pairing hallucinated and truthful responses are expensive to construct and may not cover the distribution of queries in production. DPO and RLHF improve behavior on the in-distribution queries they were trained on; out-of-distribution queries remain risky. Models trained with strong anti-hallucination objectives may also develop excessive refusal behavior, being unhelpful on queries where they have sufficient knowledge to answer correctly. The tension between reducing hallucination and maintaining helpfulness is a fundamental design challenge in RLHF and DPO training.

Uncertainty expression is only valuable if it is calibrated: if the model says "I'm not sure" on every answer, it provides no useful signal. Calibrated uncertainty requires both training signal and rigorous evaluation on held-out data. Measuring calibration requires ground truth at scale, which is costly to collect. Most deployed models are overconfident: they express high certainty on claims that turn out to be false more often than a well-calibrated model would.

Post-hoc verification adds a latency cost and introduces its own error sources. An NLI model used for consistency checking has its own failure modes; an LLM used as a verifier can hallucinate corrections. Every layer added to a mitigation pipeline is an additional source of latency, cost, and potential failure. The engineering tradeoff between mitigation thoroughness and deployment practicality is real and depends on the stakes of the application.

Despite these limitations, the impact of combining these techniques in production has been substantial. Modern deployed LLMs that combine retrieval, self-consistency, RLHF-based anti-hallucination training, and uncertainty expression are measurably more reliable than base models. Benchmarks like TruthfulQA, FELM, and FActScore show consistent improvements when mitigation techniques are applied. The most impactful deployments use retrieval augmentation as the primary lever (addressing the knowledge gap problem directly) and uncertainty expression as the user-facing communication layer (replacing confident hallucinations with honest admissions of uncertainty).

The research frontier has moved toward more intrinsic solutions: training objectives that directly optimize for factuality rather than relying on the indirect signal of human preferences, calibration-aware training that teaches the model to model its own uncertainty, and interpretability tools that identify factual versus confabulated content in the model's internal representations. We will explore calibration in more depth in the upcoming chapter on Uncertainty Quantification.

Hallucination mitigation combines strategies across the generation pipeline, from knowledge sourcing through to output communication:

• Retrieval-augmented generation addresses the knowledge gap by supplying relevant documents at inference time. It is the most impactful single mitigation technique, but requires a high-quality knowledge base and attention to context faithfulness. Its failure modes include retrieval failure, context ignoring, and unfaithful extension beyond retrieved content.
• Decoding strategies modify generation behavior without changing model weights. Temperature tuning, top-p filtering, and self-consistency decoding improve factual accuracy by reducing variance and requiring cross-sample agreement. Contrastive decoding amplifies what the large model has uniquely learned relative to a smaller model.
• Training approaches include SFT on factual data, RLHF with factuality-aware human preferences, DPO on hallucination/truthful response pairs, knowledge-grounded fine-tuning, and surgical fact editing. These modify the model's intrinsic behavior but require quality training data and do not eliminate the problem.
• Prompt engineering provides low-cost gains by asking for uncertainty expression, requiring citations, and decomposing complex questions into sub-questions before answering.
• Post-hoc verification uses NLI models, LLM-based fact checking, or structured frameworks like FActScore to verify generated claims against reference knowledge after generation.
• Uncertainty expression treats communication of uncertainty as a feature. Verbalized hedges, calibrated abstention, token-level entropy signals, and selective prediction help users understand when to trust model output.

Effective hallucination mitigation in production typically combines all of these categories. RAG reduces factual errors for covered queries; decoding strategies add consistency guarantees; training reduces systematic biases; uncertainty expression handles the remainder by surfacing rather than concealing the model's limitations. The next chapter on Attribution and Citation examines how to make RAG-grounded answers verifiable, closing the loop from mitigation to accountability.

QuizLink Copied