Hallucination Detection: NLI, Self-Consistency

Article links

Make inline references clickable

Language models generate fluent, confident prose. The problem is they sometimes generate fluent, confident prose about things that never happened. A model can cite a paper that does not exist, attribute a quote to the wrong person, or invent a plausible-sounding clinical statistic with no factual basis. As we explored in the previous chapter on hallucination types, these failures range from mild imprecision to outright fabrication. Detection is the diagnostic side of the hallucination problem: given a model's output, how do you tell whether it can be trusted?

The stakes depend heavily on the domain. A hallucinated movie recommendation is inconvenient; a hallucinated drug interaction could be fatal. Medical, legal, and financial systems that deploy language models at scale need automated tools for flagging potentially false claims before they reach users. Even in lower-stakes settings, teams iterating on model quality need ways to measure factual accuracy at scale, since human review of every generated response is impractical once generation volumes reach thousands of responses per day.

Detecting hallucinations turns out to be significantly harder than it sounds. You cannot simply compare the output to "the truth," because you usually do not have ground truth available at inference time. Instead, detection methods must reason about consistency, entailment, and knowledge coverage, often using other models as judges. The field has converged on four major families of approaches: entailment-based detection, knowledge base verification, self-consistency checks, and learned detection models. Each has different strengths, failure modes, and computational costs.

One useful lens for comparing these approaches is the reference they rely on. Entailment methods use a provided source document as the authority. Knowledge base methods use a curated external database. Self-consistency methods use the model's own output distribution as an implicit reference. Learned detectors use a labeled training set of human judgments. The absence of a universally available ground truth is why hallucination detection remains an open problem: each reference type has gaps, and no single approach covers all of them.

This chapter works through each approach in depth, building up from the mathematical foundations to working code. By the end, you will have a practical toolkit for measuring factual reliability in LLM outputs. This toolkit is the foundation for the mitigation strategies covered in the next chapter on hallucination mitigation.

The most principled approach to hallucination detection frames the problem as a Natural Language Inference (NLI) task. NLI is the problem of deciding, given a premise and a hypothesis , whether entails , contradicts , or is neutral with respect to . This framing captures exactly what we care about in hallucination detection: we want to know whether the model's claims are supported by, contradicted by, or simply absent from some reference text.

Applied to hallucination detection, the premise is the source document or retrieved context, and the hypothesis is a claim from the model's generated output. If the source entails the claim, the claim is likely grounded. If the source contradicts the claim, the claim is almost certainly a hallucination. Neutral relationships are more ambiguous: the claim may be true but unsupported by the available source, or it may be false but simply not addressed by the source. For high-stakes applications, both contradiction and neutral often warrant further inspection.

Why use NLI rather than simpler similarity measures like cosine similarity or ROUGE? Because similarity is not the same as factual support. A generated summary could be semantically similar to a source document while still containing false claims, simply because it uses the same vocabulary and covers the same topic. NLI models are explicitly trained to reason about logical relationships, not surface overlap. They can recognize that "Curie won the Nobel Prize in 1905" contradicts "She won the Nobel Prize in 1903" even though both sentences are about the same topic and share most of their words. The NLI framing captures this directional, logical relationship that similarity metrics miss.

An NLI model assigns probabilities over three mutually exclusive labels. Given a source document and a generated claim , the model produces:

where:

• : the source document or reference passage used as the premise
• : the generated claim being evaluated as the hypothesis
• : probability that the source logically implies the claim
• : probability that the source neither implies nor contradicts the claim
• : probability that the source implies the negation of the claim
• by construction

The factual consistency score for claim is typically taken as the entailment probability, since this directly measures how well the source supports the claim:

where a score near 1 means the source strongly supports the claim, and a score near 0 means the claim is unsupported or contradicted. In cases where you want to specifically flag contradictions rather than just low entailment, you can threshold on directly, which behaves differently from because neutral claims will not be caught by the contradiction signal.

For a full generated summary or response decomposed into sentences , you compute a document-level score by aggregating the per-sentence scores. Two common aggregation strategies are the minimum and the mean:

where:

• : number of sentences in the generated output
• : the -th sentence of the generated output
• : entailment probability for that sentence against the source

The minimum-based score is more conservative: a single unsupported sentence tanks the overall score, making it well-suited for high-stakes applications where any fabrication is unacceptable. The mean is more forgiving and often correlates better with human judgment on longer documents where minor imprecisions are expected. A practical approach is to report both and use the minimum as an alert threshold while using the mean as a quality signal.

There is also a third option, weighted aggregation, where sentences containing named entities, numbers, or dates receive higher weight because they carry the highest hallucination risk. This requires a lightweight tagger to identify high-stakes claims before scoring but can improve correlation with human judgments on documents where factual precision matters unevenly across sentences.

SummaC (Summarization Factual Consistency) formalizes the entailment idea into a practical detection framework that handles long source documents more gracefully. Rather than scoring each generated sentence against the full source concatenated together, SummaC scores each generated sentence against each source sentence individually and builds a score matrix.

The key motivation for this design is that source documents are often long and structured. A question-and-answer document might have dozens of paragraphs, but a particular generated claim may only be supported by one specific paragraph. If you concatenate the entire source and pass it to an NLI model, you run into the model's context limit and dilute the signal: the entailment score for a specific claim gets averaged across many irrelevant source sentences. SummaC solves this by finding the single source sentence that best supports each generated sentence, which is both more accurate and more interpretable. It also sidesteps the context length problem entirely, since each NLI call processes only a single pair of sentences.

Let the source be split into sentences and the generated output into sentences . The score matrix captures all pairwise entailment scores:

where:

• : entailment probability that source sentence supports generated sentence
• : the -th sentence of the source document, used as the premise
• : the -th sentence of the generated output, used as the hypothesis
• : the matrix has one row per generated sentence and one column per source sentence

For each generated sentence , the row score is the maximum entailment score across all source sentences:

where:

• : how well the best-matching source sentence supports generated sentence
• The maximum is taken because a claim only needs to be supported by one part of the source to be grounded. Averaging would penalize claims that are tightly supported by a specific section simply because most other source sentences are irrelevant to that claim.

The final document-level score is the mean of row scores:

where:

• : the overall factual consistency score for the generated text
• Values near 1 indicate all generated sentences are well-supported by the source
• Values near 0 indicate many generated sentences lack source support

The construction of the score matrix also gives you interpretability for free. When a sentence receives a low row score, you can inspect the row to see which source sentences were checked and confirm that none of them supports the claim. This makes the detection system auditable, which is valuable in applications where you need to explain why a response was flagged.

SummaC significantly outperformed prior metrics like ROUGE on factual consistency benchmarks, particularly on cases where the summary is fluent and semantically similar to the source but contains subtle factual errors. The improvement is most pronounced on long source documents and on numerical claims, where surface similarity is a poor proxy for factual accuracy.

The quality of entailment-based detection depends heavily on which NLI model you use as the backbone. This choice affects accuracy, generalization across domains, sensitivity to numerical claims, and handling of long inputs. Three models are worth understanding in depth because they represent different points on the generalization-precision tradeoff.

DeBERTa-v3-large-mnli is the strongest general-purpose option. DeBERTa improves on BERT by using disentangled attention, which encodes content and positional information separately and allows the model to better capture fine-grained semantic relationships, including the kind of subtle logical dependencies that distinguish entailment from neutral. Trained on the MultiNLI dataset, which covers fiction, telephone transcripts, travel guides, government documents, and other genres, this model generalizes well but was not specifically optimized for factuality in machine-generated text. It is the right choice for new domains where no fine-tuned model exists and you are willing to accept slightly lower precision in exchange for broad coverage.

FactCC trades generalization for precision on document-summarization factuality. By training on synthetically perturbed news articles, FactCC learns the specific kinds of errors that summarization models make, including entity swaps, pronoun confusions, and subtle numerical changes. On standard summarization factuality benchmarks covering XSum and CNN/DailyMail, FactCC significantly outperforms generic NLI models. However, because it was trained entirely on news data, it can underperform on domains with very different writing styles, such as scientific papers or dialogue. If your application involves news summarization specifically, FactCC is likely the best choice; if not, it may be over-specialized.

TRUE is the best choice when you need to generalize across diverse input types. By training on a mixture of NLI, dialogue, question-answering, and fact-checking datasets, TRUE builds a broader understanding of what "factual support" means in different contexts. If your system handles both document summarization and conversational grounding, TRUE generalizes better than a single-domain model. Its multi-task training effectively regularizes the model against overfitting to any single dataset's distributional quirks.

In practice, the right approach is to evaluate two or three backbone options on a small labeled sample from your specific domain and use the model that best matches human judgments on that data. Swapping out the backbone is straightforward because the interface is always the same: a (premise, hypothesis) pair in, a three-way probability distribution out. The modular design of entailment-based detection means you can improve accuracy by simply upgrading the backbone without changing any other part of the pipeline.

Entailment-based detection works when you have a reference document to check against. But what if the model is making claims about the world at large, not about a specific source? In an open-domain question-answering system, users ask about historical events, scientific facts, and biographical details for which there is no single "source document" to use as a premise. In that case, you need a different approach: verifying claims against a structured knowledge base that encodes general world knowledge.

Knowledge base (KB) verification is conceptually the most direct approach to hallucination detection. Rather than asking whether a claim is supported by some available text, it asks whether the claim is true according to a structured database of known facts. This is a much stronger standard. It can catch hallucinations that happen to be consistent with vague or ambiguous source documents, and it can verify claims about facts that have never been mentioned in any retrieved context.

The general pipeline involves three steps. First, you parse the generated text into discrete, verifiable atomic claims. Second, you map the mentions in those claims to canonical entities in the knowledge base. Third, you check whether the claims hold according to the knowledge base records. Each step is a non-trivial NLP problem in its own right, and errors compound across steps.

Knowledge bases like Wikidata store information as subject-predicate-object triples. For example, the fact that Marie Curie received the Nobel Prize in Physics is stored as:

where:

• The first element is the subject: the entity the claim is about
• The second element is the predicate: the relation or property being asserted
• The third element is the object: the value of that property for the subject

To verify claims, you need to convert natural language claims into the same triple format. A claim extractor maps a sentence to a set of triples , where:

• : the natural language claim to be verified
• : the number of distinct factual assertions in the claim
• : the -th extracted triple, representing one factual assertion

This extraction step is itself a hard NLP problem. Common approaches include information extraction pipelines that use rule-based or neural models to identify entity mentions and their relations, LLM-based extraction that prompts a language model to enumerate the factual claims in a sentence in triple form, and Open IE systems that extract triples without a predefined ontology.

LLM-based extraction has become increasingly popular because it handles complex sentences and implicit relations more gracefully than rule-based systems. A simple prompt like "List all factual claims in the following sentence as subject-predicate-object triples" can reliably decompose sentences into atomic claims, especially for well-formed biographical or encyclopedic text. The downside is that LLM-based extraction introduces another model into the pipeline, adding latency and cost.

Each extracted triple is then verified by querying the knowledge base for whether the triple holds. For Wikidata, this means translating the triple into a SPARQL query and executing it against the Wikidata SPARQL endpoint.

Before querying, entity mentions must be resolved to canonical identifiers. "Curie" in one sentence and "Marie Curie" in another both need to map to the same Wikidata entity (Q7186). Without this normalization, a query for the person "Curie" will find nothing, and the triple will be incorrectly flagged as unverifiable.

The disambiguation problem is more subtle than it first appears. Consider the sentence "Jordan played for the Bulls." Without context, "Jordan" could refer to Michael Jordan the basketball player or the country of Jordan, but the phrase "played for the Bulls" strongly implies the basketball player. A naive entity linker that matches based solely on the most common entity for a surface form will fail on ambiguous cases. The context of the surrounding text, particularly other named entities and the semantic field of the sentence, is essential for correct disambiguation.

Learned entity linkers address this using a two-stage approach that has become the dominant paradigm. In the first stage, a bi-encoder embeds the mention in context and retrieves the top-K candidate entities from a dense index using approximate nearest-neighbor search. The mention encoder processes the surface form plus surrounding context, while the entity encoder processes the entity's name and description from the knowledge base. The top-K candidates are retrieved by computing the dot product between the mention embedding and all entity embeddings. In the second stage, a cross-encoder re-ranks the candidates by jointly encoding the mention context and the entity description. This two-stage design is efficient enough for real-time systems while achieving high disambiguation accuracy on standard benchmarks.

For claims about highly specific or niche entities, entity linking often fails entirely because the entity simply does not exist in the knowledge base's entity catalog. A language model generating text about a newly formed startup, a novel scientific concept, or a person who is not famous enough to have a Wikidata entry will produce entity mentions that no linker can resolve. This coverage limitation is one of the fundamental constraints of the KB verification approach and is why it works best for well-known entities in stable domains.

Once triples are extracted and entities are linked, you query a knowledge base to check consistency. For Wikidata, this means using SPARQL queries to retrieve the known relations for a given entity pair. If the expected predicate exists with the expected object, the claim is verified; if not, the claim is flagged as potentially false.

The limitation of this approach is coverage. Wikidata contains roughly 100 million triples, which sounds large but is a tiny fraction of all human knowledge. Claims about niche topics, recent events, or specialized domains often have no relevant entries. In those cases, the verifier must either abstain (producing no score) or fall back to a different approach.

A critical failure mode is the temporal gap. Wikidata reflects the state of the world as of when entries were last updated, which means verifying claims about recent events, people, or discoveries will frequently return no match, even when the claim is true. This issue is particularly pronounced for rapidly changing domains like sports statistics, corporate leadership, and ongoing scientific research. A claim that "Dr. X is the current CEO of Company Y" may be true but unverifiable if Wikidata's entry for Company Y has not been updated recently. Treating unverifiable claims as hallucinations inflates the false positive rate in temporally sensitive applications.

Knowledge base verification also struggles with relational claims that involve implicit temporal qualifications. "Einstein was a professor at Princeton" is technically true for a specific period of his life, but a KB query might find a different affiliation if the record was indexed at a different time. Properly handling temporal predicates requires additional logic to check the validity period of each triple, which most systems do not implement.

A hybrid approach that has gained significant traction is FActScore (Factual Precision Score). Rather than using a structured knowledge base, FActScore retrieves relevant passages from Wikipedia using a dense retrieval model, then checks each atomic fact against the retrieved passages using an NLI model. This approach captures the benefits of KB verification (checking against a broad knowledge source) while avoiding the coverage and temporal limitations of structured databases.

The key design insight is that Wikipedia contains far more information than any structured KB, covers a much wider range of entities, and is updated more frequently. Dense retrieval allows the system to find the most relevant passages for any given claim, even for niche topics not covered by Wikidata. The NLI step then checks whether those passages support or contradict the claim, using the same entailment reasoning we saw in the previous section.

The process for a generated biography or factual response :

1. Decompose into atomic facts using an LLM prompted to enumerate one claim per fact
2. For each fact , retrieve relevant Wikipedia passages using a retrieval model such as DPR or BM25
3. Score each fact as supported or unsupported using an NLI model applied to the retrieved passages
4. Report the proportion of supported facts:

where:

• : total number of atomic facts extracted from the generated text
• : indicates that fact is verified as supported by retrieved passages
• : cardinality (count) of the set of supported facts
• : proportion of facts that are supported; 1.0 means all facts are grounded, 0.0 means none are

FActScore was proposed alongside a large benchmark of human-written factuality annotations for LLM-generated biographies, making it one of the most carefully evaluated detection methods available. The benchmark revealed that popular LLMs at the time of publication had FActScore values between 0.3 and 0.6 on biographical text, meaning between 40% and 70% of their factual claims were not verifiable against Wikipedia. This quantification was influential in motivating investment in hallucination mitigation research.

The main cost of FActScore is its pipeline complexity. Each fact requires a retrieval call, which involves encoding the query with a dense retrieval model and running approximate nearest-neighbor search over a large index. Then each retrieved passage must be scored with an NLI model. For a single paragraph with a dozen facts, this results in dozens of model calls. The atomic fact decomposition step itself requires an LLM call. Total inference cost can be 50 to 100 times higher than entailment-only scoring against a fixed source document.

The approaches above all require some external reference, whether a source document, a knowledge base, or retrieved passages. But what if you do not have access to any of these? Self-consistency methods exploit a property of language models themselves: if a model is uncertain about a fact, its outputs will vary across different samples. If it confidently knows a fact, it should produce consistent outputs regardless of how the question is asked.

The intuition comes from thinking about the model's internal state. A well-trained language model encodes factual information in its parameters as probability distributions over tokens. For facts the model has seen many times in training, the distribution will be sharply peaked, and multiple samples will agree. For facts the model has encountered infrequently or inconsistently, the distribution will be flatter, and samples will diverge. Hallucinations often arise from the model generating plausible-sounding continuations rather than retrieving specific memorized facts. These plausible-sounding outputs tend to be more varied because there are many plausible-sounding falsehoods, but typically only one true answer.

This connection to Bayesian uncertainty is more than just intuition. Self-consistency scores have been shown to correlate with model calibration in several studies. Models that are well-calibrated (meaning their expressed confidence matches their accuracy) also tend to have higher self-consistency on facts they know and lower self-consistency on facts they do not. This suggests that consistency checking is tapping into epistemic uncertainty rather than just measuring output diversity.

SelfCheckGPT formalizes this intuition into a practical detection algorithm. Given a generated passage , the method works as follows:

1. Sample additional independent passages from the same model with the same prompt, using non-zero temperature to ensure diversity
2. For each sentence in , measure how consistently 's content appears across the sampled passages

The key point is that sampling is done independently: each is generated from scratch, not conditioned on . If the model repeatedly generates the same fact across independent samples, that fact is likely something the model has encoded reliably. If the model contradicts itself across samples, the claim is less trustworthy. Independence is essential because conditioning on would introduce strong coherence pressure that would cause the model to repeat the same claims, masking the model's uncertainty.

The consistency score for sentence can be computed in three ways, each with different tradeoffs between cost and accuracy.

NLI-based consistency measures how often the sampled passages contradict the claim. Higher scores indicate more frequent contradiction, which signals hallucination:

where:

• : number of independently sampled passages
• : the -th sampled passage, generated independently from
• : probability that contradicts claim according to an NLI model
• A high score means the claim is frequently contradicted by other outputs, suggesting hallucination

BERTScore-based consistency measures semantic similarity between the claim and sampled passages, treating low similarity as a signal of inconsistency:

where:

• : the BERTScore F1 between the claim and the sampled passage, ranging from 0 to 1
• Subtracting from 1 converts the similarity score into an inconsistency score: low similarity becomes high inconsistency
• A high score means the claim is semantically dissimilar from the sampled passages, suggesting the sampled outputs do not confirm the claim

N-gram overlap provides a lighter-weight alternative that requires no additional models. For each sampled passage , the method computes the proportion of n-grams in the claim that also appear in , then averages across all samples. The inconsistency score is 1 minus this overlap, so claims whose exact wording or phrasing rarely appears in sampled passages score higher. This approach trades some accuracy for simplicity: it cannot reason about meaning, so paraphrases of the same fact will be treated as dissimilar even if they express identical information. But its speed advantage is significant when the primary bottleneck is inference cost rather than detection precision.

The three variants span a spectrum of computational cost and accuracy. NLI-based scoring is the most accurate but requires running an NLI model for every (sampled passage, claim) pair, which is NLI calls per sentence. BERTScore is moderately cheaper and handles semantic paraphrases well. N-gram overlap is fastest and imposes no additional model dependencies. In practice, start with n-gram overlap for rapid prototyping, then upgrade to NLI-based scoring if you observe unacceptable false-negative rates on your validation set.

Self-consistency approaches have a fundamental limitation: they can only detect hallucinations that stem from model uncertainty. If a model has confidently encoded a false belief, it will produce that false belief consistently across all samples, and the consistency check will give it a high score indicating ground truth, when in reality it is a systematic error.

This is not a hypothetical concern. Language models are known to have systematic biases toward plausible-sounding but incorrect information in domains where training data is sparse or skewed. Studies of factuality errors in large language models have found that models confidently produce incorrect birth dates, wrong award years, and mistaken nationality claims for less-famous individuals, precisely because the pretraining data contained inconsistent information that the model averaged over in a systematically biased way. Consistency checks will not catch these cases, and may even inflate their apparent reliability.

Additionally, self-consistency requires multiple forward passes through the model. For a passage of sentences checked against samples, you need full generations. At samples (a common setting in the original SelfCheckGPT paper), this is an 11x increase in generation cost, not counting the NLI scoring step. For large models or high-throughput applications, this cost is often prohibitive.

There is also a subtle failure mode related to topic breadth. When the prompt is broad and the model can respond in many valid ways, the samples will naturally vary in which facts they mention, not because any of them are hallucinating but simply because the space of relevant true facts is large. A sentence about an obscure historical event might receive a high inconsistency score simply because other samples chose to discuss different aspects of the same event. Calibrating consistency thresholds on this kind of prompt is difficult without domain-specific labeled data.

Rather than reasoning from first principles about entailment or consistency, a different approach is to train a dedicated model specifically for the task of detecting hallucinations. These learned detectors treat factual inconsistency detection as a supervised classification problem, using human-annotated or synthetically generated training examples to learn what factual errors look like.

The general setup is:

• Input: A source document (or context) and a generated claim
• Output: A score or binary label indicating whether is supported by
• Training signal: Human-annotated examples of supported and unsupported claims, or synthetically generated perturbations of true claims

The appeal of this approach is that the learned detector can exploit patterns that are specific to how language models fail, which may not be fully captured by general NLI reasoning. A model trained on thousands of examples of LLM hallucinations will develop representations tailored to the distribution of real hallucination errors, rather than the distribution of NLI training data (which often consists of human-written sentences with artificial contradictions designed to probe logical reasoning). The distribution shift between human-authored NLI examples and LLM hallucinations is significant enough that dedicated training can yield meaningful accuracy gains.

FactCC (Factual Consistency Checking) was one of the first dedicated models for summarization factuality. It uses a BERT-based classifier trained on synthetic data generated by perturbing source sentences in specific ways designed to mimic the failure modes of summarization models:

• Entity swap: Replace a named entity with a different entity of the same type, simulating the common error of attributing actions or properties to the wrong person or organization
• Pronoun swap: Change pronouns to introduce coreference errors that flip the subject of an action
• Number perturbation: Modify numerical values slightly, simulating the tendency of summarization models to change quantities while keeping surrounding text intact
• Noise injection: Insert or delete words at random, creating grammatically plausible but factually inconsistent text

These perturbations create realistic-looking hallucinations that share the surface form and vocabulary of the original text. The model learns to distinguish original sentences (factually consistent) from perturbed ones (hallucinations). The key advantage is that synthetic data generation is cheap, allowing for large-scale training without expensive human annotation.

FactCC formulates detection as binary classification. Given a (source, claim) pair , the model predicts a consistency score:

where:

• : the source document and claim concatenated with a separator token, fed as a single input sequence
• : the CLS token representation output by BERT after processing the full concatenated input, which encodes a summary of the relationship between and
• : a learned weight matrix that maps the CLS embedding to a scalar logit
• : the sigmoid function, which maps the logit to a probability
• near 1 means the model predicts the claim is factually consistent with the source

Training uses binary cross-entropy loss over the labeled examples. The model learns to associate the entity swaps, number perturbations, and other synthetic errors with the "inconsistent" label, building up internal representations that capture what factual disagreement looks like. The resulting model transfers reasonably well to real summarization hallucinations because real errors frequently take the same forms as the synthetic perturbations.

A more recent approach, MiniCheck, addresses the practical challenge that larger NLI models are expensive to run at scale. MiniCheck fine-tunes a small language model (7 billion parameters or fewer) specifically for the task of claim verification against a grounding document, achieving performance competitive with GPT-4 on factuality benchmarks at a fraction of the cost.

MiniCheck casts the problem as a yes/no question: "Is this claim supported by the document?" The model processes the source document and claim together and outputs a binary probability. It is trained on a curated mixture of factual consistency datasets from summarization, question-answering faithfulness datasets, and synthetic perturbations similar to those used in FactCC.

By combining diverse supervision signals, MiniCheck generalizes better across domains than models trained on a single dataset. It is also designed to handle long-form inputs using a windowed approach when source documents exceed the model's context window: the source is split into overlapping windows, each window is scored independently against the claim, and the maximum entailment score across all windows is reported. This windowed design mirrors the logic of SummaC but uses a single model for both the window comparison and the final scoring step.

The cost advantages of MiniCheck are substantial. Replacing GPT-4-based factuality checks with MiniCheck in a production pipeline reduces inference cost by roughly two orders of magnitude, making it practical to check every generated response rather than only sampling a subset.

TRUE (Towards a Unified Model for Factual Precision Evaluation) trains a single model on many fact-checking tasks simultaneously to build a general-purpose factuality detector that works across diverse input types. The training datasets include:

• Summarization factuality (XSum, CNN/DailyMail) covering news summaries
• Dialogue factuality (Wizard of Wikipedia, FaithDial) covering conversational grounding
• Question answering faithfulness (NaturalQuestions, TriviaQA) covering open-domain QA
• Fact verification (FEVER) covering explicit claim verification against Wikipedia

TRUE frames all tasks as textual entailment and uses a shared NLI objective. The motivation is that these tasks are all instances of the same underlying question, "Does text A support text B?", but they vary in their input structure, domain, and the nature of the claims being checked. Training on a mixture forces the model to learn a general notion of factual support rather than overfit to the stylistic patterns of any single dataset.

The multi-task training signal acts as a form of regularization. A model that can perform well on both news summarization and dialogue grounding must have learned more fundamental representations of factual consistency rather than surface features that correlate with one domain's error patterns. On the TRUE benchmark, a model trained on the full mixture outperformed models trained on individual tasks across all evaluation domains, showing that the multi-task training provides a measurable benefit rather than just averaging the performance of individual models.

TRUE is particularly valuable when you are building a system that must handle diverse input types, such as a general-purpose LLM assistant that answers questions, summarizes documents, and engages in dialogue. A domain-specific model like FactCC would be optimized for only one of these use cases.

A fourth paradigm that has emerged alongside learned detector models is using large language models themselves as judges of factual consistency. Rather than training a dedicated classifier, you prompt a powerful LLM to evaluate whether a claim is supported by a source document or is factually correct.

A typical prompt follows this structure:

Source: [source document] Claim: [generated claim] Is the claim fully supported by the source? Respond with "supported", "contradicted", or "unsupported". Explain your reasoning briefly.

This approach has several advantages. It requires no fine-tuning, can be applied immediately to new domains without any labeled data, and can produce natural language explanations of why a claim is flagged. Larger LLMs like GPT-4 have shown strong correlation with human judgments on factuality benchmarks, often outperforming smaller dedicated models.

The disadvantages are cost and reproducibility. Each evaluation requires a full LLM forward pass, which is expensive at scale. Results also depend on the specific prompt and model version, making them hard to reproduce exactly. There is also a concern about using a model to evaluate the outputs of the same model family, which can introduce systematic blind spots if both models share the same factual errors.

Despite these limitations, LLM-as-judge has become the de facto evaluation approach for factuality in many research papers and production systems, particularly when the goal is measuring aggregate quality across a large set of responses rather than real-time per-response detection.

Let's implement the three major detection approaches and compare their outputs on a set of example claims. We will build an entailment-based detector, a self-consistency checker, and visualize the comparison.

Entailment-Based DetectionLink Copied

The faithful claims score high on entailment, while the hallucinated claims score high on contradiction or neutral. Notice that "Marie Curie was born in France" scores as contradiction because the source says she was Polish (though it also says she was naturalised-French, making this claim partially correct but directly at odds with the primary characterization in the source). "She won the Nobel Prize in Physics in 1905" scores high on contradiction because the source explicitly states 1903.

The neutral score for "Curie was the only woman ever to win the Nobel Prize" illustrates an important edge case. The claim is false historically, but the source document only asserts that she was the first woman to win. It does not say anything about whether others have won since. The NLI model correctly identifies that the source is insufficient to contradict the claim directly, even though the claim is wrong.

The SummaC approach scores each generated sentence against each source sentence, then takes the maximum. This helps when the source is long and only one section supports a given claim. Let's implement it.

Self-Consistency ScorerLink Copied

The key parameters for configuring the entailment-based and self-consistency detectors are:

• model_name: The HuggingFace model identifier for the NLI backbone. Larger models (e.g., cross-encoder/nli-deberta-v3-large) are more accurate; smaller ones (e.g., cross-encoder/nli-deberta-v3-small) are faster and better for batch processing.
• max_length: Maximum token length for the NLI input sequence. Set to 512 for most models. Longer source documents must be truncated or processed in windows.
• threshold: The entailment score cutoff above which a claim is considered grounded. A common starting point is 0.5, but optimal values depend on your domain and tolerance for false positives versus false negatives.
• n (samples): Number of independent passages sampled in SelfCheckGPT. More samples improve reliability of the consistency estimate but multiply inference cost. Typical values range from 3 to 20.

Visualizing Detection MethodsLink Copied

Before deploying any hallucination detector, you need to know how well it performs on data that resembles your actual use case. A rich ecosystem of benchmarks has emerged for this purpose, each targeting different aspects of the detection problem.

AggreFact is a benchmark for summarization factuality that aggregates annotations from multiple prior datasets, covering both abstractive summarizers and more recent language models. It distinguishes between different types of factual errors, such as entity errors and relational errors, and provides a large pool of human judgments for calibrating detector thresholds. Models that perform well on AggreFact tend to transfer reasonably to real production settings for news summarization tasks.

TRUE Benchmark evaluates detectors across multiple tasks simultaneously, covering summarization, dialogue, QA, and fact verification. Because it tests the same model on all tasks, it penalizes models that specialize in one domain at the expense of others. This makes it particularly useful for evaluating general-purpose detectors like TRUE or MiniCheck.

FELM (Fine-grained Evaluation of Language Model Factuality) focuses on a broader range of factual categories, including world knowledge, reasoning, and instructions, rather than just document-grounded claims. It exposes failure modes that document-only benchmarks miss, particularly on claims that require multi-hop reasoning or combining facts from multiple sources.

FActScore Benchmark is the gold standard for open-domain biographical factuality. By collecting detailed human annotations on LLM-generated biographies of people at different levels of fame, it reveals how model accuracy degrades as subjects become less prominent in training data. Models are much more accurate about famous historical figures than about minor figures who appeared infrequently in pretraining corpora.

When evaluating a detector, report area under the ROC curve (AUC) in addition to accuracy at a fixed threshold, since the optimal threshold will vary across applications. Also report false positive and false negative rates separately, because the cost of each error type differs: a false positive wastes a user's trust by flagging a correct response, while a false negative damages trust by letting a hallucination through.

Hallucination detection has advanced considerably in recent years, but the problem remains unsolved in several important ways. Understanding these limitations is not just academic: they directly determine when you can trust a detection system and when you need to add human oversight.

The deepest limitation is the reference problem. Most detection methods require some form of ground truth or reference against which to check generated claims. Entailment methods need a source document; knowledge base methods need structured facts; even self-consistency methods implicitly treat the model's own distribution as a reference. When no reference is available and the model is being asked to generate novel content, detection becomes nearly impossible without full external verification. This is often the expensive human process you were trying to automate in the first place. For truly open-domain generation, the honest answer is that automated detection is still insufficient and human review remains necessary for high-stakes content.

Scalability is a serious practical constraint that shapes system design. The most accurate methods, particularly knowledge base verification and multi-sample self-consistency, are computationally expensive. FActScore requires decomposing outputs into atomic facts, retrieving relevant Wikipedia passages for each, and running NLI on each pair. For a single paragraph with a dozen facts, this could require hundreds of model calls. Production systems must balance detection accuracy against the latency and cost constraints of real applications. A common engineering solution is to run lightweight detection (n-gram consistency or a small NLI model) at inference time for every response, and reserve expensive methods (FActScore, full KB verification) for periodic audits and offline quality measurement.

Generalization across domains remains an open challenge. Models fine-tuned on summarization factuality datasets often perform poorly on dialogue, question answering, or code generation contexts. The nature of what constitutes a "hallucination" differs across domains: a confident numerical approximation that is off by 5% might be acceptable in one context and catastrophic in another. Calibrating detection thresholds appropriately for each domain requires labeled data that is often not available. Teams deploying in specialized domains such as medicine, law, or finance should expect to collect domain-specific evaluation sets rather than relying on benchmark performance from general-purpose datasets.

Systematic model errors are fundamentally invisible to consistency-based detection. When a model has confidently internalized a false belief, it will produce that belief consistently across all samples, and consistency checks will rate it as highly reliable. This failure mode is not rare: studies have found that large language models systematically misrepresent less-famous individuals, invent plausible-sounding but incorrect citations, and confidently state wrong dates and statistics in domains where training data was sparse. The only reliable defense against systematic errors is external ground truth, which brings us back to the reference problem.

Entailment cascade errors are another subtle limitation of NLI-based methods. The entailment score for a claim depends on whether the claim is true and on how the source document expresses its content. A source that uses hedged language ("it is thought that...") or that describes uncertainty ("the date is disputed") may not strongly entail a confident generated claim even when the claim is correct. Conversely, a confidently stated claim in the source may entail a generated paraphrase even if the claim is oversimplified. NLI models are trained on clean, direct premise-hypothesis pairs and sometimes misfire on the messier rhetorical structures that appear in real documents.

Despite these limitations, detection methods have had significant practical impact. They enable automatic evaluation pipelines that can assess factual reliability at scale, making it feasible to run A/B tests on model changes and detect regressions in factual accuracy before deployment. They underpin retrieval-augmented generation systems that use detection scores to decide when to retrieve additional context. When an entailment score drops below a threshold, the system can trigger an additional retrieval step rather than returning the low-confidence answer directly. And they form the feedback signal for reinforcement learning approaches to hallucination mitigation, where a detected hallucination triggers a negative reward that discourages the generation of similar patterns.

The development of reliable detection methods has also sharpened our understanding of what makes hallucinations hard to prevent at training time. The patterns that detection models learn to recognize, such as overspecific numerical claims, entity attribute confusions, and event date errors, have informed the design of better data curation strategies and pretraining objectives. In this sense, the detection problem and the prevention problem are deeply intertwined: you cannot measure what you cannot detect, and you cannot prevent what you cannot measure.

Hallucination detection uses four main approaches, each suited to different contexts:

• Entailment-based detection frames the problem as NLI, scoring whether each generated claim is entailed by a source document. The SummaC framework extends this to document level by scoring each generated sentence against each source sentence and taking the maximum. It requires a reference document but is efficient and interpretable, making it the natural choice for retrieval-augmented generation systems.

• Knowledge base verification extracts atomic claims as subject-predicate-object triples and checks them against structured databases like Wikidata. FActScore uses a hybrid approach combining dense retrieval with NLI scoring against Wikipedia passages. It can verify open-domain claims but is limited by knowledge base coverage, temporal gaps, and the cost of triple extraction and entity linking.

• Self-consistency checks exploit the principle that confident, accurate knowledge produces consistent outputs across independent samples. SelfCheckGPT measures the average contradiction score between a claim and multiple sampled passages using NLI, BERTScore, or n-gram overlap. It requires no external reference but is computationally expensive, requires multiple generation passes, and cannot detect confident systematic errors.

• Learned detection models such as FactCC, MiniCheck, and TRUE train dedicated classifiers using synthetic perturbations and human-annotated factuality datasets. They are fast at inference time and transfer across domains, but require careful training data curation to avoid overfitting to specific hallucination patterns. The emerging LLM-as-judge paradigm uses large language models as zero-shot evaluators, trading cost for flexibility.

In practice, the most reliable production systems combine multiple detection signals. A fast entailment check flags responses for which the retrieved context provides insufficient support, a consistency check catches claims that the model appears uncertain about, and periodic offline FActScore audits measure aggregate factuality trends across the system. The next chapter on hallucination mitigation examines how these detection signals can drive active interventions, from retrieval augmentation to training-time corrections.

QuizLink Copied