Probing Classifiers: Decoding What Language Models Learn

Article links

Make inline references clickable

When a language model reads the sentence "The cats chased the mice," it generates a sequence of hidden-state vectors, one per token. These vectors don't come labeled with grammatical roles or semantic properties. But if you train a simple logistic regression on top of them to predict part-of-speech tags, and that logistic regression achieves high accuracy, you have learned something important: the model's representations encode syntactic information, even though the model was never explicitly trained to do so.

This is the core idea behind probing classifiers. Rather than looking inside a model's weights directly, you extract its intermediate representations and ask: does this representation contain enough information to solve a specific linguistic task? The answer, measured by a simple classifier's accuracy, tells you what information is encoded where.

Probing classifiers have become one of the most widely used tools in NLP interpretability research. They address three persistent questions: What do language models learn? Where in the network is linguistic knowledge stored? And how can we move beyond accuracy numbers to understanding of internal representations?

The appeal of probing is partly its simplicity. You don't need to modify the model, retrain anything, or understand the internal mechanics of attention and feed-forward layers. You just freeze the model, extract vectors, and see what a simple classifier can do with them. But this simplicity is also a source of interpretive danger, because a method that is easy to apply is also easy to misinterpret. The bulk of this chapter is about using probing correctly, not just running it.

To understand why probing classifiers became central to NLP interpretability, it helps to understand what alternatives were available before them, and why those alternatives fell short.

The earliest approach to understanding what neural networks learned was weight visualization. In computer vision, you could visualize the convolutional filters of early layers and see that they resembled edge detectors and Gabor filters, which matched well-understood properties of the primate visual cortex. For language models, the equivalent approach would be to look at the weight matrices in word embedding layers. You could find that word vectors clustered semantically in embedding space, that "king - man + woman = queen" held approximately, and that syntactic analogies also worked. But this only worked for the embedding layer. The intermediate layers of a transformer, with their 768-dimensional hidden states produced by interleaved attention and MLP computations, are not interpretable by inspection.

A second approach was ablation: remove parts of the model and see what breaks. If removing an attention head hurts dependency parsing performance on a downstream task, maybe that head is doing dependency-related computation. But ablation is crude. It confounds the information encoded in representations with the model's ability to route and use that information. An attention head might be carrying syntactic information without being the sole processor of it; ablating it might cause minimal performance degradation because other heads compensate.

A third approach was analysis of attention patterns. Much early work visualized which tokens attended to which other tokens and argued that certain heads performed identifiable syntactic operations. Heads that attended strongly from a verb to its subject seemed to be computing subject-verb agreement. This was qualitatively interesting but hard to quantify: how much syntactic information is in the attention pattern? Is the pattern causing the downstream behavior or coincidental?

Probing classifiers offered something different. They provided a quantitative, reproducible method: train a classifier, measure accuracy, report a number. If the number is high, the representations encode the probed property. The method generalizes across any linguistic property for which you have labeled data. It does not require modifying the model, so it can be applied to any pretrained model without retraining.

The first systematic probing studies appeared around 2018 and 2019, coinciding with the rise of BERT. Alain and Bengio (2016) had already proposed the basic methodology for computer vision, and the NLP community quickly adapted it. By 2019, there were dozens of papers probing BERT for POS, syntactic dependency, named entity type, coreference, semantic role labeling, and many other properties. The field had a new vocabulary: probes, probing tasks, probing profiles, and selectivity scores.

The mechanics are straightforward. You take a pretrained model and freeze its weights entirely. You then pass a dataset of labeled examples through the model and collect the hidden-state representations at some intermediate layer. Finally, you train a small classifier, typically logistic regression, on those frozen representations to predict the labels.

The classifier is intentionally simple. A linear model (logistic regression) is the standard choice because it can only succeed if the information is linearly decodable from the representations. If the probe must itself solve the task through complex nonlinear computation, you can no longer attribute the success to the base model's representations. Using a small multilayer perceptron (MLP) as a probe is sometimes done, but it introduces ambiguity: did the probe find the information in the representations, or did it compute it from scratch?

The workflow is:

1. Choose a linguistic task with labeled data (e.g., part-of-speech tagging, syntactic dependency labeling, named entity type prediction).
2. Pass each example through the pretrained model, extracting the representation at a specific layer for a specific token.
3. Train a linear classifier on those representations with the linguistic labels as targets.
4. Evaluate on a held-out test set.
5. Interpret the accuracy as evidence of what the representations encode.

This workflow looks like standard supervised learning because it is. The key departure from normal supervised learning is that the features (the representations) come from a frozen pretrained model, not from the task-specific training process. Everything the representations can tell you was determined before the probe was trained.

The term "encoding" deserves scrutiny. When researchers say that BERT "encodes" POS information, what exactly do they mean?

The formal definition traces back to information theory. A representation encodes a property if the mutual information is greater than zero:

where:

• is the entropy of the label distribution, measuring how uncertain the label is without any representation
• is the conditional entropy of the label given the representation, measuring residual uncertainty after seeing the representation
• is the reduction in uncertainty: how much knowing the representation tells you about the label

If mutual information is zero, the representation tells you nothing about the label. The property is completely absent. If mutual information equals , the representation determines the label perfectly.

The problem is that computing mutual information exactly for high-dimensional continuous representations is intractable. Probing accuracy is used as a proxy. A probe that achieves perfect accuracy corresponds to zero conditional entropy, which corresponds to perfect mutual information. In practice, probe accuracy gives you a lower bound on mutual information, not an exact estimate. Pimentel et al. (2020) analyze this relationship formally and show that the connection between probe accuracy and information content is subtle: different probe architectures and training procedures can give different accuracy estimates for the same underlying mutual information.

The choice of probe architecture matters more than it might seem. A linear probe is interpretable in a strong sense: it succeeds only if the linguistic property is accessible as a linear function of the representation space. This maps onto a meaningful geometric intuition. If part-of-speech classes are linearly separable in the representation space, a hyperplane exists that correctly partitions noun vectors from verb vectors. You can visualize this, compute feature importances, and reason about what dimensions of the representation the probe exploits.

Nonlinear probes, such as shallow MLPs with ReLU activations, relax this requirement. They can succeed even when the information is encoded in a nonlinear way. But this creates an interpretability problem: the probe's success might reflect its own computational power, not the structure of the representations. The worry is that a sufficiently powerful probe would succeed on random representations, defeating the purpose of the experiment.

This concern has led to a general norm in the field: use the simplest probe that can answer your question. If a linear probe achieves 95% accuracy on POS tagging, there is no reason to use an MLP. If a linear probe achieves only 60% and an MLP achieves 85%, the gap tells you that the information exists but is not linearly organized, which is itself informative.

The geometric intuition is worth making explicit. Consider a simplified 2D representation space. If a linear probe can separate POS classes, each class occupies a roughly convex region of the space and a straight line (or hyperplane in higher dimensions) suffices to partition them. The probe's weight vector defines the orientation of this hyperplane. If classes overlap in a way that only a curved boundary can separate them, a linear probe fails but an MLP might succeed. The following visualization demonstrates both scenarios using synthetic 2D data, illustrating why the choice between linear and nonlinear probes encodes a meaningful claim about the structure of the representation space.

The value of a probing experiment lives or dies by the quality of its task design. The task must be test linguistic information, have a clear definition, and matched to the kind of representation being probed.

Probing tasks roughly divide into three categories based on the level of linguistic abstraction they target. Syntactic tasks ask about grammatical structure, the kind of information that traditional NLP systems modeled explicitly with parsing algorithms. Semantic tasks ask about meaning-level properties that depend on context and world knowledge. Surface tasks ask about superficial statistical features that a model could plausibly encode without deep understanding. Comparing probe performance across these three levels tells you something important about the nature of the representations: models that encode only surface features are very different from models that capture linguistic structure.

The most studied probing tasks involve syntax. Part-of-speech tagging asks whether a token's representation encodes its grammatical category (noun, verb, adjective, etc.). Syntactic dependency labels ask whether the relation between a head and its dependent (subject, object, modifier) is encoded. Both tasks have gold-standard annotated corpora (Penn Treebank, Universal Dependencies) that provide large amounts of labeled data.

Constituent tree depth probing asks a more subtle question: given the representation of a token, can a probe predict how deep that token sits in the syntactic parse tree? A noun embedded inside a prepositional phrase inside a relative clause sits deeper than a main-clause subject. The finding that BERT representations encode tree depth (Jawahar et al., 2019) suggests the model builds something like an internal parse tree even when trained only on raw text with masked-language modeling.

Grammatical number agreement is another well-studied syntactic probe. Given a sentence like "The keys on the table are on the shelf," does the representation of "are" encode that "keys" (not "table") is the subject that determines plural agreement? Probes for subject-verb number agreement test whether long-distance syntactic dependencies are represented, not just local bigram-level patterns. The surprising finding is that BERT representations support this kind of long-range agreement even across intervening noun phrases, suggesting that something functionally analogous to syntactic structure guides how representations are formed.

Syntactic dependency probing goes further by asking about the labeled arc structure of a parse. For a sentence parsed as a dependency tree, each word has a head (the word it depends on) and a dependency label (the grammatical function of the relation, such as subject, object, or modifier). Probing whether the representation of a word predicts its dependency label, independent of its POS tag, tests whether syntactic function is encoded separately from syntactic category. The answer, across multiple studies, is that it is: dependency labels are accessible, and they carry information beyond what POS alone would predict.

Semantic probing tasks ask about meaning-level properties. Named entity type prediction is common: does the representation of "Paris" distinguish it from "Microsoft" as a location versus an organization? Semantic role labeling probes ask whether the representation of a word in a sentence encodes its thematic role as agent, patient, or instrument.

Coreference probes examine whether the model represents that two mentions in a text refer to the same entity. A pair of representations for "John" and "he" in the same paragraph should, if coreference is encoded, be more similar when "he" refers to "John" than when it refers to a different individual. Word sense disambiguation probes ask whether the representation of a polysemous word like "bank" reflects which sense is active in context. These tasks are harder to probe successfully because they depend on long-range context and compositional semantics.

Semantic similarity probing tests whether the cosine distance between two word representations correlates with human judgments of semantic relatedness. This is a regression task rather than a classification task, and it measures something slightly different: not whether a discrete property is encoded, but whether the geometry of the representation space reflects semantic relationships.

Sentiment-relevant feature probing has also been studied: given a movie review, does the representation of a word encode whether it is more likely in positive or negative reviews? This connects probing to interpretability for NLP applications rather than just linguistic science.

There is a real risk in probing research of probing surface statistical regularities rather than linguistic structure. If your training corpus is dominated by financial news, models might learn that "bank" in financial contexts tends to appear near "interest rate" and "deposits." A probe might detect "financial institution" sense not because the model understands word sense disambiguation, but because it has memorized co-occurrence statistics.

This motivates careful task design that controls for surface confounds. Good probing tasks:

• Have balanced class distributions across varied contexts
• Are tested on out-of-domain examples to check for generalization
• Separate linguistic structure from corpus-level statistics
• Are defined precisely enough that two annotators would agree on the correct label

The position-in-sentence probe is a canonical example of a surface task. A probe trained to predict whether a token appears in the first or second half of the sentence can succeed simply because BERT's positional embeddings encode position directly. This has nothing to do with deep linguistic understanding. Including such tasks alongside linguistic tasks lets you calibrate what "high accuracy" means: if the surface task achieves 99% accuracy and the syntactic task achieves 88%, the 88% looks more impressive. If the surface task achieves only 65%, the 88% is even more striking.

Token length is another surface task. Short function words (the, a, it, in) have different syntactic roles on average than long content words (subsequently, particularly, generalization). A probe for token length might accidentally proxy for syntactic category without capturing grammatical structure.

Another design choice involves granularity. POS tagging at the Universal POS level has 17 coarse categories. Penn Treebank POS has 45 fine-grained categories. Syntactic function labels number in the dozens. The coarser the task, the higher the probe accuracy, but the less informative the result. Knowing that a model can linearly separate nouns from verbs is less interesting than knowing it can distinguish 17 syntactic roles in complex embedded clauses.

Researchers often run probes at multiple granularities to understand how easily different levels of linguistic structure can be extracted. The pattern typically shows that coarse distinctions are encoded linearly and accessibly, while fine-grained distinctions require either deeper layers or more expressive probes.

This granularity tradeoff has a theoretical interpretation. Coarse POS categories (noun, verb, adjective) are highly predictive of surface distributional patterns: nouns appear after determiners, before verbs, and in positions where noun phrases occur. Models trained on raw text with any prediction objective will learn to distinguish these categories because their surface distributions are so different. Fine-grained categories (proper noun vs. common noun, main verb vs. auxiliary) are less predictable from local distributional statistics and require more sophisticated contextual analysis. The difficulty of probing fine-grained categories thus tells you something about the depth of the representations.

High probe accuracy is necessary but not sufficient for strong conclusions about what a model encodes. Interpreting probing results correctly requires careful reasoning about three failure modes.

This interpretive challenge is more subtle than it might appear. When you run a probing experiment and find that a linear classifier achieves 90% accuracy on POS tagging using BERT's layer 6 representations, the number raises more questions than it answers. Does it mean BERT "knows" syntax? Does it mean the representations are useful for downstream syntactic tasks? Or does it just mean that logistic regression is good at this, given 768 dimensions of input? Working through these questions carefully is what separates scientifically valid probing conclusions from naive overinterpretation.

The most significant interpretive trap is confusing "the information is accessible in the representation" with "the model uses this information when making predictions." A representation might contain linearly decodable syntactic information that the model's downstream attention and MLP layers never consult. Pimentel et al. (2020) show formally that mutual information between representations and labels is the correct measure of information content, but probe accuracy is only an estimator of this, and an imperfect one.

Consider: if you train a probe on the final layer representations and achieve 90% accuracy on dependency labels, does the model use those dependency labels when predicting the next word? Not necessarily. The syntactic information might be a byproduct of the model learning other statistical patterns. The model might solve its actual task (next-token prediction) through attention patterns that do not respect the syntactic structure you found.

This is why probing results should be interpreted as measuring information content in representations, not as measuring model behavior. The distinction matters for claims about model capabilities. A model that contains syntactic information in its representations is not the same as a model that solves syntactic tasks, and it is not the same as a model that uses syntax to make better predictions.

This gap between information presence and information use has motivated the development of causal interpretability methods, which you will encounter in the chapter on activation patching. Causal methods ask not "is the information there?" but "does the information causally affect predictions?" These are different questions that require different experiments.

A subtler issue is that different probing classifiers have different capacities to extract information, independent of how much information is there. A larger MLP with more parameters might extract more because it is doing more computation itself. A linear probe with regularization might fail to extract information that is clearly present because the regularization is too aggressive.

Voita and Titov (2020) formalize this as the minimum description length (MDL) approach to probing. Instead of measuring accuracy, they measure how many bits are needed to transmit the probe's predictions given the representations. A good probe needs few bits because the representations already encode the task. A poor probe, or one learning from scratch, needs many bits. This gives a more principled measure of how easily extractable the information is, controlling for the probe's own computational power.

A third problem is that the training data for the probe may be statistically dependent on factors that correlate with the labels, independent of what the model learned. If the probe training set contains many sentences where nouns are always followed by verbs, the probe might learn to exploit the position of the extracted token as a proxy for the POS label. This is especially problematic when the text used for probing comes from the same corpus as the model's pretraining data.

Good experimental design mitigates this by shuffling, stratifying, and using train/test splits that control for superficial correlates. Cross-domain evaluation is particularly important: train the probe on one domain (news text) and test on another (Wikipedia), making sure that corpus-specific statistics don't substitute for linguistic structure.

The most important methodological innovation in probing research is the control task, introduced by Hewitt and Lisserman (2019). Control tasks address a fundamental question: if a probe achieves 90% accuracy, does that tell us something interesting about the model, or does it just tell us that logistic regression is good at this task regardless of what representations you give it?

To understand why this question is sharp, consider a 768-dimensional representation space with only 80 training examples and 6 classes. The probe has parameters plus biases, nearly 60 parameters per training example. In this regime, the probe has enough capacity to perfectly separate any 80 points in 768 dimensions by random chance. If the probe can memorize arbitrary labels with high accuracy, then high accuracy on real labels tells you nothing specific about what BERT learned. The control task is designed to measure this memorization capacity directly.

The idea behind a random control task is elegant. Assign random labels to the training examples, drawn from the same label distribution as the real task but with no semantic connection to the input. Then train the probe on these random labels using the model's representations. Measure the accuracy.

If the probe achieves high accuracy even with random labels, it means either the representations are so high-dimensional that the probe can memorize the training set (overfitting), or the probe architecture is too expressive and can extract arbitrary patterns from the representations.

If accuracy on random labels is near chance (i.e., the probe can't overfit), but accuracy on real labels is high, you have stronger evidence that the probe is extracting meaningful information from the representations.

The control task thus defines a baseline. The selectivity metric captures how much of the probe's accuracy comes from linguistic signal rather than from the probe's capacity to memorize:

where:

• : probe accuracy trained on real linguistic labels (e.g., POS tags)
• : probe accuracy trained on randomly assigned labels drawn from the same label distribution

A high selectivity score means the probe succeeds specifically because of what the real labels capture about the input, not because of general memorization capacity. If the probe could memorize arbitrary labels equally well, the two accuracies would be similar and selectivity would approach zero.

The selectivity score has an intuitive interpretation in information-theoretic terms. The control accuracy measures how much the representations help with any classification task of this type (because they span a rich 768-dimensional space that a linear classifier can overfit). The real task accuracy measures how much they help with the specific linguistic task. The difference is specifically what those linguistic labels capture above and beyond what any labels would capture.

A related control is label remapping, where the actual labels are deterministically shuffled by a fixed permutation. Unlike random assignment, this preserves the label distribution but destroys the semantic relationship between input and label. Checking that the probe achieves lower accuracy under remapping than under real labels confirms the probe is learning something beyond label frequency statistics.

Label remapping is particularly useful when the label distribution is skewed. If nouns make up 40% of tokens and the probe achieves 60% accuracy on a random control (by predicting "noun" everywhere), the selectivity metric might be misleading without careful accounting for the majority class.

Another control is to train the probe on representations from a randomly initialized (untrained) model. If a probe achieves comparable accuracy using random model representations as using pretrained model representations, it suggests the task is trivially easy from input features, not that the model has learned anything meaningful.

This is particularly important for tasks like named entity type prediction, where the surface form of words (capitalization, character patterns) already carries strong signal. A randomly initialized model's embedding layer might encode capitalization information in a simple lookup-table way that is sufficient to solve the task. Comparing pretrained model accuracy against random model accuracy tells you how much the pretraining contributed beyond what character-level features can provide.

Understanding control tasks geometrically is illuminating. The representation space for BERT-base is 768-dimensional. A logistic regression probe defines a hyperplane in this space. In 768 dimensions, you can always find a hyperplane that separates a small number of points, regardless of their labels. This is a consequence of the fact that points in general position in -dimensional space are linearly separable as long as .

With 80 training examples and 768 dimensions, we are nowhere near the regime where linear separability fails. The probe can memorize any labeling. The control task explicitly measures this memorization capacity. The selectivity gap tells you: beyond the general separability of points in high-dimensional space, how much does the specific geometry of BERT's representations (shaped by pretraining on English text) help predict linguistic labels?

Ordinary probing classifiers ask whether a discrete property (POS tag, dependency label) is encoded as a linear function of a single token's representation. Structural probing, introduced by Hewitt and Manning (2019), asks a more ambitious question: is the metric geometry of the representation space isomorphic to the metric geometry of syntactic parse trees?

The paper's core claim is striking. For any sentence, the syntactic parse tree defines a metric on word pairs: the tree distance between two words is the number of edges on the shortest path connecting them in the tree. Hewitt and Manning hypothesized that BERT's representations encode this metric structure as a global geometric property of the representation space rather than as isolated features for each word.

To test this, they learned a linear transformation such that the squared Euclidean distance between transformed word representations correlates with parse tree distances:

where:

• is the tree distance between words and in the gold parse tree
• and are their BERT representations
• is a learned linear transformation (the structural probe)

The key architectural choice is that is a shared transformation, applied identically to every word representation. It is a global metric space transformation rather than a per-word classifier. This is a much stronger structural claim than a standard probe: each word's representation encodes parse tree depth, and the entire parse tree is encoded as a metric geometry in representation space.

The transformation projects representations into a lower-dimensional subspace (they use rank 64 out of 768 dimensions) in which tree distances are recovered. The fact that a 64-dimensional linear subspace of BERT's 768-dimensional space is sufficient to recover parse tree distances suggests that syntactic structure is encoded in a small, structured subspace of the representations, not diffusely spread across all dimensions.

This finding was influential for several reasons. It gave a geometric interpretation to what BERT learns: the model doesn't just classify words by POS, it organizes the representation space so that syntactically related words are geometrically close, in a way that exactly mirrors the parse tree structure. The structural probe also scales naturally to larger models and other architectures: any model that produces token-level representations can be probed for structural encoding.

Let us build a complete probing experiment from scratch. We will probe BERT representations for part-of-speech information, implement control tasks, and measure selectivity. The experiment uses a small corpus to stay tractable, but the methodology scales to any size.

We start with imports and a small POS-tagged dataset. In practice you would use a standard corpus like the Penn Treebank or Universal Dependencies; here we construct a compact synthetic dataset with varied grammatical structures.

With representations in hand, we split the data and train a logistic regression probe. The key is that the BERT representations are completely frozen; only the logistic regression parameters are trained.

Now we implement the critical control: random label assignment. This tests whether the high accuracy is specific to POS structure or whether the probe could memorize arbitrary labels equally well.

The selectivity score isolates what the real POS labels contribute over and above arbitrary label assignment. A large gap means the probe succeeded specifically because of the linguistic structure in the labels. A small gap means you cannot confidently attribute the probe's accuracy to POS encoding, regardless of how high the absolute accuracy is.

Note the subtlety in the control design. We assign random labels to both train and test independently, so the probe cannot memorize train-to-test mappings. The test accuracy for the control probe reflects how well a probe trained on random labels generalizes to new random labels, which, in 768 dimensions with regularization, is lower than its training accuracy. This is precisely the right comparison: we want to know whether the representations contain information about POS labels, not whether the probe can overfit on train.

One of the most revealing analyses in probing research is comparing probe accuracy across all layers of a model. This produces a probing profile that shows where in the network different types of information are most linearly accessible.

VisualizationLink Copied

It is also informative to compare multiple probing tasks at a fixed layer to understand what kinds of information that layer encodes most strongly. We compare POS tagging against two surface-level tasks: position (is the token in the first half of the sentence?) and token length (is the token short or long?).

Interpreting the ResultsLink Copied

The key insight from multi-task probing is that different types of information have different accessibility profiles across layers. Surface features like token length tend to be decodable from shallow layers. Syntactic structure peaks in middle layers where attention patterns have had time to integrate context. Semantic properties often peak in higher layers where representations have been transformed by many attention and MLP operations.

The position probe result is particularly informative about BERT's architectural inductive biases. Because BERT adds sinusoidal positional embeddings to the input, position information is explicitly injected into the representation from the very first layer. It persists through the network because positional information is useful for many downstream tasks. The fact that a linear probe can decode position from BERT's representations is thus less surprising than the fact that it can decode POS, because position is explicitly encoded while POS is emergent.

The selectivity framework addresses an important technical concern. Suppose your probe achieves 90% accuracy but the control probe also achieves 80% accuracy. Does the 10% selectivity gap represent "real" syntactic knowledge? Or does it just reflect that the real labels happen to cluster in the representation space slightly better than random labels due to some incidental correlation?

The honest answer is that selectivity is a relative measure, not an absolute one. It tells you how much better the probe does on the real task compared to random labels, controlling for the probe's capacity to memorize. A high selectivity score provides positive evidence. A low selectivity score is more ambiguous: it could mean the information is absent, or it could mean the probe can memorize both real and random labels equally well.

This has led researchers to use selectivity alongside other measures. Cross-linguistic generalization tests whether a probe trained on English generalizes to French syntactic labels, which is less likely to be exploiting idiosyncratic English corpus statistics. Counterfactual analysis checks whether removing the relevant syntactic information from the input drops probe accuracy while removing irrelevant features does not. MDL probing (Voita and Titov 2020) measures compression gain, giving a cleaner signal than accuracy alone.

The MDL framework is worth understanding in more depth. The codelength of a probe is the number of bits it takes to transmit the labels given the representations under the probe's model. A probe that assigns probability to the correct label for each example uses bits for that example. The total codelength is:

The compression ratio compares this to the uniform code (which assigns equal probability to all labels):

where:

• is the number of classes
• is the number of examples
• A compression ratio greater than 1 means the representations help; a ratio close to 1 means they do not

This ratio is invariant to the number of examples and classes, making it directly comparable across tasks and datasets.

Setting aside methodological subtleties, probing research has produced a consistent picture of what BERT-style models encode.

Syntactic structure is robustly encoded in middle layers. Tenney et al. (2019) probed for ten different linguistic tasks across BERT layers and found a clear stratification: basic features like POS and constituent labels peak in early-middle layers, while semantic tasks like coreference and semantic role labeling peak in later layers. The finding that the structure of language is layered inside BERT in roughly the same order as the NLP pipeline was striking enough to prompt the paper title "BERT Rediscovers the Classical NLP Pipeline." This result was not obvious: there was no reason in principle why the layers of a model trained on masked language modeling should correspond to the layers of the NLP processing hierarchy. The finding suggests that the hierarchical structure of language reflects how information needs to be processed to predict masked tokens effectively, rather than being a linguist.s organizational choice.

Syntactic trees are encoded in a geometric sense. Hewitt and Manning (2019) showed that the squared distance between words' representations in a transformed linear space correlates with their distance in the syntactic parse tree. This is a stronger claim than ordinary probing: syntactic information is accessible and encoded as a metric geometry in representation space. The structural probe they introduced finds a low-dimensional linear subspace of BERT's representation space (rank 64 out of 768 dimensions) in which tree distances are recovered with high accuracy. This means parse tree structure is concentrated in a structured subspace rather than distributed across all 768 dimensions, which has implications for how we think about where syntactic knowledge is "stored."

Semantic roles are partially encoded, but with more noise and ambiguity than syntax. The difficulty of semantic role labeling probes relative to POS probes reflects a property of these representations: semantics is less deterministic and more context-dependent than syntax. A verb like "run" takes different semantic arguments in "She runs a company" and "She runs a marathon," and the probe must somehow distinguish these uses. The partial success of semantic probes suggests that context-sensitive semantic information is present but harder to extract with a linear classifier.

World knowledge is surprisingly accessible via probing. Petroni et al. (2019) showed that filling in factual blanks ("The Eiffel Tower is in [MASK]") accesses information distributed across the transformer's representations. This is different from linguistic structure probing but uses the same methodology, and it demonstrates that the scope of what "encoding" means for large models extends beyond grammar to encyclopedic knowledge. The LAMA benchmark (LAnguage Model Analysis) quantified this: BERT's representations encode enough world knowledge to answer many factual questions in a cloze format, even though it was never trained on explicit question answering.

Number and gender agreement probes have also produced interesting findings. For languages with grammatical gender (French, German, Spanish), probes for noun gender consistently find that gender is encoded linearly at early-to-middle layers. This is less surprising than syntactic tree encoding, because gender in these languages is largely lexically determined and a model can learn gender from the distributional patterns of determiners and adjectives that agree with each noun. More interesting is the finding that agreement features are propagated through long dependency chains: the representation of a participle in a sentence with a complex noun phrase encodes the gender of the distant head noun, rather than the nearest noun.

The probing methodology is not specific to BERT. It has been applied to GPT-2, GPT-3, T5, and more recent models, with some consistent findings and some surprises.

Larger models generally encode linguistic properties more robustly. Probing accuracy for syntactic tasks tends to be higher in BERT-large than BERT-base, and higher again in models like GPT-3. This is consistent with the general finding that larger models trained on more data develop richer representations that support a wider range of downstream tasks.

The layer structure of probing profiles shifts in larger models. In BERT-base with 12 layers, syntactic information peaks around layer 6. In larger models with more layers, the peak shifts proportionally: in BERT-large with 24 layers, syntactic peaks appear around layers 10 through 14. This suggests that the fraction of total depth devoted to syntactic processing is roughly constant across scales.

GPT-2 and other causal language models show a different profile from BERT. Because GPT-2 uses causal (unidirectional) attention, each token can only attend to previous tokens. This architectural difference affects what information is accessible in individual token representations. Probing GPT-2's representations for syntactic information shows that syntactic structure is still encoded, but less uniformly than in BERT, because some information about future tokens (which a token's syntactic role may depend on) is unavailable.

Instruction-tuned models show somewhat different probing profiles than the base models they derive from. The fine-tuning process modifies the representations in ways that sometimes increase and sometimes decrease the linear accessibility of specific linguistic properties, depending on whether those properties are relevant to the fine-tuning objective.

Probing classifiers have transformed how researchers think about what language models learn, but the methodology has real limits that practitioners should understand.

The most important limitation is the gap between information accessibility and information use. Even if you prove that syntactic tree structure is linearly encoded in BERT's middle layers, this does not tell you whether BERT's attention heads or MLP layers consult this structure when making predictions. It is possible that the syntactic information is a byproduct of the model solving its pretraining task and is never downstream-relevant.

This matters for practical interpretability. If you are trying to understand why a model made a specific prediction, finding that some representation encodes some property is not enough. You need to show that the property causally influenced the prediction. Methods like activation patching (which we explore in the Activation Patching chapter) and causal mediation analysis address this gap directly. Probing tells you about the information in the representation; causal methods tell you about the information used in the computation.

The distinction also matters for model comparison. If Model A encodes syntactic structure and Model B does not, but both perform identically on downstream tasks, what can you conclude? You cannot conclude that syntactic structure is irrelevant, because Model B might have found a different route to the same performance. But you also cannot conclude that Model A is "better" at syntax in any practical sense, because the syntax it encodes may not be consulted.

Another limitation is the probe-complexity confound: more powerful probes can extract more information, but it becomes unclear whether the information was in the representations or whether the probe learned it. Choosing a linear probe is a partial solution, but the choice of regularization strength also affects what the probe can memorize. Weak regularization allows memorization; strong regularization prevents extraction of complex structure.

There is no universally agreed-upon probe architecture or regularization strategy. Papers use different settings, which makes comparing results across papers hazardous. The MDL probing framework is a promising step toward a principled measure, but it has not yet become the default.

Probing experiments require labeled datasets, and linguistic annotations are expensive to produce. Many probing papers use relatively small datasets (thousands of examples) compared to the training data of the models being probed (billions of tokens). This creates a mismatch: the model may have learned subtle linguistic structure from massive data that small probing datasets cannot detect.

Label imbalance is also a problem. If 80% of tokens are nouns and verbs, a probe that learns to output only those labels achieves high accuracy without capturing the full POS system. Stratified sampling and macro-averaged F1 scores help but do not fully solve the problem. Researchers should report both overall accuracy and per-class accuracy or macro-F1 to ensure that rare categories are evaluated fairly.

Dataset construction choices also matter. If all sentences in the probing dataset come from the same source as the model's pretraining corpus (news articles, Wikipedia), the probe may exploit corpus-specific statistical patterns rather than test linguistic information structure. Ideally, probing datasets should be drawn from a different distribution than the pretraining data to test whether the representations generalize.

A more subtle limitation is that most probing research has focused on English. When probing is extended to morphologically rich languages (Finnish, Turkish, Arabic), the picture changes. These languages encode far more information in the form of words, including case, number, gender, tense, and aspect, and the question of whether models trained primarily on English data encode this information is both practically important and theoretically interesting.

Cross-lingual models like mBERT (multilingual BERT) have been probed for their encoding of linguistic properties across languages. The findings show that mBERT encodes some universal linguistic properties consistently across languages, while language-specific properties are encoded in a more language-specific subspace of the representation. This has implications for transfer learning: properties encoded in language-universal subspaces are the most transferable to low-resource languages.

Despite these limitations, probing classifiers have had substantial impact. They gave researchers their first systematic tool for asking "what did this model learn?" without modifying the model or its training procedure. The finding that BERT's representations contain POS, syntactic, and semantic information was not obvious and changed how practitioners thought about pretraining. It explained why fine-tuning on a small labeled dataset could achieve strong performance: the model had already learned much of the linguistic structure it needed, and fine-tuning only needed to align that structure to the target task.

Probing also clarified the design space for model improvements. If probing shows that a model fails to encode some linguistic property, you can diagnose whether to add pretraining objectives, modify tokenization, or change the model architecture. The finding that standard BERT fails to encode certain types of temporal reasoning motivated pretraining objectives that expose the model to more temporal structure in the text.

The broader intellectual contribution is a conceptual vocabulary for discussing what language models learn. Before probing, the dominant framework was input-output behavior: does the model perform well on task X? After probing, researchers could ask more specific questions: which layer encodes property Y, how robustly, with what selectivity, and in what geometric structure? This vocabulary has shaped both academic research and practical interpretability work in industry.

Probing classifiers are a methodological foundation for understanding what language models encode in their representations. By training a simple classifier on frozen model representations and measuring its accuracy on linguistic tasks, you can determine what information is linearly accessible at each layer.

The key ideas from this chapter:

• A probe is intentionally simple (usually logistic regression) so that its success reflects properties of the representations, not the probe's own computation.
• Probing tasks span syntactic (POS, dependency, tree depth), semantic (NER, SRL, coreference), and surface-level (position, length) properties. Good task design includes balanced classes, varied contexts, and controls for surface confounds.
• Probe accuracy alone is insufficient; selectivity (accuracy minus control accuracy) is the correct measure. A high selectivity score means the probe's success is specific to the real labels, not general memorization.
• Control tasks establish a baseline by replacing real labels with random ones, measuring how much of the probe.s success comes from linguistic signal versus memorization capacity.
• The minimum description length framework provides a more principled alternative to accuracy-based evaluation, measuring how much the representations compress the labels.
• Structural probing (Hewitt and Manning 2019) extends beyond classification to ask whether the metric geometry of representation space mirrors syntactic parse tree structure.
• Probing tells you about information accessibility, not causal relevance to model predictions. Methods like activation patching provide the causal analysis that probing cannot.
• BERT and similar models robustly encode POS in early-middle layers, syntactic structure in middle layers, and semantic properties in later layers, recapitulating the classical NLP pipeline in their layer structure.

In the next chapter, we extend probing to compare across layers systematically, building probing profiles that reveal where different types of linguistic knowledge are concentrated in the network.

QuizLink Copied