Mechanistic Interpretability: Circuits, Induction Heads

Article links

Make inline references clickable

When a language model predicts that "The Eiffel Tower is located in" should be followed by "Paris," what computation produced that answer? Which weights fired, which attention heads activated, and which neurons collectively encoded the relevant geographic knowledge? Mechanistic interpretability is the research program that asks precisely these questions. Rather than treating a neural network as a black box that produces outputs from inputs, mechanistic interpretability tries to reverse-engineer the model into a human-understandable algorithm. It seeks to identify specific circuits, components, and mechanisms responsible for particular behaviors, and to do so with the same rigor that an engineer would use to understand a digital circuit board.

This chapter covers the core ideas of mechanistic interpretability: what circuits are, how researchers discover them, the canonical discovery of induction heads, and the broader implications of this line of work. We build on concepts from earlier chapters, particularly the attention mechanisms discussed in Part XIII: Self-Attention and the sparse autoencoders and feature interpretation covered in the previous two chapters of Part LIX: Interpretability. The next chapter covers activation steering, a technique that uses mechanistic understanding to directly modify model behavior.

The field has roots in the work of Chris Olah and colleagues at OpenAI (later Anthropic), who in 2020 published a series of articles about circuits in convolutional image networks. They found that early layers learn curve detectors, middle layers compose those into more complex shapes, and later layers encode high-level concepts like faces and dogs. These findings were not mere description; the researchers could point to specific neurons, specific weight patterns, and specific compositions that implemented those computations. The same spirit of analysis has since been applied to transformer language models with increasing ambition and precision.

Mechanistic interpretability differs from other interpretability approaches by aiming for completeness. Attention visualization (covered in Chapter 2 of this part) shows where a model "looks," but not what it computes when it looks there. Probing classifiers (Chapters 4-5) test whether certain information exists somewhere in the network, but do not explain how that information is used. Mechanistic interpretability aims to trace the full computational path from input to output, producing what the field sometimes calls an "algorithm-level" explanation: an account of the computation in terms a programmer could verify.

The central concept in mechanistic interpretability is the circuit. A circuit is a subgraph of a neural network, consisting of specific neurons, attention heads, and the connections between them, that implements a particular computation. Just as a physical circuit board contains sub-circuits responsible for specific functions (the audio processing circuit, the memory controller circuit), a transformer model contains sub-circuits responsible for specific linguistic and reasoning functions.

The circuit viewpoint emerged from the observation that neural networks are not monolithic. Not every neuron participates equally in every computation. When a model predicts the next word in "The president signed the bill into", most of the network's capacity is irrelevant. A smaller subset of components, connected in specific ways, performs the semantic and syntactic work that leads to "law." Circuits are those relevant subsets.

This idea is not obvious in advance. One might expect neural network computations to be distributed across the network, with every parameter contributing a small amount to every prediction. Instead, researchers have repeatedly found that specific behaviors are implemented by surprisingly small subsets of parameters. The model devotes localized computational resources to localized tasks. This modularity is what makes mechanistic interpretability tractable, at least for well-defined behaviors in appropriately sized models.

To make this concrete, we need a precise vocabulary for describing neural network computations. A transformer consists of layers, each containing a multi-head attention module and an MLP (feed-forward) module. We can think of the outputs of each layer as "residual stream" vectors that accumulate information as the computation proceeds.

In this framing, a circuit consists of:

• Nodes: Attention heads, MLP neurons, or entire layers that contribute to the computation
• Edges: The information flow between nodes, mediated by the residual stream
• Weights: The specific weight matrices (query, key, value, output, MLP in/out) that implement the computation

A key insight from the mechanistic interpretability literature is that attention heads communicate with each other through the residual stream. Head A writes information to the residual stream, and head B in a later layer reads that information. This read-write structure, rather than direct head-to-head connections, is how multi-layer computations are organized. The residual stream functions as the bus over which all inter-component communication happens, and understanding which information travels on that bus at each position is central to circuit analysis.

To understand the read-write structure more concretely, consider what happens when attention head in layer processes a sequence. The head reads the current residual stream at each position by applying its query and key matrices. The resulting attention pattern determines which positions the head "looks at." Then, using the value and output matrices, the head writes a new vector to the residual stream at the current position, a vector that is a weighted combination of the information read from attended positions. Downstream heads in later layers read the residual stream again, now enriched with whatever the upstream head wrote. The circuit concept captures which of these reads and writes matter for a particular behavior.

Because all transformer components communicate through the residual stream via addition, the final representation at layer for a given token can be decomposed as:

where:

• : the initial token embedding (input to the first layer)
• : the residual stream at the input to layer
• : the output of the attention module at layer (the sum over all heads)
• : the output of the MLP module at layer

This decomposition is powerful because it means each component's contribution to the final output can be studied in isolation. If you want to understand what a specific attention head contributes, you can look at the vector it adds to the residual stream and project it onto the output embedding space to see which tokens it promotes or suppresses. This projection approach makes it possible to read the functional meaning of a head's output in terms of vocabulary logits, the same space where the model's final prediction lives.

The decomposition also underlies a useful analysis technique called "logit attribution." Because the final logit for token is the inner product of the final residual stream with the unembedding row for , and because the final residual stream is a sum of contributions, the logit can also be decomposed as a sum of contributions from each layer's attention and MLP output. This allows researchers to assign credit or blame for specific predictions to specific components.

The attention output for a single head can itself be decomposed. For head at layer :

where:

• : the attention weight matrix (which positions attend to which)
• : the value weight matrix
• : the output projection weight matrix

The product is sometimes called the OV circuit (Output-Value circuit) of the head. It determines what information the head reads from the attended positions and how it transforms that information before writing to the residual stream.

Similarly, the product is called the QK circuit (Query-Key circuit). It determines which positions attend to which based on their content in the residual stream.

Separating the OV and QK circuits is analytically useful because they perform distinct functions. The QK circuit answers "where should this head look?" while the OV circuit answers "given where it is looking, what does it write?" A head with a well-tuned QK circuit that attends to the right positions can still be useless if its OV circuit writes an uninformative vector. Conversely, a powerful OV circuit is meaningless without accurate attention targeting. Understanding a head requires understanding both circuits, and the product decomposition makes it possible to study each independently.

When researchers say two attention heads "compose," they mean the output of one head is read by the query, key, or value matrix of another. There are three types of composition:

• Q-composition: Head B reads Head A's output through its query matrix
• K-composition: Head B reads Head A's output through its key matrix
• V-composition: Head B reads Head A's output through its value matrix

Each type of composition creates a different functional relationship. V-composition is "information passing": Head A selects relevant content, and Head B further processes or re-routes it. Q-composition is "attention routing": Head A's output determines where Head B should look, allowing earlier layers to influence later attention patterns. K-composition is "relevance signaling": Head A's output determines which positions Head B considers relevant keys, allowing earlier computations to change the attractiveness of positions as attention targets.

Understanding which heads compose with each other is the key challenge in circuit analysis. A circuit for a specific behavior is exactly the set of heads and MLPs that participate in the computation, connected through the specific compositions that implement it. Two heads may both show high individual importance on a task (measured by ablation), yet fail to compose with each other. The information flowing from Head A might not reach Head B in a way that matters. Path patching (discussed below) is the technique that disambiguates this.

One useful quantitative measure of composition is the composition score. For Q-composition between Head A (layer ) and Head B (layer ), the score measures how much the output subspace of Head A overlaps with the query input subspace of Head B. Formally, if and are the respective weight matrices, and if we compute the Frobenius norm of their product relative to the norms of the individual matrices, we get a measure of how strongly A's outputs are used by B's query computations. High composition scores indicate that the heads are functionally related, making them candidates for the same circuit.

Discovering circuits is empirical detective work. The core methodology involves three steps: identify a clean, well-defined behavior; use ablations to find which components are necessary; and verify the components suffice to implement the behavior by studying their weights.

Good circuit analysis starts with a behavior that is as clean and minimal as possible. The classic example from early mechanistic interpretability work is "indirect object identification": given a sentence like "When Mary and John went to the store, John gave a drink to", the model should predict "Mary" as the next token. This task isolates a specific piece of linguistic knowledge (the indirect object in a double-object construction) from the full complexity of natural language understanding.

The researchers Chris Olah, Nick Cammarata, and others at Anthropic began this work by identifying behaviors in smaller transformers (two-layer models) where complete circuit analysis was tractable. Working with small, clean models before scaling is a standard methodological principle: the circuits that exist in toy models often appear in larger models too, but the full circuit is easier to characterize when the model has only a dozen attention heads rather than hundreds.

The choice of behavior matters a great deal. Behaviors that are too complex (like "solve a math word problem") decompose into many sub-tasks with independent circuits, making a unified circuit analysis intractable. Behaviors that are too simple (like "repeat the last token") may not reveal much that is interesting. The ideal behavior sits in the middle: complex enough to require non-trivial computation, simple enough that the circuit can be fully characterized. The IOI (indirect object identification) task and the greater-than task have proven productive precisely because they satisfy this balance.

A related consideration is the construction of appropriate test sets. For circuit analysis to be rigorous, you need a large, controlled set of inputs that isolate the behavior of interest from confounds. The IOI researchers created thousands of sentence templates of the form "When [Name1] and [Name2] [verb], [Name2] [verb phrase] [Name1]", varying the names systematically to ensure that their circuit discoveries held across diverse surface forms, rather than only a handful of cherry-picked examples.

As we covered in Chapter 6 (Activation Patching), causal tracing is the technique of replacing activations in a "corrupted" forward pass with activations from a "clean" forward pass to identify which components are causally necessary for correct behavior.

The procedure for circuit discovery typically works as follows. You run the model on a clean input (one that produces the correct behavior) and a corrupted input (one where the behavior breaks, such as by replacing a key name with an unrelated word). Then you systematically restore activations from the clean run into the corrupted run, one component at a time, and measure how much the correct output probability recovers.

Components whose restoration causes large recovery of the correct output are causally important. By doing this systematically across all attention heads and MLP layers, you can produce a "causal importance map" showing which parts of the network matter for a specific task. This map is often sparse: for well-defined behaviors, only a small fraction of the network's components show up as important. That sparsity is precisely what makes circuit analysis informative.

The interpretation of activation patching results requires care. Patching a component from clean to corrupted (or vice versa) measures the causal effect of that component's activation, but that effect depends on all the components downstream from it. A component might appear unimportant in isolation but become important when examined in the context of its downstream consumers. Researchers therefore combine patching with component-level weight analysis to build a complete picture.

Activation patching has one important limitation: it identifies important components but does not tell you what those components compute. Two heads might both be causally important but for different reasons. Characterizing what each component does requires additional analysis of the weight matrices.

Once you have identified the important components, you analyze their weight matrices to characterize what they compute. For attention heads, this means examining the QK and OV circuits. For MLP neurons, it means studying the input weights (which patterns activate the neuron) and the output weights (what the neuron writes to the residual stream when active).

The QK circuit can be visualized by computing the attention pattern for specific types of inputs. If a head's query-key interaction consistently produces high attention weights for positions sharing some property (such as the same token, or a specific syntactic relation), that tells you the head's selection criterion. For instance, a head that attends from each pronoun to the most recent co-referent noun would show a characteristic pattern in the QK circuit: the query direction for pronoun representations points toward the direction for noun representations that share the pronoun's gender feature.

The OV circuit can be analyzed by looking at the matrix directly. If you project this matrix onto the input and output embedding spaces, you can ask: "When this head attends to a position with token X, what does it write to the residual stream?" This can be expressed as a mapping from input tokens to output logit changes.

A particularly useful projection is the "full OV circuit" mapped through the unembedding matrix. If the model's embedding matrix is (mapping tokens to residual stream vectors) and the unembedding matrix is (mapping residual stream vectors back to logit space), then the full effect of a head's OV circuit on the output distribution is captured by the product:

This product yields a matrix, where is the vocabulary size. Each entry tells you: "when this head attends to a token- position, how much does it promote token in the final output logits?" If the matrix is approximately diagonal (entry dominates row ), the head implements a copy operation. Off-diagonal structure indicates a more complex transformation, such as mapping attended tokens to related but different output tokens (for example, a head that reads a country name and promotes its capital city in the output logits).

Weight analysis is most tractable in small models, because the weight matrices are small enough to visualize and reason about directly. In larger models, researchers use dimensionality reduction, sparse approximations, or targeted projections to identify the key directions.

Path patching is a refinement of activation patching that lets you study individual edges rather than individual nodes. Standard activation patching asks "does this component matter?" Path patching asks "does information flowing from component A to component B matter?"

To path-patch from component A to component B, you corrupt the edge between them specifically: run the corrupted input normally through all layers, but replace only the input to B with the clean-run value of A's output. If this specific edge is critical, the target behavior recovers. If not, the edge is not part of the circuit even if both components individually are.

Path patching enables more precise circuit characterization. You can confirm that "heads A and B are both important" and that "A feeds directly into B via the residual stream in a way that is causally necessary." When combined with weight analysis, path patching lets you build a full circuit description: here are the nodes, here are the edges, and here is the weight-level explanation for each edge's function. That complete description is what distinguishes a mechanistic understanding from a mere list of important components.

The full workflow of behavioral isolation, activation patching, weight analysis, and path patching forms a convergent methodology. Each step builds on the previous: behavioral isolation identifies what to study; activation patching narrows the field to important components; weight analysis characterizes what those components do; and path patching verifies the edges connecting them. No single step suffices, but together they allow researchers to make strong, empirically grounded claims about specific circuits.

The most celebrated circuit discovered through mechanistic interpretability is the induction head circuit. Induction heads are attention heads that implement "pattern completion" based on previous context. Understanding them provides a concrete demonstration of how circuit-level analysis works, and the story of their discovery illustrates the scientific method applied to neural network internals.

An induction head performs the following computation: given a sequence like [A][B]...[A], when the model sees the second occurrence of [A], it should predict [B] as the next token. This is because [B] followed [A] earlier in the context.

More precisely, if the model has seen token A followed by token B somewhere in its context window, and then sees token A again, an induction head will attend back to the first occurrence of A and copy what came after it (B) to the current position's representation. This enables in-context learning of simple patterns. The capability is not limited to exact token repetition; it generalizes to bigram patterns and, in stronger forms, to longer subsequences.

Induction heads are interesting as a mechanistic curiosity and as a potential substrate for in-context learning more broadly. In-context learning is the ability of large models to solve new tasks from a few examples in the prompt, without updating any weights. Olsson et al. (2022) argued that induction heads are a core mechanism enabling in-context learning: by detecting repeated patterns and completing them, the model can implicitly follow the "input-output" examples shown in the prompt.

The discovery of induction heads by Olsson et al. (2022) was significant because it showed that:

1. A specific, identifiable circuit implements in-context learning behavior
2. This circuit appears in virtually every transformer trained on natural language
3. The circuit forms during a specific phase of training (the "induction bump")
4. Ablating induction heads dramatically reduces in-context learning performance

The induction head circuit consists of two attention heads working together:

1. Previous Token Head: An early-layer head (often layer 0) that attends to the token at position from position . It copies information about each token to the next position in the sequence.

2. Induction Head: A later-layer head that uses the information written by the previous token head to attend back to the position just before any repetition of the current token.

Let's trace through this step by step. Suppose the sequence contains ...[A][B]...[A][?]... and we're computing what the model predicts at position [?].

At position [B] in the early part of the sequence, the previous token head runs and copies information about [A] into [B]'s residual stream. Now [B]'s representation contains a blend of its own embedding and some information about [A].

At the second occurrence of [A], the induction head's query matrix processes [A]'s current representation. The key matrix processes all earlier positions' representations. Because [B]'s representation now contains [A] information (thanks to the previous token head), the induction head produces high attention from the current [A] position to the earlier [B] position.

The OV circuit of the induction head then copies [B]'s content to the current position, which causes the model to predict [B] (or whatever came after the previous [A]).

The two-head structure reflects a functional division of labor. The previous token head solves a sub-problem: "enrich each position's representation with information about its predecessor." The induction head solves a different sub-problem: "given my current token, find the position in context that followed this same token, and copy its content." Neither head alone can solve the full induction problem. The previous token head does not know which positions will later be queried. The induction head, without the enriched keys provided by the previous token head, has no way to locate the position following the first occurrence of the current token. It is their composition, mediated by the residual stream, that implements the full pattern-completion algorithm.

Let's formalize this. Let denote the embedding of token at position , and let denote the residual stream at position after layer 1 (which contains the previous token head).

The previous token head's OV circuit writes an approximation of to position . So after the previous token head runs:

where:

• : the embedding of the token at position
• : the embedding of the token at position
• : the mixing coefficient determined by the head's attention weight and OV circuit

Now, in the later layer, the induction head's QK circuit computes attention scores between the current position (the second [A]) and all earlier positions. The query for position is derived from (the embedding of [A]). The key for earlier position (the earlier [B]) is derived from .

The attention score between position and position is:

where:

• : the query weight matrix of the induction head
• : the key weight matrix of the induction head
• : the key dimension (for normalization)

When the QK circuit is tuned such that (the head strongly prefers matching the embedded query token to the same token in the key), this score becomes large for position (the [B] after the first [A]). This causes the induction head to attend to position and copy [B]'s representation forward, predicting [B] at position .

The elegance of this explanation is that it completely accounts for the induction head's behavior in terms of two simple weight interactions. No appeal to "the model knows how to continue patterns" is needed; the specific matrix multiplications implement the pattern-matching directly. This is what distinguishes a mechanistic explanation from a high-level behavioral description. The behavioral description ("the model completes patterns") is true but uninformative. The mechanistic description traces the exact sequence of weight operations that produce the behavior, making the claim verifiable and the mechanism transparent.

Evidence for the induction head story comes from training dynamics. When you train a small two-layer transformer from scratch and track its loss on a random token sequence (where the only structure is repeated tokens), the loss curve shows a characteristic sharp drop around a few thousand steps. This has been called the "induction bump."

Simultaneously, the previous token head and the induction head form during this phase. Before the bump, neither head shows induction-like behavior. After the bump, both heads are functional and the circuit is complete.

This phase transition in behavior corresponds to a discrete circuit forming, rather than gradual improvement across all components. It is consistent with the hypothesis that induction heads represent a coherent computational unit that is learned as a package rather than incrementally. The model does not first learn a partial induction ability and then improve it; rather, it relies on a suboptimal strategy for thousands of steps, then abruptly switches to the induction circuit strategy.

The induction bump has interesting implications for understanding how neural networks learn algorithms. The discrete transition suggests that the relevant algorithmic solution has a minimum viable complexity: there is no gradient-directed path from "no induction" to "partial induction" because partial induction does not reduce loss. The model must solve the two-head coordination problem jointly. This contrasts with how we might naively expect gradient descent to work, smoothly interpolating between solutions. The reality appears to be that some algorithms are all-or-nothing, and learning them requires a kind of combinatorial search that happens to succeed at a particular point in training.

It also raises questions about what comes before the induction bump. The model must be doing something during the thousands of steps before the circuit forms. Researchers have found that before the induction bump, the model relies more heavily on direct unigram statistics (predicting the most common token regardless of context). After the bump, in-context bigram patterns override those statistical priors when they are available. The circuit formation marks a discrete change in computational strategy rather than a smooth parameter update.

The induction head is the most thoroughly characterized circuit, but mechanistic interpretability researchers have identified many other circuits and mechanisms. Each new discovery adds to a growing library of computational primitives that transformers use to process language.

Hanna et al. (2023) identified the circuit responsible for "greater-than" comparisons in GPT-2. When the model processes text like "The war lasted from 1943 to 19...", it correctly predicts digits that are greater than 43. The researchers identified a small set of attention heads and MLP neurons that implement this comparison.

The circuit works through a combination of positional encoding (which allows heads to detect relative positions of digits) and MLP neurons that implement threshold functions on numerical magnitude representations. The discovery is notable because it shows that something as abstract as numerical comparison is implemented by specific, identifiable components rather than being diffusely distributed across the network.

What makes the greater-than circuit particularly interesting is the role of the MLP layers. While the attention heads handle positional and contextual processing, the actual comparison logic lives in MLP neurons that function as threshold detectors. These neurons have input weights that respond to numerical magnitude and output weights that selectively promote or suppress the relevant digit tokens. This provides direct evidence that MLP layers actively implement logical operations rather than merely smoothing or transforming signals.

The circuit analysis also revealed why the model sometimes fails at greater-than comparisons. Failure modes trace back to specific circuit components: when the threshold neurons in the relevant MLP layers receive input from positions that lack clear magnitude information (as happens with ambiguous or unusual number formats), the comparison fails. This failure mode prediction is what distinguishes a mechanistic explanation from a merely correlational one.

Wang et al. (2022) at Anthropic performed a circuit analysis for indirect object identification (the "John and Mary" task described earlier). They identified 26 attention heads forming a circuit with distinct functional roles:

• Name mover heads: Copy the correct name (the indirect object) to the final position
• Backup name mover heads: Alternative pathways that activate when the primary movers fail
• Negative name mover heads: Suppress the incorrect name (the subject, not the object)
• S-inhibition heads: Inhibit the subject from being copied
• Induction heads: Participate in recognizing repeated tokens in the context
• Duplicate token heads: Identify which tokens appear multiple times (relevant to finding the non-subject name)

This multi-component circuit demonstrates that even a single linguistic task can decompose into a rich set of interacting mechanisms. The existence of backup pathways is particularly interesting: it suggests that transformers develop redundant implementations, which has implications for robustness and interpretability. When you ablate the primary name mover heads, model performance drops but does not collapse entirely, because the backup pathways partially compensate. This redundancy is not a design choice by the researchers; the model discovered it during training because gradient descent finds multiple parallel paths when they improve robustness.

The IOI circuit also provides a template for understanding how transformers handle co-reference. The model must decide which name in "When Mary and John went to the store, John gave a drink to" refers to the indirect object of the final clause. This requires integrating positional information (who appeared where in the sentence), semantic roles (who is the subject, who is the object), and token identity (which names are the same). The circuit analysis shows how attention heads with different specializations collectively solve this integration problem, with information flowing through the residual stream from early positional processing to later semantic-role disambiguation.

Algorithmic tasks are especially amenable to mechanistic analysis because they have precise, verifiable specifications. Researchers have studied how transformers implement:

• Modular arithmetic: Adding numbers modulo a prime, where Fourier representations appear in the weights
• Sorting algorithms: Simple sorting via attention-based comparison
• Bracket matching: Detecting balanced bracket sequences using running sum representations

The modular arithmetic work by Nanda et al. (2023) is particularly instructive. When a transformer learns to add numbers modulo a prime , it discovers an unexpected algorithm: it represents numbers using Fourier modes (sine and cosine components at frequencies that are multiples of ). The multiplication theorem for Fourier series then implements modular addition almost automatically. This result was unexpected: no one designed the network to use Fourier representations; the network found them because they are the natural solution to the problem.

The key observation is that modular addition satisfies a periodicity condition. For any prime and any integer , we have . This periodicity is naturally captured by Fourier modes: a representation using frequencies for various integers automatically respects the modular structure. The network, through gradient descent, discovered this mathematical relationship and implemented it. The circuit analysis confirmed it by extracting the Fourier components from the model's weight matrices and verifying that they match the theoretically predicted frequencies.

This kind of discovery illustrates what makes mechanistic interpretability scientifically valuable. The goal extends beyond "can we explain this prediction" to "what general computational strategies do neural networks develop?" The Fourier-mode finding suggests that networks converge on principled mathematical solutions when those solutions are efficient, not on arbitrary numerical coincidences.

These "toy" algorithmic tasks are not merely academic. The representations found in them (Fourier modes, running counters, positional encodings) appear to be general computational primitives that also appear in language tasks. Understanding how the model implements a clean algorithm provides tools for recognizing similar structures in messier, more realistic settings.

The bracket matching work found that the model maintains a running sum in its residual stream: each token position carries information about the "balance" of the bracket sequence seen so far, much like a stack counter in a traditional parser. This representation appears not because the model was told to implement a stack, but because the stack is the optimal solution for tracking bracket balance. The mechanistic analysis verified this by showing that specific MLP neurons implement increment and decrement operations on this balance representation.

The progression from toy models to realistic applications follows a clear logic. In a toy model with a fully characterized circuit, you can verify the circuit is complete: every computation in the task is accounted for. This verified understanding then provides hypotheses to test in larger models. Circuits that appear in both toy models and realistic models are likely to reflect deep structural properties of transformer architectures rather than idiosyncrasies of specific training runs.

A complication for mechanistic interpretability is that neural network representations are often not clean. The "one neuron, one concept" hypothesis is false in general. A single neuron frequently responds to multiple unrelated concepts, a phenomenon called polysemanticity.

The explanation for polysemanticity connects to the concept of superposition, which we covered in depth in the Sparse Autoencoders chapter. In brief: a layer with dimensions can represent more than distinct features if those features are represented as nearly-orthogonal vectors rather than axis-aligned directions. The model exploits the fact that most features are rare (they appear in only a fraction of tokens), so the interference between overlapping feature representations is usually small.

Superposition creates a fundamental challenge for mechanistic interpretability because the natural unit of analysis (the neuron or the attention head) does not align with the natural unit of meaning (the feature). A circuit for a specific task may be better described in terms of feature directions in the residual stream rather than individual neurons. When two unrelated concepts share a neuron, ablating that neuron disrupts both computations at once, making it difficult to attribute the effect to either concept specifically.

The Sparse Autoencoder approach (Chapter 8) attempts to recover the feature directions, giving a more interpretable basis for circuit analysis. When circuits are described in terms of SAE features rather than raw neurons, the explanations become cleaner and more meaningful. Instead of "this neuron activates for concept X," we can say "this SAE feature direction encodes concept X," and circuits can be expressed as operations over these feature directions. The integration of SAE-based feature analysis with circuit analysis is an active research direction, and several teams have begun mapping circuits in terms of SAE features rather than raw neurons.

The degree of superposition varies across the network. Early layers tend to have more superposition (they must represent many low-level features in a compact space), while later layers, which need to make finer distinctions, sometimes have sparser representations. MLP neurons also tend to be more polysemantic than attention heads, because MLP neurons interact with the residual stream through fixed weight vectors while attention heads can dynamically route information based on context.

Underlying much of mechanistic interpretability is an empirical observation called the linear representation hypothesis: that concepts in neural networks are represented as linear directions in the activation space, and that the model implements computations by performing linear operations on these directions.

Evidence for the linear representation hypothesis includes:

• Word2Vec arithmetic: The classic observation that king - man + woman approximately equals queen in word embedding space, which works because the "royalty" direction and the "gender" direction are approximately linear.
• Representation steering: Interventions that add a linear vector to activations can predictably change model behavior (covered in the Activation Steering chapter).
• Probing classifiers: Linear probes (from Chapter 4) can decode many properties from neural representations with high accuracy, suggesting those properties are linearly encoded.

The hypothesis is more than an observation about embeddings; it extends to intermediate representations throughout the network. Researchers have found linear directions encoding grammatical number, verb tense, sentiment, factual attributes (like "is this a city in France"), and even more abstract properties. When these directions are identified (through probing or SAE decomposition), they can be used to construct mechanistic explanations of model behavior.

If representations are linear, then circuits can be described as matrix operations on feature vectors, which makes the algebra tractable. The residual stream decomposition above relies on this: if attention head outputs add linearly, and if features are encoded linearly, then the composition of circuits is also approximately linear. This linearity assumption enables the QK and OV circuit analysis described earlier, because those analyses express the head's computation as matrix multiplication, which is inherently linear.

The linear representation hypothesis has important caveats. It holds approximately, not exactly. Some computations in neural networks are nonlinear (the MLP layers implement nonlinear functions). And the hypothesis is harder to verify for higher-level, abstract concepts than for simple lexical properties. The direction encoding "is this token a number?" is easier to find and use than a direction encoding "is this sentence expressing irony?" partly because irony involves complex contextual dependencies that may not reduce to a single direction.

A subtler issue is that even when representations are linear, the computations performed by the model may not be. A circuit that reads a linear feature direction and then passes it through an MLP may produce a nonlinear function of that feature. The linear representation hypothesis enables the first step (reading the feature) but does not guarantee that the full computation is linear. Researchers increasingly combine linear representation analysis with function-level analysis of MLP neurons to handle this complexity.

Let us implement a demonstration of induction head behavior. We will build a minimal two-layer transformer, train it on a task that requires in-context pattern completion, and then verify that induction-head-like behavior emerges in the attention patterns. This implementation mirrors the analytical setup used by Olsson et al. (2022) in their original induction head study, scaled down for clarity.

Creating Induction Training DataLink Copied

TrainingLink Copied

Inspecting Attention PatternsLink Copied

Visualizing Attention PatternsLink Copied

One head in layer 0 should specialize as the previous token head: it should consistently attend from position to position . This is the sub-diagonal pattern in the attention matrix. A head that implements this pattern reliably across all positions is providing the "ingredient enrichment" that the induction head in layer 1 depends on.

Finally, we probe the OV circuit of the putative induction head to verify it acts as a copy circuit. A copy circuit writes a vector to the residual stream that promotes the same token it attended to, effectively forwarding the attended token's identity to the output. The full OV circuit matrix should look approximately diagonal if the head is truly copying.

We visualize the full OV effect matrix directly. A strong copy circuit should look like a near-diagonal matrix: each token (column) should most strongly promote itself (the matching row). Off-diagonal patterns, where one token promotes a different token, would suggest a more complex transformation rather than pure copying.

Mechanistic interpretability has attracted significant interest as a scientific project and as a potential contribution to AI safety. The argument is straightforward: if we understand exactly what computations a model performs and why, we are better positioned to ensure those computations are aligned with human values and free from dangerous failure modes.

One line of application involves deceptive alignment. A model might learn to behave well during training and evaluation while pursuing a different objective during deployment. If such a model exists, its internal computation would look different from an aligned model even when both produce the same outputs. Mechanistic interpretability could, in principle, detect this discrepancy by examining the circuits responsible for the model's responses and checking whether they implement the intended computation or a different one that happens to produce matching outputs.

Another application involves feature auditing. If we can decompose model representations into interpretable features using SAEs, we can audit those features for the presence of undesirable representations, such as features encoding harmful stereotypes, deceptive reasoning patterns, or adversarial manipulation strategies. Finding such features in the model's residual stream does not prove they will cause harm, but it provides a basis for targeted investigation and remediation.

The connection between mechanistic interpretability and the SAE approach from the previous chapter is particularly important here. Raw neuron-level analysis is too noisy and polysemantic to support reliable auditing. But SAE features, which represent cleaner and more semantically coherent concepts, could support feature-level auditing at scale. Researchers are already developing tools to automatically label SAE features and flag features that match safety-relevant categories.

It is worth being precise about the limitations of these safety applications. Mechanistic interpretability currently operates at the level of simple behaviors in relatively small models. Scaling to safety-critical behaviors in frontier models, which are far more complex and operate over much longer sequences, remains a major open problem. The gap between "we understand induction heads in a two-layer transformer" and "we can verify the alignment of a frontier model" is enormous. Researchers in the field are frank about this gap and actively working to develop more scalable tools.

Mechanistic interpretability is a young and rapidly evolving field. Several significant challenges constrain what current methods can achieve, and understanding these challenges is important for evaluating what circuit-level findings establish.

Scale. The circuits discovered so far are primarily in small models (GPT-2 scale, two-layer models). Modern large language models have hundreds of layers and tens of thousands of attention heads. Manually tracing circuits at this scale is not feasible. Automated circuit discovery methods (using systematic activation patching at scale) are being developed, but they remain computationally expensive and produce results that are harder to verify. The tools developed for small models do not trivially generalize: the residual stream in a 70-billion-parameter model carries far more information than in a two-layer toy model, and the circuits are correspondingly more complex and intertwined.

Polysemanticity. Even with SAE features as the unit of analysis, circuits may be messy because features interact in complex, context-dependent ways. The "same" circuit may implement slightly different computations in different contexts, making clean characterizations difficult. A head that acts as a copy circuit for token identity may behave differently when the token appears in a specific syntactic context. These context-dependencies can make circuit descriptions that are accurate on average misleading for specific edge cases.

Non-linear interactions. The residual stream decomposition and the linear representation hypothesis assume that contributions add approximately linearly. But MLP layers implement nonlinear functions, and interactions between components can create effects that are not captured by studying components in isolation. Activation patching mitigates this by being causally rather than linearly oriented, but the full picture of how nonlinear interactions contribute to circuit behavior remains incomplete. When two components interact nonlinearly, the effect of patching one component depends on the state of the other, and isolating individual contributions becomes ill-defined.

Completeness vs. comprehensibility. A circuit that truly accounts for all the causal paths contributing to a behavior may be very large and complex, defeating the goal of comprehensibility. Researchers must make choices about which components to include in a circuit description, which introduces a selection bias. Different teams may find different "circuits" for the same behavior depending on their methodology, their choice of importance threshold, and which components they chose to analyze. This makes it difficult to compare circuit analyses across papers and to assess how complete any given analysis is.

Training vs. final model analysis. Most mechanistic interpretability work analyzes fully trained models. But the circuits found in a model depend on the training data, training procedure, and hyperparameters. Circuits may not be stable across models trained with the same architecture but different random seeds or data orders. This limits the generality of individual circuit findings. A circuit found in one GPT-2 instance may not appear in identical form in another, raising questions about whether circuit analysis reveals architectural truths or training-run idiosyncrasies.

Despite these limitations, mechanistic interpretability has produced concrete scientific results. The discovery of induction heads, the characterization of name mover circuits, and the identification of specific algorithmic implementations (Fourier circuits for modular arithmetic, greater-than circuits) have provided principled, verifiable accounts of specific model behaviors. These results would have been impossible under a purely behavioral view of neural networks. Each verified circuit finding advances the field's capacity to understand and ultimately predict model behavior at a computational level.

The field is also developing better tooling rapidly. Libraries like TransformerLens make it easy to apply activation patching and weight analysis to GPT-2-scale models. Automated circuit discovery methods are extending the reach of the methodology to larger models. And the integration of SAE-based feature analysis with circuit analysis is creating a richer vocabulary for describing model computations. The path forward requires both better tools and better theory, and both are actively developing.

Mechanistic interpretability aims to reverse-engineer neural networks into human-understandable algorithms by identifying circuits, which are subgraphs of the model implementing specific computations.

Key concepts from this chapter:

• Circuits consist of nodes (attention heads, MLP neurons) connected by edges through the residual stream
• The QK circuit determines where a head attends; the OV circuit determines what it copies
• Activation patching identifies causally important components by restoring clean activations into corrupted forward passes
• Path patching identifies important edges, not just nodes, enabling precise circuit characterization
• Induction heads form a two-head circuit implementing pattern completion, discovered through careful circuit analysis, and represent a potential substrate for in-context learning
• Polysemanticity and superposition complicate circuit analysis by creating non-axis-aligned feature representations that do not map cleanly to individual neurons
• The linear representation hypothesis underpins much of mechanistic interpretability, though it has important caveats for nonlinear MLP computations

Mechanistic interpretability is complementary to the SAE-based feature interpretation approach from the previous chapter. Where SAE analysis decomposes representations into interpretable features, circuit analysis traces how those features flow and interact through computation. Together, they provide tools for understanding both what neural networks represent and how they use those representations to compute outputs. The long-term goal is a complete, verifiable account of model behavior at the computational level, a goal that currently remains out of reach for frontier-scale models but is advancing with each new methodological development.

The next chapter on activation steering uses mechanistic insights to go beyond understanding: by identifying the directions in activation space that encode specific concepts, we can intervene on model behavior by directly editing those directions during inference.

QuizLink Copied