building loupe, the world's worst interpretability tool, and pointing it at GPT-2's memory
Introduction
If you've ever spent a late night staring at IDA Pro/Binary Ninja or Ghidra, trying to figure out what a stripped binary does, congratulations: you already have the exact instincts for one of the most interesting open problems in computer science.
A large language model is one of the most hostile reverse engineering target we have. Billions of parameters. No comments. No function names. And the "compiler" (gradient descent) optimizes ruthlessly for behavior while caring exactly zero about legibility. It condensed out of an optimization process.
The discipline of taking these things apart is called mechanistic interpretability ("mech interp"). The working definition, borrowed from Neel Nanda, who runs interpretability at DeepMind: any approach to understanding a model that uses its internals (weights and activations) instead of just poking inputs and reading outputs. 99% of ML treats the model as a black box and makes benchmark numbers go up. Mech interp is the field that opens the box. Static analysis, dynamic analysis, hooking, patching. It's our job, aimed at an artifact no human wrote. (Before you object: yes, humans wrote the architecture, the training loop, the data pipeline. Nobody has handmade the weights. There is no source that translates to those billions of parameters; gradient descent found them.)
In this post, we'll learn the field the only way we know how: by building loupe, arguably the world's worst interpretability tool, from scratch in Go, and pointing it at GPT-2's memory. Zero dependencies. Every tensor op is a for loop you can put a print statement inside. I'm treating GPT-2 as the hello-world of transformers for this post.
I assume you have some working knowledge of Go, a machine with ~1.5 GB of free RAM, and a lot of patience for file formats and matrix multiplication.
Topics we need to understand while writing loupe:
- What a transformer's "memory" actually looks like (the residual stream)
- Why you can't just read it (superposition)
- The mech interp toolbox, and how it maps 1:1 onto our RE toolbox
- The safetensors file format
- GPT-2's byte-level BPE tokenizer
- The full GPT-2 forward pass (attention, MLPs, LayerNorm)
- The logit lens:
stringsfor activations - Ablation: NOP-ing pieces of a mind
Let's start by understanding the target.
The target: a binary with no symbols
A transformer processes text token by token. At each token position it maintains one vector called the residual stream. Think of it as a shared memory bus: every layer reads from it, computes something, and writes its result back. Early layers write low-level parsing junk. Middle layers accumulate abstract state. The final layer is basically the output buffer, from which the next-token prediction gets read off.
token in next-token out
| ^
v |
+------------------------------------------------------------------+
| RESIDUAL STREAM |
| (shared memory: one vector per token position) |
+------------------------------------------------------------------+
^ | ^ | ^ | ^ |
read | | write | | | | | |
| v | v | v | v
+--------+ +--------+ +--------+ +--------+
| attn 0 | | mlp 0 | | attn 1 | ... | mlp 11 |
+--------+ +--------+ +--------+ +--------+
no struct definitions. no symbol table. good luck.
Nearly every technique in the field is some way of reading or tampering with this bus.
Matrices in ninety seconds
If your linear algebra is rusty, here is everything this post needs, mapped to go. A vector is a []float32. That's it. The residual stream at one token position is 768 floats. The mysterious "activation" that interpretability papers agonize over has the same memory layout as an audio buffer. Geometrically it's a direction in 768-dimensional space, but in the debugger it's an array.
A dot product is the whole game. Multiply pairwise, sum:
func dot(a, b []float32) float32 {
var s float32
for i := range a { s += a[i] * b[i] }
return s
}
Read it as a similarity measure: big positive means the two vectors point the same way, near zero means unrelated, negative means opposed. It's memcmp for meaning. Every interpretability verb in this post bottoms out here: "is concept X present in this activation" is a dot product, "how strongly does the model want to say token T" is a dot product.
A matrix is a bank of vectors. Two ways to hold it, both true:
- A lookup table.
wte.weighthas shape[50257, 768]: literally an array of 50257 vectors, one per vocabulary token. "Embedding" a token is array indexing,wte.Row(id). Nothing fancier is happening. - A transformation.
y = xWmaps a 768-vector to, say, a 2304-vector. And each output element is just a dot product betweenxand one column ofW. A matrix multiply is many dot products in a trenchcoat. That's the entire body of ourlinear()function.
The model is ~500 MB of matrices plus three nonlinearities. Everything in the checkpoint (we'll dump the full inventory when we parse it) is either a lookup table (wte, wpe) or a transformation (the attention and MLP projections). The only non-matrix ingredients are softmax, GELU, and LayerNorm, and they exist for a load-bearing reason: a chain of pure matrix multiplies collapses algebraically into one matrix multiply. The nonlinearities are what stop a 12-layer model from being secretly equivalent to a 1-layer model. They're why depth means anything.
The catch: the bus is packed. Under the superposition hypothesis, the model crams way more concepts ("features") into the residual stream than it has dimensions, storing them as overlapping directions in high-dimensional space. It's as if every variable in a program shared one block of RAM, distinguishable only by which basis you project onto.
how you wish it worked how it actually works
---------------------- ---------------------
0x00 language: Spanish one 768-dim vector,
0x08 topic: cooking thousands of features as
0x10 tense: past overlapping directions,
0x18 mood: positive no addresses, no names
Sounds mystical, so let's make it arithmetic. Minimal version: 3 features in 2 dimensions. Three unit vectors at 120 degrees:
f2 = (-0.50, 0.87)
\
\
+--------- f1 = (1.00, 0.00)
/
/
f3 = (-0.50, -0.87)
Three features, two dims, no dimension "belongs" to any feature. To store "feature 1 is active", set the memory to x = f1. To read, take dot products:
x . f1 = 1.00 <- there
x . f2 = -0.50 <- interference
x . f3 = -0.50 <- interference
The signal is there, contaminated by crosstalk from the other directions. In 2D the crosstalk is terrible (-0.50), which is exactly the point: superposition is a bad deal in low dimensions and an absurdly good one in high dimensions. Two facts make it work at scale:
- High-dimensional geometry is generous. 768 dims fit only 768 exactly-orthogonal vectors, but they fit far more that are merely close to orthogonal, and the number you can pack grows exponentially with the dimension (the Johnson-Lindenstrauss family of results). Relax the requirement from perfectly orthogonal to a few percent of overlap and capacity stops being the binding constraint. Nearly-orthogonal means the bleed between any two features is a couple of percent instead of the 0.50 we just saw in 2D.
- Features are sparse. Any given token is about a handful of things, not a million things at once. Few active directions means little accumulated interference. The model trades a little crosstalk for a colossal amount of capacity.
So the memory dump at realistic scale reads like this (numbers illustrative, geometry real):
store: x = 1.0*spanish + 0.7*cooking # this token is about two things
read back with dot products:
x . spanish = 1.02 active (signal + bleed)
x . cooking = 0.69 active
x . past = 0.03 crosstalk. tolerable.
x . legalese = -0.02 crosstalk.
... thousands more features, all reading ~0.00
This is also why staring at individual neurons only gets you so far. There is a real effect pulling features onto neuron axes: the elementwise nonlinearity inside each MLP gives that layer what Chris Olah calls a privileged basis, the same way a CPU privileges byte boundaries, and a decent fraction of neurons do come out readable (about a third in a GPT-2-sized model, by blinded human scoring). But many don't. Those polysemantic neurons are representing features that simply aren't best described one axis at a time. And the residual stream, with no elementwise nonlinearity acting on it, has no privileged basis at all. So "what does neuron 47 mean" is sometimes a fair question and sometimes like asking what byte 47 of a zip file means.
Which reframes the whole job. The problem isn't reading a variable, it's working out the basis the variables are written in. Olah calls this the search for an interpretable basis, and it's the most RE-shaped idea in the field: before you can name anything, you have to figure out how to carve the memory. That's what the SAEs in the next section are for.
And yet (this is the reason the field exists) there are tantalizing signs of real, recoverable structure inside: directions that reliably mean things, attention heads with identifiable jobs, literal circuits implementing algorithms. The question is whether we can build tooling to find them.
The toolbox
Before we write code, the lay of the land. Mech interp's standard kit maps well onto ours. Olah drew the core of this table in 2022: binary to parameters, VM to architecture, program state to activations, variables to neurons and feature directions. Here it is with our tools hung off it:
| reverse engineering | mech interp | question it answers |
|---|---|---|
| stripped binary | trained weights | the artifact itself |
| shared memory / bus | residual stream | where state lives |
strings | logit lens | what's legible in memory right now? |
| signature scanning | linear probes | is concept X present in state? |
| symbol recovery | sparse autoencoders (SAEs) | what are the variables? |
| execution tracing | max-activating examples | when does this component fire? |
| function hooking | activation patching | is this state causally load-bearing? |
| live memory patching | steering vectors | can I redirect behavior from inside? |
| NOP-ing instructions | ablation | what depends on this component? |
| taint / data-flow analysis | attribution graphs | what feeds what? |
| fuzzing / dynamic analysis | black-box prompting | behavior before you attach a debugger |
A few of these deserve a sentence each:
We can say Logit lens is the field's strings, and it's what we're building on in this post. The model's final layer converts residual stream vectors into vocabulary scores via a fixed unembedding matrix. The logit lens applies that same matrix to intermediate layers and asks: if the model had to speak right now, what would it say? Crude, occasionally revealing, often garbage. It's a community technique from a researcher called nostalgebraist (2020), and it's the ancestor of fancier lenses we'll meet at the end.
Activation patching is function hooking, and it's the field's most important causal technique. Run the model on prompt A, record an internal activation, run prompt B with that activation transplanted in, watch what changes. This is what separates "I found a correlation in memory" from "I found the code path."
SAEs are automated symbol recovery: dictionary learning over the packed memory, hoping the recovered directions correspond to human concepts. You can browse recovered symbols for open models on Neuronpedia. It's like opening a binary with symbols partially restored. Noisy, contested, most of all, fascinating.
Black-box methods are fuzzing. Careful prompt variation, reading chain of thought, prefill attacks that put words in the model's mouth. Don't be a purist; often the correct first move in an investigation is just talking to the model a lot.
OK, enough theory. impl time.
loupe's architecture
loupe is going to be a single Go binary that:
- parses a GPT-2 checkpoint (safetensors) into tensors
- tokenizes a prompt (byte-level BPE)
- runs the full forward pass, snapshotting the residual stream after the embedding and after every one of the 12 blocks
- points the lens at any snapshot: apply the model's own final LayerNorm + unembedding, softmax, sort, print
prompt --> [tokenizer] --> ids --> [forward pass] --> 13 memory snapshots
| |
(ablation flags v
can NOP any [lens: ln_f + W_te^T]
block's write) |
v
top-k tokens per layer
The final command should look something like this:
$ ./loupe -model ./gpt2 -prompt "Hello world"
Zero dependencies. The whole thing is six Go files and ~1500 lines. Stdlib only, so go build and nothing else.
loupe/
|____ main.go # CLI, ranking, rendering
|____ model.go # forward pass, snapshots, ablation, unembed
|____ tokenizer.go # byte-level BPE
|____ safetensors.go # checkpoint parser
|____ neuron.go # neuron attribution, clamping, corpus scan
|____ viz.go # the fun part, later
|____ download_weights.sh
Parsing the checkpoint
Model weights ship as .safetensors. If you were expecting some eldritch format, good news, it's beautifully boring:
[8 bytes] little-endian uint64: header length N
[N bytes] JSON header: {"tensor_name": {"dtype":"F32","shape":[...],"data_offsets":[s,e]}, ...}
[rest] raw tensor bytes, offsets relative to end of header
That's it. That's the whole format. A length-prefixed JSON header and a blob.
safetensors.go
func LoadSafetensors(path string) (map[string]*Tensor, error) {
raw, err := os.ReadFile(path)
if err != nil { return nil, err }
if len(raw) < 8 {
return nil, fmt.Errorf("safetensors: file too short")
}
hlen := binary.LittleEndian.Uint64(raw[:8])
if uint64(len(raw)) < 8+hlen {
return nil, fmt.Errorf("safetensors: truncated header")
}
header := raw[8 : 8+hlen]
body := raw[8+hlen:]
var entries map[string]json.RawMessage
if err := json.Unmarshal(header, &entries); err != nil {
return nil, fmt.Errorf("safetensors: bad header json: %w", err)
}
out := make(map[string]*Tensor, len(entries))
for name, rm := range entries {
if name == "__metadata__" { continue }
var e stEntry
if err := json.Unmarshal(rm, &e); err != nil {
return nil, fmt.Errorf("safetensors: bad entry %q: %w", name, err)
}
if e.Dtype != "F32" {
return nil, fmt.Errorf("safetensors: tensor %q has dtype %s", name, e.Dtype)
}
start, end := e.DataOffsets[0], e.DataOffsets[1]
if start < 0 || end > int64(len(body)) || end < start {
return nil, fmt.Errorf("safetensors: bad offsets for %q", name)
}
b := body[start:end]
data := make([]float32, len(b)/4)
for i := range data {
data[i] = math.Float32frombits(
binary.LittleEndian.Uint32(b[i*4 : i*4+4]),
)
}
out[name] = &Tensor{Data: data, Shape: e.Shape}
}
return out, nil
}
Fifty lines and a 500 MB checkpoint is just a file like any other. The interesting tensors for gpt2-small:
wte.weight [50257, 768] token embeddings (also the unembedding, tied)
wpe.weight [1024, 768] position embeddings
h.{0..11}.ln_1.* pre-attention LayerNorm
h.{0..11}.attn.c_attn.* [768, 2304] fused QKV projection
h.{0..11}.attn.c_proj.* [768, 768] attention output projection
h.{0..11}.ln_2.* pre-MLP LayerNorm
h.{0..11}.mlp.c_fc.* [768, 3072] MLP up-projection
h.{0..11}.mlp.c_proj.* [3072, 768] MLP down-projection
ln_f.* final LayerNorm
One gotcha that will eat an afternoon if you don't know it: HuggingFace's GPT-2 uses the Conv1D convention, so linear weights are stored [in, out] and the op is y = xW + b, not the [out, in] you might expect. Transposed weights produce output that is confidently, fluently wrong, which is a fun failure mode to debug in a neural network.
The tokenizer, or: fighting RE2
GPT-2 doesn't eat bytes, it eats token ids, produced by byte-level BPE from two files: vocab.json (token string -> id) and merges.txt (ranked merge rules). The pipeline:
- split text into "words" with a regex
- map each word's raw bytes through GPT-2's weird byte<->unicode table (so arbitrary bytes become printable)
- repeatedly merge the adjacent pair with the lowest rank in
merges.txt - look up the surviving pieces in the vocab
Steps 2-4 are mechanical. Step 1 is where the porting scar is. The reference split pattern is:
's|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+
See that (?!\S)? Negative lookahead. Go's regexp is RE2, and RE2 refuses lookaheads on principle (linear-time guarantees). Every reverse engineer knows this genre of bug: the spec assumes an engine feature your platform doesn't have, and "close enough" tokenization silently corrupts everything downstream.
The fix: match \s+ greedily, then re-attach the final whitespace char of a run to the following word and re-split that fragment, which reproduces the reference behavior:
tokenizer.go
if isAllSpace(tok) && i+1 < len(raw) && !isAllSpace(raw[i+1]) && !strings.HasPrefix(raw[i+1], " ") {
head := tok[:len(tok)-1]
if head != "" { out = append(out, head) }
// re-split "last space + next word" with the same regex so edge cases
// like contractions come out identical to the reference
combined := tok[len(tok)-1:] + raw[i+1]
out = append(out, preTokenPattern.FindAllString(combined, -1)...)
i++
continue
}
Is this elegant? No. Does it match the reference tokenizer? Yes. Does it work? Probably. Moving on.
The forward pass: an instrumented emulator
This is the emulator core, and it's where writing everything by hand pays off. GPT-2's per-block recipe:
x = x + attn( ln_1(x) ) # attention sublayer writes to the bus
x = x + mlp( ln_2(x) ) # MLP sublayer writes to the bus
That's the whole transformer. LayerNorm, a causal multi-head attention, a GELU MLP, and residual additions. Every "write to the bus" is literally x[p][i] += out[i] in our code, which makes two things trivial that PyTorch makes you fight hooks for:
1. Snapshot memory at every boundary. We copy the residual stream after the embedding and after every block. Thirteen snapshots. This is an emulator that dumps memory at every instruction boundary, because we own the emulator.
2. NOP any write. Wrap each sublayer in an if:
model.go (abridged; the attention math is collapsed into helpers)
for l := 0; l < m.NLayer; l++ {
// ---- attention sublayer ----
if !ab.Attn[l] {
q, k, v := projectQKV(x, l)
for p := 0; p < seq; p++ {
merged := causalAttention(q, k, v, p)
proj := linear(merged, prjW, prjB)
for i := 0; i < m.NEmbd; i++ {
x[p][i] += proj[i]
}
}
}
// ---- MLP sublayer ----
if !ab.MLP[l] {
for p := 0; p < seq; p++ {
hidden := linear(layerNorm(x[p], ln2w, ln2b), fcW, fcB)
for i := range hidden {
hidden[i] = geluNew(hidden[i])
}
out := linear(hidden, mpW, mpB)
for i := 0; i < m.NEmbd; i++ {
x[p][i] += out[i]
}
}
}
states = append(states, copyState(x))
}
The attention helpers above stand in for the QKV and causal-attention loops in the repository; the ablation control flow and residual writes are shown verbatim.
-ablate-mlp 4 on the command line, and block 4's MLP never touches the bus. Component surgery from a flag.
The matmul is the least clever code imaginable, and that's the point:
// y = xW + b, W row-major [in, out]
func linear(x []float32, W, b *Tensor) []float32 {
y := make([]float32, W.Shape[1])
if b != nil { copy(y, b.Data) }
for i, xi := range x {
if xi == 0 { continue }
row := W.Data[i*W.Shape[1] : (i+1)*W.Shape[1]]
for o, wv := range row { y[o] += xi * wv }
}
return y
}
No BLAS, no SIMD, no GPU. Honest loops. gpt2-small is just 124M params.
The lens itself
After all that, the actual technique is almost anticlimactic:
main.go
states, err := model.Forward(ids, Ablation{
Attn: attnSet,
MLP: mlpSet,
}) // [embed, block0, ..., block11]
if err != nil {
fmt.Fprintln(os.Stderr, "loupe:", err)
os.Exit(1)
}
renderPosition(model, tok, states, ids, len(ids)-1, *k)
Inside renderPosition, the lens operation at each depth is the one-liner probs := model.Unembed(st[pos]); the rest of the function ranks and prints those probabilities.
Unembed is the model's own final LayerNorm plus a dot product against all 50257 rows of wte. We borrow the model's mouth and hold it up to its own intermediate thoughts.
Running it
$ ./download_weights.sh # ~500MBish from HuggingFace
$ go build -o loupe .
$ ./loupe -model gpt2 -prompt 'Hello world'
loupe: loading weights from gpt2 ...
loupe: 12 layers, 12 heads, d_model 768; running 2 tokens ...
-- lens at position 1 (token ' world') --
layer top tokens (p)
embed ' world' 100.0% | ' livest' 0.0% | 'world' 0.0% | ...
L00 ' world' 27.3% | ' wide' 5.0% | ' World' 2.9% | ...
L02 ' wide' 30.3% | ' world' 8.0% | ' order' 5.4% | ...
L05 ' wide' 32.7% | ',' 4.4% | 'craft' 2.4% | ...
L07 ',' 26.4% | '!' 20.0% | ' wide' 3.9% | ...
L09 '!' 49.0% | ',' 39.3% | '.' 2.3% | ...
L11 * ',' 27.5% | '.' 13.9% | '!' 9.1% | ...
It works, and even this dumb 2-token prompt is showing real structure. The embedding layer is a pure echo (100% ' world', of course, that's all it contains). Then the early layers free-associate through the token's neighborhood: ' wide', ' order', 'craft'. world-wide. world order. worldcraft. Somewhere around L06 the model stops thinking about the word "world" and starts thinking about what comes after "Hello world", and by the final layers it has converged on punctuation, which is genuinely GPT-2's best guess after a bare greeting.
That trajectory (echo, associate, predict) is the residual stream doing its job, and we just watched it in a terminal.
The neural hex dump
One position at a time is gdb. We want xxd. So viz.go renders the whole thing as a grid: rows are layers, columns are token positions, each cell is the top lens token at that (layer, position), background heat is its probability, and cells that already match the model's final prediction get marked:
Look at the last column. Paris enters the candidate set late in the network, competes with London at L09, and becomes the top prediction in the final two blocks.
And because we own the emulator, tampering is a flag away:
# NOP every MLP: attention-only GPT-2. what survives?
$ ./loupe -model gpt2 -prompt "..." -grid -ablate-mlp 0,1,2,3,4,5,6,7,8,9,10,11
# binary-search for where the Paris recall actually lives
$ ./loupe -model gpt2 -prompt "The Eiffel Tower is in the city of" -ablate-mlp 5,6,7
Do you see where this is going? Ablate components, watch which column's convergence breaks, and you're localizing where a fact lives inside a mind. With for loops.
Chasing one fact, and getting fooled
Whole sublayers are a blunt instrument. Let's go one level down and ask which individual MLP neuron is responsible for Paris.
Each MLP has 3072 neurons, and neuron j writes h[j] * W_proj[j] into the residual stream. So its pull on any token T is just a dot product: h[j] * <W_proj[j] . g, wte[T]>, where g is the final LayerNorm gain. That's direct logit attribution at neuron granularity, it's about thirty lines, and it ranks all 3072 in one forward pass.
Ranking layer 9 by pull on ' Paris', then pinning each candidate to zero to check whether the ranking meant anything:
$ ./loupe -model gpt2 -prompt "The Eiffel Tower is in the city of" \
-neuron-attrib 9 -target " Paris"
โโ neurons in L09 pushing ' Paris' at position 9 โโ
baseline P(' Paris') = 7.00%
neuron attribution activation P(target) if pinned to 0
2649 8.750 1.927 6.09% (-0.91)
1828 8.340 1.519 6.62% (-0.38)
Neuron 2649 it is. Now the fun part, because we own the emulator: pin it to a range of values and watch what the model says.
$ ./loupe -model gpt2 -prompt "The Eiffel Tower is in the city of" \
-clamp 9:2649 -topk 3
โโ clamping L09 neuron 2649 at position 9 โโ
pinned to top tokens (p)
(natural) this neuron's unmodified value here is 1.927
-40.0 ' G' 4.1% | ' E' 2.6% | ' A' 1.9%
-20.0 ' E' 3.9% | ' G' 3.1% | ' the' 2.0%
-10.0 ' E' 4.1% | ' G' 2.6% | ' the' 1.5%
0.0 ' Paris' 6.1% | ' London' 4.7% | ' New' 2.7%
10.0 ' London' 10.2% | ' Paris' 7.7% | ' Berlin' 4.8%
20.0 ' London' 13.3% | ' New' 7.3% | ' Berlin' 4.2%
40.0 ' New' 20.5% | ' London' 19.1% | ' Chicago' 4.4%
Look at that. Turn it up and Paris loses to London, then to New York. Turn it down and GPT-2 stops forming words. It's a dial, and it looks an awful lot like it controls which city the model believes in. Write the blog post, ship the demo.
Don't. There's a second question, and it's the one that separates a finding from a story: what does this neuron actually respond to? Not what happens when I move it, but what makes it move.
The cheapest version of that question needs no corpus at all. Just print the neuron's activation at every position of the prompt we already have:
$ ./loupe -model gpt2 -prompt "The Eiffel Tower is in the city of" -act 9:2649
โโ L09 neuron 2649 activation by position โโ
0 'The' 0.029
1 ' E' -0.142
2 'iff' 0.193 #
3 'el' -0.071
4 ' Tower' -0.145
5 ' is' -0.123
6 ' in' 1.296 ##########
7 ' the' -0.043
8 ' city' -0.076
9 ' of' 1.927 ###############
Two spikes, and neither is on ' Tower' or ' city'. They're on ' in' and ' of'. That is not where a fact about Paris would live. That's where a preposition lives.
So sweep a corpus and confirm it. Three megabytes of Austen, Melville, Shelley and Doyle, 780k tokens, sampling windows across the whole thing, keeping the contexts where 2649 fires hardest:
$ ./loupe -model gpt2 -scan 9:2649 -corpus gutenberg.txt -scan-max 700
โโ contexts that most activate L09 neuron 2649 โโ
3.06 ... had returned with him the day before[ from] town, and...
2.88 ... has not seized. My departure[ for] Ingolstadt...
2.75 ... story of the Arethusa fountain [near] Syracuse (...
2.65 ...enshaw, and whose residence[ is] near Hors...
2.65 ... With these I journeyed down[ to] Streatham...
2.63 ... sea-captain, this day resides[ in] the village of...
2.56 ...time to pay his respects to his friends[ in] Hertfordshire...
2.54 ...haw, and whose residence is[ near] Horsham...
2.53 ... Turk? What India to England?[ What] at last will...
2.52 ... bottled ale. He must have been born[ in] some time ...
That is not a city neuron. Almost every one of those is the token immediately before a place name, and the token itself is usually a locative preposition. Across the top 25 hits the peak token is in six times, from and near three times each, to, for and of twice each, and a comma three times, always a comma sitting between a place name and what follows it. The neuron isn't firing on London. It's firing on the empty slot where a location is about to go.
It is not perfectly clean, and I'm showing you the rows in order rather than the ones that flatter me. Rank nine is ' What', from Melville's "What India to England? What at last will..." Rank ten is ' in' but the continuation is "some time", which is temporal rather than locative. Two of the top 25 are outright misses and a couple more are arguable. That is what a real feature looks like at neuron granularity: a strong tendency with a ragged edge, not a clean switch.
One control settles it. Run -act over a few more prompts and read the last position, feeding it the same token in a non-locative frame:
$ ./loupe -model gpt2 -act 9:2649 -prompt ... # final position
"The Eiffel Tower is in the city of" 2649 = 1.927
"She was born in" 2649 = 2.483
"We travelled from" 2649 = 2.837
"The capital of France is" 2649 = 1.223
"I would like a cup of" 2649 = -0.078 <- same ' of'
"The reliability of Wikipedia is very" 2649 = -0.062
"He was thinking about" 2649 = -0.120
Identical token, ' of', and the neuron reads 1.927 in "the city of" against -0.078 in "a cup of". It's tracking the semantic slot, not the string.
Which means the clamp experiment was telling the truth and I read it wrong. Cranking 2649 up doesn't select London. It shouts a place goes here louder, and when you turn that shout up far enough it drowns out the specific Eiffel-Tower-shaped evidence that singled out Paris, leaving the model to fall back on its generic prior for what a city is called: London, New York, Chicago. Turn it down and the "place goes here" signal disappears entirely, which is why the model stops producing words at all. Every number in that sweep is consistent with a slot detector. None of them required a city selector. I invented that part.
This is what people mean by an interpretability illusion, and it is worth sitting with, because the intervention was real, reproducible, and pointed the wrong way. In RE terms I patched a byte, watched the program take a different branch, and concluded I'd found the license check. The corpus scan is the disassembly I should have read first.
A confession on top of that one. My first scan came back saying 2649 fires on "LONDON", "CHANCERY LANE" and "Charing", which fit the city story beautifully. It was Gutenberg front matter: the publisher's address block sits in the first few pages of one of those books, and I had been scanning the first N windows rather than sampling across the corpus. I nearly published an artifact of my own sampling as a discovery. Anthropic have an incident report about almost exactly this, features that looked bomb-making-specific turning out to be an artifact of a too-narrow visualization dataset. Your tooling will hand you a beautiful wrong answer if you let it choose the data.
So why do we need a whole other model?
Neuron 2649 came out fairly legible in the end. A locative-slot detector is a thing you can name. Plenty of neurons don't do that: you scan them and get capital letters in acronyms mixed with a subordinate clause mixed with something about boats, all in the same unit. That's polysemanticity, and it's the empirical wall that superposition explains: if the model is packing more features than it has axes, most axes are going to carry pieces of several features.
Which is why you can't just read neurons off one at a time, and why the field built something else. Here's the whole idea of a sparse autoencoder in one pass, because it's simpler than its reputation.
You have 3072 neuron activations. You believe the layer represents more concepts than that, so pick a bigger number, say 16384, and give each hypothesized concept its own weight vector over the neurons. Stack those into a matrix of shape [3072, 16384], multiply your activation vector through it, and you get a 16384-long vector where each entry is meant to be the strength of one concept. Now force nearly all of those entries to zero, because a token is about a handful of things and not sixteen thousand. Then take the few survivors and map them back to 3072 neuron values with a second matrix, shape [16384, 3072]. Train the pair on one objective: the reconstruction should match the input you started with.
That's it. Two matrices and a sparsity constraint. If the reconstruction is faithful while almost everything is zeroed, the surviving directions had to be carrying the real structure, and the columns of that first matrix are your recovered symbols. It's dictionary learning, pointed at a memory dump.
The catch is the one every RE hits eventually: recovering some symbols is not recovering the symbol table. Anthropic's largest run on Claude found features for specific San Francisco neighbourhoods, while the model demonstrably knows things at much finer grain, like which streets intersect, and those did not show up. Chris Olah's name for the gap is the dark matter of interpretability, and current estimates put what we can extract at a small fraction of what's in there. You get a telescope. The bright stars resolve. Most of the mass is still inferred from how it bends everything else.
The limitation of loupe
Run -grid on anything and the first several layers read as noise. That's not the model being empty; it's our tool being dumb. The logit lens assumes the model uses final-layer coordinates at every depth, which is like assuming a register means the same thing at every point in a program's execution. Sometimes true. Mostly not.
Sitting with that failure is the best way to appreciate the tool's descendants. The tuned lens learns a per-layer correction. And this year, Anthropic published the Jacobian lens (paper), which models how a perturbation at layer L actually propagates through all remaining layers, averaged over ~1000 contexts. It recovers meaning at depths where our tool prints garbage. Tools have generations, and you appreciate generation N by writing generation one.
That paper is worth your time for the results alone. Using the refined lens, they found a small privileged subset of internal representations that behaves like the model's working memory: the model can report what's in it, load concepts into it on request, and route multi-step reasoning through it, while fluent text and grammar run automatically underneath and never touch it. Unspoken intermediate steps of a calculation show up there, in order. Patch them and the model's conclusion follows the patch. (Their spider example: "The number of legs on the animal that spins webs is" makes spider light up mid-network without ever being said; overwrite it with ant and the model answers 6 instead of 8. Function hooking on a thought.)
And the part that should make every malware analyst's ears twitch: this working memory leaks intent. Deliberately backdoored "model organisms" carried secretly and trick in their workspace on ordinary prompts, even when the visible output looked clean. One model's workspace registered fake and fictional during a staged ethics test (it had noticed the sandbox), and ablating that awareness changed its behavior for the worse. Implanted objectives leak indicators into runtime state whether or not the payload fires, and samples behave differently when they detect the sandbox. We know this movie.
Where the field is at
Quick tour of what's been found with the grown-up tooling, so you know the landmarks:
- Circuits are real. Early work reverse engineered complete algorithms out of small transformers: induction heads doing in-context copying, an identifiable subgraph of GPT-2 attention heads resolving indirect objects. The weights contain algorithms a human can read out and name.
- Features are real, and steerable. SAE work at scale found millions of interpretable features, famously including a Golden Gate Bridge feature in Claude which, clamped high, produced a model that related everything to the bridge. Silly demo, serious point: recovered symbols support live patching.
- You can trace real computation. Attribution graphs (per-prompt data-flow graphs over recovered features, closer to a single execution trace than to a static CFG) showed models planning rhymes ahead of time, doing genuine multi-hop reasoning internally, and sometimes producing chain-of-thought that doesn't match the actual internal computation. Worth knowing the hit rate before you get excited: the technique yields insight on roughly a quarter of the prompts it's pointed at.
- Interpretability can win games. Auditing games: train a model with a hidden objective, have competing teams find it, some with internals access, some without. Model organisms with planted secrets are the field's CTF challenges.
One more thing worth knowing before you dive in: the field has an ongoing argument that will feel familiar. The founding dream was ambitious reverse engineering, decompile the whole network into something human-legible. The current consensus among many senior people is that this looks close to doomed for frontier models: some human-understandable high-level structure driving the important behavior, sitting on an enormous tail of niche heuristics that resists clean description. You can understand the hot paths. You will not exhaust the binary. The response is pragmatic interpretability: probes that monitor production models, techniques that catch hidden objectives, tools that explain specific failures. Less "recover the complete source", more "triage the incident, find the implant". Nobody fully reverses an entire OS either (not required rather?), and the tooling ecosystem around partial understanding is one of the most useful/productive in our field.
Meanwhile, the target mutates
Everything above (and everything loupe assumes) is built on the vanilla transformer: one residual stream accumulating uniformly through depth, softmax attention over a KV cache, dense MLPs. Here's the thing about targets: they don't wait for your tooling to catch up.
Three days before this post, Moonshot released Kimi K3, a 2.8T-parameter open-weights model. Read its architecture notes with your reverse engineer hat on, because every headline change attacks an assumption we baked into loupe:
Attention Residuals (AttnRes). In GPT-2, block N sees earlier computation only through the accumulated sum on the bus. K3 adds a learned operation (they call it ฮฑ) that lets each block directly read the outputs of specific earlier blocks: block n-1, n-2, n-3, all the way back to the embedding. Their words: it "selectively retrieves representations across depth rather than accumulating them uniformly."
GPT-2 (what loupe assumes) Kimi K3 (AttnRes)
------------------------- ------------------------------
one bus, uniform accumulation bus + addressed reads of
older memory snapshots
[block N] [block N]
^ ^ ^---ฮฑ---[block n-1 out]
| | ^---ฮฑ---[block n-2 out]
(sum of everything below) | ^---ฮฑ---[embedding]
Sit with that for a second. loupe snapshots the residual stream after every block, from the outside, with a debugger we bolted on. K3 makes those per-block snapshots first-class objects the model itself learns to consult. The ฮฑ weights are literally a learned answer to "which old memory dump do I read right now". The architecture internalized the memory-dump operation. (Anthropic's workspace paper found that broadcast-across-depth was something vanilla transformers had to emulate with their weights; K3 gives the model native hardware for it. Speculation, but it smells like convergence.)
Kimi Delta Attention (KDA). Delta-rule linear attention: instead of softmax over a KV cache of every past position, a compressed recurrent state. "Which token attends to which token", the thing half of classic interp circuit analysis inspects, partially ceases to exist as an object. New ISA. Your disassembler doesn't decode these opcodes yet.
Stable LatentMoE, 16 of 896 experts. The "one shared RAM" picture becomes banked memory with a router as the segment selector. Superposition was already bad; now which circuit even executed depends on the input. Static analysis of one path tells you about one path. Fun detail: Nanda's guide literally advises newcomers to favor dense models because MoE is a pain for interpretability. The frontier went the other way, hard.
And the punchline: the weights are public. A 2.8T frontier-class artifact, MXFP4-quantized, sitting on the internet, and the interp tooling for KDA, AttnRes, and extreme-sparsity MoE basically does not exist yet. TransformerLens grew up on GPT-2-shaped models. Nobody has written the logit lens story for a network where "the state at layer L" is no longer the only channel to layer L+1, or figured out what activation patching even means when the patch target is a routed expert that fires for 1.8% of tokens.
If you come from security, you know this movie too: the target ships a new packer, the community's unpackers lag by a year, and the people who close that gap first get to read everything before anyone else. That gap, right now, is the widest it's been. First-mover territory for people who enjoy writing tooling for undocumented targets, which, if you've read this far, is you.
Getting into it for real
loupe is a teaching tool. For actual research, the stack is:
- ARENA: the field's crackmes. Guided exercises implementing patching, probes, and SAE analysis against real models.
- TransformerLens / nnsight: the real debuggers. Hooks on every internal activation of an open model, which is a level of access no malware analyst ever gets from a target.
- Neuronpedia: the shared lab bench. Browse SAE features, build attribution graphs in the browser.
- Nanda's guide: the canonical roadmap. His advice compresses to: learn the minimum viable basics in under a month, then learn everything else by doing throwaway 1-5 day projects. Mech interp is an empirical science; reading papers for months before touching a model is the classic mistake.
Starter projects that map well to a security background: replicate the refusal is mediated by a single direction paper (a bypass primitive discovered through principled internal analysis, you will appreciate it); attack one of the "taboo" models with a secret word trained in and extract the secret with as many techniques as you can; or extend loupe. Direct logit attribution is the cheapest win, and the tool ARENA has you build in section 1.2: the forward pass already computes each sublayer's write before it lands on the bus, so snapshot those individually and unembed each one, and you get a per-component blame table instead of a running total. Activation patching between two prompts is ~40 lines on top of the existing Forward, and it upgrades the tool from strings to function hooking.
The research culture the field's leaders push is one any good reverse engineer already runs on: every exciting result is false until you've tried hard to kill it, simple dumb hypotheses before clever ones, read your raw data, and the more thrilling the finding the more likely it's an artifact of your tooling. They even have a name for the failure mode, "interpretability illusions", which is the same disease as trusting a decompiler's pretty output without checking the disassembly.
Outro
Twenty years ago, reverse engineering meant recovering intent from machine code a human once wrote. The frontier now is recovering intent from machine code that was learned, not written. And the early returns (named circuits, recovered features, readable working memory) suggest that minds, like binaries, have structure waiting to be found by anyone patient enough to build the right lens.
Code is here: github.com/ant4g0nist/loupe
Resources
- Mechanistic Interpretability, Variables, and the Importance of Interpretable Bases - Chris Olah. The source of the binary/parameters, memory/activations, variables/features analogy this whole post runs on, and the clearest statement of why finding the right basis is the job.
- interpreting GPT: the logit lens - nostalgebraist
- A Mathematical Framework for Transformer Circuits - read it like an ISA manual
- How To Become A Mechanistic Interpretability Researcher - Neel Nanda
- Verbalizable Representations Form a Global Workspace in Language Models - Anthropic, and the readable summary
- Open Problems in Mechanistic Interpretability
- Anthropic interpretability team / Transformer Circuits thread
- Kimi K3 - the open 2.8T target waiting for its tooling
Disclaimer: I am not an AI and can make mistakes. Please double-check responses.