The Fog Is the Feature

Ask a large language model to define a chair and it won’t hesitate. I just tried it. Here’s what I got back:

A chair is a piece of furniture for one person to sit on, typically with a seat raised on legs and a back to lean against.

Clean, sharp, confident — exactly the kind of line we expect from a precision machine. And it springs a leak the moment we press. A stool has no back. A beanbag has no legs and sits flat on the floor, yet it’s sold as a chair. A tree stump we perch on does a chair’s whole job without being furniture at all. Every patch we add only invites the next exception — four legs? Except office chairs roll on one.

Notice what didn’t happen, though: the model didn’t hedge, didn’t stall, didn’t refuse. It handed over the definition with total confidence. The fuzziness wasn’t a failure of nerve. It was hiding inside the confident answer: in the plain fact that no hard-edged definition of “chair” survives contact with an actual chair. What it handed us wasn’t so much a definition as a prototype, one that shades off in various directions toward beanbags, tree stumps, and so on.

We have names for this — fuzziness, vagueness — and a settled opinion: it’s the flaw, the soft spot, the thing the next model, or the one after that, will finally engineer away. Let’s consider an alternative. The fuzziness is not a bug. It’s the engine — and it’s the same engine running our own minds.

Start with how one of these systems actually holds a concept, because it’s stranger, and simpler, than people expect. It doesn’t store “cat” as a word, or as a definition, or as a rule. It stores it as a kind of place.

The picture to hold onto is color. A color is a point in a space with three dimensions — red, green, blue — and every color we can name is a location in that space. “Reddish” is a direction; “how red” is how far we travel along it; teal sits between blue and green. Nobody loses sleep over the lack of a sharp line where blue ends and green begins. A concept in a neural network is the same kind of thing, only with thousands of dimensions instead of three: a position in a vast space, where nearness means similarity and the relationships between meanings show up as distances and directions.

Once we see concepts that way, the soft edges stop looking like defects and start looking inevitable. If a concept is like a region, and belonging to it is a matter of nearness to the cluster of clear cases, then things shade from clearly-inside, through a crowded and ambiguous middle, to clearly-outside. A sharp boundary appears only if we impose one — draw a line at some arbitrary distance and declare everything past it “not a cat.” But that line is something we add on top. The representation itself doesn’t express it. The fog is what the medium produces on its own; the crisp edge has to be painted on after the fact.

It’s even less tidy than “one concept, one neat region,” because concepts share the same underlying machinery — a single neuron (biological or artificial) helps represent many things at once, and any one thing is smeared across many neurons. There’s no particular cell we can point to and say the cat lives here. When researchers at Anthropic recently managed to extract the cleanest individual “concepts” they could from a working model, they found, among many thousands of others, a Golden Gate Bridge feature. It didn’t behave like an on/off detector. It lit up most strongly for the bridge itself, but also, more faintly, for fog over the bay, for Alcatraz, for the bridge named in other languages and shown in pictures. They had gone in with a microscope to find as sharp a concept as they could, and even that one had no clean edges.

I should say plainly what this does not mean. It does not mean the inside of these systems is mush. The representations have real, usable structure: concepts correspond to directions we can identify and even nudge the model along. The point is not that there’s no structure; it’s that the structure is built out of soft materials rather than hard ones — graded similarity rather than crisply defined symbols. Soft substrate, real structure. And in case this seems a mere quirk of engineering: cognitive scientist Peter Gärdenfors has argued that human concepts are exactly this — regions in a similarity space, with color as his model case. The geometry is not a machine artifact. It may be what thought is.

We know this isn’t the only way to try, because we tried the other way first, for decades, and it’s worth remembering how that went.

The dream was to build a mind out of crisply defined pieces: explicit symbols, logical rules, definitions with clean necessary-and-sufficient conditions. Its great monument was a project called Cyc, into which Douglas Lenat and his colleagues poured nearly forty years, hand-writing common-sense facts into a vast logical database one careful rule at a time. By the end, there were on the order of tens of millions of them.

It never reached the general, common-sense intelligence it was built for, and not because the people were short on brilliance or patience. The trouble was structural. Logic runs in closed systems, where a condition holds or it doesn’t and an inference follows or it doesn’t; but the world keeps generating cases the rules never anticipated, and a thing made of hard edges doesn’t bend when it meets them — it breaks. We can always write another rule, but we’ll never write all the ones we need, and the system stays brittle right up to the moment it shatters.

What eventually worked was the opposite move. Neural networks didn’t win by sharpening their concepts better than Cyc did. They won by giving sharpness up: by representing the world in soft, graded, overlapping similarities, the way, it turns out, brains do. The fuzziness wasn’t a compromise on the road to intelligence. It was the breakthrough.

The fuzziness everyone complains about in the machine is not a machine quirk at all. It’s the architecture of concepts as such — and we found it in ourselves first, long before there were any machines to blame.

In the 1970s the psychologist Eleanor Rosch showed that human categories show typicality: a robin strikes people as a better example of “bird” than a penguin does, and they will even confirm “a robin is a bird” faster than they confirm it of the penguin. But there’s a subtlety. That fact alone shows less than it appears to — because people will also rate 3 a “better” odd number than 447, and “odd number” has a perfectly crisp definition — 447 is exactly as odd as 3. If even a sharp-edged concept produces the same gradient, the gradient can’t be the mark of a fuzzy edge; it’s tracking something else, like how readily a familiar example comes to mind, not whether the concept itself has a boundary. So typicality on its own is not the evidence.

The real evidence is harder to wave away, and it’s something “odd number” conspicuously lacks: borderline cases. Nobody hesitates over whether 447 is odd, or changes their mind about it on a Tuesday. But ask people whether a tomato is a vegetable, a rug furniture, an olive a fruit, a hot dog a sandwich — and two things happen. They disagree with each other, and they disagree with themselves, giving different answers on different days. That instability isn’t a gap in anyone’s grasp of some definition; the concept itself has a soft edge, the same edge the Golden Gate feature had. Wittgenstein saw the shape of it without any of the apparatus: “game,” he noticed, has no strict definition, no single feature common to chess and catch and solitaire and the Olympics — only a web of overlapping resemblances. The old idea that a concept is a definition failed in psychology for precisely the reason it failed in artificial intelligence. Real concepts don’t have definitions. They have neighborhoods.

Let’s mark a limit. All of this concerns how single concepts are represented. How concepts combine into structured thoughts — what separates “the dog bit the man” from “the man bit the dog” — is a harder and still-unsettled question, and it is exactly where classical critics like the philosopher Jerry Fodor argued that similarity-based minds must run aground. That argument isn’t over. But the claim I'm making here is narrower: at the level of individual concepts, both we and the machines we have built run on graded similarity. And since the machines manage to combine those vague concepts into structured thought anyway, it must be possible somehow — even if just how, and how reliably, is part of what's still unsettled. We designed thinking machines in our own image, and got our soft edges back.

This very softness, this very imprecision, is what makes either kind of mind work.

The give in a concept is what lets a mind handle something it’s never seen before. Show it a thing it has no rule for and it can still place it: by what it resembles, by where it falls among the things it does know. It can stretch a concept to fit, carry an analogy, hear a metaphor and not choke. A crisp rule can do none of that. It can only match or fail to match. The flexibility we most prize in a mind — meeting novelty, bending without breaking, finding the loose fit that is good enough — is bought with exactly the fuzziness we complain about. Crisp, rule-bound, logical thought is not the ground floor of a mind like ours. It’s a late, hard-won, effortful achievement we build on top, the way we learn long division. The fog comes first.

It gets more interesting: the thing everyone points to as the worst flaw in these systems — hallucinations, confident fabrications — is the same faculty seen from another side. A crisp database either has the record or returns “not found.” A fuzzy similarity-space has no “not found.” Asked about a gap where there is no stored fact, it does the only thing it knows how to do: it interpolates from the neighbors and hands us something plausible. The capacity that lets it generalize gracefully into the unknown is the very capacity that lets it fabricate with a straight face. We can’t have the reach without the risk of reaching into thin air. (This isn’t the whole story of why models confabulate. Training incentives and the plain absence of a built-in fact-checker play their parts too. And a fuzzy concept is not the same thing as a false fact. But the common root is real.)

And we do it too. Our memories are not recordings we replay; they are reconstructions we assemble, freshly, each time. This is why people can confidently “remember” things that never happened, and fill gaps in half-seen events with whatever seems to fit. Same architecture, same bargain: a mind that interpolates is a mind that sometimes confabulates.

The fog is not something to be fixed. It’s the condition of meeting a complicated world with finite means — in silicon and in us alike. The dream of well-defined concepts is exactly that, a dream: not the grown-up form of thinking that fuzziness falls short of, but a narrow, specialized trick laid carefully over something softer and older.

There’s one question I’ve kept just out of frame, and I’ll leave it there for now, because it deserves its own essay. When one of these systems hesitates at the border between two concepts — when we do — is it merely unsure, groping after a fact of the matter it hasn’t yet pinned down? Or is there, at the genuine borderline, simply no fact of the matter to be had? That’s a more strictly philosophical question than this one, no less interesting for all that, but left for another day.

But the more immediate claim stands on its own, and it’s the one worth carrying away. There never were sharp edges in our concepts. And that was never a bug, it was the main feature.

• The interpretability work behind the Golden Gate feature — extracting interpretable “features” from a production model, and the short-lived “Golden Gate Claude” — is Anthropic’s Scaling Monosemanticity (2024). The gradedness I describe (a feature that fires by degree, and bleeds into associations, images, and other languages) is visible in their examples.

• Concepts as regions in a similarity space, with color as the model case: Peter Gärdenfors, Conceptual Spaces: The Geometry of Thought (MIT Press, 2000).

• That human category membership is genuinely graded — and that the borderline instability matters more than typicality alone — runs from Eleanor Rosch’s original work (1973–75) through Gregory Murphy’s The Big Book of Concepts (2002) to James Hampton’s studies tying graded membership directly to vagueness. The caution that typicality by itself proves little — because even “odd number” shows it — is Armstrong, Gleitman & Gleitman, “What some concepts might not be” (Cognition, 1983).

• The crisp-rules road and its limits: Douglas Lenat (1950–2023) and the Cyc project. Stephen Wolfram’s remembrance is a vivid account.

• On composition — the open question of whether graded, similarity-based representation can combine concepts into structured thoughts — the classic challenge is Fodor & Lepore, “The red herring and the pet fish” (Cognition, 1996); for where today’s models actually stand, see Dziri et al., “Faith and Fate: Limits of Transformers on Compositionality” (2023). It remains unsettled.

• This essay stands on its own, but it also sits beneath a longer series I’ve been writing on vagueness and the failed dream of perfect precision. If it interests you, that’s where it goes next.

Doug Smith holds a PhD in philosophy of mind and is a scholar of early Buddhism. He is the creator of Doug’s Dharma on YouTube. This essay was written in collaboration with an instance of Claude — which is, when you think about it, a good way to write an essay about the commonalities between minds that run on vagueness.