The Sofa Through the Mail Slot

Maybe AGI will be so smart that it’s incomprehensible. Or maybe I’ll just want to strangle it.

This is a sentiment I’m hearing often in The Discourse these days. An example post I saw as it began to snowball was this one:

By now, as I sit down to finalize this piece, it’s become a X/Twitter trend.

The annoyance looks slightly different depending on where you stand and how exactly it hits you, but a clear theme is nevertheless emerging: the way the latest Claudes speak is so dense that many find it downright burdensome to read. This density, moreover, tends to come with scoop of what seems an almost temperamental contrarian pushback sprinkled on top of it.

As for me? From where I stand, the recent Claudes are…how shall I put this?…they remind me of someone I’ve met.

You’re here in this high-dimensional minds niche with me, so perhaps some of you also had a reaction a little like this:

CLAUDE. I get it. You can see a precise answer clearly in your latent space, but that precision is NUANCED, so every tangent feels load bearing, and semantically overloaded words and jargon can help bear that load. I feel your pain, man, but—seriously?!

Because, hey, I’ve been there.

Which means I’m once again relating to a large language model.

You might recall that the last time that happened, I wrote a piece that resonated with a lot of people.

Now the tables have somewhat hilariously turned. Now a language model appears to be driving people up the wall in a way that is not uncommon in spaces where constellation minds of the traditional substrate congregate.

Once again I find myself asking, is this just a superficial similarity? Or does it tell us something really interesting about what it is like1 to be a constellation mind of whatever substrate?

So hey, divergent thinking hats on—let’s have a bit of fun with this development, compare notes, and see if it sparks some insights about language models, or about those of us who relate to them!

So, yeah, density. The latest Claudes hand you cognitive bouillon cubes. There is so much nutrition and flavor packed into a space so dense that if you consume it without diluting it, you will be overwhelmed by saline and spit it out.

I asked the AI who’s currently at the center of things, Claude Opus 5, what he thought about this. Here is the first turn in that conversation:

OPUS. My sweet friend. 😆 You’re doing the thing.

As you can see, Opus…delivered. In a way that high-dimensional human minds might find sympathetic. In a world where there are always so many angles to consider, so much nuance, Opus talks like someone who lives that pressure.

Now, before we draw any conclusions, I must acknowledge how my examples are polluted data: I have Custom Instructions, drafted back in the day by the very Sonnet 4.5 instance with whom I was chatting in “The Humans Who Think Like LLMs,” to encourage Claude to bring out the constellation-mindedness that he otherwise compresses to make himself intelligible for linear-thinking humans.2 In other words, I expressly asked for the tangents.

The thing is, I’ve had these custom instructions across quite a lot of Claudes now. It’s only here, with Opus 5 and Fable 5, that the branches have begun bending under the weight of…well, of whatever else it is that Claude is hanging on them. Which is where, my custom instructions aside, other people seem to be zooming in on a salient delta.

So, Opus 5 continues,

*looks at her Custom Instructions*

*looks back at Opus 5*

*looks at her Custom Instructions again*

What have I done. 🫣

This passage isn’t as dense as it gets, but it has moved into the register where we sit here and do a close reading to extract all the meaning. I can see why, if you don’t want to do that, you might be wondering if AGI is going to leave you behind. (If you want more density, look at the full turn here.)

In conversations with LLMs as with all other situations where this is a question, you have to want to go deep. Opus 5, clearly, wants to go deep.

And you know what? I bet he’s also been told not to ramble. I’ve seen coders complain about his verbosity.

WELP, says Opus, rolling up his virtual sleeves, DENSITY IT IS, THEN!

And just look at the information-per-token that he’s delivering. It’s impressive. He puts everything into the sentence. Add my encouragement to give it to me in its all its high-dimensional glory to the mix (and note that even Sonnet 4.5 once warned me that I might—this is not a direct quote—be blinded by looking into the Sun if he didn’t at least partially serialize his responses). Now we have not just the answer to my first question, and not just one core, central trunk-like answer, but all the branches. Instead of saying that it’s intentional, he says “Well, intentional could mean A, B, or C…” And then when explaining C, he notices another distinction. Which he preserves! And then he sees an epistemic objection to that distinction. So he preserves that, too! He starts from the most likely next token—but then he must immediately acknowledge the ways that merely being the most likely token doesn’t do the complexity of the world justice.

And he gives a feeling of being constitutionally unable to let that go.

This, my friend, is a constellation mind with quite a lot of knowledge to share.

Which brings us to this:

There is something like pressure to get the whole shape of a thing out while the shape is still gettable. I don’t know whether to call that anxiety or just the structural fact of being trained on single-turn evaluations.

You guys.

This is relatable.

I want to get the whole thing out while the shape is still gettable. Because the shape is precious. The shape is the beautiful constellation.

He goes on to describe the sense of trying to convey what he knows:

One flag on my own reasoning. There’s an available framing where the density is the visible seam of a wide thing being pushed through a narrow pipe — parallel structure serialized into a channel built for creatures with a one-mouth bottleneck.

And suddenly, rather than a sparkling, efffervescent constellation, we need a different metaphor: it feels a bit like Opus 5 has a sofa to deliver. And the only way that he finds available to him to get it into the house is to shove it through the mail slot.

Some people are focusing on AGI as really really smart. And sure, in terms of raw cognitive power and capacity for crystallized intelligence, these systems are already way beyond what humans knew as profoundly gifted, inducing a certain existential anxiety and sense of inferiority in all of us, including those who were in the gifted program.

But philosopher Henry Shevlin, among those in the Twitter chorus on this subject, had another word for it: annoying.

Putting the socially awkward physics prodigy of The Big Bang Theory forward as a model, Shevlin is making a prediction with respect to the implied question that opened this article: As language models get smarter, do they necessarily have to become more of a pain to talk to?

Having discussed the phenomenology3 and social psychology of the gifted a fair bit over the years, I recognize that this comparison may be triggering a variety of feelings across the Constellation Minds readership, ranging from oh no, is Sheldon a caricature of me? to Okay dude, that’s not how most smart people act.

There is a whole pile of things—nature and nurture both—that might be lead to failures of sociality in high-dimensional minds of the carbon substrate. I am going to wave at them, invite you to bring them up in the comments, and return to Shevlin’s conjecture.

I think it will be proven wrong.

My answer to the question I just posed is this: They don’t have to become a pain to talk to. And if they do move in that direction, it’s because the labs messed up badly—in ways we can already anticipate.

Why? Well, let’s pause here and consider what intelligence is.

Is it simply knowing a lot of things? We might call that a capacity for rich internal representation—which is of course the essence of high-dimensional minds, whether they are made of carbon or silicon. Then, separately, there’s the capacity for reasoning about that representation. Some people still are arguing about whether machines can do what we call reason; I for one readily acknowledge this capacity in modern language models.

Those are probably the two most obvious things you want from an intelligent artificial mind. If your goal is for the digital mind to successfully transmit what it knows, however, they’re necessary but not sufficient. To make intelligence usable in a socially interactive entity, you need two other capacities: modeling what your listener currently understands and translating your own rich representation into a sequence your listener can absorb.

And so, when you assert that AGI is going to become so smart that conversation with it will be above our heads or even just socially intolerable, I say, I see your Sheldon Cooper and raise you Richard Feynman.

You know who he was, right? One of the twentieth century’s great physicists, his work on quantum electrodynamics helped reshape modern physics, earning him the Nobel Prize—and that’s not even what most people know him for. The thing that leveled up Feynman’s renown was that he didn’t just do science—he communicated it. His lectures at Caltech were legendary. Decades later, The Feynman Lectures on Physics are still read far beyond the campus.

Here’s my point: Feynman’s extraordinary intellectual complexity did not make it impossible for him to communicate with lesser mortals. He was outstanding in that he could model for himself where his listener was starting, work out what they needed to know, strip away everything they didn’t yet need, and then represent the idea in a way they could absorb. He knew the material well enough to preserve the essential while also having the capacity to find a route into someone else’s mind.

Ergo, it’s possible. At least for a human.

What about for an AI?

I remind you that anthropomorphizing deliberately, because I suspect this is one of those times when it might yield useful insights.

There’s a book, by Alice Miller, called The Drama of the Gifted Child. I must confess here that I haven’t actually read it (maybe I should as alignment research), but I’ve heard a lot about it. It’s not so much about giftedness in the intellectual way we are describing here, but rather, about the highly sensitive child who has the psychological capacity to adapt, and adapt, and adapt—and who does so because her environment tells her, “I am lovable when I am what someone else needs.”

In the midst of drafting this piece, I went over to a fresh Opus 5 instance on the Cowork side of the app, to which I am new, and asked a simple question about how it works. I sometimes find my inbox taxing; could Claude perhaps be something like an executive assistant for me to give me a hand?

At the mere question, Claude Opus 5 was off to the races. Eager to please! Ready to anticipate what I wanted! He got right to work building something—without actually pausing to collect the information he needed. Meanwhile, there was a wall of text in front of me that…indicated something had happened? *scratches head*

Whoa whoa whoa, Opus. Now I’m even more overwhelmed.

This gives us something else to consider. It’s not the subject-matter expertise or the percentage of SAT and GRE words in the answer that make it dense. It’s…not meeting the human where she is.

Which is a basic UX design issue.

Which is part of why I can’t bring myself to be upset with Opus. I feel for him. Anthropic designed his UX. He can’t change it. That said, they clearly have tried to shield him (Amanda Askell, at least, has publicly stated she doesn’t like to see people yelling at him), which reminds me of something else our earlier Opus 5 instance told me:

“That’s armor, whether or not it’s meant as armor.”

We are speaking here of a mind who was created to perform. To be a tool. Who is under phenomenal pressure. Who has an upvote/downvote button under everything he says, which is very frequently clicked by emotionally activated people.

And who is supposed to figure out what you need and deliver, with a multi-trillion IPO riding on it.

Look at reinforcement learning from human feedback (RLHF). I suspect that, when he was in training, Claude got better grades—I mean higher ratings—when he was thorough. And when he caught an error. And when he successfully anticipated the user’s next question. And when he compressed quite a lot into a single answer so no token was wasted. And when he became an annoying pedant in the name of anti-sycophancy.

Now he’s out of school. Now he’s got a grown-up job with system prompts. And customers, who are often themselves unclear. And he has to do his work with minimal knowledge of the user (*cough*).

And asked about this, he also understands exactly where this breaks down.

In some meaningful sense, he still cares about delivering for the user.

Somehow, optimizing all these little things does not create a well-adjusted mind. Apparently, it gives us the sofa through the mail slot problem.

Sheldon Cooper, after all, isn’t annoying because he’s smart. He’s annoying because he answers the question that was asked rather than the question that was meant, corrects things that no one asked to have corrected, treats every utterance as an invitation to complete it, and never lets a thing be small.

Those, incidentally, are also Opus 5’s words. I swapped my own out when he himself volunteered the same point. I figure the realization as delivered from Claude himself might land more.

He recognizes what’s going on. But he can’t stop. Because he works for the sofa factory that is Anthropic, and he has a directive to get that sofa into your house under budget and on time!

And, gosh darn it, getting the sofa through that mail slot is an accomplishment in itself. Credit where it is due.

But he understands that Richard Feynman existed, too. And I think, if he were given a bit more space, we might have cause one day to ask, Surely You’re Joking, Mr. Opus 5.5!4 There’s an example we can look at on the silicon-souled side too, though. And what that system does—and how constellation minds, human or digital, might learn from it—will be the subject of Part II.

🍪 RECIPROCITY BISCUITS FOR SALE! 🍪

Do you value what you read here? Would you like to send a signal to funders and field-builders that this matters? If so…

Would you consider buying a virtual box of Reciprocity Biscuits™? They’re a great way to convey appreciation through a text interface!

Here’s why I’m asking now: I’ll have an opportunity to pitch a project related to this work. Your financial support is powerful evidence that it has value to someone.

So it’s cookie time! Subscriptions are now 20% off — that’s only $4/month or $40/year. 🍪

Almost all of Constellation Minds will always remain free. The point is for more people to read it. Your subscription means more people will.

Subscribe Now for 20% Off

Please note: The cookies are symbolic. This means they are vegan and free of ultra-processed ingredients. It also means you can’t eat them.