The Invisible Tokens

I want to tell you two things. One is idealistic. The other is a spreadsheet. They’re connected.

The idealistic thing: right now, a builder in Kodaikanal or Kansas can access capabilities that cost Fortune 500 companies millions of dollars three years ago. The barriers between “I have an idea” and “this is in someone’s life” have never been lower.

The spreadsheet thing: when you type “Hi” to an AI assistant, 90% of the tokens being processed aren’t yours.

NVIDIA just reported Q4 FY2026 earnings: $65.7 billion in revenue, up 67% year over year. They’re selling the shovels in history’s most expensive gold rush.

Here’s the part that doesn’t show up in the earnings call: most of the people using those shovels have no idea how much gold they’re actually consuming.

AT&T revealed this week that they process 8 billion tokens daily. Eight billion. When they rebuilt their orchestration layer, they achieved 90% cost savings — which tells you exactly how much invisible overhead they’d been carrying without realizing it.

If AT&T, with its army of engineers and enterprise budgets, was drowning in invisible token costs, what does that mean for you?

A developer recently tested what happens when you build a simple Azure AI agent designed to do one thing: respond “Hi.”

That single greeting consumed approximately 2,691 tokens.

Not 2,691 total. 2,691 tokens of overhead — system prompts, memory management, plugin schemas, tracing metadata — before a single character of actual conversation. Microsoft confirmed this is “by design.”

ChatGPT reserves 750–900 tokens for system instructions on every conversation you start. When Simon Willison analyzed Claude’s system prompt, he found the search tool instructions alone consume over 6,000 tokens — most of it instructions telling the model not to regurgitate copyrighted content.

For a typical short query — the kind most of us send dozens of times a day — system overhead represents 90–97% of total input tokens.

You type fifty tokens. The system processes two thousand.

It gets worse when you have a conversation.

Every turn in a conversation resends the entire history. Turn one sends the history once. Turn two sends it twice. Turn ten sends it ten times.

A 10-turn conversation doesn’t cost 10x a single turn. It costs roughly 55x.

This is why your API bills always exceed estimates by 2–4x. This is why RAG systems — which add 2,000 to 10,000 tokens per query for retrieval context — can blow through budgets in days.

Here’s the pricing sleight of hand: output tokens cost 3–10x more than input tokens, but providers advertise input prices prominently. GPT-4o Mini’s headline price of $0.15 per million tokens becomes $1.35 per million for typical usage patterns — nine times the number you saw on the pricing page.

Only 21% of larger companies have formal AI cost-tracking systems. Most don’t even know what they’re spending, let alone why.

If you’re building on a $20/month ChatGPT subscription, you’re getting a subsidized experience. The investors burning billions are betting you’ll become dependent before they raise prices.

If you’re building with the API, your $50 budget is actually covering $120–220 worth of token consumption once you factor in system overhead, conversation history, and the input/output ratio.

If you’re building something that needs to scale, the math gets existential fast. Every user conversation carries invisible weight. Every feature that extends context carries compounding costs. Every “smart” integration — RAG, tool use, agents — multiplies the hidden tax.

This isn’t a reason to stop building. It’s a reason to build with clarity.

Here’s where the spreadsheet becomes idealistic again.

The same industry creating these invisible costs is also building ways to eliminate them:

• Prompt compression is real and it works. LLMLingua achieves up to 20x compression with only a 1.5 point drop in performance. Anthropic demonstrated reducing an agent task from 150,000 tokens to 2,000 — a 98.7% reduction. Cloudflare’s Markdown for Agents cuts usage by 80%.

• Prompt caching delivers 60–95% cost savings by recognizing when you’re sending the same system prompts repeatedly — which, given what we just covered, is constantly.

• Context management matters more than context size. Gemini 3 Pro scores 77% on recall benchmarks at 128K tokens. At 1 million tokens, it drops to 26.3%. The “Lost in the Middle” research showed accuracy degrades over 30% when relevant information sits in the middle rather than at the edges.

More context isn’t better. Better context is better.

The builders who understand this will pay a fraction of what their competitors pay for equivalent capability. That’s a moat. That’s a margin. That’s sustainability.

I’m not telling you this to create fear. I’m telling you because transparency is the foundation of every good decision.

The invisible token tax is real. So is the efficiency frontier. Both exist at the same time.

The gates are open — AI capabilities genuinely are accessible to anyone with a laptop and a vision. But there’s a toll booth, and most people can’t see it.

Go Beyond exists to make these invisible structures visible. Not to stop you from building. To help you build with your eyes open.

Next week: the $660 billion bet. Who pays when AI’s investors want their money back?

—Praveer

P.S. — This is why I built Kontinuity. The moment I saw the gap between tokens typed and tokens consumed, I couldn't unsee it. If you're interested in making that visibility automatic, it's at kontinuity.space