How Much of Your LLM Bill Is the Same Question Twice? I Measured It.

Here’s a question worth a line item: of all the prompts your app sends to an LLM this month, how many are the same question you already paid to answer — just worded differently?

“How do I reset my password?” and “i forgot my password how do i change it” are one question wearing two outfits. An exact-match cache — hash the string, look it up — sees two different strings and pays twice. A semantic cache asks a better question: have I already answered something that means this? If yes, return the stored answer for the price of a vector lookup instead of a model call.

The pitch is irresistible: delete a chunk of your LLM spend and latency for free. So I built one, on Postgres alone, and measured it on real data — including the part the pitch never mentions, which is what happens when “close enough” isn’t actually the same question.

The setup: one Postgres, real questions

The whole cache is one table and one index. No vector database, no Redis, no separate service — if you already run Postgres, you already have the infrastructure.

CREATE EXTENSION vector;

CREATE TABLE cache (
  id         bigserial PRIMARY KEY,
  question   text NOT NULL,
  answer     text,                       -- whatever the LLM returned last time
  embedding  vector(768) NOT NULL,       -- nomic-embed-text
  hits       int NOT NULL DEFAULT 0
);

-- the index that turns "have I answered this?" into a sub-millisecond lookup
CREATE INDEX ON cache USING hnsw (embedding vector_cosine_ops);

For traffic I needed questions with known duplicate relationships, so I could score not just whether the cache fired but whether it was right to. The Quora Question Pairs dataset is exactly that: 404,302 real question pairs, each labeled by a human as duplicate or not. I union-found the duplicate pairs into intent clusters — each cluster is one underlying question asked many ways — which gave 12,191 well-formed intents, and I used the 800 largest.

From those I synthesized a stream of 6,000 requests shaped like real support traffic: a few intents are very popular (Zipf-distributed), most requests are paraphrases of a known intent, and a quarter are genuinely novel questions the cache has never seen. To be clear about what’s real and what’s modeled: the questions and the duplicate labels are real human-annotated data; the traffic mix is my model of what a support queue looks like. Embeddings come from nomic-embed-text (768-dim) running locally.

The cache starts empty and warms as it runs — every miss calls the “LLM,” stores the answer, and the next paraphrase of that question can hit. That’s the real lifecycle, not a pre-baked index.

The baseline everyone ships first: exact match

Before vectors, the honest baseline. Normalize the string, hash it, look it up. It’s free, it’s correct by construction — an identical string is unarguably the same question — and on this traffic it caught:

27.1% of requests, with zero false hits.

That’s not nothing. More than a quarter of the stream was verbatim repeats. But three out of four repeats slipped through, because people don’t ask the same question the same way twice. Every one of those was a paraphrase an exact cache is blind to — and a model call I paid for.

The semantic cache: one query

Here’s the entire lookup. Embed the incoming question, find the nearest stored question by cosine distance, and accept it only if it’s similar enough:

SELECT id, answer, 1 - (embedding <=> %(q)s::vector) AS similarity
FROM cache
ORDER BY embedding <=> %(q)s::vector
LIMIT 1;
-- serve the cached answer iff similarity >= threshold; otherwise call the LLM and store the result

That <=> is pgvector’s cosine-distance operator, and the HNSW index makes the ORDER BY ... LIMIT 1 a sub-millisecond operation instead of a scan. The whole thing rides on one number: the threshold. Too low and unrelated questions count as matches. Too high and paraphrases slip through like they did with exact match. So the only honest way to report a semantic cache’s performance is to sweep it.

At a sensible 0.92 threshold, the semantic cache caught 39% of requests — versus 27% for exact match. Same traffic, half again as many calls deleted, just by matching meaning instead of bytes:

That gap is the paraphrases — the “i forgot my password” requests that an exact cache pays full price for and a semantic cache answers for free. If the post ended here it would be the usual semantic-cache pitch. It doesn’t, because that 0.92 was a choice, and the choice has a cost.

The number nobody benchmarks: false hits

A semantic cache can be wrong in a way an exact cache never can. When it decides two questions are “close enough” and they aren’t, it doesn’t fail loudly — it serves a confidently wrong answer from the cache and never calls the model that would have gotten it right. “How do I cancel my subscription?” answered with the steps for “How do I change my subscription?” That’s worse than a cache miss; it’s a wrong answer delivered with zero latency and full confidence.

So I scored every hit against the ground-truth labels. A hit is a false hit when the matched cache entry belongs to a different intent than the request. Here’s the whole picture — the same threshold knob, plotted for what it gives you (hit rate, in blue) and what it costs you (false-hit rate, in red):

Read it from the left. At a greedy 0.80 threshold the cache deletes a stunning 60.7% of LLM calls — and serves a wrong answer on 20.6% of its hits. One in five “savings” is a customer getting the wrong answer. Crank the threshold up and the risk falls off fast: by 0.92 the false-hit rate is 3.5%, by 0.96 it’s 1.1%, and at 0.98 it’s 0.5% — but the hit rate has decayed back toward what exact match gave you for free.

There is no free threshold. Hit rate and false-hit rate are the same dial read from two ends, and where you set it isn’t an engineering decision — it’s a product one. A cache in front of “what are your office hours?” can tolerate the occasional near-miss. A cache in front of billing or medical or legal answers cannot, and belongs up at 0.97+ where it’s barely more aggressive than exact match. The right move in practice is usually a band: above a high threshold, serve from cache; in a middle zone, call the model anyway but log the near-miss; below it, miss. The benchmark that matters is the false-hit column, and almost nobody publishes it.

What a hit actually saves

When the cache does fire correctly, the win is real and large. Answering a question two ways:

• The model: generate the answer. Measured against my local LLM, a median 3,299 ms and ~244 output tokens per answer.
• The cache: embed the question (~31 ms) and run the pgvector lookup. That index scan, measured over the run, was 0.37 ms at the median:

Limit  (actual time=0.213..0.213 rows=1 loops=1)
  ->  Index Scan using cache_embedding_hnsw on cache  (actual time=0.213..0.213 rows=1)
        Order By: (embedding <=> $1)
Execution Time: 0.245 ms

So a cache hit returns in about 31 ms against the model’s 3.3 seconds — roughly 106× faster — and the lookup itself is rounding error next to the embedding call:

The dollars are simpler than vendors make them sound. A cache hit deletes one model call, so your cost reduction is just your hit rate times your per-call cost. At the safe 0.92 threshold that’s ~39% of your generation spend gone — and the cache’s own cost is negligible, because an embedding is a thousand times cheaper than a completion (here, ~270 tokens through a tiny embedding model versus a full generation). On a frontier model at a million requests a month, deleting 39% of calls is real money; the exact figure is yours to compute, but the multiplier is “your bill × your hit rate,” and the latency win comes along for free.

Where it earns its keep — and where it bites

Caching is a correctness decision dressed as a performance one. Before you put one in front of a model, the honest checklist:

• Cache stable, shared answers; never personalized or fresh ones. “How do I export my data?” caches beautifully. “What’s my current balance?” must never be served from someone else’s hit — the answer depends on who and when, and a semantic cache only knows what. When in doubt, don’t cache.
• Isolate per tenant. In a multi-tenant app the cache key is the question and the tenant. The clean way to enforce that in Postgres is the same one I use for retrieval — row-level security, so a cache hit can never cross a tenant boundary even by accident.
• Invalidation is the hard part, as always. When the underlying answer changes — a policy update, a new price — the cached answer is now confidently stale. Tie cache entries to the source documents they were generated from and evict on change; a created_at TTL is the crude backstop.
• Watch for embedding drift. Change the embedding model and every stored vector is now in a different space than your incoming queries. Version the cache by embedding model and re-embed (or start fresh) on a swap.
• The false-hit rate is a monitoring metric, not a one-time benchmark. Real traffic drifts. Sample cache hits, spot-check them against a fresh model answer, and alert if the disagreement rate climbs. The eval discipline that catches a drifting RAG pipeline is the same one that catches a cache quietly getting too aggressive.

None of this needs a new datastore. It’s a table, an HNSW index, and a threshold you treat with the respect a product decision deserves — sitting in the database you already run.

The pattern under the pattern

This is the third time on this blog that the same boring move has won: the recommendation engine where collaborative filtering beat embeddings, the geospatial search that fused PostGIS and pgvector in one query, and now a semantic cache — all of them one Postgres instance doing a job people reflexively buy a second system for. A semantic cache is retrieval with a confidence gate, and Postgres has been good at retrieval the whole time.

The result that stuck with me is the one I didn’t expect to have to draw: that the cache’s savings and its wrongness are the same number read from opposite ends. Semantic caching isn’t free money — it’s a dial between money and risk, and the only way to set it responsibly is to measure both ends, which is exactly the column most benchmarks leave out.

I build and measure systems like this — retrieval, RAG, search, and the cost/latency layers around them — on the database teams already run, so there’s one well-understood system to operate instead of four. If that’s the problem you’re staring at, let’s talk.