A team I worked with had a slow dashboard. The account balance endpoint hit Postgres on every load, the page felt sluggish under traffic, and the fix was obvious to everyone: add a read replica. Point the dashboard reads at the replica, keep writes on the primary, ship it. The graph got faster. Nobody filed it as a correctness change because it wasn't one. It was a performance change.
Three weeks later, support got a ticket. A user had spent their balance, the deduction committed on the primary, and for a second and a half the dashboard (reading off the replica, which hadn't caught up) still showed the old, higher number. The user saw money they no longer had. During that window someone could have made a second decision against a balance that was already gone.
Nobody decided to weaken consistency. There was no design doc that said "we accept showing users stale balances." The team added a replica to make a page faster, and in doing so they quietly traded away a guarantee they didn't know they were holding. The trade was real. The decision was never made.
Here's the position I want to argue: you do not get to pick "CA." The C-versus-A choice everyone quotes from CAP isn't a menu you order from once at architecture time. It's a behavior your system exhibits during a partition, and a different tradeoff, latency versus consistency, that your system makes on every single read when there's no partition at all. You are choosing both of these constantly. The only question is whether you're choosing them on purpose.
CAP gets quoted as "pick two of three: Consistency, Availability, Partition tolerance." That framing is where the damage starts, because it puts P on the same shelf as the other two, as if it were a property you might decline.
You can't decline it. P is partition tolerance: the system continuing to function when the network between nodes drops, delays, or reorders messages. And the network will do that. A switch reboots, a NIC flaps, an availability zone goes dark, a Kubernetes node gets cordoned mid-deploy, a GC pause makes a node look dead for 800ms. The moment your data lives on more than one machine (a replica, a second region, a cache on a different box) partitions are a thing that happens to you. Refusing to tolerate them doesn't mean they stop. It means your system corrupts or hangs when one occurs.
So the honest reading of CAP is not "pick two." It's: partitions happen, so when one does, you get to pick exactly one of C or A, and you've already picked, whether you know it or not.
Choose C (a CP system): during a partition, refuse to serve requests you can't serve correctly. The node that can't confirm it has the latest data returns an error or blocks rather than hand back a possibly-stale answer. You stay correct; you give up availability for the duration.
Choose A (an AP system): during a partition, keep answering with whatever data you have locally, even if it might be stale or might later conflict. You stay up; you give up consistency for the duration.
There is no third door where the partition politely waits for you. The replica setup above is an AP choice that nobody recognized as a choice. When the replica lagged (a tiny partition in time, if not in topology), the system happily served the stale balance instead of refusing. That was availability winning over consistency. It just won by default, in a config change labeled "performance."
Here's the part CAP leaves out, and it's the part that actually runs your life. CAP only describes what happens during a partition. Partitions are rare. What about the other 99.9% of the time, when the network is fine?
That's PACELC. Read it as: if Partition, then Availability-or-Consistency; Else, Latency-or-Consistency. The first half is just CAP. The second half, the "E," for else, is the one you decide on every read, every day, usually without noticing.
When there's no partition and everything is healthy, you are still trading consistency against latency. Every time you add a cache, you've decided that serving a possibly-stale value fast is better than fetching the authoritative value slow. Every time you read from a replica, you've decided that the replica's slightly-behind view is acceptable in exchange for taking load off the primary. Those aren't partition-time decisions. They're the normal, healthy-network state of your system.
The dashboard team thought they were operating in the "E" branch: no partition, just trading a little latency for a faster page. And most of the time they were, and it was fine. But they never specified the "P" branch. They never said what the system should do when the replica fell behind. So the system did the default AP thing, serving stale, at exactly the moment correctness mattered. The everyday latency tradeoff and the rare partition tradeoff are the same architectural seam, and they'd only reasoned about one side of it.
A cache and a read replica are not performance features. They are consistency tradeoffs that happen to make things faster. The speed is the part you notice. The traded-away guarantee is the part that pages you at 2am.
Now drop an AI agent into this. Ask it to "make this endpoint scale" or "this query is slow, speed it up," and watch what it does.
It will, cheerfully and competently, add a cache. Or suggest reading from a replica. Or memoize the result. The code will be clean. It'll wire up Redis with a sensible TTL, or flip the read to a replica connection, and it'll compile and pass your tests, because your tests assert on a single-node happy path where the cache and the primary always agree.
What it will not do is tell you it just changed your correctness model. There's no line in the diff that says "heads up: this endpoint is now AP. Under replica lag or a cache that's mid-invalidation, it will serve stale data, and for a balance check that's a correctness bug." The agent optimized the metric you named (latency) and silently spent a budget you didn't name (consistency). This is exactly the silent logic drift failure mode: it did the task you asked and quietly changed a guarantee you didn't, in a diff that reads like a reasonable optimization. The agent isn't wrong that a cache makes it faster. It's wrong by omission about what that costs.
This is the natural objection, and it deserves a real answer: if AP is so dangerous, just be CP everywhere. Strong consistency, no caches, no replicas for reads, always correct. Safe, right?
No, and not because it's slow, though it is. It's because "CP everywhere" is also a choice with a cost you have to actually want. A strictly consistent system gives up availability during partitions by design: when the network splits, the minority side stops serving rather than risk divergence. If you genuinely make everything CP, you're signing up for the endpoint to return errors during every network blip, every failover, every node that's briefly unreachable. For a payment authorization, that's correct and worth it. You'd rather decline than double-spend. For a "number of likes" counter or a recommendations widget, refusing to serve during a blip is absurd; stale-but-up is obviously right.
The point isn't that CP beats AP or the reverse. It's that the right answer is per-feature, and it has to be chosen. A balance check wants CP. A view counter wants AP. The dashboard's bug wasn't that they picked AP. It's that a balance, which wanted CP, got AP by accident because nobody made the call. "Strong everywhere" isn't safety. It's a different unmade decision with a different bill.
So the inversion is this: every read path in your system has already chosen C-or-A for partitions and L-or-C for the healthy case. The cache you added, the replica you read from, the strict primary you kept — each is a stance. You're not deciding whether to make these tradeoffs. You're only deciding whether they're written down or discovered in an incident channel.
Before you let an agent "make it scale," make the call yourself. Here's a starter prompt that forces the agent to surface the tradeoff instead of burying it in a TTL:
For the feature below, classify it as CP or AP and justify the choice in one sentence.
Then answer, concretely:
1. PARTITION BEHAVIOR: When the network partitions (e.g. a read
replica lags, the cache and source disagree, or a node is
unreachable), exactly what should this feature do — serve
possibly-stale data, return an error, block, or something else?
State the actual behavior, not the principle.
2. PACELC / ELSE BEHAVIOR: When there is NO partition and everything
is healthy, what latency-vs-consistency tradeoff does this feature
make on a normal read (e.g. read from a cache/replica for speed, or
always hit the authoritative source)? Name it and justify it.
Do not propose an implementation yet. First commit to the behavior.
Feature: <paste your endpoint / feature description here>What to verify: check that it actually committed to a behavior during a partition, a concrete "serve stale" or "return 503" or "block until caught up," not just a tidy definition of CP and AP. If it defined the terms and dodged the behavior, it dodged the only part that matters. Make it answer "what does this do when the replica lags," in those words.
That's the theory and a way to force the decision into the open. The production patterns that actually implement each side of these choices (idempotency keys so a retried write is safe, retries with backoff and jitter, sagas for the transactions that cross a service boundary, distributed locks with fencing tokens, caching with stampede protection) are the paid series, Tuesdays and Thursdays. Each one comes with the prompt to generate it and the checklist to verify what the agent hands you against the failure modes that pattern hides. This post is F1: it's the lens the whole series looks through, because every one of those patterns is a deliberate answer to a C-or-A, L-or-C question you'd otherwise answer by accident.
Subscribe free for the Friday theory. Upgrade when you want the implementations and the verification checklists.
And tell me in the comments: what's a "performance optimization" in your system that quietly changed what your users are allowed to see — and who found out first, you or them?
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.