Day 47: Keypair Management — Moving Beyond Password Databases

Here’s the lesson every junior engineer learns wrong: “authentication is a solved problem, just plug in Keycloak or Auth0’s SDK.” So they wire up a REST call — POST /v1/verify — that hits a centralized auth service backed by Postgres, storing bcrypt or argon2 hashes in a users table. Every request that touches your edge (an XDP hook, a sidecar, a gateway) triggers a synchronous round trip to that service before the request is allowed to proceed.

This works fine in a demo. It works fine at 200 requests/sec. It falls over catastrophically once you’re running a multi-tenant WASM component mesh doing 100M+ req/s, because the entire design assumes authentication is rare and out of band. At hyperscale, authentication is neither — it’s on the hot path, and it happens on nearly every connection establishment across tens of thousands of tenants sharing the same kernel.

The abstraction hides three things from you:

• The cost of a database round trip under contention.

• The cost of the hashing algorithm itself, deliberately tuned to be slow (that’s the whole point of bcrypt) — which is a security feature for password storage and an operational liability when it’s on your request path.

• The cost of context switching every time a per-tenant process or thread has to wake up to do the check.

None of this is visible in the framework’s happy-path documentation. It’s only visible once you’ve paged someone at 3 AM because pg_stat_activity shows 4,000 connections in waiting state.

Let’s quantify it instead of hand-waving.

Connection pool exhaustion. A standard Postgres deployment tuned for OLTP workloads caps out around 100–500 usable connections before context-switch overhead in the backend itself dominates. If your auth check requires a DB round trip and you’re fielding 50,000 new connections/sec across a tenant fleet, you need a connection available roughly every 20 microseconds. You don’t have one. Requests queue, p99 latency goes from 2ms to 400ms, and it cascades: retries pile onto an already-saturated pool.

Scheduler thrashing. The “process-per-tenant” or “thread-per-tenant” pattern for isolation means the CFS (Completely Fair Scheduler) is juggling thousands of runnable tasks, most of which are blocked on I/O (that DB call) for a few milliseconds at a time. Each wake-up is a context switch: save registers, flush the pipeline, reload the memory management unit state. On x86_64 that’s roughly 1,500–3,000 cycles of pure overhead before your auth logic even runs, and every context switch invalidates a chunk of the TLB (Translation Lookaside Buffer), forcing a page-table walk on the next few memory accesses. At 50k auth checks/sec with thread-per-tenant isolation, you can lose double-digit percentages of a core’s total cycles to nothing but TLB shootdowns and scheduler bookkeeping — before a single byte of actual verification work happens.

Hashing CPU burn. bcrypt with a cost factor of 12 takes roughly 250ms by design on commodity hardware — that’s the point, it’s meant to resist offline brute force. But if you’re checking a bearer credential on every request instead of once at login, you’ve turned a deliberately-expensive security primitive into your primary bottleneck. This is the single most common mistake in systems that “grew” from session-cookie auth into service-to-service auth without redesigning the primitive.

The naive fix — “just cache the auth result in Redis” — buys you time but doesn’t fix the architecture. You’ve added a second network hop and a second point of failure, and you still pay full verification cost on every cache miss, which under tenant churn is constant.