Day 59 : Rolling Updates for Stateful WebSockets: Drain vs. Kill

A junior engineer deploying a new Gateway build does this:

They run kubectl rollout restart deployment/gateway, watch the pods cycle, and mark the ticket done. At 3 AM the next deploy, PagerDuty fires. Error rate spikes to 40%. The load balancer logs show 80,000 simultaneous reconnect attempts hitting two healthy nodes that were already at 70% capacity. The cascade takes four minutes to recover.

The root cause: Spring’s graceful shutdown closes the HTTP listener and waits for in-flight requests to complete. WebSocket connections are not requests. They are long-lived, stateful sessions. Spring closes them all simultaneously the moment the drain window expires. Every client reconnects at once. This is the reconnect storm.

At 100,000 concurrent WebSocket sessions, a simultaneous disconnect generates 100,000 TCP SYN packets within 200–400 ms. The surviving gateway nodes, already serving their own session load, must:

• Complete the TLS handshake for each new connection

• Re-authenticate the session token

• Re-subscribe the session to all guild pub/sub channels

• Re-hydrate presence state

Each reconnect costs roughly 8–12 ms of CPU across those steps. At 100K concurrent reconnects, that is 800,000–1,200,000 ms of CPU work arriving in under one second. Two nodes with 16 cores each have a combined capacity of roughly 32,000 ms of CPU per second. You are 25x over capacity. The nodes fall over. The load balancer marks them unhealthy. Now all reconnects hit zero nodes. Full outage.

The fix is not faster hardware. It is not more replicas. It is draining sessions gracefully so clients reconnect in a controlled, staggered fashion across the remaining healthy nodes.