The Deployment Strategy Ladder: From Recreate to Canary

Every deploy is the same physical act: replace the code that’s running with new code. What differs — and what the words recreate, rolling, blue/green, canary, and shadow actually name — is how you make that swap. And every one of those choices is just a position on a dial between four things you can’t maximize all at once: how fast you release, how stable the system stays, how much infrastructure you pay for, and how big the blast radius is when a release is bad.

So there’s no “best” strategy, only the right rung for how much an outage costs you. The strategies form a ladder: each step up detects problems earlier and shrinks the damage, in exchange for more infrastructure and automation. Here’s the whole ladder, what each rung buys, and the one thing that can sink any of them.

Recreate: turn it off and on again

The simplest strategy is the one almost every system starts with: stop the old version, start the new one.

docker compose up -d --force-recreate
# or, even barer:
systemctl restart myapp

While the new process boots, the service is either fully down or throwing errors. If startup takes a few hundred milliseconds, nobody notices. But the moment startup stretches into tens of seconds — because the app warms a cache, runs database migrations, reconnects to queues, or waits on an external service — you have real downtime and real “who dropped the API again” messages.

For all the talk of progressive delivery and service meshes, an enormous number of services still deploy exactly this way, and that’s often fine. There’s no reason to build automated blue/green traffic switching for an internal admin panel three people open twice a week. Recreate’s whole virtue is that it needs nothing: no per-version routing, no load balancing between versions, no extra infrastructure. One process, one restart.

Rolling Update: replace instances one at a time

When downtime stops being acceptable, the next rung is the rolling update: instead of killing every instance at once, you update them a few at a time, so the ones still on the old version keep serving while the others cut over. The service stays available throughout. This is the sensible default for most services — it gives you near-zero-downtime without a second full environment (blue/green) or traffic splitting (canary).

It’s the built-in default for a Kubernetes Deployment, where maxUnavailable and maxSurge tune how aggressively pods are cycled. The same idea predates Kubernetes by years — Ansible’s serial: 1 updates hosts one at a time — and behind a plain load balancer it’s just “drain one backend, update it, health-check it, return it, repeat”:

upstream backend {
    server app1:8080;
    server app2:8080;
    server app3:8080;
}

Two things make or break a rolling deploy. The first is health checks and graceful shutdown. If the balancer starts sending traffic to an instance that’s up but not ready — hasn’t connected to the database yet — users get errors. A proper readiness endpoint returns 200 only after the app has fully started and its dependencies are live, and a graceful shutdown lets in-flight requests finish before an instance leaves rotation.

The second, more insidious one is version compatibility. During a rolling update, some instances run the old version and some the new — at the same time. That means the two versions must coexist: their APIs have to be compatible, and a database migration must not break the version that hasn’t updated yet. Which is the thread we’ll pull at the end, because it runs through the next rung too.

Blue/Green: keep the old version one switch away

Rolling solves downtime. Blue/green solves risk — specifically, how fast you can undo a bad release. Instead of mutating your running servers, you stand up two complete environments: blue is serving users now; green gets the new version and runs in parallel, taking no user traffic until it’s deployed, verified, and declared ready. Then you flip the load balancer to green.

            ┌── Blue v1   (live)                    ┌── Blue v1   (standby)
LB ─────────┤                   ──[ switch ]──►  LB ─┤
            └── Green v2  (staged)                   └── Green v2  (live)

The payoff appears at the worst moment. If green turns out broken, rollback is nearly instant — you don’t rebuild containers or wait out a rolling update, you just point traffic back at blue. That’s why blue/green has long been the favorite for systems where the cost of a bad minute dwarfs the cost of extra servers: banking, telecom, big enterprise platforms.

You pay for it three ways. You roughly double the infrastructure during the deploy (ten servers need ten more). You inherit state-sync problems if the app keeps sessions, files, or caches locally — which is why blue/green pairs best with stateless apps whose state lives in external databases, Redis, object storage, and queues. And — the trap that catches teams who think they’ve bought a risk-free release — both environments share one database. If green ships a schema migration the blue code can’t handle, flipping traffic back to blue won’t save you, because blue is now talking to a database it no longer understands. Implementation-wise it’s two HAProxy backend pools, or in Kubernetes switching a Service’s selector between two Deployments (Argo Rollouts adds a first-class BlueGreen type so you don’t hand-roll it).

Canary: let a few users test it for you

Blue/green answers “how fast can I roll back?” Canary answers a different question: how do I know the new version works before everyone sees it? Because staging is never production — only production has the real traffic, the real edge cases, the bugs that never reproduce on a test stand.

So instead of flipping all traffic at once, you send the new version a small slice — 5%, then 10%, 25%, 50%, 100% — watching at each step. If it’s healthy, you widen; if it misbehaves, you pull traffic back and stop.

        ┌── v2  (5%)
LB ─────┤
        └── v1  (95%)

The name comes from 19th-century miners who carried a canary underground: if dangerous gas built up, the bird succumbed first, buying the miners time to get out. Your first 5% of users are that bird — they hit the problem first, giving you time to react before it reaches everyone. The catch is that canary only works if you’re watching the bird: it demands real monitoring — error rate, latency, CPU and memory, new exceptions, business metrics — and ideally automated rollback when a threshold trips. A canary nobody is measuring is just a slow rollout.

You can do the crude version with load-balancer weights:

backend app
  server app_v1 10.0.0.1:8080 weight 95 check
  server app_v2 10.0.0.2:8080 weight 5  check

But the reason canary feels heavy is that doing it well wants traffic-shaping and metric analysis. A service mesh (Istio, Linkerd) sets the split declaratively and can watch Prometheus metrics to auto-roll-back; on Kubernetes, tools like Argo Rollouts and Flagger take over the gradual shift, the metric analysis, and the abort. That’s real complexity — overkill for a small service, and the safest option there is for a high-traffic one.

What partial rollouts quietly assume

Canary (and to a lesser extent rolling) only works if two things are true, and both catch teams out.

You can see what the new version is doing. A canary you don’t measure is just a slow full rollout — the whole point is to catch trouble at 5% instead of 100%. That means real telemetry on the new version, and the four signals worth watching are the classic ones: latency, traffic, errors, and saturation (CPU/memory/connections). Wire those into the rollout so the ramp pauses or reverses automatically when error rate or latency crosses a threshold; otherwise you’re relying on a human to be staring at a dashboard at the right moment.

A single user shouldn’t bounce between versions. During a partial rollout, two versions serve traffic at once, so without care the same person can hit v2 on one request and v1 on the next — and if the versions render or behave differently, that’s a confusing experience or an outright bug. Session affinity (sticky routing by cookie or user ID) keeps a given user pinned to one version for the duration, which matters most when there’s client-side state or a visible UI change. It’s the same version-skew problem as the rolling-update API-compatibility rule, just felt by one user across requests instead of across instances.

Shadow: real traffic, zero users

There’s one more rung, for special cases. Traffic mirroring (shadow deployment) sends every real request to the current version and a copy to the new version — but only the current version’s response ever reaches the user. The new version processes real production traffic while being completely invisible.

location / {
    mirror /shadow;
    proxy_pass http://production;
}
location /shadow {
    internal;
    proxy_pass http://new-version;
}

This is the safest way to validate a big change under genuine load — you rewrote a service from one language to another, or swapped the ORM, and staging looks fine but you need to know how it behaves at real scale. It’s most powerful paired with a diffing component that compares production and shadow responses and logs the divergences, catching regressions before any user is exposed.

Two hard limits, though. First, you double the load — 10,000 RPS becomes 10,000 mirrored RPS the shadow must also handle (mirror 1–10% to start). Second, and more dangerous, side effects. Mirror a POST /api/payment and the shadow might charge the card a second time, send a duplicate email, or create a duplicate order. Real shadow deployments therefore require care: disable external integrations, point at test queues and separate databases, and filter the request types that mutate the outside world. (Helpfully, Nginx doesn’t wait on the shadow backend, so a slow or dead shadow never affects the user — but that also means you tune its timeouts separately.)

The rung that isn’t a rung: your database

Notice what kept reappearing: the database breaks the neat story of every strategy above the first. Rolling update runs old and new code at once; blue/green keeps the old version one switch away — but both share a single schema, and an instant traffic rollback doesn’t roll back a migration. No deployment strategy, however sophisticated, saves you from a schema change the other version can’t tolerate.

The fix isn’t a deployment strategy at all; it’s how you migrate. Make schema changes backward-compatible using the expand-contract (a.k.a. parallel-change) pattern, in separate deploys:

1. Expand — add the new column/table, nullable or with a default. The old code ignores it; the new code can use it. Both versions are happy.
2. Migrate & deploy the code that writes/reads the new shape, while still tolerating the old.
3. Contract — only after every instance runs the new code, drop the old column.

Renaming a column becomes “add new → backfill → write both → read new → stop writing old → drop old,” never a single breaking ALTER. It’s more steps, but it’s what makes rolling and blue/green actually safe — and it’s the half of “zero-downtime deploys” that the deployment tooling can’t do for you.

Deploy is not release: feature flags

There’s a second axis that sits alongside the whole ladder and is easy to conflate with canary. Every strategy above moves traffic at the infrastructure level. Feature flags move exposure at the application level — a conditional in the code that decides, per request or per user, whether a feature is on.

That distinction lets you decouple two things people usually fuse: deploying the code and releasing the feature. You can ship the new code to 100% of servers with a plain rolling update — flag off, so it’s dark and inert — and then turn it on later: for internal users first, then 5% of customers, then everyone, all without another deploy. If something breaks, you flip the flag off instantly — a kill switch that doesn’t need a rollback or a redeploy.

It also clarifies what canary versus A/B testing really are. Canary is an infrastructure question (“is this build healthy on real traffic?”); A/B testing is a product question (“does variant B convert better than A?”) — and both are often implemented with the same flag machinery (your own, or tools like Unleash, Flagsmith, or the OpenFeature standard). In practice the strongest setups combine them: rolling or canary to get the binary out safely, and flags to control who actually sees each feature inside it.

Which rung to stand on

Lined up by risk and cost, the ladder reads cleanly — each step detects problems earlier and risks less, while demanding more infrastructure and automation:

The practical guidance: start at Rolling — it closes most of the gap for most services. Drop to Recreate when the service is internal and simple. Climb to Blue/Green when a fast rollback is worth a duplicate environment, to Canary when you need confidence on live traffic, and to Shadow for the special job of proving a big change under load before anyone touches it. And whichever rung you pick, keep your migrations backward-compatible — because the strategy controls the code, and the database has a mind of its own.