Zero-downtime updates

Now I Get It! runs on about 34 Lambda functions. Deploying new code used to mean updating them one at a time across a 30-to-60-second window in which some functions ran new code and others ran old. There was no rollback, no health check, and the CloudFront invalidation for the frontend was fire-and-forget.

A tiered sequential deploy -- non-critical functions first, then critical, then frontend -- doesn't eliminate the mixed-version window. It just moves it around. Lambda aliases do eliminate it, by collapsing the traffic switch to a single atomic operation per function.

How it works

Every Lambda in the stack gets a live alias. Every consumer -- API Gateway integrations, SQS event source mappings, the Cognito post-signup trigger, EventBridge targets -- invokes the alias, not the function directly. A deploy runs in five phases:

1. Upload new code to $LATEST. Zero traffic impact because the alias still points at the old version.
2. Publish numbered version snapshots.
3. Pre-flight health check against the new versions by qualifier.
4. Flip every alias to the new version in a single pass.
5. Post-flight health check through the live alias.

The alias flip takes about 22 seconds across the whole stack. Rollback is the same operation in reverse, targeting version numbers saved to an S3 manifest.

The CloudFormation edit

Every consumer reference had to route through the alias. That meant 29 API Gateway integrations, 31 Lambda permissions, 4 SQS event source mappings, a Cognito trigger, an EventBridge target, two Lambda-to-Lambda environment variables, and a policy ARN. A Python script did the bulk transformation; a regex backreference bug required a second pass to catch a few stragglers.

Aliases are deliberately not managed by CloudFormation. Defining alias resources in the template would have CFN reset the version to $LATEST on every stack update, fighting the deploy script's alias flip. The template hardcodes :live suffixes in ARN references; the deploy script owns alias creation and updates.

Migration chicken-and-egg

The updated template references :live aliases that must exist before the stack update runs, or API Gateway returns 500 on every request. But aliases can't exist for Lambdas that haven't been created yet, including a new health-check function added in the same deploy. A two-step migration handles this: a script creates aliases for all existing functions first, then the stack update runs. New functions get their alias during the first update-loop pass.

Five bugs caught during the test deploy

Bash 3.x on macOS. The deploy script used declare -A for an associative array. macOS ships bash 3.x, which doesn't support it. Replaced with a flat temp file, looked up with grep.

Arithmetic counter under set -e. The migration script used ((created++)) to count aliases. When created goes from 0 to 1, bash evaluates the expression as false, and set -e kills the script. Switched to created=$((created + 1)).

Stack name mismatch. The migration script used a distinct stack name for the test environment. Test and prod actually share the same stack name -- they're differentiated by AWS account.

Permission chicken-and-egg. A new Lambda's API Gateway permission referenced its :live alias, which didn't exist at stack-update time. Lambda permissions on the unqualified function don't cover alias invocations. Fix: use the unqualified reference in CloudFormation for the base permission, then have the deploy script add an alias-qualified permission after the flip with || true for idempotency.

Missing DescribeTable IAM action. The health-check Lambda calls DescribeTable to verify DynamoDB connectivity. The action wasn't in the shared role's policy. One-line template fix.

What else shipped

A health-check endpoint returning deploy version (git SHA plus timestamp), DynamoDB connectivity, and maintenance mode status. The deploy script hits it before and after the alias flip. It returns degraded with an error message if any check fails; always HTTP 200, so API Gateway doesn't mask the body.

A maintenance mode driven by an SSM parameter. Upload and confirm endpoints return 503 with a Retry-After header. Read-only endpoints keep working. The frontend checks health on page load and shows an amber banner when maintenance is on.

A pre-deploy checklist that prints stack, region, domain, branch, and SHA, and warns on uncommitted changes. Prod gets a 5-second countdown. Post-deploy verification hits health, domain root, and a library endpoint through CloudFront. Deploy version is written to SSM and the frontend config file. The deploy log is append-only.

CloudFront invalidation now blocks until completion, so the new frontend is visible the moment the deploy finishes.

The numbers

14 files changed, +2,306/-113. Ten new integration tests covering the health endpoint, the maintenance gate, CORS bypass, and degraded-state returns. 375 tests passing. Full deploy including CloudFormation, backend, and frontend takes 473 seconds, most of it waiting on the CloudFront invalidation. The mixed-version window is now measured in hundreds of milliseconds.