Week 3: Distributed Locking (Days 11–15)

You already have a catalog of schedules and a binder that turns rows into ScheduledFutures. That is a control plane. Horizontal scale breaks it the moment two JVMs share the same catalog: each node still wakes on the same cron, and both happily “reconcile invoices” at 02:00. Payment platforms call that double-settlement. Warehouse systems call it double-ship. Schedulers call it the reason locks exist.

Airflow’s executors, Kubernetes leader election for controllers, and Redis-backed queue claim patterns all solve variants of the same question: among N healthy processes, which one may enter this critical section right now? integrated lab — Aegis — makes that question tangible. You will watch unprotected ticks collide, then enforce exclusion with database leases, optimistic claims, Redis SET NX, and a Redlock quorum — one dashboard, no toy mocks.

The production insight that separates junior from senior designs is not “use Redis.” It is knowing which failure mode your lock actually covers. A Postgres SELECT FOR UPDATE and a Redis key with TTL fail differently under network partitions. Product managers feel those differences as duplicate emails; SREs feel them as pager storms; architects must name them before choosing a store.

1. The single-instance illusion after scale-out

2. Where locks sit in the Task Scheduler architecture

3. Control flow: tick → acquire → run → release

4. Data flow across RDBMS and Redis

5. Lock and task state machines

6. Four real coordination strategies (plus the “none” baseline)

7. Build, test, and demo Aegis

8. Production trade-offs and homework

Static @Scheduled methods are process-local. Persist the schedule in a database and every replica will still fire. Without a distributed lock, your audit table fills with two EXECUTED rows for the same logical tick — different instanceId, same business effect.

That is not a logging bug. It is a missing mutual-exclusion boundary around work that is not idempotent. Even idempotent handlers benefit from locks: you save CPU, cache stampedes, and load on downstream banks or SaaS APIs that rate-limit by account.