Week 5: Task Lifecycle & Recovery (Days 21–25)

Gmail still delivers your mail when a SMTP relay blips. Stripe still settles payments after a network timeout. Your bank’s nocturnal batch does not “forget” mid-run because one pod restarted. They share a boring superpower: every attempt is recorded, every status change is legal, every failure is classified, and retries are bounded with backoff.

A scheduler without a lifecycle is a stopwatch with amnesia. At scale, amnesia is an outage you cannot explain.

1. Execution ledger (who ran what, when, with what outcome)

2. Status state machine (illegal transitions rejected)

3. Error capture (message + stack persisted)

4. Durable exponential backoff (RETRYING + nextRetryAt)

5. Spring Retry for in-process transient slices

6. Dead letter when hope is gone

7. Build, test, demo

Before you optimize throughput, you need a flight recorder. Each attempt gets a UUID executionId, start/end timestamps, duration, node id, and either an outcome or an error+stack. Netflix-style ops dashboards are useless without this row-level truth.

Insight: Metrics tell you rates; the ledger tells you stories. When a VP asks “what happened to job X at 02:14?”, you open one row—not a haystack of logs.