System Design - How to Design a Distributed Task Scheduler

Every production system has jobs that need to run on a schedule.

On a single server, this is trivial: add a cron job. One line in crontab.

At scale — millions of jobs, thousands of servers, strict reliability requirements — cron becomes catastrophic. If the server running cron dies, the job doesn’t run. If you add more servers and run cron on all of them, every job runs twice. There’s no visibility, no retry, no audit trail.

A distributed task scheduler solves all of this. It’s one of the most common system design interview questions at senior level — and one of the most underestimated in complexity.

Functional requirements:

• Register and manage scheduled jobs (one-time and recurring)

• Execute jobs at or very close to their scheduled time

• Support cron expressions and fixed-delay intervals

• Retry failed jobs with configurable backoff

• Provide execution history and status visibility

Non-functional requirements:

• Exactly-once execution — a job must run exactly once per scheduled slot, not zero times and not twice

• Fault tolerance — scheduler nodes can die without losing job execution

• Horizontal scalability — handle millions of jobs across thousands of worker nodes

• Low latency — jobs should fire within seconds of their scheduled time

• High availability — the scheduler itself cannot be a single point of failure

Scale:

Total registered jobs:      10 million
Jobs triggered per second:  10,000 at peak
Job execution time:         seconds to hours
Worker nodes:               thousands
Scheduler nodes:            small cluster (3–10)
Acceptable trigger delay:   < 10 seconds from scheduled time

The hardest problems:

1. Exactly-once — prevent double execution when multiple nodes are running

2. Leader election — who decides which jobs fire when, without a single point of failure

3. Failure recovery — a worker dies mid-job; how does the job get rescheduled

4. Clock skew — different servers have slightly different clocks; a job scheduled for 09:00:00 might fire at 09:00:03 on one node

Before designing the solution, understand exactly why the naive approach fails:

Problem 1: Single point of failure

• Server A: runs cron → fires job at 9am ✓

• Server A crashes at 10am
  → job scheduled for 11am: never runs ✗

Problem 2: Duplicate execution

• Server A: runs cron → fires job at 9am

• Server B: also runs cron → fires same job at 9am
  → charge_subscription.py runs twice
  → customer charged twice ✗

Problem 3: No observability Cron has no built-in logging beyond stdout. No history, no retry, no alerting on failure. At scale you need to know: did the job run? How long did it take? What failed?

Problem 4: No coordination Jobs can’t depend on other jobs. No way to say “run job B only after job A completes.” No workflow.

Problem 5: Static scheduling Adding a new server means manually editing crontab on that server. No dynamic registration, no API.

20% Summer Discount