Design a Job Scheduler
Practice a distributed job scheduler with time semantics, leases, missed jobs, retries, idempotency, observability, and safe execution ownership.
What you'll learn
Run the architecture drill
- How to separate schedule evaluation from job execution.
- How leases, idempotency, and retry policy prevent duplicate or lost work.
- How to make time semantics and operational visibility explicit.
A job scheduler seems like a table of cron expressions until time zones, clock changes, worker failures, long-running work, missed executions, duplicate claims, and operational recovery enter the picture. A robust scheduler is a durable coordination system with clear ownership of every execution attempt.
Start with the architecture drill, then follow the request path through scale, failure, and a defensible interview answer.
Make your assumptions and decision path easy to inspect.
ATOFF architecture reasoning canvas
Design a Job Scheduler: the architecture drill
Short answer
A job scheduler is a durable coordination service that translates time rules into safely owned execution attempts despite clock changes and worker failure.
10× evolution
Partition due-time scans and make lease ownership easy to inspect.
01 Decision checkpoint
Design brief
Specify time zone, daylight-saving behavior, missed-run policy, and whether schedules mean fixed instants or local wall-clock occurrences.
02 Decision checkpoint
First request path
A worker should claim a due run with an expiring lease and a stable run ID so a crashed worker can be recovered without uncontrolled parallel execution.
03 Decision checkpoint
Failure drill
Persist attempt state, backoff, and idempotency keys so operators can distinguish an overdue job from a duplicate side effect.
Say it in the interview
I would define time and missed-run semantics first, then use durable run records and leases to make worker recovery safe.
- 1
Schedule definitions
next occurrence: Time evaluator zone + due logic
- 2
Time evaluator zone + due logic
create once: Durable run intent
- 3
Durable run intent
claim: Atomic claim + expiring lease
- 4
Atomic claim + expiring lease
execute: Idempotent worker
- 5
Idempotent worker
complete or fail: Run history + terminal state
- 6
Run history + terminal state
inspect state: Recovery scanner missed + retryable
- 7
Recovery scanner missed + retryable
requeue safely: Durable run intent
- Schedule definitions flows to Time evaluator zone + due logic via next occurrence.
- Time evaluator zone + due logic flows to Durable run intent via create once.
- Durable run intent flows to Atomic claim + expiring lease via claim.
- Atomic claim + expiring lease flows to Idempotent worker via execute.
- Idempotent worker flows to Run history + terminal state via complete or fail.
- Run history + terminal state flows to Recovery scanner missed + retryable via inspect state.
- Recovery scanner missed + retryable flows to Durable run intent via requeue safely.
Walk the design under pressure
Define what time means for each schedule
Clarify one-time versus recurring jobs, time zone ownership, daylight-saving behavior, allowed lateness, concurrency, and what should happen after downtime. "Run every day at 9" is ambiguous without a zone and a policy for a skipped or repeated local time.
Store the schedule definition separately from the generated run. A run should be a durable record with a scheduled timestamp, unique key, state, and attempt history so operators can explain whether work was skipped, delayed, or duplicated.
Use a lease to assign work, not a hopeful poll
Multiple scheduler instances can evaluate due work for availability, but only one should own a specific run at a time. An atomic claim with an expiring lease lets another worker recover if the owner dies. The worker still needs idempotent execution because it can complete an external action just before losing its lease or acknowledgement.
Separate a short scheduling transaction from long execution. Holding a database lock while a job calls an external service creates poor failure behavior and prevents recovery from being independent.
Treat retries and missed runs as product policy
Some jobs should run once later if missed; others should skip stale work; still others should coalesce many missed intervals into one recomputation. Retry rules need backoff, limits, error categories, and a terminal dead-letter state with a human owner. A retry is not a resolution unless the effect is safe to repeat.
Track queue delay, lease expiry, scheduled-versus-start time, retry count, and execution duration. These turn a vague "cron is slow" report into a diagnosable capacity or failure signal.
Staff-level insight: scheduling is an operational contract
At staff scope, the hard questions are multi-tenant fairness, surge behavior at common schedule boundaries, deployment safety, and whether teams can replay or cancel jobs without violating business invariants. Stagger default schedules and make job ownership, on-call routing, and retry cost visible.
In an interview, distinguish system-at-least-once execution from business-exactly-once outcomes. Use idempotency keys or a transactional outbox around external effects, then explain how a user or operator sees an execution that is genuinely uncertain.
Keep this with you
Key takeaways
- A scheduler needs durable run records, explicit time semantics, and a recovery policy.
- Leases assign execution ownership but do not remove the need for idempotent work.
- Missed jobs and retries are product decisions that must be observable and owned.
Practice aloud
Interview questions to explore
- 1.What happens to a daily job during a daylight-saving time change?
- 2.How does another worker recover a run whose owner died?
- 3.Which jobs should be skipped, coalesced, or replayed after downtime?
Common follow-ups
Frequently asked questions
Can a scheduler guarantee exactly-once execution?
It can reduce duplicates with leases and durable state, but external side effects still need idempotency or transactional coordination because a worker can fail between performing work and recording success.
Why use a queue for scheduled jobs?
A durable queue decouples schedule evaluation from worker capacity, exposes backlog, and enables retry and recovery policies without holding the scheduler itself hostage to long-running work.
Already an Elite member? Open the complete walkthrough.
Need the broader preparation context? Go back to Interview Preparation for behavioral readiness, question practice, and the larger AceTheOffer preparation framework.