Job Scheduler System Design Template (Reference Design)

Distributed scheduler: cron, delayed and one-off jobs, leases and visibility timeouts, fencing, idempotent handlers, retries, DLQ

Arquitectura y diseño de sistemasCompletadesign job scheduler mathActualizada 20 oct 2026

Abrir en el editorDescargar .md

Cómo usar: Use when replacing ad hoc cron and delay queues. Write the at-least-once guarantee first and size for the top-of-the-hour spike. The main pitfall is non-idempotent handlers, which turn lease expiry into duplicate side effects.

Vista previa

System Design: [Job scheduler name]

Reference design for a distributed job scheduler that runs cron, delayed and one-off jobs for many teams. It covers job definitions, due-time indexing, leasing with visibility timeouts, at-least-once execution with idempotent handlers (and why exactly-once execution is not promised), sharding, priorities, retries with dead-letter queues, and observability. It follows the structure of the System Design Document template and is pre-filled with a worked design to adapt. Values marked Example are assumptions.

Document control

Field	Value
Document ID	[SDD-SCHED-001]
System / feature	[Distributed job scheduler]
Version	[0.1]
Status	[Draft / In review / Approved / Implemented / Superseded]
Author(s)	[@author]
Reviewers	[@reviewer-1], [@reviewer-2]
Last updated	YYYY-MM-DD
Related	[Platform roadmap], [ADR: storage for job queue], [Runbook]

Approvals

Role	Name	Decision	Date
Tech lead	[@name]	[Approved / Changes requested]	YYYY-MM-DD
Staff / principal engineer	[@name]	[Decision]	YYYY-MM-DD
SRE / operations	[@name]	[Decision]	YYYY-MM-DD
Security	[@name]	[Decision]	YYYY-MM-DD
Representative job owner team	[@name]	[Decision]	YYYY-MM-DD

Revision history

Version	Date	Author	Change summary
0.1	YYYY-MM-DD	[@name]	First draft for design review
0.2	YYYY-MM-DD	[@name]	Example: Added deterministic jitter for cron jobs to flatten the :00 spike
0.3	YYYY-MM-DD	[@name]	Example: Fencing tokens on leases
1.0	YYYY-MM-DD	[@name]	Example: Approved

Overview
Requirements
Capacity estimation
API design
Data model
High-level design
Detailed design
Scalability and performance
Reliability, security and operations
Trade-offs and alternatives
Delivery plan and risks
Appendices

1. Overview

1.1 Problem statement

[Example: Teams run cron on individual VMs and ad hoc delay queues. When a VM dies its jobs silently stop; two VMs once ran the same billing job and double-invoiced 3 000 customers; nobody can see which jobs failed overnight.]

1.2 Proposed solution

[Example: Teams register job definitions (cron, delayed or one-off) through an API. Definitions and runs are stored in a sharded relational store with an index on due time. Scheduler nodes, each owning a set of shards through leader leases, materialise due runs and move them to priority ready queues. Workers lease runs with a visibility timeout, heartbeat while working and report the result. Expired leases make the run available again, so execution is at-least-once; fencing tokens and idempotent handlers make the effects exactly-once. Failed runs retry with backoff and land in a dead-letter queue after the last attempt.]

1.3 Decisions requested from reviewers

ID	Question	Recommended option	Section
Q-01	Example: Guarantee offered to job owners?	At-least-once + idempotency key per run	7.4
Q-02	Example: Store for due runs?	Sharded PostgreSQL with `SKIP LOCKED` dequeue	5.2, 10
Q-03	Example: Missed cron runs after an outage?	Configurable: run once (default), run all, or skip	7.2
Q-04	Example: Apply jitter to cron by default?	Yes, deterministic up to 60 s unless pinned	7.2
[Q-nn]	[Question]	[Option]	[Section]

2. Requirements

2.1 Functional requirements

ID	Requirement	Priority
FR-01	Example: Cron jobs with time zone (5-field cron syntax, IANA zones, DST rules defined)	Must
FR-02	Example: Delayed jobs (run at time T or after duration D)	Must
FR-03	Example: One-off immediate jobs with payload up to 256 KB	Must
FR-04	Example: Priorities: critical, default, bulk	Must
FR-05	Example: Retries with exponential backoff and per-job maximum	Must
FR-06	Example: Dead-letter queue with inspect, replay and discard	Must
FR-07	Example: Concurrency limit per job definition (for example max 1 running)	Should
FR-08	Example: Pause, resume and cancel jobs and runs	Must
[FR-nn]	[Requirement]	[Must / Should / Could]

2.2 Non-functional requirements

ID	Quality	Target	Measured as
NFR-01	Timeliness	Example: Run started within 5 s of due time at p99 (critical), 60 s (bulk)	Start time minus due time
NFR-02	No loss	Example: Every due run reaches a terminal state	Reconciliation of runs
NFR-03	Availability	Example: 99.95% for scheduling and dispatch	Synthetic canary jobs
NFR-04	Throughput	Example: 20 000 run starts/s burst	Load test
NFR-05	Isolation	Example: One team's backlog does not delay others' critical jobs	Per-tenant fairness test
[NFR-nn]	[Quality]	[Target]	[Measurement]

2.3 Out of scope

[Example: DAG workflows with dependencies between jobs (use the workflow engine)]
[Example: running the job code itself; workers are owned by teams]
[Example: sub-second scheduling precision]
[Item deferred, with the release that will take it]

3. Capacity estimation

3.1 Assumptions

Input	Symbol	Value	Source
Recurring (cron) definitions	$J_c$	Example: 2 million	Inventory of existing crons x growth
Share of cron definitions that run hourly at :00	$h$	Example: 30%	Inventory
Delayed jobs per day	$J_d$	Example: 30 million	Team forecasts
One-off jobs per day	$J_o$	Example: 15 million	Team forecasts
Cron runs per day (all schedules)	$R_c$	Example: 5 million	Inventory
Run record size	$s$	Example: 500 bytes (payload stored separately)	Schema in 5.2
Run history retention	$T$	Example: 30 days	Decision

3.2 Average throughput

R_{day} = R_c + J_d + J_o = 5 \times 10^{6} + 3 \times 10^{7} + 1.5 \times 10^{7} = 5 \times 10^{7} \ \text{runs/day}

QPS_{avg} = \frac{5 \times 10^{7}}{86\,400} \approx 580 \ \text{runs/s}

3.3 The top-of-the-hour spike

Without jitter, every hourly cron becomes due in the same second:

R_{spike} = h \times J_c = 0.3 \times 2 \times 10^{6} = 600\,000 \ \text{runs at :00}

With deterministic jitter spread uniformly over 60 s:

QPS_{spike} = \frac{600\,000}{60} = 10\,000 \ \text{runs/s}

which is why NFR-04 targets 20 000 run starts/s (2x headroom) and why jitter is on by default (Q-04).

3.4 Storage

S = R_{day} \times s \times T = 5 \times 10^{7} \times 500 \ \text{B} \times 30 = 750 \ \text{GB}

Definitions: $2 \times 10^{6} \times 2$ KB $= 4$ GB. Payloads above 4 KB are stored in object storage and referenced by key.

3.5 Summary

Quantity	Estimate	Design consequence
Runs	~580/s average, ~10 000/s at :00 with jitter	Sharded store, batch claim of due runs
Run history	~750 GB for 30 days	Partition by day; drop old partitions
Definitions	~4 GB	Fits any relational store
Timeliness	5 s p99 for critical	Scheduler polls each shard at least every second

4. API design

4.1 Endpoints

Method	Path	Purpose	Auth	Idempotent
PUT	`/v1/jobs/{name}`	Create or update a job definition	Team token	Yes
POST	`/v1/jobs/{name}/runs`	Enqueue a one-off or delayed run	Team token	With `Idempotency-Key`
POST	`/v1/jobs/{name}:pause` / `:resume`	Pause or resume a definition	Team token	Yes
POST	`/v1/lease`	Worker leases up to N runs for given queues	Worker token	No (returns new leases)
POST	`/v1/runs/{id}/heartbeat`	Extend lease	Worker token, fencing token	Yes
POST	`/v1/runs/{id}/complete`	Report success or failure	Worker token, fencing token	Yes
GET	`/v1/dlq?job=&cursor=`	List dead-lettered runs	Team token	Yes
POST	`/v1/dlq/{id}:replay`	Re-enqueue a dead-lettered run	Team token	Yes

4.2 Request and response examples

A job definition for a nightly export, with the policies that matter:

{"name": "billing.export-invoices", "schedule": {"cron": "0 2 * * *", "timeZone": "Europe/Berlin", "jitterSeconds": 0, "missedRunPolicy": "run_once"}, "queue": "billing", "priority": "critical", "timeoutSeconds": 900, "retry": {"maxAttempts": 5, "baseSeconds": 30, "maxSeconds": 1800}, "concurrency": 1, "payload": {"format": "csv"}}

A lease response gives the worker everything it needs, including the idempotency key it must use for side effects:

{"leases": [{"runId": "run_8K2", "job": "billing.export-invoices", "attempt": 2, "idempotencyKey": "billing.export-invoices@2026-03-15T02:00:00+01:00", "fencingToken": 41877, "leaseExpiresAt": "2026-03-15T01:02:30Z", "payload": {"format": "csv"}}]}

4.3 Errors

Status	Problem type	When
400	`invalid-schedule`	Cron expression or time zone invalid
409	`stale-lease`	Fencing token older than the current lease (another worker owns the run)
409	`concurrency-limit`	Definition already has the maximum running
413	`payload-too-large`	Payload over 256 KB

5. Data model

5.1 Entities

Definitions describe what and when; runs are concrete executions with leases and attempts.

5.2 Storage choices

Data	Store	Partition key	Main queries	Why
Definitions	Relational database (single primary)	`job_name`	Next due definitions	Small, transactional
Runs	Example: 16 PostgreSQL shards	`shard = hash(job_name) mod 16`	Due runs by `(state, priority, due_at)`; claim with `FOR UPDATE SKIP LOCKED`	Transactional claims without a separate lock service
Attempts and history	Same shards, partitioned by day	`run_id`	History per job	Cheap retention by dropping partitions
Shard ownership	Coordination service (etcd or ZooKeeper class)	Shard ID	Who owns shard N	Leader leases with fencing
Large payloads	Object storage	Run ID	Fetch by worker	Keeps rows small

6. High-level design

Schedulers decide when runs become ready; dispatchers hand ready runs to workers; the store is the source of truth for both.

Component	Responsibility	Scales by
Scheduler API	Validate and store definitions and ad hoc runs	Stateless replicas
Scheduler nodes	For owned shards: compute next cron times, insert runs, flip due runs to `ready`	Shards per node
Dispatcher API	Claim ready runs by priority and fairness, issue leases and fencing tokens	Stateless replicas
Lease reaper	Return runs with expired leases to `ready` or `retry_wait`	Per shard, inside scheduler nodes
Team workers	Execute handlers, heartbeat, report results	Owned by teams
[Component]	[Responsibility]	[Scaling unit]

7. Detailed design

7.1 Lease and execute (critical path)

A worker claims runs in one transaction, then must heartbeat before the lease expires; every call after the claim carries the fencing token.

7.2 Cron materialisation, jitter and missed runs

The scheduler for a shard selects definitions with next_due_at in the next 60 s, inserts a RUN for each with a unique (job_name, due_at) constraint (so a double insert after failover is harmless), and advances next_due_at.
Cron times are computed in the definition's time zone. During a DST jump a non-existent local time runs at the next valid time; a repeated local time runs once (first occurrence). State this to users; it surprises people.
Jitter: unless jitterSeconds is 0, the run is offset by hash(job_name) mod jitterSeconds, so each job keeps a stable time while the fleet spreads out.
Missed runs after an outage follow missedRunPolicy: run_once (default) inserts one catch-up run, run_all inserts every missed time, skip inserts none.

7.3 Run state machine

Every run follows the same states; the dashboard and API use these names.

Backoff before attempt $n$ (1-based retries) uses full jitter:

\text{delay}_n = \text{rand}\left(0,\ \min(\text{max},\ \text{base} \times 2^{\,n-1})\right)

With base 30 s and max 1 800 s: up to 30 s, 60 s, 120 s, 240 s, then capped growth toward 30 minutes.

7.4 Delivery guarantee and consistency model

At-least-once execution. A worker can finish the work and crash before complete; the lease expires and another worker runs it again. Network partitions can also leave a slow worker running while a new lease is issued. Exactly-once execution is not achievable in this model and is not promised.
Exactly-once effects, by contract. Each run carries an idempotency key that is stable across attempts (job_name@due_at for cron, caller key for ad hoc runs). Handlers must use it: unique constraints, upserts, or the downstream system's idempotency header.
Fencing. Every lease gets a monotonically increasing token. The dispatcher rejects heartbeats and completions with an older token, and handlers that write to a shared resource should pass the token so the resource can reject stale writers.
Store consistency. Claims, state changes and lease extensions are single-shard transactions, so a run is never leased to two workers with the same token. Cross-shard views (dashboards, DLQ listing) are eventually consistent.

7.5 Priority and fairness

Dispatchers serve queues in weighted order (Example: critical 70%, default 25%, bulk 5% of claims when all have work), and within a priority they round-robin across teams so one team's 1 million bulk runs cannot starve another team's 10. Per-definition concurrency is enforced at claim time by counting leased runs of that definition.

7.6 Sharding and scheduler ownership

Runs are partitioned into 16 shards by hash(job_name). Each scheduler node holds leases on shards in the coordination service; when a node dies its shards are reassigned within the lease TTL (Example: 10 s). A new owner starts by running the reaper and the materialiser, both of which are idempotent. Resharding doubles the shard count by splitting hash ranges during a maintenance window.

8. Scalability and performance

Concern	Approach	Limit before redesign
Claim rate	`SKIP LOCKED` batch claims, 16 shards	Example: ~2 000 claims/s per shard, ~32 000/s total
:00 spike	Deterministic jitter; materialise 60 s ahead	Jobs pinned to exact :00 (no jitter) above 50 000
Table bloat	Daily partitions for history; aggressive vacuum on run tables	Monitor dead tuples
Many tiny jobs	Batch leases (up to 100 per call)	[Limit]
Long-running jobs	Heartbeats; maximum lease extension of 6 h	Use workflow engine above that

Latency budget from due time to handler start (critical, target p99 5 s):

Step	Budget
Materialiser poll interval	1 000 ms
Flip to `ready`	100 ms
Worker long-poll wait for next claim	2 000 ms
Claim transaction	50 ms
Payload fetch and handler start	500 ms
Headroom	1 350 ms

9. Reliability, security and operations

9.1 Failure modes

Failure	Impact	Detection	Mitigation
Worker crash mid-run	Run stuck until lease expiry	Lease expiry count	Reaper returns run to `retry_wait`; idempotent handler on rerun
Scheduler node dies	Shard not materialised	Shard ownership gap	Lease reassignment within 10 s; catch-up with missed-run policy
Shard primary down	Runs on that shard delayed	DB health	Synchronous standby promotion; other shards unaffected
Poison job (always fails)	Retry noise, wasted capacity	DLQ growth per job	Max attempts then DLQ; alert owner
Clock skew	Early or late runs	NTP monitoring	Due-time comparisons use database time, not worker time
Split brain on shard ownership	Double materialisation	Duplicate key errors	Unique `(job_name, due_at)`; fencing on scheduler leases
[Failure]	[Impact]	[Signal]	[Mitigation]

9.2 Security and abuse

Threat	Control
Team A modifies or leases Team B's jobs	Tokens scoped to team and queues; authorisation on every call
Secrets in payloads	Payloads may reference secrets by name only; scanner rejects obvious credentials
Runaway enqueue loops	Per-team quotas on enqueue rate and pending runs
Audit gaps	Definition changes and DLQ replays logged with actor
[Threat]	[Control]

9.3 Observability

Signal	Metric or log	Alert threshold
Schedule lag	Start time minus due time, per priority	Critical p99 above 5 s for 5 min
Ready backlog	Ready runs per queue and age of oldest	Oldest critical above 60 s
Lease expiries	Expired leases per job	Above 1% of runs for a job
DLQ	New dead runs per job	Any for critical jobs
Canary	Synthetic job every minute per shard	Missed twice

9.4 Cost estimate

Item	Unit cost	Quantity	Monthly cost
Database shards (primary + standby)	[$/instance-month]	Example: 32	[$]
Scheduler and dispatcher nodes	[$/node-month]	Example: 12	[$]
Coordination service	[$/month]	3 or 5 nodes	[$]
Object storage for payloads	[$/TB-month]	[TB]	[$]
Total			[$]

10. Trade-offs and alternatives

Decision	Options	Chosen	Why	What we give up
Queue store	Message broker with delayed delivery; Redis sorted sets; relational with `SKIP LOCKED`	Relational	Transactions, queries for dashboards, cancel and replay are easy	Lower ceiling than a broker; vacuum tuning
Guarantee	At-most-once; at-least-once + idempotency; "exactly-once" via distributed transactions	At-least-once + idempotency	Honest, achievable, no loss	Every handler must be idempotent
Timer mechanism	In-memory hierarchical timing wheel; database polling	Database polling every second	Durable, simple, adequate for second-level precision	Not suitable for sub-second scheduling
Ownership	Single leader for all; sharded leaders	Sharded leaders	Horizontal scale, smaller blast radius	Coordination service dependency
Build or adopt	Existing open-source or managed scheduler; build	[Adopt / Build]	[Reason: multi-tenancy, fairness, integration]	[Cost]

11. Delivery plan and risks

11.1 Milestones

Milestone	Scope	Exit criterion	Target date
M1	One-off and delayed jobs, leases, retries, DLQ	Chaos test (kill workers, kill DB primary) loses no runs	YYYY-MM-DD
M2	Cron with time zones, jitter and missed-run policies	DST test suite passes	YYYY-MM-DD
M3	Priorities, fairness, concurrency limits	Fairness test: critical lag unaffected by bulk flood	YYYY-MM-DD
M4	Migration of VM crons	All critical crons migrated; old crons disabled	YYYY-MM-DD

11.2 Rollout and rollback

[Example: Migrate each cron by running it in the scheduler in "dry-run" mode (handler logs, no effects) alongside the VM cron for a week, then switch. Rollback re-enables the VM cron and pauses the definition.]

11.3 Risks

ID	Risk	Likelihood	Impact	Mitigation	Owner
R-01	Example: Handlers are not idempotent and duplicate side effects	High	High	Idempotency review in onboarding; key provided by platform	[@name]
R-02	Example: Database becomes the bottleneck	Medium	Medium	Shard doubling plan; batch claims	[@name]
R-03	Example: Teams pin all jobs to :00	Medium	Medium	Jitter default; quota on pinned jobs	[@name]
[R-nn]	[Risk]	[L / M / H]	[L / M / H]	[Mitigation]	[@name]

12. Appendices

12.1 Open questions

ID	Question	Owner	Due
OQ-01	Example: Do we support second-level cron fields?	[@name]	YYYY-MM-DD
OQ-02	Example: How long are DLQ entries kept?	[@name]	YYYY-MM-DD
[OQ-nn]	[Question]	[@name]	YYYY-MM-DD

12.2 Glossary

Term	Meaning
Lease	Time-limited claim of a run by a worker
Visibility timeout	Time a leased run is hidden from other workers
Fencing token	Increasing number that lets systems reject a stale lease holder
DLQ	Dead-letter queue: runs that exhausted retries
Jitter	Deliberate random or hashed offset to spread load
[Term]	[Meaning]

12.3 References

Reference	Use
IANA Time Zone Database	Time zones for cron
PostgreSQL documentation, `SELECT ... FOR UPDATE SKIP LOCKED`	Claim query in 7.1
IETF draft "The Idempotency-Key HTTP Header Field"	Idempotent enqueue
[Internal handler idempotency guide]	Contract for job owners

Standard and licence note

Based on common industry practice for distributed job scheduling and queueing systems. This is a reference design to adapt, with illustrative numbers. Original text written for this collection. Template licence: MIT.

Plantillas relacionadas

Arquitectura y diseño de sistemas