System Design: [Job scheduler name]

Reference design for a distributed job scheduler that runs cron, delayed and one-off jobs for many teams. It covers job definitions, due-time indexing, leasing with visibility timeouts, at-least-once execution with idempotent handlers (and why exactly-once execution is not promised), sharding, priorities, retries with dead-letter queues, and observability. It follows the structure of the System Design Document template and is pre-filled with a worked design to adapt. Values marked Example are assumptions.

Document control

Approvals

Revision history

Contents

1. Overview
2. Requirements
3. Capacity estimation
4. API design
5. Data model
6. High-level design
7. Detailed design
8. Scalability and performance
9. Reliability, security and operations
10. Trade-offs and alternatives
11. Delivery plan and risks
12. Appendices

1. Overview

1.1 Problem statement

[Example: Teams run cron on individual VMs and ad hoc delay queues. When a VM dies its jobs silently stop; two VMs once ran the same billing job and double-invoiced 3 000 customers; nobody can see which jobs failed overnight.]

1.2 Proposed solution

[Example: Teams register job definitions (cron, delayed or one-off) through an API. Definitions and runs are stored in a sharded relational store with an index on due time. Scheduler nodes, each owning a set of shards through leader leases, materialise due runs and move them to priority ready queues. Workers lease runs with a visibility timeout, heartbeat while working and report the result. Expired leases make the run available again, so execution is at-least-once; fencing tokens and idempotent handlers make the effects exactly-once. Failed runs retry with backoff and land in a dead-letter queue after the last attempt.]

1.3 Decisions requested from reviewers

2. Requirements

2.1 Functional requirements

2.2 Non-functional requirements

2.3 Out of scope

• [Example: DAG workflows with dependencies between jobs (use the workflow engine)]
• [Example: running the job code itself; workers are owned by teams]
• [Example: sub-second scheduling precision]
• [Item deferred, with the release that will take it]

3. Capacity estimation

3.1 Assumptions

3.2 Average throughput

3.3 The top-of-the-hour spike

Without jitter, every hourly cron becomes due in the same second:

With deterministic jitter spread uniformly over 60 s:

which is why NFR-04 targets 20 000 run starts/s (2x headroom) and why jitter is on by default (Q-04).

3.4 Storage

Definitions: $2 \times 10^{6} \times 2$ KB $= 4$ GB. Payloads above 4 KB are stored in object storage and referenced by key.

3.5 Summary

4. API design

4.1 Endpoints

4.2 Request and response examples

A job definition for a nightly export, with the policies that matter:

{"name": "billing.export-invoices", "schedule": {"cron": "0 2 * * *", "timeZone": "Europe/Berlin", "jitterSeconds": 0, "missedRunPolicy": "run_once"}, "queue": "billing", "priority": "critical", "timeoutSeconds": 900, "retry": {"maxAttempts": 5, "baseSeconds": 30, "maxSeconds": 1800}, "concurrency": 1, "payload": {"format": "csv"}}

A lease response gives the worker everything it needs, including the idempotency key it must use for side effects:

{"leases": [{"runId": "run_8K2", "job": "billing.export-invoices", "attempt": 2, "idempotencyKey": "billing.export-invoices@2026-03-15T02:00:00+01:00", "fencingToken": 41877, "leaseExpiresAt": "2026-03-15T01:02:30Z", "payload": {"format": "csv"}}]}

4.3 Errors

5. Data model

5.1 Entities

Definitions describe what and when; runs are concrete executions with leases and attempts.

5.2 Storage choices

6. High-level design

Schedulers decide when runs become ready; dispatchers hand ready runs to workers; the store is the source of truth for both.

7. Detailed design

7.1 Lease and execute (critical path)

A worker claims runs in one transaction, then must heartbeat before the lease expires; every call after the claim carries the fencing token.

7.2 Cron materialisation, jitter and missed runs

• The scheduler for a shard selects definitions with next_due_at in the next 60 s, inserts a RUN for each with a unique (job_name, due_at) constraint (so a double insert after failover is harmless), and advances next_due_at.
• Cron times are computed in the definition's time zone. During a DST jump a non-existent local time runs at the next valid time; a repeated local time runs once (first occurrence). State this to users; it surprises people.
• Jitter: unless jitterSeconds is 0, the run is offset by hash(job_name) mod jitterSeconds, so each job keeps a stable time while the fleet spreads out.
• Missed runs after an outage follow missedRunPolicy: run_once (default) inserts one catch-up run, run_all inserts every missed time, skip inserts none.

7.3 Run state machine

Every run follows the same states; the dashboard and API use these names.

Backoff before attempt $n$ (1-based retries) uses full jitter:

With base 30 s and max 1 800 s: up to 30 s, 60 s, 120 s, 240 s, then capped growth toward 30 minutes.

7.4 Delivery guarantee and consistency model

• At-least-once execution. A worker can finish the work and crash before complete; the lease expires and another worker runs it again. Network partitions can also leave a slow worker running while a new lease is issued. Exactly-once execution is not achievable in this model and is not promised.
• Exactly-once effects, by contract. Each run carries an idempotency key that is stable across attempts (job_name@due_at for cron, caller key for ad hoc runs). Handlers must use it: unique constraints, upserts, or the downstream system's idempotency header.
• Fencing. Every lease gets a monotonically increasing token. The dispatcher rejects heartbeats and completions with an older token, and handlers that write to a shared resource should pass the token so the resource can reject stale writers.
• Store consistency. Claims, state changes and lease extensions are single-shard transactions, so a run is never leased to two workers with the same token. Cross-shard views (dashboards, DLQ listing) are eventually consistent.

7.5 Priority and fairness

Dispatchers serve queues in weighted order (Example: critical 70%, default 25%, bulk 5% of claims when all have work), and within a priority they round-robin across teams so one team's 1 million bulk runs cannot starve another team's 10. Per-definition concurrency is enforced at claim time by counting leased runs of that definition.

7.6 Sharding and scheduler ownership

Runs are partitioned into 16 shards by hash(job_name). Each scheduler node holds leases on shards in the coordination service; when a node dies its shards are reassigned within the lease TTL (Example: 10 s). A new owner starts by running the reaper and the materialiser, both of which are idempotent. Resharding doubles the shard count by splitting hash ranges during a maintenance window.

8. Scalability and performance

Latency budget from due time to handler start (critical, target p99 5 s):

9. Reliability, security and operations

9.1 Failure modes

9.2 Security and abuse

9.3 Observability

9.4 Cost estimate

10. Trade-offs and alternatives

11. Delivery plan and risks

11.1 Milestones

11.2 Rollout and rollback

[Example: Migrate each cron by running it in the scheduler in "dry-run" mode (handler logs, no effects) alongside the VM cron for a week, then switch. Rollback re-enables the VM cron and pauses the definition.]

11.3 Risks

12. Appendices

12.1 Open questions

12.2 Glossary

12.3 References

Standard and licence note

Based on common industry practice for distributed job scheduling and queueing systems. This is a reference design to adapt, with illustrative numbers. Original text written for this collection. Template licence: MIT.