# System Design: [Job scheduler name]

> Reference design for a distributed job scheduler that runs cron, delayed and
> one-off jobs for many teams. It covers job definitions, due-time indexing,
> leasing with visibility timeouts, at-least-once execution with idempotent
> handlers (and why exactly-once execution is not promised), sharding,
> priorities, retries with dead-letter queues, and observability. It follows
> the structure of the System Design Document template and is pre-filled with
> a worked design to adapt. Values marked *Example* are assumptions.

<!--
Guidance: How to use this template
- Write the delivery guarantee (7.4) in plain words before anything else and
  make every job owner read it. Most scheduler incidents are handlers that
  were not idempotent.
- Size for the top-of-the-hour spike, not the daily average (3.3). Cron users
  all pick :00.
- Fill the run state machine (7.3) with your real states and make the
  dashboard use the same names.
- Keep the failure-mode and trade-off tables filled; reviewers check them first.
-->

## Document control

<!-- Guidance: A design is a proposal until it is approved; record who decided and when. -->

| Field | Value |
|---|---|
| Document ID | [SDD-SCHED-001] |
| System / feature | [Distributed job scheduler] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Implemented / Superseded] |
| Author(s) | [@author] |
| Reviewers | [@reviewer-1], [@reviewer-2] |
| Last updated | YYYY-MM-DD |
| Related | [Platform roadmap], [ADR: storage for job queue], [Runbook] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Tech lead | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| Staff / principal engineer | [@name] | [Decision] | YYYY-MM-DD |
| SRE / operations | [@name] | [Decision] | YYYY-MM-DD |
| Security | [@name] | [Decision] | YYYY-MM-DD |
| Representative job owner team | [@name] | [Decision] | YYYY-MM-DD |

## Revision history

<!-- Guidance: Note what changed in the design, not the wording. -->

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft for design review |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Added deterministic jitter for cron jobs to flatten the :00 spike |
| 0.3 | YYYY-MM-DD | [@name] | *Example:* Fencing tokens on leases |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved |

## Contents

<!-- Guidance: Keep aligned with the headings. -->

1. [Overview](#1-overview)
2. [Requirements](#2-requirements)
3. [Capacity estimation](#3-capacity-estimation)
4. [API design](#4-api-design)
5. [Data model](#5-data-model)
6. [High-level design](#6-high-level-design)
7. [Detailed design](#7-detailed-design)
8. [Scalability and performance](#8-scalability-and-performance)
9. [Reliability, security and operations](#9-reliability-security-and-operations)
10. [Trade-offs and alternatives](#10-trade-offs-and-alternatives)
11. [Delivery plan and risks](#11-delivery-plan-and-risks)
12. [Appendices](#12-appendices)

## 1. Overview

<!-- Guidance: One page that a busy reviewer can read alone: problem, proposed solution in a paragraph, and the decisions that need their attention. -->

### 1.1 Problem statement

[*Example:* Teams run cron on individual VMs and ad hoc delay queues. When a
VM dies its jobs silently stop; two VMs once ran the same billing job and
double-invoiced 3 000 customers; nobody can see which jobs failed overnight.]

### 1.2 Proposed solution

[*Example:* Teams register job definitions (cron, delayed or one-off) through
an API. Definitions and runs are stored in a sharded relational store with an
index on due time. Scheduler nodes, each owning a set of shards through
leader leases, materialise due runs and move them to priority ready queues.
Workers lease runs with a visibility timeout, heartbeat while working and
report the result. Expired leases make the run available again, so execution
is at-least-once; fencing tokens and idempotent handlers make the effects
exactly-once. Failed runs retry with backoff and land in a dead-letter queue
after the last attempt.]

### 1.3 Decisions requested from reviewers

| ID | Question | Recommended option | Section |
|---|---|---|---|
| Q-01 | *Example:* Guarantee offered to job owners? | At-least-once + idempotency key per run | 7.4 |
| Q-02 | *Example:* Store for due runs? | Sharded PostgreSQL with `SKIP LOCKED` dequeue | 5.2, 10 |
| Q-03 | *Example:* Missed cron runs after an outage? | Configurable: run once (default), run all, or skip | 7.2 |
| Q-04 | *Example:* Apply jitter to cron by default? | Yes, deterministic up to 60 s unless pinned | 7.2 |
| [Q-nn] | [Question] | [Option] | [Section] |

## 2. Requirements

<!-- Guidance: Functional requirements describe behaviour; non-functional requirements describe qualities with numbers. Mark what is out of scope so nobody designs for it. -->

### 2.1 Functional requirements

| ID | Requirement | Priority |
|---|---|---|
| FR-01 | *Example:* Cron jobs with time zone (5-field cron syntax, IANA zones, DST rules defined) | Must |
| FR-02 | *Example:* Delayed jobs (run at time T or after duration D) | Must |
| FR-03 | *Example:* One-off immediate jobs with payload up to 256 KB | Must |
| FR-04 | *Example:* Priorities: critical, default, bulk | Must |
| FR-05 | *Example:* Retries with exponential backoff and per-job maximum | Must |
| FR-06 | *Example:* Dead-letter queue with inspect, replay and discard | Must |
| FR-07 | *Example:* Concurrency limit per job definition (for example max 1 running) | Should |
| FR-08 | *Example:* Pause, resume and cancel jobs and runs | Must |
| [FR-nn] | [Requirement] | [Must / Should / Could] |

### 2.2 Non-functional requirements

| ID | Quality | Target | Measured as |
|---|---|---|---|
| NFR-01 | Timeliness | *Example:* Run started within 5 s of due time at p99 (critical), 60 s (bulk) | Start time minus due time |
| NFR-02 | No loss | *Example:* Every due run reaches a terminal state | Reconciliation of runs |
| NFR-03 | Availability | *Example:* 99.95% for scheduling and dispatch | Synthetic canary jobs |
| NFR-04 | Throughput | *Example:* 20 000 run starts/s burst | Load test |
| NFR-05 | Isolation | *Example:* One team's backlog does not delay others' critical jobs | Per-tenant fairness test |
| [NFR-nn] | [Quality] | [Target] | [Measurement] |

### 2.3 Out of scope

- [*Example:* DAG workflows with dependencies between jobs (use the workflow engine)]
- [*Example:* running the job code itself; workers are owned by teams]
- [*Example:* sub-second scheduling precision]
- [Item deferred, with the release that will take it]

## 3. Capacity estimation

<!-- Guidance: Back-of-the-envelope numbers decide the shape of the design. Show inputs, formulas and results; round generously and say which way you rounded. -->

### 3.1 Assumptions

| Input | Symbol | Value | Source |
|---|---|---|---|
| Recurring (cron) definitions | $J_c$ | *Example:* 2 million | Inventory of existing crons x growth |
| Share of cron definitions that run hourly at :00 | $h$ | *Example:* 30% | Inventory |
| Delayed jobs per day | $J_d$ | *Example:* 30 million | Team forecasts |
| One-off jobs per day | $J_o$ | *Example:* 15 million | Team forecasts |
| Cron runs per day (all schedules) | $R_c$ | *Example:* 5 million | Inventory |
| Run record size | $s$ | *Example:* 500 bytes (payload stored separately) | Schema in 5.2 |
| Run history retention | $T$ | *Example:* 30 days | Decision |

### 3.2 Average throughput

$$
R_{day} = R_c + J_d + J_o = 5 \times 10^{6} + 3 \times 10^{7} + 1.5 \times 10^{7} = 5 \times 10^{7} \ \text{runs/day}
$$

$$
QPS_{avg} = \frac{5 \times 10^{7}}{86\,400} \approx 580 \ \text{runs/s}
$$

### 3.3 The top-of-the-hour spike

Without jitter, every hourly cron becomes due in the same second:

$$
R_{spike} = h \times J_c = 0.3 \times 2 \times 10^{6} = 600\,000 \ \text{runs at :00}
$$

With deterministic jitter spread uniformly over 60 s:

$$
QPS_{spike} = \frac{600\,000}{60} = 10\,000 \ \text{runs/s}
$$

which is why NFR-04 targets 20 000 run starts/s (2x headroom) and why jitter
is on by default (Q-04).

### 3.4 Storage

$$
S = R_{day} \times s \times T = 5 \times 10^{7} \times 500 \ \text{B} \times 30 = 750 \ \text{GB}
$$

Definitions: $2 \times 10^{6} \times 2$ KB $= 4$ GB. Payloads above 4 KB are
stored in object storage and referenced by key.

### 3.5 Summary

| Quantity | Estimate | Design consequence |
|---|---|---|
| Runs | ~580/s average, ~10 000/s at :00 with jitter | Sharded store, batch claim of due runs |
| Run history | ~750 GB for 30 days | Partition by day; drop old partitions |
| Definitions | ~4 GB | Fits any relational store |
| Timeliness | 5 s p99 for critical | Scheduler polls each shard at least every second |

## 4. API design

<!-- Guidance: Define the contract first: resources, methods, request and response shapes, errors, idempotency, pagination, rate limits and versioning. Use RFC 9457 problem details for errors. -->

### 4.1 Endpoints

| Method | Path | Purpose | Auth | Idempotent |
|---|---|---|---|---|
| PUT | `/v1/jobs/{name}` | Create or update a job definition | Team token | Yes |
| POST | `/v1/jobs/{name}/runs` | Enqueue a one-off or delayed run | Team token | With `Idempotency-Key` |
| POST | `/v1/jobs/{name}:pause` / `:resume` | Pause or resume a definition | Team token | Yes |
| POST | `/v1/lease` | Worker leases up to N runs for given queues | Worker token | No (returns new leases) |
| POST | `/v1/runs/{id}/heartbeat` | Extend lease | Worker token, fencing token | Yes |
| POST | `/v1/runs/{id}/complete` | Report success or failure | Worker token, fencing token | Yes |
| GET | `/v1/dlq?job=&cursor=` | List dead-lettered runs | Team token | Yes |
| POST | `/v1/dlq/{id}:replay` | Re-enqueue a dead-lettered run | Team token | Yes |

### 4.2 Request and response examples

A job definition for a nightly export, with the policies that matter:

```json
{"name": "billing.export-invoices", "schedule": {"cron": "0 2 * * *", "timeZone": "Europe/Berlin", "jitterSeconds": 0, "missedRunPolicy": "run_once"}, "queue": "billing", "priority": "critical", "timeoutSeconds": 900, "retry": {"maxAttempts": 5, "baseSeconds": 30, "maxSeconds": 1800}, "concurrency": 1, "payload": {"format": "csv"}}
```

A lease response gives the worker everything it needs, including the
idempotency key it must use for side effects:

```json
{"leases": [{"runId": "run_8K2", "job": "billing.export-invoices", "attempt": 2, "idempotencyKey": "billing.export-invoices@2026-03-15T02:00:00+01:00", "fencingToken": 41877, "leaseExpiresAt": "2026-03-15T01:02:30Z", "payload": {"format": "csv"}}]}
```

### 4.3 Errors

| Status | Problem type | When |
|---|---|---|
| 400 | `invalid-schedule` | Cron expression or time zone invalid |
| 409 | `stale-lease` | Fencing token older than the current lease (another worker owns the run) |
| 409 | `concurrency-limit` | Definition already has the maximum running |
| 413 | `payload-too-large` | Payload over 256 KB |

## 5. Data model

<!-- Guidance: Choose storage by access pattern, not by habit. Show entities, keys, partitioning and the queries each store must answer. -->

### 5.1 Entities

Definitions describe what and when; runs are concrete executions with leases
and attempts.

```mermaid
erDiagram
  JOB_DEFINITION {
    string job_name PK
    string team_id
    string cron_expr
    string time_zone
    string queue
    string priority
    int max_attempts
    int timeout_s
    int concurrency
    boolean paused
    datetime next_due_at
  }
  RUN {
    string run_id PK
    int shard
    string job_name FK
    datetime due_at
    string state
    int attempt
    string idempotency_key
    bigint fencing_token
    string worker_id
    datetime lease_expires_at
  }
  ATTEMPT {
    string run_id PK
    int attempt PK
    datetime started_at
    datetime ended_at
    string outcome
    string error
  }
  DLQ_ENTRY {
    string run_id PK
    string job_name
    string last_error
    datetime dead_at
  }
  JOB_DEFINITION ||--o{ RUN : "produces"
  RUN ||--o{ ATTEMPT : "has"
  RUN ||--o| DLQ_ENTRY : "may end in"
```

### 5.2 Storage choices

| Data | Store | Partition key | Main queries | Why |
|---|---|---|---|---|
| Definitions | Relational database (single primary) | `job_name` | Next due definitions | Small, transactional |
| Runs | *Example:* 16 PostgreSQL shards | `shard = hash(job_name) mod 16` | Due runs by `(state, priority, due_at)`; claim with `FOR UPDATE SKIP LOCKED` | Transactional claims without a separate lock service |
| Attempts and history | Same shards, partitioned by day | `run_id` | History per job | Cheap retention by dropping partitions |
| Shard ownership | Coordination service (etcd or ZooKeeper class) | Shard ID | Who owns shard N | Leader leases with fencing |
| Large payloads | Object storage | Run ID | Fetch by worker | Keeps rows small |

## 6. High-level design

<!-- Guidance: One diagram that shows every component on the request paths and where state lives. Name components by responsibility. -->

Schedulers decide when runs become ready; dispatchers hand ready runs to
workers; the store is the source of truth for both.

```mermaid
flowchart LR
  teams([Team services]) --> api[Scheduler API]
  api --> defs[(Definitions DB)]
  api --> runs[(Run shards)]
  coord[(Coordination service)] --- sched[Scheduler nodes]
  sched --> defs
  sched -->|materialise due runs| runs
  workers([Team workers]) -->|lease, heartbeat, complete| disp[Dispatcher API]
  disp --> runs
  reaper[Lease reaper] --> runs
  runs --> dlq[(Dead-letter view)]
  runs --> metrics[[Run events and metrics]]
```

| Component | Responsibility | Scales by |
|---|---|---|
| Scheduler API | Validate and store definitions and ad hoc runs | Stateless replicas |
| Scheduler nodes | For owned shards: compute next cron times, insert runs, flip due runs to `ready` | Shards per node |
| Dispatcher API | Claim ready runs by priority and fairness, issue leases and fencing tokens | Stateless replicas |
| Lease reaper | Return runs with expired leases to `ready` or `retry_wait` | Per shard, inside scheduler nodes |
| Team workers | Execute handlers, heartbeat, report results | Owned by teams |
| [Component] | [Responsibility] | [Scaling unit] |

## 7. Detailed design

<!-- Guidance: Go deep on the parts that are new, risky or contested. Show the sequence of calls and the failure behaviour of each step. -->

### 7.1 Lease and execute (critical path)

A worker claims runs in one transaction, then must heartbeat before the lease
expires; every call after the claim carries the fencing token.

```mermaid
sequenceDiagram
  participant W as Worker
  participant D as Dispatcher
  participant S as Run shard
  participant X as Downstream system
  W->>D: POST /lease (queues=billing, max=10)
  D->>S: begin, select 10 ready runs by priority and due_at, FOR UPDATE SKIP LOCKED
  D->>S: set state leased, new fencing token, lease expiry now + 30 s, commit
  D-->>W: leases with idempotency keys and tokens
  loop every 10 s while working
    W->>D: heartbeat(run, token)
    D->>S: extend lease if token matches
  end
  W->>X: side effect with idempotency key
  X-->>W: ok (or already done)
  W->>D: complete(run, token, success)
  D->>S: state=succeeded if token matches
  alt token stale
    D-->>W: 409 stale-lease, discard result
  end
```

### 7.2 Cron materialisation, jitter and missed runs

- The scheduler for a shard selects definitions with `next_due_at` in the next
  60 s, inserts a `RUN` for each with a unique `(job_name, due_at)` constraint
  (so a double insert after failover is harmless), and advances `next_due_at`.
- Cron times are computed in the definition's time zone. During a DST jump a
  non-existent local time runs at the next valid time; a repeated local time
  runs once (first occurrence). State this to users; it surprises people.
- Jitter: unless `jitterSeconds` is 0, the run is offset by
  `hash(job_name) mod jitterSeconds`, so each job keeps a stable time while the
  fleet spreads out.
- Missed runs after an outage follow `missedRunPolicy`: `run_once` (default)
  inserts one catch-up run, `run_all` inserts every missed time, `skip`
  inserts none.

### 7.3 Run state machine

Every run follows the same states; the dashboard and API use these names.

```mermaid
stateDiagram-v2
  [*] --> scheduled
  scheduled --> ready: due time reached
  scheduled --> cancelled: cancel
  ready --> leased: worker claims
  leased --> succeeded: complete ok
  leased --> retry_wait: failure or lease expired, attempts left
  leased --> dead: failure, no attempts left
  retry_wait --> ready: backoff elapsed
  dead --> ready: replay from DLQ
  succeeded --> [*]
  cancelled --> [*]
  dead --> [*]
```

Backoff before attempt $n$ (1-based retries) uses full jitter:

$$
\text{delay}_n = \text{rand}\left(0,\ \min(\text{max},\ \text{base} \times 2^{\,n-1})\right)
$$

With base 30 s and max 1 800 s: up to 30 s, 60 s, 120 s, 240 s, then capped
growth toward 30 minutes.

### 7.4 Delivery guarantee and consistency model

- **At-least-once execution.** A worker can finish the work and crash before
  `complete`; the lease expires and another worker runs it again. Network
  partitions can also leave a slow worker running while a new lease is issued.
  Exactly-once execution is not achievable in this model and is not promised.
- **Exactly-once effects, by contract.** Each run carries an idempotency key
  that is stable across attempts (`job_name@due_at` for cron, caller key for
  ad hoc runs). Handlers must use it: unique constraints, upserts, or the
  downstream system's idempotency header.
- **Fencing.** Every lease gets a monotonically increasing token. The
  dispatcher rejects heartbeats and completions with an older token, and
  handlers that write to a shared resource should pass the token so the
  resource can reject stale writers.
- **Store consistency.** Claims, state changes and lease extensions are
  single-shard transactions, so a run is never leased to two workers with the
  same token. Cross-shard views (dashboards, DLQ listing) are eventually
  consistent.

### 7.5 Priority and fairness

Dispatchers serve queues in weighted order (*Example:* critical 70%, default
25%, bulk 5% of claims when all have work), and within a priority they
round-robin across teams so one team's 1 million bulk runs cannot starve
another team's 10. Per-definition `concurrency` is enforced at claim time by
counting `leased` runs of that definition.

### 7.6 Sharding and scheduler ownership

Runs are partitioned into 16 shards by `hash(job_name)`. Each scheduler node
holds leases on shards in the coordination service; when a node dies its
shards are reassigned within the lease TTL (*Example:* 10 s). A new owner
starts by running the reaper and the materialiser, both of which are
idempotent. Resharding doubles the shard count by splitting hash ranges during
a maintenance window.

## 8. Scalability and performance

<!-- Guidance: Show how the design meets the numbers in section 3 and where it will break first. -->

| Concern | Approach | Limit before redesign |
|---|---|---|
| Claim rate | `SKIP LOCKED` batch claims, 16 shards | *Example:* ~2 000 claims/s per shard, ~32 000/s total |
| :00 spike | Deterministic jitter; materialise 60 s ahead | Jobs pinned to exact :00 (no jitter) above 50 000 |
| Table bloat | Daily partitions for history; aggressive vacuum on run tables | Monitor dead tuples |
| Many tiny jobs | Batch leases (up to 100 per call) | [Limit] |
| Long-running jobs | Heartbeats; maximum lease extension of 6 h | Use workflow engine above that |

Latency budget from due time to handler start (critical, target p99 5 s):

| Step | Budget |
|---|---|
| Materialiser poll interval | 1 000 ms |
| Flip to `ready` | 100 ms |
| Worker long-poll wait for next claim | 2 000 ms |
| Claim transaction | 50 ms |
| Payload fetch and handler start | 500 ms |
| Headroom | 1 350 ms |

## 9. Reliability, security and operations

<!-- Guidance: Name the failure modes, the threats and the signals operators watch. Each mitigation should be testable. -->

### 9.1 Failure modes

| Failure | Impact | Detection | Mitigation |
|---|---|---|---|
| Worker crash mid-run | Run stuck until lease expiry | Lease expiry count | Reaper returns run to `retry_wait`; idempotent handler on rerun |
| Scheduler node dies | Shard not materialised | Shard ownership gap | Lease reassignment within 10 s; catch-up with missed-run policy |
| Shard primary down | Runs on that shard delayed | DB health | Synchronous standby promotion; other shards unaffected |
| Poison job (always fails) | Retry noise, wasted capacity | DLQ growth per job | Max attempts then DLQ; alert owner |
| Clock skew | Early or late runs | NTP monitoring | Due-time comparisons use database time, not worker time |
| Split brain on shard ownership | Double materialisation | Duplicate key errors | Unique `(job_name, due_at)`; fencing on scheduler leases |
| [Failure] | [Impact] | [Signal] | [Mitigation] |

### 9.2 Security and abuse

| Threat | Control |
|---|---|
| Team A modifies or leases Team B's jobs | Tokens scoped to team and queues; authorisation on every call |
| Secrets in payloads | Payloads may reference secrets by name only; scanner rejects obvious credentials |
| Runaway enqueue loops | Per-team quotas on enqueue rate and pending runs |
| Audit gaps | Definition changes and DLQ replays logged with actor |
| [Threat] | [Control] |

### 9.3 Observability

| Signal | Metric or log | Alert threshold |
|---|---|---|
| Schedule lag | Start time minus due time, per priority | Critical p99 above 5 s for 5 min |
| Ready backlog | Ready runs per queue and age of oldest | Oldest critical above 60 s |
| Lease expiries | Expired leases per job | Above 1% of runs for a job |
| DLQ | New dead runs per job | Any for critical jobs |
| Canary | Synthetic job every minute per shard | Missed twice |

### 9.4 Cost estimate

| Item | Unit cost | Quantity | Monthly cost |
|---|---|---|---|
| Database shards (primary + standby) | [$/instance-month] | *Example:* 32 | [$] |
| Scheduler and dispatcher nodes | [$/node-month] | *Example:* 12 | [$] |
| Coordination service | [$/month] | 3 or 5 nodes | [$] |
| Object storage for payloads | [$/TB-month] | [TB] | [$] |
| **Total** | | | **[$]** |

## 10. Trade-offs and alternatives

<!-- Guidance: For each major decision list the options, what each costs and why you chose one. Reviewers will propose the alternatives anyway; answer them here. -->

| Decision | Options | Chosen | Why | What we give up |
|---|---|---|---|---|
| Queue store | Message broker with delayed delivery; Redis sorted sets; relational with `SKIP LOCKED` | Relational | Transactions, queries for dashboards, cancel and replay are easy | Lower ceiling than a broker; vacuum tuning |
| Guarantee | At-most-once; at-least-once + idempotency; "exactly-once" via distributed transactions | At-least-once + idempotency | Honest, achievable, no loss | Every handler must be idempotent |
| Timer mechanism | In-memory hierarchical timing wheel; database polling | Database polling every second | Durable, simple, adequate for second-level precision | Not suitable for sub-second scheduling |
| Ownership | Single leader for all; sharded leaders | Sharded leaders | Horizontal scale, smaller blast radius | Coordination service dependency |
| Build or adopt | Existing open-source or managed scheduler; build | [Adopt / Build] | [Reason: multi-tenancy, fairness, integration] | [Cost] |

## 11. Delivery plan and risks

<!-- Guidance: How the design reaches production safely, and what could stop it. -->

### 11.1 Milestones

| Milestone | Scope | Exit criterion | Target date |
|---|---|---|---|
| M1 | One-off and delayed jobs, leases, retries, DLQ | Chaos test (kill workers, kill DB primary) loses no runs | YYYY-MM-DD |
| M2 | Cron with time zones, jitter and missed-run policies | DST test suite passes | YYYY-MM-DD |
| M3 | Priorities, fairness, concurrency limits | Fairness test: critical lag unaffected by bulk flood | YYYY-MM-DD |
| M4 | Migration of VM crons | All critical crons migrated; old crons disabled | YYYY-MM-DD |

### 11.2 Rollout and rollback

[*Example:* Migrate each cron by running it in the scheduler in "dry-run"
mode (handler logs, no effects) alongside the VM cron for a week, then switch.
Rollback re-enables the VM cron and pauses the definition.]

### 11.3 Risks

| ID | Risk | Likelihood | Impact | Mitigation | Owner |
|---|---|---|---|---|---|
| R-01 | *Example:* Handlers are not idempotent and duplicate side effects | High | High | Idempotency review in onboarding; key provided by platform | [@name] |
| R-02 | *Example:* Database becomes the bottleneck | Medium | Medium | Shard doubling plan; batch claims | [@name] |
| R-03 | *Example:* Teams pin all jobs to :00 | Medium | Medium | Jitter default; quota on pinned jobs | [@name] |
| [R-nn] | [Risk] | [L / M / H] | [L / M / H] | [Mitigation] | [@name] |

## 12. Appendices

<!-- Guidance: Supporting detail, glossary and open questions. -->

### 12.1 Open questions

| ID | Question | Owner | Due |
|---|---|---|---|
| OQ-01 | *Example:* Do we support second-level cron fields? | [@name] | YYYY-MM-DD |
| OQ-02 | *Example:* How long are DLQ entries kept? | [@name] | YYYY-MM-DD |
| [OQ-nn] | [Question] | [@name] | YYYY-MM-DD |

### 12.2 Glossary

| Term | Meaning |
|---|---|
| Lease | Time-limited claim of a run by a worker |
| Visibility timeout | Time a leased run is hidden from other workers |
| Fencing token | Increasing number that lets systems reject a stale lease holder |
| DLQ | Dead-letter queue: runs that exhausted retries |
| Jitter | Deliberate random or hashed offset to spread load |
| [Term] | [Meaning] |

### 12.3 References

| Reference | Use |
|---|---|
| IANA Time Zone Database | Time zones for cron |
| PostgreSQL documentation, `SELECT ... FOR UPDATE SKIP LOCKED` | Claim query in 7.1 |
| IETF draft "The Idempotency-Key HTTP Header Field" | Idempotent enqueue |
| [Internal handler idempotency guide] | Contract for job owners |

## Standard and licence note

Based on common industry practice for distributed job scheduling and queueing
systems. This is a reference design to adapt, with illustrative numbers.
Original text written for this collection. Template licence: MIT.