# System Design: [Workflow orchestration engine name]

> Reference design for a durable workflow orchestration engine: a platform
> that runs long-lived business processes written as ordinary code (order
> fulfilment, account onboarding, payment sagas, data pipelines) and
> guarantees that they resume exactly where they stopped after crashes,
> deploys and outages. It covers the event history and replay model,
> deterministic workflow code, activities, durable timers, task queues and
> matching, retries and timeouts, workflow code versioning, visibility
> search and sharding of history. It is distinct from the Job Scheduler
> System Design Template (`system-design/design-job-scheduler`), which runs
> independent jobs on a schedule; this engine coordinates multi-step stateful
> processes. It follows the structure of the System Design Document template
> and is pre-filled with a worked design to adapt. Replace every
> `[bracketed]` value and every row marked *Example*; all numbers are
> illustrative assumptions, not benchmarks.

<!--
Guidance: How to use this template
- Start from the workload shape: workflow starts per second, steps per
  workflow and how long workflows stay open. Open workflows and pending
  timers, not starts, usually size the system (section 3).
- Write the determinism rules (7.1) before the SDK. Every production
  incident in this class of system eventually traces back to workflow code
  that behaves differently on replay.
- Decide the versioning strategy (7.4) before the first workflow ships;
  retrofitting it onto running workflows is painful.
- If your processes are independent jobs with no state between steps, use
  the job scheduler design instead; this engine costs more to run.
-->

## Document control

<!-- Guidance: A design is a proposal until it is approved; record who decided and when. -->

| Field | Value |
|---|---|
| Document ID | [SDD-WF-001] |
| System / feature | [Durable workflow orchestration platform] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Implemented / Superseded] |
| Owner | [@owner] |
| Author(s) | [@author] |
| Reviewers | [@reviewer-1], [@reviewer-2] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal] |
| Build or adopt | [Build / adopt open-source engine / managed service] |
| Related | Job Scheduler System Design Template (`system-design/design-job-scheduler`), Message Queue System Design Template (`system-design/design-message-queue`), Event Sourcing and CQRS Template (`architecture/event-sourcing-cqrs`) |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Tech lead, platform | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| Staff / principal engineer | [@name] | [Decision] | YYYY-MM-DD |
| SRE / operations | [@name] | [Decision] | YYYY-MM-DD |
| Data / storage owner | [@name] | [Decision] | YYYY-MM-DD |
| Security | [@name] | [Decision] | YYYY-MM-DD |

## Revision history

<!-- Guidance: Note what changed in the design, not the wording. -->

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft for design review |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Raised shard count from 1,024 to 4,096 after open-workflow estimate |
| 0.3 | YYYY-MM-DD | [@name] | *Example:* Added sticky task queues to cut replay cost |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved |

## Contents

<!-- Guidance: Keep aligned with the headings. -->

1. [Overview](#1-overview)
2. [Requirements](#2-requirements)
3. [Capacity estimation](#3-capacity-estimation)
4. [API design](#4-api-design)
5. [Data model](#5-data-model)
6. [High-level design](#6-high-level-design)
7. [Detailed design](#7-detailed-design)
8. [Scalability and performance](#8-scalability-and-performance)
9. [Reliability, security and operations](#9-reliability-security-and-operations)
10. [Trade-offs and alternatives](#10-trade-offs-and-alternatives)
11. [Delivery plan and risks](#11-delivery-plan-and-risks)
12. [Appendices](#12-appendices)

## 1. Overview

<!-- Guidance: One page that a busy reviewer can read alone: problem, proposed solution in a paragraph, and the decisions that need their attention. -->

### 1.1 Problem statement

[*Example:* Order fulfilment spans payment, inventory, warehouse and
shipping services and can take from minutes to 14 days. Today the state
machine is spread across 9 queues, 4 cron jobs and a status column in the
orders table. After a broker outage last month, 3,200 orders were stuck in
"payment captured, not reserved" and took two engineers four days to
repair by hand. Every new process repeats the same plumbing: retries,
timeouts, compensation and status pages.]

### 1.2 Proposed solution

[*Example:* A shared engine stores each workflow execution as an
append-only event history. Workflow code runs in worker processes owned by
product teams; it is replayed from the history to rebuild state, so it must
be deterministic. Side effects run as activities with their own retry and
timeout policies. The engine provides durable timers, signals and queries,
and dispatches workflow and activity tasks through partitioned task queues.
Histories are sharded by workflow ID across a fixed number of shards, each
owned by one history host at a time. A visibility store indexes executions
for search, and closed histories are archived to object storage after a
retention period.]

### 1.3 Decisions requested from reviewers

| ID | Question | Recommended option | Section |
|---|---|---|---|
| Q-01 | *Example:* Build, adopt open source or use a managed service? | Adopt an open-source engine or managed service unless a hard constraint rules both out | 10 |
| Q-02 | *Example:* How many history shards? | 4,096, fixed at cluster creation | 3.4, 7.2 |
| Q-03 | *Example:* How do teams change workflow code safely? | Version markers in code plus worker build versioning | 7.4 |
| Q-04 | *Example:* What is the delivery guarantee for activities? | At least once; activities must be idempotent | 7.5 |
| [Q-nn] | [Question] | [Option] | [Section] |

### 1.4 45-minute interview path

<!-- Guidance: Use this order in an interview or a time-boxed review. Spend the most time on one deep dive rather than touching all of them. -->

| Minutes | Cover | Key point to land |
|---|---|---|
| 0 to 5 | Scope: long-running multi-step processes, not cron jobs | Durability across crashes is the product |
| 5 to 10 | Core requirements and guarantees: workflow state never lost, activities at least once | Exactly-once side effects are not promised |
| 10 to 15 | Estimates: starts per second, events per workflow, open workflows, pending timers | Open workflows times duration sizes storage and timers |
| 15 to 20 | Model: workflow, activity, event history, task queue, timer | History is the source of truth |
| 20 to 25 | High-level diagram: frontend, history shards, matching, workers, visibility | Engine never runs user code |
| 25 to 38 | One deep dive: event history and replay with determinism (7.1) is the core; sharding and the transfer queue (7.2) or versioning (7.4) are good alternatives | Explain why non-deterministic code breaks replay |
| 38 to 42 | Failure modes: shard host loss, worker crash, poisoned workflow | Shard ownership moves; workers retry |
| 42 to 45 | Trade-offs and what changes at 10x | History store write rate binds first |

## 2. Requirements

<!-- Guidance: Separate core requirements (the design must meet them) from below-the-line items (explicitly out of scope). State guarantees precisely; "exactly once" needs a definition or it will be misread. -->

### 2.1 Core functional requirements

| ID | Requirement | Priority |
|---|---|---|
| FR-01 | *Example:* Start a workflow by type and ID; reject or reuse duplicates per policy | Must |
| FR-02 | *Example:* Run activities with retry policy (initial interval, backoff, maximum attempts) and timeouts | Must |
| FR-03 | *Example:* Durable timers from seconds to 365 days | Must |
| FR-04 | *Example:* Signals (external events into a running workflow) and queries (read workflow state) | Must |
| FR-05 | *Example:* Child workflows and cancellation that propagates to children | Must |
| FR-06 | *Example:* Search executions by type, status, start time and custom attributes | Must |
| FR-07 | *Example:* Safe deployment of changed workflow code while old executions are running | Must |
| FR-08 | *Example:* Reset a workflow to an earlier point after a bug fix | Should |
| FR-09 | *Example:* Schedules (cron-style starts) for workflows | Should |
| [FR-nn] | [Requirement] | [Must / Should / Could] |

### 2.2 Non-functional requirements

| ID | Quality | Target | Measured as |
|---|---|---|---|
| NFR-01 | Durability | *Example:* No acknowledged workflow event is lost (RPO 0 within a region) | Fault injection on storage |
| NFR-02 | Start latency | *Example:* p99 below 100 ms to acknowledge a start | Frontend timing |
| NFR-03 | Dispatch latency | *Example:* p99 below 50 ms from task scheduled to task delivered when a worker is polling | Matching metrics |
| NFR-04 | Throughput | *Example:* 5,000 starts/s and 100,000 history events/s at peak | Load test |
| NFR-05 | Timer accuracy | *Example:* Timers fire within 1 s of due time at p99 | Timer lag metric |
| NFR-06 | Isolation | *Example:* One namespace's surge does not raise another's dispatch latency above target | Multi-tenant load test |
| [NFR-nn] | [Quality] | [Target] | [Measurement] |

### 2.3 Below the line (out of scope)

- [*Example:* running user code inside the engine (workers run in teams' own services)]
- [*Example:* exactly-once side effects in external systems (activities are at least once; idempotency is the caller's job)]
- [*Example:* high-throughput stream processing (use the message queue and a stream processor)]
- [*Example:* simple scheduled jobs without multi-step state (Job Scheduler System Design Template, `system-design/design-job-scheduler`)]
- [*Example:* a visual workflow designer for non-engineers]

## 3. Capacity estimation

<!-- Guidance: Every estimate must end in a design consequence. Use Little's law (open items = arrival rate x time in system) for open workflows and timers. -->

### 3.1 Assumptions

| Input | Value | Source |
|---|---|---|
| Workflow starts per second, peak | *Example:* 5,000 | Product forecasts summed across teams |
| History events per workflow, average | *Example:* 20 | Pilot workflows |
| Event size, average | *Example:* 1 KB | Pilot payloads (large payloads by reference) |
| Average time a workflow stays open | *Example:* 2 hours (7,200 s) | Mix of short and multi-day workflows |
| Pending timers per open workflow | *Example:* 2 | Pilot workflows |
| Workflow tasks plus activity tasks per workflow | *Example:* 8 + 6 = 14 | Pilot workflows |
| Closed-history retention in primary store | *Example:* 7 days | Support needs |
| Storage replication factor | *Example:* 3 | Storage design |

### 3.2 History write rate

5,000 starts/s x 20 events = 100,000 events/s; at 1 KB that is 100 MB/s of
history, 300 MB/s of disk writes with 3 replicas.

**Consequence:** the history store must sustain six-figure appends per
second with low latency, which rules out a single relational primary. Use a
horizontally partitioned store and keep events small: payloads above
*Example:* 256 KB go to object storage and the event carries a reference.

### 3.3 Open workflows and timers

Open workflows = 5,000 starts/s x 7,200 s = 36 million. Pending timers =
36 million x 2 = 72 million.

**Consequence:** timers cannot live only in memory. Each shard keeps a
durable timer queue ordered by fire time and loads only the next
*Example:* few seconds of timers into memory.

### 3.4 Shards

With 4,096 shards, the average shard takes 100,000 / 4,096 = about 24
events/s and holds 36 million / 4,096 = about 8,800 open workflows. On 64
history hosts each host owns 64 shards, about 1,560 events/s.

**Consequence:** shard count is fixed at cluster creation because workflow
IDs hash to shards permanently. Pick a count that leaves room for 10x growth
(4,096 allows hosts to grow to several hundred before a shard becomes the
unit of contention).

### 3.5 Task dispatch

5,000 workflows/s x 14 tasks = 70,000 task dispatches/s.

**Consequence:** a single task queue partition cannot serve the busiest
queues. Partition each task queue (*Example:* 4 to 16 partitions for busy
queues) and prefer synchronous matching to a waiting poller over persisting
the task.

### 3.6 Retention

100 MB/s x 86,400 s x 7 days = about 60.5 TB of history, about 181 TB with 3
replicas.

**Consequence:** the primary store keeps only open workflows plus 7 days of
closed ones; closed histories are archived to object storage, where a year
costs much less per TB.

### 3.7 Summary

| Quantity | Estimate | Design consequence |
|---|---|---|
| History events | 100,000/s | Partitioned history store; payloads by reference |
| Open workflows | 36 million | Durable mutable state per workflow, cached on owner host |
| Pending timers | 72 million | Durable per-shard timer queues |
| Shards | 4,096 on 64 hosts | Fixed shard count, ownership via membership |
| Task dispatch | 70,000/s | Partitioned task queues, sync match |
| Retention | about 181 TB replicated for 7 days | Archive closed histories |

## 4. API design

<!-- Guidance: Define the client API (start, signal, query, cancel), the worker protocol (poll, complete) and the developer-facing SDK separately. -->

### 4.1 Interfaces

| Interface | Operation | Purpose |
|---|---|---|
| Client API | `StartWorkflow(namespace, workflow_id, type, task_queue, input, id_reuse_policy)` | Start an execution |
| Client API | `SignalWorkflow(workflow_id, signal_name, payload)` | Deliver an external event |
| Client API | `QueryWorkflow(workflow_id, query_name)` | Read state from the workflow (served by a worker) |
| Client API | `CancelWorkflow` / `TerminateWorkflow` | Graceful cancel versus hard stop |
| Client API | `ListWorkflows(query)` | Search visibility records |
| Worker API | `PollWorkflowTask(task_queue)` / `RespondWorkflowTaskCompleted(commands)` | Run workflow code and return commands |
| Worker API | `PollActivityTask(task_queue)` / `RespondActivityTaskCompleted(result)` | Run an activity |
| Worker API | `RecordActivityHeartbeat(details)` | Liveness and progress for long activities |

### 4.2 Workflow code example

```python
# Example: workflow and activity in an SDK of this engine (pseudocode)
@workflow
def fulfil_order(order):
    payment = activity(capture_payment, order, retry=Retry(initial=1s, backoff=2.0, max_attempts=10),
                       start_to_close=30s)
    try:
        activity(reserve_stock, order, start_to_close=60s)
    except ActivityFailed:
        activity(refund_payment, payment)          # compensation
        raise
    shipped = wait_for_signal("shipped", timeout=days(14))   # durable timer + signal
    if not shipped:
        activity(open_support_case, order)
    return "done"
```

Workflow code may only call engine APIs for anything non-deterministic:
time (`workflow.now()`), random values, UUIDs, sleeps and I/O all go through
the SDK so they are recorded in history.

### 4.3 Errors and idempotency

| Situation | Behaviour |
|---|---|
| Start with an ID that is already running | `ALREADY_STARTED` error, or return the running execution, per reuse policy |
| Signal to a closed workflow | `NOT_FOUND`; caller decides whether to start a new one |
| Activity completion after timeout | Rejected; the retry already owns the attempt |
| Duplicate completion from a retried worker call | Ignored using the task token and attempt number |

## 5. Data model

<!-- Guidance: The event history is the source of truth; everything else (mutable state, queues, visibility) is derived and must be rebuildable from it. -->

### 5.1 Entities

Each execution has one history and one mutable state row, both in the same
shard so they can be updated in one transaction.

```mermaid
erDiagram
  SHARD {
    int shard_id PK
    string owner_host
    bigint range_id
  }
  EXECUTION {
    string namespace PK
    string workflow_id PK
    string run_id PK
    int shard_id FK
    string status
    bigint next_event_id
    bigint db_version
  }
  HISTORY_EVENT {
    string run_id PK
    bigint event_id PK
    string event_type
    datetime event_time
    blob attributes
  }
  TIMER_TASK {
    int shard_id PK
    datetime fire_time PK
    bigint task_id PK
    string run_id
  }
  TRANSFER_TASK {
    int shard_id PK
    bigint task_id PK
    string run_id
    string target
  }
  VISIBILITY_RECORD {
    string run_id PK
    string workflow_type
    string status
    datetime start_time
    string search_attributes
  }
  SHARD ||--o{ EXECUTION : owns
  EXECUTION ||--|{ HISTORY_EVENT : "recorded as"
  SHARD ||--o{ TIMER_TASK : schedules
  SHARD ||--o{ TRANSFER_TASK : queues
  EXECUTION ||--|| VISIBILITY_RECORD : "indexed by"
```

### 5.2 Storage choices

| Data | Store | Why |
|---|---|---|
| History events, mutable state, timer and transfer tasks | *Example:* Wide-column or distributed SQL store, partitioned by shard | High append rate; per-shard transactions |
| Task queues (backlog only) | Same store, partitioned by task queue partition | Persist only when no poller is waiting |
| Visibility records | *Example:* Search index (fed asynchronously) | Flexible queries without loading the history store |
| Large payloads | Object storage, referenced from events | Keeps events small |
| Archived histories | Object storage, compressed | Cheap long retention |

### 5.3 Access patterns

| Query | Served by | Key or index | Partition rationale |
|---|---|---|---|
| Append events and update state for workflow W | History store | (`shard_id`, `workflow_id`, `run_id`) | Shard = hash(namespace, workflow_id) mod 4,096, so one workflow's writes stay in one shard transaction |
| Read full history for replay | History store | (`run_id`, `event_id` range) | Sequential read within a partition |
| Timers due in the next N seconds on shard S | Timer queue | (`shard_id`, `fire_time`, `task_id`) | Range scan per shard; no global timer index |
| Tasks to hand to matching or visibility | Transfer queue | (`shard_id`, `task_id`) | Written in the same transaction as the event (outbox) |
| Next task for task queue Q | Matching backlog | (`task_queue`, `partition`, `task_id`) | Partitions spread hot queues |
| Workflows of type T failed in the last hour | Visibility index | (`namespace`, `workflow_type`, `status`, `close_time`) | Search index, eventually consistent |

## 6. High-level design

<!-- Guidance: Show the engine services, the workers owned by product teams, and which component owns each kind of state. -->

The engine schedules work; workers in product teams' services execute it.

```mermaid
flowchart LR
  app([Client services]) --> fe[Frontend]
  subgraph engine["Engine cluster"]
    fe --> hist[History service, shards 0..4095]
    fe --> match[Matching service, task queues]
    hist --> match
    hist --> vis[Visibility indexer]
    hist --> arch[Archiver]
  end
  hist --> hdb[(History store)]
  match --> mdb[(Task backlog)]
  vis --> idx[(Search index)]
  arch --> obj[(Object storage)]
  subgraph team["Team services"]
    w1[Workflow worker]
    w2[Activity worker]
  end
  w1 -->|long poll| fe
  w2 -->|long poll| fe
  w2 --> ext[External APIs]
  ring[[Membership ring]] -.shard ownership.-> hist
```

| Component | Responsibility | Scales by |
|---|---|---|
| Frontend | Authentication, rate limits per namespace, routing to the owning shard | Stateless instances |
| History service | Owns shards; appends events, runs timer and transfer queues | Hosts; shards move between hosts |
| Matching service | Task queue partitions; matches tasks to long-polling workers | Partitions per queue |
| Workers | Run workflow code (replay) and activities | Owned and scaled by teams |
| Visibility indexer | Feed search index from transfer tasks | Index cluster |
| Archiver | Move closed histories to object storage after retention | Batch workers |
| Membership ring | Decide which host owns which shard | Gossip or coordination service |

## 7. Detailed design

<!-- Guidance: Go deep on the parts that are new, risky or contested. For each deep dive list the options, the choice and the reason. -->

### 7.1 Deep dive 1: event history, replay and determinism

| Option | How state survives a crash | Developer experience | Verdict |
|---|---|---|---|
| A. Explicit state machine in a database | Persist state after each step | Plumbing in every service | Status quo; error-prone |
| B. Checkpoint serialised program state | Snapshot memory periodically | Fragile across code changes | Rejected |
| C. Event history plus deterministic replay | Rebuild state by re-running code against recorded results | Plain code; strict rules | Chosen |

**Choice:** C. **Reason:** the history records every decision and result,
so state can always be rebuilt and audited, and the code reads like a normal
function. The cost is the determinism contract.

How one workflow task works:

```mermaid
sequenceDiagram
  participant H as History shard
  participant M as Matching
  participant W as Workflow worker
  H->>M: add workflow task (via transfer queue)
  W->>M: long poll task queue
  M-->>W: task with new events since last task
  W->>W: replay code from history (or resume cached state)
  W->>W: run until blocked, collect commands
  W->>H: complete task with commands (schedule activity, start timer)
  H->>H: append events, update state, write timer and transfer tasks in one transaction
```

Determinism rules for workflow code:

- Same history in, same commands out. Never read the system clock, random
  numbers, environment variables or configuration directly; use SDK
  equivalents that record their values.
- No direct I/O (network, files, database). Wrap it in an activity.
- No native threads or unordered iteration that changes command order; use
  SDK coroutines and ordered collections.
- On replay, the SDK compares produced commands with recorded events and
  fails the task with a non-determinism error on mismatch rather than
  corrupting state.
- Keep histories bounded: when a workflow loops (for example, a
  subscription that runs for years), restart it as a fresh run with its
  current state ("continue as new") before the history grows past a
  configured cap (*Example:* 10,000 events warning, 50,000 events hard
  limit).

Sticky execution: after a task completes, the worker keeps the workflow's
state in memory and the next task is offered first to that worker's own
queue, so the full history is replayed only after a cache miss or worker
restart.

### 7.2 Deep dive 2: history sharding and the transfer queue

| Option | Concurrency control | Verdict |
|---|---|---|
| A. Any host handles any workflow, locking in the database | Row locks per workflow | Lock contention and no ordering of timers |
| B. Partition by workflow, one owner per shard | Owner serialises updates; fencing token on writes | Chosen |
| C. One actor per workflow placed dynamically | Fine-grained placement | Too many placement records at 36 million open workflows |

**Choice:** B. **Reason:** one owner per shard makes appends ordered and
lets the owner cache mutable state. Each write carries the shard's
`range_id`; when ownership moves, the new owner increments it so writes
from the old owner fail (fencing).

Every state change that must cause work elsewhere (dispatch a task, update
visibility, signal a child) is written as a transfer task in the same
transaction as the events. A per-shard processor reads the transfer queue
and calls matching or the indexer, retrying until acknowledged. This is the
transactional outbox pattern: it avoids the dual-write problem where the
event is stored but the task is never dispatched. The timer queue works the
same way, keyed by fire time.

### 7.3 Deep dive 3: task queues and matching

- Workers long-poll a task queue; matching hands a new task directly to a
  waiting poller (sync match) and persists it only if no poller is waiting.
- Busy queues are split into partitions; pollers and tasks are spread across
  partitions, and idle partitions forward to a root partition so a single
  poller still receives every task.
- Rate limits per task queue protect downstream systems that activities
  call; they are configured by the owning team.
- Backlog age per queue is the main capacity signal for workers: a growing
  backlog means the team needs more workers, not that the engine is slow.

| Option for dispatch | Latency | Storage writes | Verdict |
|---|---|---|---|
| A. Always persist, workers poll storage | Higher | One write and delete per task | Simple but costly at 70,000 tasks/s |
| B. Push to workers | Low | None | Engine must track worker capacity |
| C. Long poll with sync match, persist on backlog | Low | Only when backlogged | Chosen |

### 7.4 Deep dive 4: versioning workflow code

Changing workflow code while executions are open breaks replay if the new
code produces different commands for an existing history.

| Option | How it works | Verdict |
|---|---|---|
| A. Never change running workflows; new type name per change | Old and new code side by side | Simple but clutters types |
| B. Version markers in code | `if workflow.version("add-fraud-check", 1) >= 1:` records which branch an execution took | Chosen for small changes |
| C. Worker build versioning | Executions stay pinned to the worker build that started them; new builds take new executions | Chosen for large changes |

**Choice:** B for small changes and C for large ones. **Reason:** markers
keep one code base and are replay-safe; build pinning avoids a branch
explosion for big rewrites. Before deploying, run a replay test that feeds
recent production histories (*Example:* 1,000 per workflow type) through the
new code and fails the build on any non-determinism error.

### 7.5 Retries, timeouts and guarantees

| Timeout | Meaning | Example |
|---|---|---|
| Schedule to start | Maximum time in the task queue before a worker picks it up | Detects missing workers |
| Start to close | Maximum time for one attempt | *Example:* 30 s for a payment call |
| Schedule to close | Maximum total time including retries | *Example:* 1 hour |
| Heartbeat | Maximum gap between heartbeats for long activities | *Example:* 20 s |

Guarantees:

- Workflow state transitions are exactly once with respect to the history:
  a command is recorded once, and replay never repeats a recorded event.
- Activities are at least once: a worker can crash after the side effect and
  before reporting completion. Activities must be idempotent; pass the
  workflow ID and activity ID as an idempotency key to external APIs.
- Failures that should not be retried (validation errors) are marked
  non-retryable by the activity so the workflow can compensate at once.

> **How real systems do it** (paraphrased from Uber Engineering, "Conducting
> Better Business with Uber's Open Source Orchestration Tool, Cadence",
> November 2019, and the Temporal documentation on workflow executions; see
> 12.4). The Uber write-up describes Cadence as multi-tenant middleware that
> brings its own persistence, queues and timers. Requests enter through a
> front-end layer; a history service owns shards of workflow state; a
> matching service hands tasks to workers; the reported storage back end was
> Cassandra, with queues kept in the same store. Teams write activities and
> combine them into workflows, and workers can be stopped and restarted
> without losing progress. The Temporal documentation explains how an
> execution resumes by replaying its event history, why workflow code must
> produce the same commands on replay, and that a single execution's history
> is limited in size, so long-lived workflows start new runs with
> continue-as-new. Lesson for this design: the engine stores and schedules,
> user code runs elsewhere, and bounded histories are part of the contract.

### 7.6 Visibility and archival

Visibility records are written from transfer tasks, so search can lag the
true state by seconds; screens that need exact status call
`DescribeWorkflow`, which reads the history store. After the retention
period, the archiver writes the compressed history to object storage and
deletes it from the primary store; the UI reads archived histories on
demand.

## 8. Scalability and performance

<!-- Guidance: Show how the design meets the numbers in section 3 and where it breaks first. -->

| Concern | Approach | Limit before redesign |
|---|---|---|
| History write rate | Partitioned store, batched appends per workflow task | *Example:* store cluster write throughput |
| Hot workflow (many signals) | Signals buffered and applied per task; cap signals per execution | *Example:* tens of events per second per workflow |
| Hot task queue | Partitions plus per-queue rate limits | *Example:* about 16 partitions per queue |
| Replay cost | Sticky execution; bounded histories | History length cap |
| Shard count | Fixed at creation | Rebuild cluster or migrate namespaces to grow beyond |
| Visibility | Separate index cluster | Index write rate |

Latency budget for a short workflow task (target p99 50 ms dispatch):

| Step | Budget |
|---|---|
| Transfer queue pickup | 10 ms |
| Matching sync match to a waiting poller | 5 ms |
| Network to worker | 5 ms |
| Worker execution with cached state | 10 ms |
| Completion and history append | 15 ms |
| Headroom | 5 ms |

## 9. Reliability, security and operations

<!-- Guidance: Name the failure modes, the threats and the signals operators watch. Each mitigation should be testable in a game day. -->

### 9.1 Failure modes

| Component | Failure | Detection | Blast radius | Mitigation | RTO |
|---|---|---|---|---|---|
| History host | Crash | Membership ring change | Its shards (about 1.6% of workflows on 64 hosts) pause | Shards move to other hosts; fencing stops stale writes | *Example:* 30 s |
| History store | Partition unavailable | Write errors per shard | Workflows in affected shards | Store replication; engine retries with backoff | Store RTO |
| Matching host | Crash | Poll errors | Queues on that host | Partitions reassigned; backlog persisted | *Example:* 30 s |
| Workers | Crash or not deployed | Schedule-to-start timeout, backlog age | That team's workflows | Tasks wait safely; alert the owning team | Team RTO |
| Workflow code | Non-determinism after deploy | Non-determinism error rate | Executions of that type | Replay tests in CI; roll back worker build; reset executions | *Example:* 15 min |
| Namespace | Runaway starts | Per-namespace rate | Other tenants | Per-namespace start and dispatch rate limits | *Example:* immediate |
| Region | Full outage | Probes | All workflows in region | [Single region / async replication of namespaces with failover] | [RTO] |
| [Component] | [Failure] | [Signal] | [Scope] | [Mitigation] | [RTO] |

### 9.2 Security

| Threat | Control |
|---|---|
| One team reading another's workflow data | Namespace isolation; authorisation on every API call |
| Sensitive data in history (permanent and replicated) | Payload codec that encrypts payloads client-side with team keys; engine stores ciphertext |
| Malicious signal or start | Authenticated clients; signal schemas validated in workflow code |
| Worker impersonation | Mutual TLS between workers and frontend; per-namespace credentials |
| Retention of personal data | Retention per namespace; archive encryption; deletion process for data-subject requests |
| [Threat] | [Control] |

### 9.3 Observability

| Signal | Source | Alert threshold |
|---|---|---|
| Task queue backlog age | Matching | *Example:* above 60 s for 5 min (team alert) |
| Schedule-to-start latency | Matching | *Example:* p99 above 1 s |
| Timer lag | History timer queue | *Example:* p99 above 1 s |
| Shard movement rate | Membership | More than [n] per hour |
| Non-determinism errors | Workers | Any after a deploy |
| Workflows stuck open beyond expected duration | Visibility | Per type, team-defined |

### 9.4 Cost estimate

| Item | Unit cost | Quantity | Monthly cost |
|---|---|---|---|
| History, matching and frontend hosts | [USD per instance-month] | *Example:* 64 + 16 + 8 | [USD] |
| History store | [USD per TB-month] | *Example:* about 181 TB replicated | [USD] |
| Search index | [USD per node-month] | [Nodes] | [USD] |
| Archive storage | [USD per TB-month] | [TB per year] | [USD] |
| **Total** | | | **[USD]** |

## 10. Trade-offs and alternatives

<!-- Guidance: For each major decision list the options, what each costs and why you chose one. Include how the design changes at ten times and one tenth of the load. -->

| Decision | Options | Chosen | Why | What we give up |
|---|---|---|---|---|
| Programming model | Declarative DSL (state-machine JSON or YAML); code with replay | Code with replay | Real language, tests, refactoring | Determinism rules to learn |
| Build or adopt | Build; adopt open-source engine; managed service | [Decide; default adopt] | Years of edge cases already solved | Less control over internals |
| State storage | Relational; wide-column; distributed SQL | [Decide] | Write rate and per-shard transactions | Operational expertise needed |
| Sharding | Fixed shards; dynamic placement | Fixed shards | Simple ownership and fencing | Shard count cannot grow in place |
| Activity guarantee | At most once; at least once | At least once | No silent loss | Activities must be idempotent |
| What changes at 10x | 50,000 starts/s, 1 million events/s | Multiple clusters with namespaces spread across them, larger history store, aggressive payload offload | History store write rate binds first | Cross-cluster routing layer |
| What changes at 0.1x | 500 starts/s | Single small cluster or managed service; a relational store is enough | Team cost dominates | Less headroom to grow |

## 11. Delivery plan and risks

<!-- Guidance: How the design reaches production safely, and what could stop it. -->

### 11.1 Milestones

| Milestone | Scope | Exit criterion | Target date |
|---|---|---|---|
| M1 | Engine cluster, one pilot workflow in shadow mode | Pilot outcomes match existing system for 14 days | YYYY-MM-DD |
| M2 | SDK guidance, replay tests in CI, versioning rules | Two teams ship workflows with replay tests | YYYY-MM-DD |
| M3 | Visibility, archival, per-namespace limits | Game day: history host loss and worker outage | YYYY-MM-DD |
| M4 | Order fulfilment migrated | Stuck-order rate below [target] | YYYY-MM-DD |

### 11.2 Rollout and rollback

[*Example:* Migrate one process at a time. New orders start in the engine
behind a flag; orders already in flight finish on the old path. Rollback
stops new starts in the engine; running executions continue to completion.]

### 11.3 Risks

| ID | Risk | Likelihood | Impact | Mitigation | Owner |
|---|---|---|---|---|---|
| R-01 | *Example:* Teams write non-deterministic code | High | High | SDK lint rules, replay tests, training | [@name] |
| R-02 | *Example:* Activities not idempotent cause duplicate charges | Medium | High | Idempotency key convention; review checklist | [@name] |
| R-03 | *Example:* Shard count too small for growth | Low | High | 4,096 shards; capacity review yearly | [@name] |
| R-04 | *Example:* Engine becomes a dependency for every team without an on-call model | Medium | Medium | Platform on-call, SLOs per namespace | [@name] |
| [R-nn] | [Risk] | [L / M / H] | [L / M / H] | [Mitigation] | [@name] |

## 12. Appendices

<!-- Guidance: Supporting detail, glossary, open questions and references. -->

### 12.1 Open questions

| ID | Question | Owner | Due |
|---|---|---|---|
| OQ-01 | *Example:* Do we need multi-region replication of namespaces in year one? | [@name] | YYYY-MM-DD |
| OQ-02 | *Example:* Which workflow types need history retention beyond 7 days in the primary store? | [@name] | YYYY-MM-DD |
| [OQ-nn] | [Question] | [@name] | YYYY-MM-DD |

### 12.2 Traceability

| Requirement | Design section | Verification |
|---|---|---|
| FR-02 | 7.5 | Activity retry tests with injected failures |
| FR-03, NFR-05 | 3.3, 7.2 | Timer lag under 72 million pending timers (scaled test) |
| FR-07 | 7.4 | Replay test in CI on production histories |
| NFR-01 | 7.2, 9.1 | Kill history host mid-transaction; verify no lost events |
| NFR-06 | 7.3, 9.1 | Noisy-namespace load test |

### 12.3 Glossary

| Term | Meaning |
|---|---|
| Workflow execution | One run of a workflow type, identified by workflow ID and run ID |
| Event history | Append-only list of events that records everything an execution did |
| Replay | Re-running workflow code against its history to rebuild state |
| Activity | A unit of side-effecting work run by a worker, with retries and timeouts |
| Task queue | Named queue from which workers poll workflow or activity tasks |
| Shard | Fixed partition of executions owned by one history host at a time |
| Continue as new | Ending a run and starting a fresh run with carried-over state to bound history |
| Transfer queue | Per-shard outbox of tasks written in the same transaction as events |

### 12.4 References

| Reference | Use |
|---|---|
| Uber Engineering blog, "Conducting Better Business with Uber's Open Source Orchestration Tool, Cadence" (12 November 2019) | Inspiration for the box in 7.5 (paraphrased) |
| Temporal documentation, "Workflow Execution" and "Workflow Execution limits" (docs.temporal.io) | Event history, replay, continue-as-new, history limits used as reference for 7.1 |
| Cadence project (cadenceworkflow.io); Uber Engineering blog, "Cadence Workflow Joins the Cloud Native Computing Foundation" (2025) | Background on the open-source engine |
| Job Scheduler System Design Template (`system-design/design-job-scheduler`) | Scheduling independent jobs |
| Message Queue System Design Template (`system-design/design-message-queue`) | Queue semantics for activities that publish events |
| Event Sourcing and CQRS Template (`architecture/event-sourcing-cqrs`) | Related pattern of event-sourced state |
| Payment Ledger System Design Template (`system-design/design-payment-ledger`) | Idempotency for payment activities |

## Standard and licence note

Based on common industry practice for durable execution engines; no formal
standard governs this class of system. The Uber and Temporal material is
used as inspiration only and is paraphrased, not copied; product-specific
limits change between releases, so check the current documentation of any
engine you adopt. All numbers are illustrative examples. Original text
written for this collection. Template licence: MIT.