System Design: [Workflow orchestration engine name]

Reference design for a durable workflow orchestration engine: a platform that runs long-lived business processes written as ordinary code (order fulfilment, account onboarding, payment sagas, data pipelines) and guarantees that they resume exactly where they stopped after crashes, deploys and outages. It covers the event history and replay model, deterministic workflow code, activities, durable timers, task queues and matching, retries and timeouts, workflow code versioning, visibility search and sharding of history. It is distinct from the Job Scheduler System Design Template (system-design/design-job-scheduler), which runs independent jobs on a schedule; this engine coordinates multi-step stateful processes. It follows the structure of the System Design Document template and is pre-filled with a worked design to adapt. Replace every [bracketed] value and every row marked Example; all numbers are illustrative assumptions, not benchmarks.

Document control

Approvals

Revision history

Contents

1. Overview
2. Requirements
3. Capacity estimation
4. API design
5. Data model
6. High-level design
7. Detailed design
8. Scalability and performance
9. Reliability, security and operations
10. Trade-offs and alternatives
11. Delivery plan and risks
12. Appendices

1. Overview

1.1 Problem statement

[Example: Order fulfilment spans payment, inventory, warehouse and shipping services and can take from minutes to 14 days. Today the state machine is spread across 9 queues, 4 cron jobs and a status column in the orders table. After a broker outage last month, 3,200 orders were stuck in "payment captured, not reserved" and took two engineers four days to repair by hand. Every new process repeats the same plumbing: retries, timeouts, compensation and status pages.]

1.2 Proposed solution

[Example: A shared engine stores each workflow execution as an append-only event history. Workflow code runs in worker processes owned by product teams; it is replayed from the history to rebuild state, so it must be deterministic. Side effects run as activities with their own retry and timeout policies. The engine provides durable timers, signals and queries, and dispatches workflow and activity tasks through partitioned task queues. Histories are sharded by workflow ID across a fixed number of shards, each owned by one history host at a time. A visibility store indexes executions for search, and closed histories are archived to object storage after a retention period.]

1.3 Decisions requested from reviewers

1.4 45-minute interview path

2. Requirements

2.1 Core functional requirements

2.2 Non-functional requirements

2.3 Below the line (out of scope)

• [Example: running user code inside the engine (workers run in teams' own services)]
• [Example: exactly-once side effects in external systems (activities are at least once; idempotency is the caller's job)]
• [Example: high-throughput stream processing (use the message queue and a stream processor)]
• [Example: simple scheduled jobs without multi-step state (Job Scheduler System Design Template, system-design/design-job-scheduler)]
• [Example: a visual workflow designer for non-engineers]

3. Capacity estimation

3.1 Assumptions

3.2 History write rate

5,000 starts/s x 20 events = 100,000 events/s; at 1 KB that is 100 MB/s of history, 300 MB/s of disk writes with 3 replicas.

Consequence: the history store must sustain six-figure appends per second with low latency, which rules out a single relational primary. Use a horizontally partitioned store and keep events small: payloads above Example: 256 KB go to object storage and the event carries a reference.

3.3 Open workflows and timers

Open workflows = 5,000 starts/s x 7,200 s = 36 million. Pending timers = 36 million x 2 = 72 million.

Consequence: timers cannot live only in memory. Each shard keeps a durable timer queue ordered by fire time and loads only the next Example: few seconds of timers into memory.

3.4 Shards

With 4,096 shards, the average shard takes 100,000 / 4,096 = about 24 events/s and holds 36 million / 4,096 = about 8,800 open workflows. On 64 history hosts each host owns 64 shards, about 1,560 events/s.

Consequence: shard count is fixed at cluster creation because workflow IDs hash to shards permanently. Pick a count that leaves room for 10x growth (4,096 allows hosts to grow to several hundred before a shard becomes the unit of contention).

3.5 Task dispatch

5,000 workflows/s x 14 tasks = 70,000 task dispatches/s.

Consequence: a single task queue partition cannot serve the busiest queues. Partition each task queue (Example: 4 to 16 partitions for busy queues) and prefer synchronous matching to a waiting poller over persisting the task.

3.6 Retention

100 MB/s x 86,400 s x 7 days = about 60.5 TB of history, about 181 TB with 3 replicas.

Consequence: the primary store keeps only open workflows plus 7 days of closed ones; closed histories are archived to object storage, where a year costs much less per TB.

3.7 Summary

4. API design

4.1 Interfaces

4.2 Workflow code example

# Example: workflow and activity in an SDK of this engine (pseudocode)
@workflow
def fulfil_order(order):
    payment = activity(capture_payment, order, retry=Retry(initial=1s, backoff=2.0, max_attempts=10),
                       start_to_close=30s)
    try:
        activity(reserve_stock, order, start_to_close=60s)
    except ActivityFailed:
        activity(refund_payment, payment)          # compensation
        raise
    shipped = wait_for_signal("shipped", timeout=days(14))   # durable timer + signal
    if not shipped:
        activity(open_support_case, order)
    return "done"

Workflow code may only call engine APIs for anything non-deterministic: time (workflow.now()), random values, UUIDs, sleeps and I/O all go through the SDK so they are recorded in history.

4.3 Errors and idempotency

5. Data model

5.1 Entities

Each execution has one history and one mutable state row, both in the same shard so they can be updated in one transaction.

5.2 Storage choices

5.3 Access patterns

6. High-level design

The engine schedules work; workers in product teams' services execute it.

7. Detailed design

7.1 Deep dive 1: event history, replay and determinism

Choice: C. Reason: the history records every decision and result, so state can always be rebuilt and audited, and the code reads like a normal function. The cost is the determinism contract.

How one workflow task works:

Determinism rules for workflow code:

• Same history in, same commands out. Never read the system clock, random numbers, environment variables or configuration directly; use SDK equivalents that record their values.
• No direct I/O (network, files, database). Wrap it in an activity.
• No native threads or unordered iteration that changes command order; use SDK coroutines and ordered collections.
• On replay, the SDK compares produced commands with recorded events and fails the task with a non-determinism error on mismatch rather than corrupting state.
• Keep histories bounded: when a workflow loops (for example, a subscription that runs for years), restart it as a fresh run with its current state ("continue as new") before the history grows past a configured cap (Example: 10,000 events warning, 50,000 events hard limit).

Sticky execution: after a task completes, the worker keeps the workflow's state in memory and the next task is offered first to that worker's own queue, so the full history is replayed only after a cache miss or worker restart.

7.2 Deep dive 2: history sharding and the transfer queue

Choice: B. Reason: one owner per shard makes appends ordered and lets the owner cache mutable state. Each write carries the shard's range_id; when ownership moves, the new owner increments it so writes from the old owner fail (fencing).

Every state change that must cause work elsewhere (dispatch a task, update visibility, signal a child) is written as a transfer task in the same transaction as the events. A per-shard processor reads the transfer queue and calls matching or the indexer, retrying until acknowledged. This is the transactional outbox pattern: it avoids the dual-write problem where the event is stored but the task is never dispatched. The timer queue works the same way, keyed by fire time.

7.3 Deep dive 3: task queues and matching

• Workers long-poll a task queue; matching hands a new task directly to a waiting poller (sync match) and persists it only if no poller is waiting.
• Busy queues are split into partitions; pollers and tasks are spread across partitions, and idle partitions forward to a root partition so a single poller still receives every task.
• Rate limits per task queue protect downstream systems that activities call; they are configured by the owning team.
• Backlog age per queue is the main capacity signal for workers: a growing backlog means the team needs more workers, not that the engine is slow.

7.4 Deep dive 4: versioning workflow code

Changing workflow code while executions are open breaks replay if the new code produces different commands for an existing history.

Choice: B for small changes and C for large ones. Reason: markers keep one code base and are replay-safe; build pinning avoids a branch explosion for big rewrites. Before deploying, run a replay test that feeds recent production histories (Example: 1,000 per workflow type) through the new code and fails the build on any non-determinism error.

7.5 Retries, timeouts and guarantees

Guarantees:

• Workflow state transitions are exactly once with respect to the history: a command is recorded once, and replay never repeats a recorded event.
• Activities are at least once: a worker can crash after the side effect and before reporting completion. Activities must be idempotent; pass the workflow ID and activity ID as an idempotency key to external APIs.
• Failures that should not be retried (validation errors) are marked non-retryable by the activity so the workflow can compensate at once.

How real systems do it (paraphrased from Uber Engineering, "Conducting Better Business with Uber's Open Source Orchestration Tool, Cadence", November 2019, and the Temporal documentation on workflow executions; see 12.4). The Uber write-up describes Cadence as multi-tenant middleware that brings its own persistence, queues and timers. Requests enter through a front-end layer; a history service owns shards of workflow state; a matching service hands tasks to workers; the reported storage back end was Cassandra, with queues kept in the same store. Teams write activities and combine them into workflows, and workers can be stopped and restarted without losing progress. The Temporal documentation explains how an execution resumes by replaying its event history, why workflow code must produce the same commands on replay, and that a single execution's history is limited in size, so long-lived workflows start new runs with continue-as-new. Lesson for this design: the engine stores and schedules, user code runs elsewhere, and bounded histories are part of the contract.

7.6 Visibility and archival

Visibility records are written from transfer tasks, so search can lag the true state by seconds; screens that need exact status call DescribeWorkflow, which reads the history store. After the retention period, the archiver writes the compressed history to object storage and deletes it from the primary store; the UI reads archived histories on demand.

8. Scalability and performance

Latency budget for a short workflow task (target p99 50 ms dispatch):

9. Reliability, security and operations

9.1 Failure modes

9.2 Security

9.3 Observability

9.4 Cost estimate

10. Trade-offs and alternatives

11. Delivery plan and risks

11.1 Milestones

11.2 Rollout and rollback

[Example: Migrate one process at a time. New orders start in the engine behind a flag; orders already in flight finish on the old path. Rollback stops new starts in the engine; running executions continue to completion.]

11.3 Risks

12. Appendices

12.1 Open questions

12.2 Traceability

12.3 Glossary

12.4 References

Standard and licence note

Based on common industry practice for durable execution engines; no formal standard governs this class of system. The Uber and Temporal material is used as inspiration only and is paraphrased, not copied; product-specific limits change between releases, so check the current documentation of any engine you adopt. All numbers are illustrative examples. Original text written for this collection. Template licence: MIT.