# System Design: [Agent orchestration platform name]

> Reference design for a multi-tenant platform that runs AI agents as durable,
> resumable executions. It covers durable execution of agent runs, checkpoints
> and resumption, tool calls over the Model Context Protocol (MCP, revision
> 2026-07-28, with the optional Tasks extension for long-running tools),
> agent-to-agent calls over the Agent2Agent (A2A) protocol 1.0.1,
> human-in-the-loop pauses, tenant isolation, budgets, observability and
> evaluation hooks. Agent behaviour design (roles, prompts, hand-offs) is
> covered by the Multi-Agent System Design Template
> (`ai-data/multi-agent-system-design`). It follows the 12-section structure of
> the System Design Document Template (`system-design/system-design-document`)
> and is pre-filled with a worked design to adapt. Replace every
> `[bracketed]` value and every row marked *Example*; all numbers are
> illustrative, not benchmarks.

<!--
Guidance: How to use this template
- Treat an agent run as a long-lived workflow, not a request. Runs last
  seconds to days, wait for people and tools, and must survive deploys and
  crashes without repeating side effects (7.2).
- Record every non-deterministic result (model output, tool result, clock,
  random value) in the run journal. Replaying a model call gives a different
  answer, so a resumed run must read the recorded one.
- Decide budgets and isolation per tenant before you design scheduling: they
  set the queue model, the worker pools and the kill switches (7.5).
- Check the current MCP revision, the status of its Tasks extension and the
  A2A specification version before approval; all three move quickly.
-->

## Document control

<!-- Guidance: A design is a proposal until approved. Record who decided and which protocol versions the design commits to. -->

| Field | Value |
|---|---|
| Document ID | [SDD-AOP-001] |
| System / feature | [Agent orchestration platform] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Implemented / Superseded] |
| Owner | [@owner] |
| Approvers / reviewers | [@reviewer-1], [@reviewer-2] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal] |
| Protocol versions | [MCP 2026-07-28 (Tasks extension: yes / no)], [A2A 1.0.1, protocol version 1.0] |
| Model access | [Through the LLM gateway (`system-design/design-llm-gateway`) / direct] |
| Related | [Multi-agent system design], [Agentic threat model], [Agent evaluation plan], [Human oversight design] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Tech lead, agent platform | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| Staff / principal engineer | [@name] | [Decision] | YYYY-MM-DD |
| Security | [@name] | [Decision] | YYYY-MM-DD |
| SRE / operations | [@name] | [Decision] | YYYY-MM-DD |
| AI governance / risk | [@name] | [Decision] | YYYY-MM-DD |

## Revision history

<!-- Guidance: Record design changes, not wording changes. -->

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft for design review |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Switched long tool calls to the MCP Tasks extension |
| 0.3 | YYYY-MM-DD | [@name] | *Example:* Added per-run budget in currency next to token and step limits |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved |

## Contents

<!-- Guidance: Keep aligned with the headings. -->

1. [Overview](#1-overview)
2. [Requirements](#2-requirements)
3. [Capacity estimation](#3-capacity-estimation)
4. [API design](#4-api-design)
5. [Data model](#5-data-model)
6. [High-level design](#6-high-level-design)
7. [Detailed design](#7-detailed-design)
8. [Scalability and performance](#8-scalability-and-performance)
9. [Reliability, security and operations](#9-reliability-security-and-operations)
10. [Trade-offs and alternatives](#10-trade-offs-and-alternatives)
11. [Delivery plan and risks](#11-delivery-plan-and-risks)
12. [Appendices](#12-appendices)

## 1. Overview

<!-- Guidance: One page a busy reviewer can read alone: problem, solution in a paragraph, decisions requested. -->

### 1.1 Problem statement

[*Example:* Twelve teams build agents with different frameworks, each running
the agent loop inside a web request or a background job. When a pod restarts,
running agents are lost and users must start again; three incidents last
quarter came from agents that repeated a side-effecting tool call (a refund,
a ticket update) after a retry. Approvals by people are implemented as
polling loops that hold a worker for hours. Nobody can cap what one run may
spend, and traces stop at the framework boundary, so a failing multi-agent
task cannot be followed end to end.]

### 1.2 Proposed solution

[*Example:* A shared platform runs each agent run as a durable execution. The
agent loop is split into steps (model call, tool call, sub-agent call, human
wait); every step is written to an append-only run journal before its result
is used. Workers are stateless: any worker can resume any run from its last
checkpoint plus the journal. Tools are reached over MCP through a tool broker
that injects scoped credentials and records each call with an idempotency key;
long tools use the MCP Tasks extension so the run waits without holding a
worker. Remote agents are called over A2A. Human approvals park the run with
zero compute until a decision arrives. Every run has a budget in tokens,
steps, wall-clock time and currency, enforced before each step.]

### 1.3 Decisions requested from reviewers

| ID | Question | Recommended option | Section |
|---|---|---|---|
| Q-01 | *Example:* Durable execution model? | Step journal with recorded results plus periodic state checkpoints | 7.2 |
| Q-02 | *Example:* Build on a workflow engine or own the journal? | Own journal on a durable store for v1; revisit an engine at 10x | 7.2, 10 |
| Q-03 | *Example:* Long tool calls? | MCP Tasks extension where the server supports it; platform-side async wrapper otherwise | 7.3 |
| Q-04 | *Example:* Isolation unit? | Tenant: separate queues, quotas, credentials and storage prefixes | 7.5 |
| Q-05 | *Example:* What happens when a budget is exhausted mid-step? | Finish the current step, then pause for approval or stop with a partial result | 7.5 |
| [Q-nn] | [Question] | [Option] | [Section] |

### 1.4 45-minute interview path

<!-- Guidance: Use this order for an interview or a time-boxed review. Pick one deep dive. -->

| Minutes | Topic | What to show |
|---|---|---|
| 0 to 5 | Scope | A run is a workflow; the platform runs agents, it does not design them |
| 5 to 10 | Requirements | Core list (2.1), durability and isolation numbers (2.2), below the line (2.3) |
| 10 to 15 | Estimates | Active runs, checkpoint write rate, parked runs (3.2 to 3.4) and their consequences |
| 15 to 20 | API | Start, stream events, approve, cancel (4) |
| 20 to 28 | High-level design | Diagram (6) and the step sequence (7.1) |
| 28 to 40 | One deep dive | Durable execution and exactly-once side effects (7.2) is the strongest choice; budgets and isolation (7.5) for multi-tenant questions |
| 40 to 45 | Wrap-up | Failure modes (9.1) and what changes at 10x (10) |

## 2. Requirements

<!-- Guidance: Split core requirements from what is explicitly below the line. Give every quality a number and a measurement. -->

### 2.1 Core functional requirements

| ID | Requirement | Priority |
|---|---|---|
| FR-01 | *Example:* Start an agent run from a registered agent definition and an input; return a run ID immediately | Must |
| FR-02 | *Example:* Stream run events (steps, partial outputs, status) to the caller and allow reconnection from the last event | Must |
| FR-03 | *Example:* Persist every step so a run resumes after a worker crash or deploy without repeating completed steps | Must |
| FR-04 | *Example:* Call tools on MCP servers registered per tenant, with per-run scoped credentials | Must |
| FR-05 | *Example:* Call remote agents over A2A and expose selected platform agents as A2A servers | Should |
| FR-06 | *Example:* Pause a run for a human decision, with timeout and escalation, and resume on approval or rejection | Must |
| FR-07 | *Example:* Per-run and per-tenant budgets in tokens, steps, wall-clock time and currency | Must |
| FR-08 | *Example:* Cancel a run; cancellation reaches in-flight tools and sub-agents | Must |
| FR-09 | *Example:* Evaluation hooks: export run trajectories and attach graders online and offline | Should |
| FR-10 | *Example:* Replay a finished run from its journal for debugging, without side effects | Should |
| [FR-nn] | [Requirement] | [Must / Should / Could] |

### 2.2 Non-functional requirements

| ID | Quality | Target | Measured as |
|---|---|---|---|
| NFR-01 | Durability | *Example:* No completed step lost after it is acknowledged | Chaos test: kill workers during runs |
| NFR-02 | Side-effect safety | *Example:* Zero duplicate executions of tools marked non-idempotent | Tool broker idempotency audit |
| NFR-03 | Resume time | *Example:* p95 at or below 5 s from worker loss to the run continuing | Chaos test timing |
| NFR-04 | Scheduling overhead | *Example:* p95 at or below 100 ms between a step finishing and the next step starting | Span timing |
| NFR-05 | Isolation | *Example:* One tenant at its limits raises other tenants' scheduling delay p95 by at most 20% | Mixed-tenant load test |
| NFR-06 | Parked cost | *Example:* A run waiting for a person or a task holds no worker and no model capacity | Worker metrics during waits |
| NFR-07 | Availability | *Example:* 99.9% for run start and event APIs | SLO on API responses |
| [NFR-nn] | [Quality] | [Target] | [Measurement] |

### 2.3 Below the line (out of scope)

- [*Example:* Agent behaviour, roles and prompts (see Multi-Agent System Design Template (`ai-data/multi-agent-system-design`) and AI Agent Specification Template (`ai-data/ai-agent-spec`))]
- [*Example:* Model serving and provider routing (see LLM Gateway System Design Template (`system-design/design-llm-gateway`))]
- [*Example:* Running untrusted generated code (see Agent Code Execution Sandbox System Design Template (`system-design/design-agent-sandbox-execution`))]
- [*Example:* Building MCP servers; the platform consumes them (see MCP Server Specification Template (`ai-data/mcp-server-spec`))]
- [*Example:* End-user chat interface (see LLM Chat Assistant System Design Template (`system-design/design-llm-chat-assistant`))]
- [Item deferred, with the release that will take it]

## 3. Capacity estimation

<!-- Guidance: Every number must end in a consequence. Show inputs, formulas and results. -->

### 3.1 Assumptions

All values are *Example* planning inputs.

| Input | Value | Source |
|---|---|---|
| Runs per day | 1.2 million | Tenant forecasts |
| Peak-to-average ratio | 3 | Hourly pattern of pilot tenants |
| Steps per run (mean) | 15 | Pilot traces |
| Share of steps that are model calls | 60% | Pilot traces |
| Mean active run duration, excluding waits | 4 min | Pilot traces |
| Checkpoint plus journal entry per step (mean) | 40 KB | Message deltas, tool results, metadata |
| Runs that pause for a human | 5% | Pilot |
| Mean human wait | 2 h | Pilot |
| Tool calls that are long-running | 10% | Tool inventory |
| Concurrent runs per worker | 200 | Load test (async I/O) |
| Mean model and tool cost per run | USD 0.04 | Gateway metering |

### 3.2 Run and step rates

| Quantity | Calculation | Result | Consequence |
|---|---|---|---|
| Average runs per second | 1,200,000 / 86,400 | about 13.9 | |
| Peak runs per second | 13.9 x 3 | about 41.7 | Run start API is light; it only writes a run record and enqueues |
| Steps per day | 1,200,000 x 15 | 18 million | |
| Peak steps per second | 41.7 x 15 | about 625 | The scheduler dispatches about 625 steps per second at peak, so a queue per tenant class with batching is enough; no exotic scheduler |
| Model calls per day | 18 million x 0.6 | 10.8 million | The platform is one large tenant of the LLM gateway and needs its own quota there |
| Peak model calls per second | 625 x 0.6 | 375 | Concurrency limits per tenant protect the gateway quota |

### 3.3 Active runs and workers

Little's law: active runs = arrival rate x active duration. Average:
13.9 x 240 s = about 3,340 active runs; peak: 41.7 x 240 s = about 10,000.
At 200 concurrent runs per worker, peak needs 50 workers at full load, or
72 at a 70% utilisation target (50 / 0.7 = 71.4, rounded up). **Consequence:**
workers must be asynchronous and stateless; a run is never pinned to a worker,
so scaling and deploys move runs freely.

### 3.4 Parked runs

Runs pausing for a person: 1.2 million x 5% = 60,000 per day. With a mean wait
of 2 hours, the number parked at any time is 60,000 x 2 / 24 = 5,000.
**Consequence:** 5,000 waiting runs would hold 25 workers if they stayed in
memory; parking must persist state and release the worker, and the wake-up
must come from a durable timer or an approval event.

### 3.5 Journal storage

| Quantity | Calculation | Result | Consequence |
|---|---|---|---|
| Journal per day | 18 million x 40 KB | 720 GB | Store deltas, not full state; large tool results go to object storage with a reference |
| Peak write rate | 625 x 40 KB per second | 25 MB/s | Partition the journal by run ID; one partition per run keeps ordering simple |
| Hot retention, 30 days uncompacted | 720 GB x 30 | 21.6 TB | Too much for the hot store: compact finished runs to a final snapshot plus an archived journal after 7 days, which keeps it near 720 GB x 7, about 5 TB |
| Long-running tool tasks per day | 7.2 million tool calls x 10% | 720,000 | Prefer task status notifications; poll with backoff otherwise |

### 3.6 Spend

1.2 million runs x USD 0.04 = USD 48,000 per day, about USD 1.44 million per
30 days. **Consequence:** a runaway loop that multiplies a run's cost by 100
is expensive within minutes, so budgets are checked before every step, not
in a nightly report.

### 3.7 Summary

| Quantity | Estimate | Design consequence |
|---|---|---|
| Peak steps | about 625 per second | Simple per-class queues with tenant fairness |
| Peak active runs | about 10,000 | 72 stateless async workers |
| Parked runs | about 5,000 at any time | Zero-compute parking with durable timers |
| Journal | 720 GB per day | Delta journal, compaction, object storage for large results |
| Spend | about USD 48,000 per day | Pre-step budget enforcement |

## 4. API design

<!-- Guidance: Define the contract: resources, streaming, errors, idempotency and versioning. -->

### 4.1 Endpoints

| Method and path | Purpose |
|---|---|
| `POST /v1/runs` | Start a run; body names the agent, version, input and optional budget; `Idempotency-Key` header required |
| `GET /v1/runs/{id}` | Run status, current step, budget used |
| `GET /v1/runs/{id}/events` | Server-sent events stream; supports `Last-Event-ID` for resumption |
| `POST /v1/runs/{id}/approvals/{approval_id}` | Approve, reject or edit a pending action |
| `POST /v1/runs/{id}/cancel` | Cooperative cancellation |
| `POST /v1/runs/{id}/replay` | Debug replay from the journal with tools stubbed |
| `GET /v1/runs/{id}/trajectory` | Export steps for evaluation |
| `POST /a2a/{agent}` | A2A endpoint for platform agents exposed to other systems |

### 4.2 Run states

```mermaid
stateDiagram-v2
  [*] --> queued
  queued --> running
  running --> waiting_tool: long tool task
  waiting_tool --> running: task completed or failed
  running --> waiting_human: approval needed
  waiting_human --> running: approved or edited
  waiting_human --> failed: rejected or timed out
  running --> completed
  running --> failed
  running --> cancelled
  waiting_tool --> cancelled
  waiting_human --> cancelled
  completed --> [*]
  failed --> [*]
  cancelled --> [*]
```

The platform maps its states to the A2A task states when a run is exposed
over A2A: `waiting_human` maps to `TASK_STATE_INPUT_REQUIRED`, `running` and
`waiting_tool` to `TASK_STATE_WORKING`, and the terminal states to
`TASK_STATE_COMPLETED`, `TASK_STATE_FAILED` and `TASK_STATE_CANCELED`.

### 4.3 Errors

| Status | Problem type | Meaning |
|---|---|---|
| 400 | `invalid-input` | Input does not match the agent's input schema |
| 403 | `agent-not-allowed` | Tenant may not run this agent or version |
| 409 | `run-not-waiting` | Approval sent to a run that is not waiting for it |
| 429 | `tenant-concurrency-exceeded` | Too many active runs for the tenant |
| 402 | `budget-exhausted` | Tenant budget exhausted; runs cannot start |

Errors use RFC 9457 problem details. A repeated `POST /v1/runs` with the same
`Idempotency-Key` returns the existing run instead of starting a second one.

## 5. Data model

<!-- Guidance: Choose storage by access pattern. List each query, the key that serves it and why the partition key was chosen. -->

### 5.1 Entities

The run journal is the source of truth; the run record is a projection that
can be rebuilt from it.

```mermaid
erDiagram
  TENANT ||--o{ AGENT_DEFINITION : registers
  AGENT_DEFINITION ||--o{ RUN : "instantiated as"
  RUN ||--o{ JOURNAL_ENTRY : records
  RUN ||--o{ CHECKPOINT : snapshots
  RUN ||--o{ APPROVAL : requests
  RUN ||--o{ TOOL_CALL : makes
  TENANT ||--o{ TOOL_REGISTRATION : allows
  RUN {
    string run_id PK
    string tenant_id
    string agent_version
    string status
    int tokens_used
    decimal cost_used
  }
  JOURNAL_ENTRY {
    string run_id PK
    int seq PK
    string step_type
    string idempotency_key
    string result_ref
  }
  CHECKPOINT {
    string run_id PK
    int at_seq PK
    string state_ref
  }
  APPROVAL {
    string approval_id PK
    string run_id FK
    string action_summary
    string decision
    datetime expires_at
  }
  TOOL_CALL {
    string idempotency_key PK
    string run_id FK
    string server
    string tool
    string task_id
    string status
  }
```

### 5.2 Access patterns

| ID | Query | Key or index that serves it | Partition key and rationale |
|---|---|---|---|
| AP-01 *(Example)* | Load a run to resume it | Latest `CHECKPOINT` by `run_id`, then `JOURNAL_ENTRY` where `seq > at_seq` | `run_id`: all of one run's entries in one partition, ordered by `seq` |
| AP-02 *(Example)* | Append a step | `JOURNAL_ENTRY (run_id, seq)` with conditional write on `seq` | Conditional write gives one writer per run (fencing) |
| AP-03 *(Example)* | Has this tool call already run? | `TOOL_CALL` by `idempotency_key` | Key derived from run ID and step number, so a replayed step finds it |
| AP-04 *(Example)* | List a tenant's active runs | Secondary index `(tenant_id, status)` | Tenant-scoped dashboards and concurrency checks |
| AP-05 *(Example)* | Pending approvals for a reviewer group | Index `(group, decision = pending, expires_at)` | Reviewer inbox sorted by deadline |
| AP-06 *(Example)* | Wake runs whose timer fired | Timer index by `fire_at` bucket (per minute) | Bucketing avoids a hot single index |
| AP-07 *(Example)* | Export a trajectory | Journal by `run_id`, joined to traces by trace ID | Read-only; served from the archive for old runs |

### 5.3 Storage choices

| Data | Store | Notes |
|---|---|---|
| Runs, approvals, tool registrations | Relational or document database | Strong consistency for status changes |
| Journal and checkpoints | Partitioned key-value or log store with conditional writes | Ordered per run |
| Large tool results and attachments | Object storage, per-tenant prefix and key | Journal holds a reference and hash |
| Timers | Durable timer service or indexed table | Wakes parked runs |
| Traces | Tracing backend | Linked by trace ID in each journal entry |

## 6. High-level design

<!-- Guidance: One diagram showing every component on the run path and where state lives. -->

```mermaid
flowchart LR
  caller([Caller or UI]) --> api[Run API]
  api --> store[(Runs and journal)]
  api --> sched[Scheduler: tenant-fair queues]
  sched --> w[Stateless workers: agent loop]
  w --> store
  w --> gw[LLM gateway]
  w --> broker[Tool broker]
  broker --> mcp1[MCP server: tickets]
  broker --> mcp2[MCP server: data, Tasks extension]
  broker --> vault[(Scoped credentials)]
  w --> a2a[A2A client]
  a2a --> remote[Remote agent]
  w --> hitl[Approval service]
  hitl --> reviewer([Reviewer])
  hitl --> timers[(Durable timers)]
  timers --> sched
  w --> events[[Run events]]
  events --> caller
  w --> otel[[Traces, metrics, trajectories]]
  otel --> evals[Evaluation hooks]
```

| Component | Responsibility | Scales by |
|---|---|---|
| Run API | Start, status, events, approvals, cancel | Horizontally |
| Scheduler | Tenant-fair queues, concurrency limits, wake-ups | Partitioned by tenant hash |
| Workers | Execute one step at a time for many runs; write the journal | Active runs (3.3) |
| Tool broker | MCP client, credential injection, idempotency, Tasks polling | Tool calls per second |
| Approval service | Approval requests, reviewer inbox, timeouts, escalation | Pending approvals |
| A2A client and server | Remote agent calls; exposes platform agents | Calls per second |
| Evaluation hooks | Online graders and trajectory export | Sampled share of runs |

## 7. Detailed design

<!-- Guidance: Go deep on the hard problems; each deep dive lists options, the choice and the reason. -->

### 7.1 Step execution

```mermaid
sequenceDiagram
  participant S as Scheduler
  participant W as Worker
  participant J as Journal
  participant B as Tool broker
  participant T as MCP server
  S->>W: run R ready at seq n
  W->>J: load checkpoint and entries after it
  W->>W: decide next step (model call via gateway)
  W->>J: append seq n+1: model result (conditional on seq)
  W->>B: tool call, idempotency key R:n+2
  B->>T: tools/call
  T-->>B: result or task handle
  B-->>W: result
  W->>J: append seq n+2: tool result
  W->>S: run R ready at seq n+3 (or parked)
```

A worker holds a lease on a run while it executes a step. The conditional
append on `seq` acts as a fencing token: if a worker lost its lease and a
second worker resumed the run, the first worker's late write fails. A
stale worker can still reach a tool before its append fails, so both workers
derive the same idempotency key for the same step and the broker returns
the recorded result instead of running the tool twice.

### 7.2 Deep dive 1: durable execution and exactly-once side effects

| Option | Strengths | Weaknesses |
|---|---|---|
| A. In-memory loop with retries | Simple; any framework | Crash loses the run; retries repeat side effects |
| B. Snapshot the full state after each step | Simple resume | Large writes; does not by itself prevent a tool running twice when the crash falls between the call and the snapshot |
| C. Event-sourced history with deterministic replay of workflow code | Proven model; resume anywhere | Agent code must be deterministic; model outputs must be recorded as activities |
| D. Step journal of recorded results plus periodic checkpoints, with idempotency keys on side effects | Fits agent loops; small writes; no determinism constraint on prompt logic between steps | The platform owns more of the engine |

**Choice:** D for v1. Every step result is appended before it is used;
checkpoints every 10 steps (*Example*) bound resume time; every tool call
carries an idempotency key derived from run ID and step number.

**Reason:** agent loops are dominated by non-deterministic steps (model
calls), so most of the value of deterministic replay is lost and its
constraints on code remain. Recording results gives the same guarantee the
journal needs. Exactly-once side effects still need the tool's help: the
broker sends the idempotency key, and tools that cannot honour one are marked
non-idempotent and require an approval or a reconciliation read before retry.

| Tool class | Retry rule |
|---|---|
| Read-only | Retry freely with backoff |
| Idempotent write (accepts a key) | Retry with the same key |
| Non-idempotent write | Never retry blindly; read the target state first, or ask a person |

> **How real systems do it: Temporal durable execution (paraphrased)**
>
> Temporal's documentation describes workflow executions whose progress is
> stored as an event history in the service's persistence layer. When a
> worker fails, the workflow code runs again and the commands it produces
> are checked against the recorded history, so execution continues from the
> last recorded event instead of starting over. External work runs as
> activities with retry policies, timers can wait without using resources,
> and a long-running execution can continue as a new run to keep its
> history bounded. Source: Temporal documentation, "Workflow Execution",
> https://docs.temporal.io/workflow-execution. LangGraph's persistence
> layer offers a related checkpoint-per-thread model (see 12.4). This box
> summarises; read the originals for details.

### 7.3 Deep dive 2: tool calls over MCP, including long-running tools

MCP revision 2026-07-28 uses stateless, self-contained JSON-RPC requests
with per-request capability negotiation. Optional extensions are opt-in on
both sides; the **Tasks** extension (`io.modelcontextprotocol/tasks`) lets a
server answer a request such as `tools/call` with a task handle instead of
blocking. The client then polls `tasks/get`, answers input requests with
`tasks/update` and can send `tasks/cancel`; task states are `working`,
`input_required`, `completed`, `failed` and `cancelled`.

| Option | Strengths | Weaknesses |
|---|---|---|
| A. Block the worker until the tool returns | Simple | Holds a worker; breaks on transport timeouts; lost on crash |
| B. Tasks extension where the server supports it; store the task ID in the journal and park the run | Durable handle survives crashes; zero-compute wait | Needs server support; polling load |
| C. Platform-side async wrapper (the broker runs the blocking call on a separate pool and parks the run) | Works with any server | The wrapper itself must be durable |

**Choice:** B when the server advertises the extension, C otherwise, A only
for tools with a p99 under 2 s (*Example*).

**Reason:** 720,000 long tool calls per day (3.5) cannot hold workers. The
task ID is written to the journal before the run parks, so a crash loses
nothing; the broker resumes polling at the server's suggested interval, or
uses status notifications when the server offers them. An `input_required`
task becomes a human or agent decision in the platform's approval flow.

Broker rules:

- Each run gets credentials scoped to its tenant, user and the tools its
  agent definition lists; tokens expire with the run.
- Tool descriptions and annotations from servers outside the tenant's
  trust list are treated as untrusted input, as the MCP specification
  advises.
- Tool results and A2A replies are untrusted data: they enter the model
  context marked as data, never as instructions, and cannot widen the
  run's tool list or credentials.
- Tool results are size-limited and stored by reference when large.
- Permission design follows the Agent Tool and Permission Design Template
  (`ai-data/agent-tool-permission-design`).

### 7.4 Deep dive 3: human-in-the-loop pauses

| Option | Strengths | Weaknesses |
|---|---|---|
| A. Worker polls an approval table | Simple | Holds compute for hours (3.4) |
| B. Park the run, emit an approval request, resume on an event or a durable timer | Zero compute while waiting; timeouts are explicit | More moving parts |

**Choice:** B. The approval request records the exact action (tool, target,
arguments summary), the agent's stated reason, the budget used so far and an
expiry. The decision (approve, reject, edit arguments) is written to the
journal as a step, so replay shows who decided what.

| Rule | *Example* value |
|---|---|
| Actions that always need approval | Payments, deletions, external messages, permission changes |
| Timeout | 24 h, then escalate to the group lead; reject after 72 h |
| Edit allowed | Arguments within the tool's schema; edits are journaled |
| Reviewer independence | The run's initiator cannot approve their own high-risk action |

Design the oversight itself with the Human Oversight Design Template
(`ai-data/human-oversight-design`).

### 7.5 Deep dive 4: tenant isolation and budgets

| Option | Strengths | Weaknesses |
|---|---|---|
| A. Shared queue, first come first served | Simple | One tenant's burst starves others |
| B. Queue per tenant with weighted fair dequeue and per-tenant concurrency caps | Isolation with shared workers | Scheduler complexity |
| C. Dedicated worker pools per tenant | Strong isolation | Poor utilisation for small tenants |

**Choice:** B for all tenants; C available for regulated tenants that need
dedicated compute.

**Reason:** NFR-05 needs bounded interference without a pool per tenant.

Budgets are checked before each step against four limits; the first limit hit
decides the outcome:

| Limit | *Example* default | On exhaustion |
|---|---|---|
| Steps per run | 50 | Stop with a partial result and a reason |
| Tokens per run | 200,000 | Pause for approval to extend, or stop |
| Wall-clock time per run, excluding human waits | 30 min | Stop; cancel in-flight tasks |
| Currency per run | USD 2.00 | Pause for approval to extend |
| Tenant daily currency | Set per contract | Reject new runs with `budget-exhausted` |

A loop detector also stops runs that repeat the same tool call with the same
arguments more than 3 times (*Example*).

### 7.6 Agent-to-agent calls over A2A

The platform acts as an A2A client towards remote agents and as an A2A server
for agents it chooses to expose. It targets A2A specification 1.0.1 (a patch
release of 1.0.0, protocol version 1.0), which defines operations such as
`SendMessage`, `SendStreamingMessage`, `GetTask`, `CancelTask` and
`SubscribeToTask` over JSON-RPC, gRPC or HTTP+JSON bindings. A remote agent
call is a step like a tool call: its task ID is journaled before the run
parks, and cancellation of the run cancels the remote task. Agent Cards
for exposed agents are specified with the A2A Agent Card and Interface
Specification Template (`ai-data/a2a-agent-card-spec`).

### 7.7 Observability and evaluation hooks

- One trace per run; one span per step, using the OpenTelemetry semantic
  conventions for generative AI (Development status) for model, tool and
  agent spans, as set out in LLM and Agent Observability Specification
  Template (`ai-data/llm-observability-spec`).
- The journal sequence number is a span attribute, so a trace and the
  journal can be joined.
- Online graders run on a sample of finished runs; offline evaluation reads
  exported trajectories. Metrics and datasets follow the AI Agent Evaluation
  Plan Template (`ai-data/agent-evaluation-plan`).

## 8. Scalability and performance

<!-- Guidance: Show how the design meets section 3 and where it breaks first. -->

| Concern | Approach | Limit before redesign |
|---|---|---|
| Step dispatch | Per-class queues, tenant-fair dequeue | Scheduler partition throughput |
| Journal writes | Partition by run ID; deltas only | Single very long run (thousands of steps): compact more often |
| Parked runs | Persisted state, durable timers | Timer index throughput at wake-up storms |
| Tool polling | Server-suggested intervals, backoff, notifications | Servers with tight polling limits |
| Model capacity | Platform quota on the LLM gateway, per-tenant caps | Gateway quota |

Scheduling overhead budget (NFR-04, p95 100 ms): dequeue 10 ms, load
checkpoint and journal tail 40 ms, budget and policy checks 10 ms, journal
append 20 ms, headroom 20 ms.

## 9. Reliability, security and operations

<!-- Guidance: Name failure modes, threats and the signals operators watch. -->

### 9.1 Failure modes

| ID | Component | Failure | Detection | Blast radius | Mitigation | RTO |
|---|---|---|---|---|---|---|
| FM-01 | Worker | Crash mid-step | Lease expiry | Runs on that worker | Another worker resumes from the journal; fencing on `seq` | 5 s |
| FM-02 | Journal store | Partition unavailable | Append errors | Runs on that partition pause | Replicated store; runs wait, no data loss | 15 min |
| FM-03 | MCP server | Down or slow | Broker error rate, circuit state | Runs using that tool | Circuit breaker; runs park and retry later; idempotency keys | Depends on the server |
| FM-04 | LLM gateway | Quota exhausted or outage | 429 and 5xx from gateway | All runs needing model calls | Back off; park runs; priority for interactive tenants | Gateway RTO |
| FM-05 | Agent version | Bad prompt or logic release | Evaluation hooks, error rate, loop detector | Runs of that version | Version pinning per run; roll back alias; stop affected runs | 10 min |
| FM-06 | Timer service | Missed wake-ups | Overdue timer metric | Parked runs | Reconciliation sweep every minute | 2 min |
| [FM-nn] | [Component] | [Failure] | [Signal] | [Scope] | [Mitigation] | [Time] |

### 9.2 Security

Mapping to the OWASP Top 10 for Agentic Applications 2026 (OWASP GenAI
Security Project); confirm entry names against the current list.

| ID | Threat (OWASP entry) | Control |
|---|---|---|
| SEC-01 | Agent Goal Hijack (ASI01) through tool results or documents | Untrusted-content marking; approval for high-risk actions; guardrails |
| SEC-02 | Tool Misuse and Exploitation (ASI02) | Per-agent tool allow-lists; argument validation in the broker |
| SEC-03 | Identity and Privilege Abuse (ASI03) | Per-run scoped, short-lived credentials; no shared service tokens |
| SEC-04 | Agentic Supply Chain Vulnerabilities (ASI04) | Registered MCP servers only; version pinning; review of server changes |
| SEC-05 | Unexpected Code Execution (ASI05) | Code runs only in the sandbox service, never in workers |
| SEC-06 | Memory & Context Poisoning (ASI06) | Provenance on memory writes; tenant-scoped memory |
| SEC-07 | Insecure Inter-Agent Communication (ASI07) | Authenticated A2A with signed Agent Cards; mutual TLS inside the platform |
| SEC-08 | Cascading Failures (ASI08) | Budgets, loop detector, circuit breakers per tool |
| SEC-09 | Rogue Agents (ASI10) | Kill switch per agent version and per tenant; audit of all actions |
| [SEC-nn] | [Threat] | [Control] |

Run the full analysis with the Agentic AI Threat Model Template
(`ai-data/agentic-threat-model`).

### 9.3 Observability

| Signal | Metric | Alert threshold |
|---|---|---|
| Scheduling delay | p95 step-to-step gap | Above 100 ms for 10 min |
| Resume time | p95 after worker loss | Above 5 s |
| Journal append errors | Rate | Any sustained rate above 0.1% |
| Parked runs | Count by reason | Sudden rise per tenant |
| Approval latency | p95 by group | Above the group's target |
| Budget stops | Count by tenant and agent | Rise after a release |
| Duplicate side-effect attempts | Idempotency hits on non-idempotent tools | Any |

### 9.4 Cost estimate

| Item | Unit cost | Quantity | Monthly cost |
|---|---|---|---|
| Model and tool spend | From gateway metering | about USD 1.44 million per 30 days (3.6) | [USD] |
| Workers | [USD per instance-month] | 72 at peak | [USD] |
| Journal and checkpoint store | [USD per TB-month] | about 5 TB hot after 7-day compaction (3.5) | [USD] |
| Object storage for results and archives | [USD per TB-month] | [TB] | [USD] |
| **Total** | | | **[USD]** |

## 10. Trade-offs and alternatives

<!-- Guidance: For each major decision list options, the choice, why and what you give up; add what changes at 10x and 0.1x. -->

| Decision | Options | Chosen | Why | What we give up |
|---|---|---|---|---|
| Execution model | In-memory; snapshots; deterministic replay; recorded-step journal | Recorded-step journal plus checkpoints | Fits non-deterministic steps | Own engine to maintain |
| Engine | General workflow engine; own journal | Own journal for v1 | Agent-specific steps (tasks, approvals, budgets) | Features a mature engine already has |
| Long tools | Block; MCP Tasks; async wrapper | Tasks where supported, wrapper otherwise | Durable handles, zero-compute waits | Polling load |
| Isolation | Shared queue; fair queues; dedicated pools | Fair queues plus optional dedicated pools | Utilisation with bounded interference | Scheduler complexity |
| Inter-agent | Custom RPC; A2A | A2A 1.0.1 | Interoperable with other vendors' agents | Spec churn |
| What changes at 10x (12 million runs per day, about 6,250 peak steps per second) | | Partition the scheduler by tenant hash; dedicated log store for the journal; consider adopting a proven durable execution engine | Scheduler partitions and journal write rate become the limits | Migration effort and a second engine to learn |
| What changes at 0.1x (120,000 runs per day) | | One database for runs and journal, a single scheduler, about 8 workers | Fixed cost of partitioned stores is not justified | Less headroom for a single noisy tenant |
| [Decision] | [Options] | [Chosen] | [Why] | [Cost] |

## 11. Delivery plan and risks

<!-- Guidance: How the design reaches production safely, and what could stop it. -->

### 11.1 Milestones

| Milestone | Scope | Exit criterion | Target date |
|---|---|---|---|
| M1 | Run API, journal, workers, model steps | Chaos test meets NFR-01 and NFR-03 | YYYY-MM-DD |
| M2 | Tool broker with MCP, idempotency keys | NFR-02 audit passes on pilot tools | YYYY-MM-DD |
| M3 | Approvals and durable timers | 5,000 parked runs hold no workers (NFR-06) | YYYY-MM-DD |
| M4 | Budgets, fair scheduling | Mixed-tenant test meets NFR-05 | YYYY-MM-DD |
| M5 | MCP Tasks, A2A client and server, evaluation hooks | Pilot multi-agent flow traced end to end | YYYY-MM-DD |

### 11.2 Rollout and rollback

[*Example:* Teams migrate one agent at a time behind a feature flag; runs
started on the old path finish there. Agent versions roll out per tenant with
a canary share and are pinned per run, so a rollback affects only new runs.]

### 11.3 Risks

| ID | Risk | Likelihood | Impact | Mitigation | Owner |
|---|---|---|---|---|---|
| R-01 | *Example:* Tools that cannot honour idempotency keys | High | High | Mark non-idempotent; reconciliation reads; approvals | [@name] |
| R-02 | *Example:* MCP Tasks support uneven across servers | High | Medium | Async wrapper fallback | [@name] |
| R-03 | *Example:* Journal growth from verbose tool results | Medium | Medium | Size limits, references, compaction | [@name] |
| R-04 | *Example:* Protocol revisions change behaviour | Medium | Medium | Pin versions; contract tests | [@name] |
| [R-nn] | [Risk] | [L / M / H] | [L / M / H] | [Mitigation] | [@name] |

## 12. Appendices

<!-- Guidance: Traceability, open questions, glossary and references. -->

### 12.1 Traceability

| Requirement | Design element | Verification |
|---|---|---|
| FR-03, NFR-01, NFR-03 | 7.1 leases and fencing, 7.2 journal | Chaos test killing workers |
| NFR-02 | 7.2 idempotency keys, tool classes | Broker audit |
| FR-06, NFR-06 | 7.4 parking and timers | Load test with parked runs |
| FR-07, NFR-05 | 7.5 fair queues and budgets | Mixed-tenant test |
| FR-04 | 7.3 broker and MCP | Contract tests per server |
| [ID] | [Element] | [Test] |

### 12.2 Open questions

| ID | Question | Owner | Due |
|---|---|---|---|
| OQ-01 | *Example:* Do we expose run state as A2A tasks for all agents or only selected ones? | [@name] | YYYY-MM-DD |
| OQ-02 | *Example:* When does adopting a general durable execution engine pay off? | [@name] | YYYY-MM-DD |
| [OQ-nn] | [Question] | [@name] | YYYY-MM-DD |

### 12.3 Glossary

| Term | Meaning |
|---|---|
| Run | One execution of an agent definition on an input |
| Step | One unit of work in a run: model call, tool call, sub-agent call or human wait |
| Journal | Append-only, ordered record of each step's result for one run |
| Checkpoint | Snapshot of run state at a journal position, used to resume faster |
| Fencing | Rejecting writes from a worker that has lost its lease |
| Parking | Persisting a waiting run and releasing its worker |
| Durable execution | Execution that survives process failure without repeating completed work |

### 12.4 References

| Reference | Use |
|---|---|
| Model Context Protocol specification, revision 2026-07-28, https://modelcontextprotocol.io/specification/2026-07-28 | Tool calls in 7.3 |
| MCP Tasks extension, https://modelcontextprotocol.io/extensions/tasks/overview (specified in the `modelcontextprotocol/ext-tasks` repository) | Long-running tools in 7.3 |
| Agent2Agent (A2A) Protocol Specification 1.0.1, https://a2a-protocol.org/latest/specification/ and https://github.com/a2aproject/A2A | Inter-agent calls in 7.6 |
| OWASP Top 10 for Agentic Applications 2026, OWASP GenAI Security Project, https://genai.owasp.org/ (CC BY-SA 4.0) | Threats in 9.2 |
| OpenTelemetry semantic conventions for generative AI (Development status), https://github.com/open-telemetry/semantic-conventions-genai | Spans in 7.7 |
| Temporal documentation, "Workflow Execution", https://docs.temporal.io/workflow-execution | Box in 7.2 (paraphrased) |
| LangChain, LangGraph persistence documentation, https://docs.langchain.com/oss/python/langgraph/persistence | Checkpoint-per-thread comparison |
| RFC 9457, Problem Details for HTTP APIs (2023) | Error format |
| Multi-Agent System Design Template (`ai-data/multi-agent-system-design`) | Agent roles and hand-offs |
| Agent Code Execution Sandbox System Design Template (`system-design/design-agent-sandbox-execution`) | Code execution |
| Job Scheduler System Design Template (`system-design/design-job-scheduler`) | Queue and timer patterns |
| Workflow Orchestration Engine System Design Template (`system-design/design-workflow-orchestration-engine`) | General durable workflow engine, the option to adopt at 10x |

## Standard and licence note

Based on the Model Context Protocol specification revision 2026-07-28 and its
Tasks extension, the Agent2Agent (A2A) Protocol Specification 1.0.1, the
OpenTelemetry semantic conventions for generative AI (Development status), RFC
9457, and the OWASP Top 10 for Agentic Applications 2026 by the OWASP GenAI
Security Project (CC BY-SA 4.0; entry names are attributed to OWASP, and any
adapted OWASP material you redistribute must carry the same licence under its
ShareAlike term). Product documentation in 7.2 is paraphrased and cited as
inspiration only. The text is original; these protocols change quickly, so
check the current official version of each source before approval. Template
licence: MIT.