System Design: [Agent orchestration platform name]

Reference design for a multi-tenant platform that runs AI agents as durable, resumable executions. It covers durable execution of agent runs, checkpoints and resumption, tool calls over the Model Context Protocol (MCP, revision 2026-07-28, with the optional Tasks extension for long-running tools), agent-to-agent calls over the Agent2Agent (A2A) protocol 1.0.1, human-in-the-loop pauses, tenant isolation, budgets, observability and evaluation hooks. Agent behaviour design (roles, prompts, hand-offs) is covered by the Multi-Agent System Design Template (ai-data/multi-agent-system-design). It follows the 12-section structure of the System Design Document Template (system-design/system-design-document) and is pre-filled with a worked design to adapt. Replace every [bracketed] value and every row marked Example; all numbers are illustrative, not benchmarks.

Document control

Approvals

Revision history

Contents

1. Overview
2. Requirements
3. Capacity estimation
4. API design
5. Data model
6. High-level design
7. Detailed design
8. Scalability and performance
9. Reliability, security and operations
10. Trade-offs and alternatives
11. Delivery plan and risks
12. Appendices

1. Overview

1.1 Problem statement

[Example: Twelve teams build agents with different frameworks, each running the agent loop inside a web request or a background job. When a pod restarts, running agents are lost and users must start again; three incidents last quarter came from agents that repeated a side-effecting tool call (a refund, a ticket update) after a retry. Approvals by people are implemented as polling loops that hold a worker for hours. Nobody can cap what one run may spend, and traces stop at the framework boundary, so a failing multi-agent task cannot be followed end to end.]

1.2 Proposed solution

[Example: A shared platform runs each agent run as a durable execution. The agent loop is split into steps (model call, tool call, sub-agent call, human wait); every step is written to an append-only run journal before its result is used. Workers are stateless: any worker can resume any run from its last checkpoint plus the journal. Tools are reached over MCP through a tool broker that injects scoped credentials and records each call with an idempotency key; long tools use the MCP Tasks extension so the run waits without holding a worker. Remote agents are called over A2A. Human approvals park the run with zero compute until a decision arrives. Every run has a budget in tokens, steps, wall-clock time and currency, enforced before each step.]

1.3 Decisions requested from reviewers

1.4 45-minute interview path

2. Requirements

2.1 Core functional requirements

2.2 Non-functional requirements

2.3 Below the line (out of scope)

• [Example: Agent behaviour, roles and prompts (see Multi-Agent System Design Template (ai-data/multi-agent-system-design) and AI Agent Specification Template (ai-data/ai-agent-spec))]
• [Example: Model serving and provider routing (see LLM Gateway System Design Template (system-design/design-llm-gateway))]
• [Example: Running untrusted generated code (see Agent Code Execution Sandbox System Design Template (system-design/design-agent-sandbox-execution))]
• [Example: Building MCP servers; the platform consumes them (see MCP Server Specification Template (ai-data/mcp-server-spec))]
• [Example: End-user chat interface (see LLM Chat Assistant System Design Template (system-design/design-llm-chat-assistant))]
• [Item deferred, with the release that will take it]

3. Capacity estimation

3.1 Assumptions

All values are Example planning inputs.

3.2 Run and step rates

3.3 Active runs and workers

Little's law: active runs = arrival rate x active duration. Average: 13.9 x 240 s = about 3,340 active runs; peak: 41.7 x 240 s = about 10,000. At 200 concurrent runs per worker, peak needs 50 workers at full load, or 72 at a 70% utilisation target (50 / 0.7 = 71.4, rounded up). Consequence: workers must be asynchronous and stateless; a run is never pinned to a worker, so scaling and deploys move runs freely.

3.4 Parked runs

Runs pausing for a person: 1.2 million x 5% = 60,000 per day. With a mean wait of 2 hours, the number parked at any time is 60,000 x 2 / 24 = 5,000. Consequence: 5,000 waiting runs would hold 25 workers if they stayed in memory; parking must persist state and release the worker, and the wake-up must come from a durable timer or an approval event.

3.5 Journal storage

3.6 Spend

1.2 million runs x USD 0.04 = USD 48,000 per day, about USD 1.44 million per 30 days. Consequence: a runaway loop that multiplies a run's cost by 100 is expensive within minutes, so budgets are checked before every step, not in a nightly report.

3.7 Summary

4. API design

4.1 Endpoints

4.2 Run states

The platform maps its states to the A2A task states when a run is exposed over A2A: waiting_human maps to TASK_STATE_INPUT_REQUIRED, running and waiting_tool to TASK_STATE_WORKING, and the terminal states to TASK_STATE_COMPLETED, TASK_STATE_FAILED and TASK_STATE_CANCELED.

4.3 Errors

Errors use RFC 9457 problem details. A repeated POST /v1/runs with the same Idempotency-Key returns the existing run instead of starting a second one.

5. Data model

5.1 Entities

The run journal is the source of truth; the run record is a projection that can be rebuilt from it.

5.2 Access patterns

5.3 Storage choices

6. High-level design

7. Detailed design

7.1 Step execution

A worker holds a lease on a run while it executes a step. The conditional append on seq acts as a fencing token: if a worker lost its lease and a second worker resumed the run, the first worker's late write fails. A stale worker can still reach a tool before its append fails, so both workers derive the same idempotency key for the same step and the broker returns the recorded result instead of running the tool twice.

7.2 Deep dive 1: durable execution and exactly-once side effects

Choice: D for v1. Every step result is appended before it is used; checkpoints every 10 steps (Example) bound resume time; every tool call carries an idempotency key derived from run ID and step number.

Reason: agent loops are dominated by non-deterministic steps (model calls), so most of the value of deterministic replay is lost and its constraints on code remain. Recording results gives the same guarantee the journal needs. Exactly-once side effects still need the tool's help: the broker sends the idempotency key, and tools that cannot honour one are marked non-idempotent and require an approval or a reconciliation read before retry.

How real systems do it: Temporal durable execution (paraphrased)

Temporal's documentation describes workflow executions whose progress is stored as an event history in the service's persistence layer. When a worker fails, the workflow code runs again and the commands it produces are checked against the recorded history, so execution continues from the last recorded event instead of starting over. External work runs as activities with retry policies, timers can wait without using resources, and a long-running execution can continue as a new run to keep its history bounded. Source: Temporal documentation, "Workflow Execution", https://docs.temporal.io/workflow-execution. LangGraph's persistence layer offers a related checkpoint-per-thread model (see 12.4). This box summarises; read the originals for details.

7.3 Deep dive 2: tool calls over MCP, including long-running tools

MCP revision 2026-07-28 uses stateless, self-contained JSON-RPC requests with per-request capability negotiation. Optional extensions are opt-in on both sides; the Tasks extension (io.modelcontextprotocol/tasks) lets a server answer a request such as tools/call with a task handle instead of blocking. The client then polls tasks/get, answers input requests with tasks/update and can send tasks/cancel; task states are working, input_required, completed, failed and cancelled.

Choice: B when the server advertises the extension, C otherwise, A only for tools with a p99 under 2 s (Example).

Reason: 720,000 long tool calls per day (3.5) cannot hold workers. The task ID is written to the journal before the run parks, so a crash loses nothing; the broker resumes polling at the server's suggested interval, or uses status notifications when the server offers them. An input_required task becomes a human or agent decision in the platform's approval flow.

Broker rules:

• Each run gets credentials scoped to its tenant, user and the tools its agent definition lists; tokens expire with the run.
• Tool descriptions and annotations from servers outside the tenant's trust list are treated as untrusted input, as the MCP specification advises.
• Tool results and A2A replies are untrusted data: they enter the model context marked as data, never as instructions, and cannot widen the run's tool list or credentials.
• Tool results are size-limited and stored by reference when large.
• Permission design follows the Agent Tool and Permission Design Template (ai-data/agent-tool-permission-design).

7.4 Deep dive 3: human-in-the-loop pauses

Choice: B. The approval request records the exact action (tool, target, arguments summary), the agent's stated reason, the budget used so far and an expiry. The decision (approve, reject, edit arguments) is written to the journal as a step, so replay shows who decided what.

Design the oversight itself with the Human Oversight Design Template (ai-data/human-oversight-design).

7.5 Deep dive 4: tenant isolation and budgets

Choice: B for all tenants; C available for regulated tenants that need dedicated compute.

Reason: NFR-05 needs bounded interference without a pool per tenant.

Budgets are checked before each step against four limits; the first limit hit decides the outcome:

A loop detector also stops runs that repeat the same tool call with the same arguments more than 3 times (Example).

7.6 Agent-to-agent calls over A2A

The platform acts as an A2A client towards remote agents and as an A2A server for agents it chooses to expose. It targets A2A specification 1.0.1 (a patch release of 1.0.0, protocol version 1.0), which defines operations such as SendMessage, SendStreamingMessage, GetTask, CancelTask and SubscribeToTask over JSON-RPC, gRPC or HTTP+JSON bindings. A remote agent call is a step like a tool call: its task ID is journaled before the run parks, and cancellation of the run cancels the remote task. Agent Cards for exposed agents are specified with the A2A Agent Card and Interface Specification Template (ai-data/a2a-agent-card-spec).

7.7 Observability and evaluation hooks

• One trace per run; one span per step, using the OpenTelemetry semantic conventions for generative AI (Development status) for model, tool and agent spans, as set out in LLM and Agent Observability Specification Template (ai-data/llm-observability-spec).
• The journal sequence number is a span attribute, so a trace and the journal can be joined.
• Online graders run on a sample of finished runs; offline evaluation reads exported trajectories. Metrics and datasets follow the AI Agent Evaluation Plan Template (ai-data/agent-evaluation-plan).

8. Scalability and performance

Scheduling overhead budget (NFR-04, p95 100 ms): dequeue 10 ms, load checkpoint and journal tail 40 ms, budget and policy checks 10 ms, journal append 20 ms, headroom 20 ms.

9. Reliability, security and operations

9.1 Failure modes

9.2 Security

Mapping to the OWASP Top 10 for Agentic Applications 2026 (OWASP GenAI Security Project); confirm entry names against the current list.

Run the full analysis with the Agentic AI Threat Model Template (ai-data/agentic-threat-model).

9.3 Observability

9.4 Cost estimate

10. Trade-offs and alternatives

11. Delivery plan and risks

11.1 Milestones

11.2 Rollout and rollback

[Example: Teams migrate one agent at a time behind a feature flag; runs started on the old path finish there. Agent versions roll out per tenant with a canary share and are pinned per run, so a rollback affects only new runs.]

11.3 Risks

12. Appendices

12.1 Traceability

12.2 Open questions

12.3 Glossary

12.4 References

Standard and licence note

Based on the Model Context Protocol specification revision 2026-07-28 and its Tasks extension, the Agent2Agent (A2A) Protocol Specification 1.0.1, the OpenTelemetry semantic conventions for generative AI (Development status), RFC 9457, and the OWASP Top 10 for Agentic Applications 2026 by the OWASP GenAI Security Project (CC BY-SA 4.0; entry names are attributed to OWASP, and any adapted OWASP material you redistribute must carry the same licence under its ShareAlike term). Product documentation in 7.2 is paraphrased and cited as inspiration only. The text is original; these protocols change quickly, so check the current official version of each source before approval. Template licence: MIT.