# System Design: [AI coding assistant name]

> Reference design for an AI coding assistant used by an engineering
> organisation from the IDE and the command line. It covers repository
> indexing and code search, context assembly, the split between inline
> completion and agentic edit loops, sandboxed tool execution and tests,
> the review and approval loop, tool integration through the Model Context
> Protocol (MCP, revision 2026-07-28) and Agent Skills (agentskills.io
> specification), latency and cost, and the security problems specific to
> code: secrets, prompt injection from repository content, and licence and
> provenance of generated code. Threats are mapped to the OWASP Top 10 for
> LLM Applications 2026. It follows the 12-section skeleton of the System
> Design Document template and is pre-filled with a worked design to adapt.
> Replace every `[bracketed]` value and every row marked *Example*; all
> numbers are illustrative examples, not benchmarks.

<!--
Guidance: How to use this template
- Treat completion and agent mode as two products on one platform. They
  differ by three orders of magnitude in tokens per request and by two in
  latency budget, so they get separate paths, models and limits (2.2, 3).
- Start the review with the 45-minute path in 1.4 if time is short; it
  says which deep dive to pick for which audience.
- Decide the permission model before the tool list. Every tool the agent
  can call is a capability an attacker can reach through a poisoned file
  (7.4, 9.2).
- Model serving lives in LLM Inference Serving System Design
  (`system-design/design-llm-inference-serving`); this document consumes it.
- Re-measure every example number with your own telemetry before approval.
-->

## Document control

<!-- Guidance: A design is a proposal until approved. Record who decided, which models and which repositories are in scope. -->

| Field | Value |
|---|---|
| Document ID | [SDD-CODE-001] |
| System / feature | [AI coding assistant: IDE extension, CLI agent, backend] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Implemented / Superseded] |
| Author(s) | [@author] |
| Reviewers | [@reviewer-1], [@reviewer-2] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal] |
| Models in scope | [Completion model, agent model, embedding model, with versions and licences] |
| Repositories in scope | [All internal repositories / listed organisations / excluded repositories] |
| Protocols | [MCP revision 2026-07-28], [Agent Skills specification] |
| Related | [AI agent specification], [Agent tool permission design], [Threat model], [LLM observability spec] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Tech lead, developer platform | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| Staff / principal engineer | [@name] | [Decision] | YYYY-MM-DD |
| Application security | [@name] | [Decision] | YYYY-MM-DD |
| Legal / open source programme office | [@name] | [Decision] | YYYY-MM-DD |
| FinOps / budget owner | [@name] | [Decision] | YYYY-MM-DD |

## Revision history

<!-- Guidance: Note what changed in the design, not the wording. -->

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft for design review |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Moved test execution from developer laptops to remote microVM sandboxes |
| 0.3 | YYYY-MM-DD | [@name] | *Example:* Added MCP server allowlist and per-tool approval policy |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved |

## Contents

<!-- Guidance: Keep aligned with the headings. -->

1. [Overview](#1-overview)
2. [Requirements](#2-requirements)
3. [Capacity estimation](#3-capacity-estimation)
4. [API design](#4-api-design)
5. [Data model](#5-data-model)
6. [High-level design](#6-high-level-design)
7. [Detailed design](#7-detailed-design)
8. [Scalability and performance](#8-scalability-and-performance)
9. [Reliability, security and operations](#9-reliability-security-and-operations)
10. [Trade-offs and alternatives](#10-trade-offs-and-alternatives)
11. [Delivery plan and risks](#11-delivery-plan-and-risks)
12. [Appendices](#12-appendices)

## 1. Overview

<!-- Guidance: One page a busy reviewer can read alone: problem, solution in a paragraph, decisions needed. Pitfall: describing the model instead of the system around it. -->

### 1.1 Problem statement

[*Example:* Developers use three unmanaged AI tools with personal accounts.
Source code leaves the company without a data processing agreement, nobody
can tell which commits were AI-assisted, and one tool executed a shell
command suggested by a README in a cloned third-party repository. Teams
want completions inside the IDE and an agent that can fix a failing test,
run the suite and open a pull request, under company controls.]

### 1.2 Proposed solution

[*Example:* One managed assistant with two clients (IDE extension and CLI)
and one backend. Inline completion goes through a low-latency path to a
small self-hosted model with a context window built from the open file,
neighbouring files and the repository index. Agent tasks run a plan, act,
observe loop with a larger model; every file edit and command runs in an
isolated remote sandbox with a copy of the repository, no production
credentials and an egress allowlist. The agent proposes a diff; a human
approves it in the IDE or as a pull request. External tools are attached
only through allowlisted MCP servers, and team procedures are packaged as
Agent Skills. All requests pass a gateway that enforces identity,
repository permissions, secret scanning and cost budgets.]

### 1.3 Decisions requested from reviewers

| ID | Question | Recommended option | Section |
|---|---|---|---|
| Q-01 | *Example:* Run agent commands on the laptop or in a remote sandbox? | Remote microVM sandbox by default; local mode only for approved repositories | 7.3 |
| Q-02 | *Example:* Index with embeddings, lexical search or both? | Both, plus a symbol graph; fuse results | 7.2 |
| Q-03 | *Example:* Which actions need human approval? | Anything outside the sandbox, any network call off the allowlist, any push | 7.4, 7.5 |
| Q-04 | *Example:* Who may add MCP servers? | Platform team allowlist; users choose from it | 7.4 |
| Q-05 | *Example:* Block completions that match public code? | Filter on by default, with a reference shown when a match is allowed | 9.2 |
| [Q-nn] | [Question] | [Option] | [Section] |

### 1.4 45-minute interview path

<!-- Guidance: One document serves interviews and real reviews. In an interview, follow this order and leave the rest as reading. -->

| Minutes | Cover | Output on the whiteboard |
|---|---|---|
| 0 to 5 | Clarify users, clients (IDE, CLI), completion versus agent, repository count | Core requirements list (2.1) and below the line (2.3) |
| 5 to 10 | Estimate completion QPS, agent tokens per day, sandbox concurrency | Three numbers and their consequences (3.6) |
| 10 to 15 | API for completion, agent task and events; data model with access patterns | 4.1 and 5.2 |
| 15 to 25 | High-level diagram: gateway, context engine, index, model pools, agent orchestrator, sandbox | Section 6 |
| 25 to 40 | One deep dive. Pick context assembly (7.2) for a retrieval-minded panel, the agent loop and sandbox (7.3) for a platform panel, or tool permissions and prompt injection (7.4, 9.2) for a security panel | Options, choice and reason |
| 40 to 45 | Failure modes, 10x row of the trade-off table, what you would build next | 9.1 and 10 |

## 2. Requirements

<!-- Guidance: Split core requirements from below-the-line items so nobody designs for the latter. Every non-functional target needs a measurement. -->

### 2.1 Core functional requirements

| ID | Requirement | Priority |
|---|---|---|
| FR-01 | *Example:* Inline completion (single and multi-line) in supported IDEs, streamed, cancellable on the next keystroke | Must |
| FR-02 | *Example:* Chat about the codebase with answers that cite files and line ranges | Must |
| FR-03 | *Example:* Agent tasks from IDE or CLI: read code, edit files, run commands and tests in a sandbox, iterate until tests pass or budget ends | Must |
| FR-04 | *Example:* Every agent change is presented as a reviewable diff; nothing is pushed without human approval | Must |
| FR-05 | *Example:* Repository index (lexical, symbol and semantic) that respects the user's repository permissions | Must |
| FR-06 | *Example:* Tool integration through MCP servers from an allowlist, with per-tool approval policies | Must |
| FR-07 | *Example:* Organisation and repository instructions and Agent Skills loaded on demand | Should |
| FR-08 | *Example:* Pull request creation with an AI-assistance trailer and a task transcript link | Should |
| FR-09 | *Example:* Usage and cost reporting per team | Should |
| [FR-nn] | [Requirement] | [Must / Should / Could] |

### 2.2 Non-functional requirements

| ID | Quality | Target | Measured as |
|---|---|---|---|
| NFR-01 | Completion latency | *Example:* p50 at or below 250 ms, p95 at or below 500 ms from request to first suggestion token in the client | Client telemetry |
| NFR-02 | Agent step latency | *Example:* p95 model turn at or below 20 s; sandbox command start at or below 2 s | Orchestrator timing |
| NFR-03 | Availability | *Example:* 99.9% monthly for completion; 99.5% for agent tasks | SLO on successful responses |
| NFR-04 | Isolation | *Example:* Sandbox has no route to production networks; zero secrets from the vault present | Quarterly penetration test, configuration audit |
| NFR-05 | Data handling | *Example:* No source code retained by the model provider; completion prompts logged only as hashes unless the team opts in; agent transcripts kept only for the audit retention in 5.3 | Contract and configuration audit |
| NFR-06 | Quality | *Example:* Completion acceptance rate tracked per language; agent task success on an internal benchmark at or above the baseline before any model change | Offline evaluation, telemetry |
| NFR-07 | Cost | *Example:* Within the per-developer budget in 3.4 | Monthly chargeback |
| [NFR-nn] | [Quality] | [Target] | [Measurement] |

### 2.3 Below the line (out of scope)

- [*Example:* model training or fine-tuning on internal code (separate project with its own data review)]
- [*Example:* autonomous merging to protected branches; the agent never merges]
- [*Example:* agents that run on a schedule without a user who started them]
- [*Example:* code review bots on every pull request (consumer of this platform, designed separately)]
- [Item deferred, with the release that will take it]

## 3. Capacity estimation

<!-- Guidance: Every estimate ends in a consequence for the design. Show inputs, arithmetic and the decision it forces. -->

### 3.1 Assumptions

All values are *Example* planning inputs.

| Input | Value | Source |
|---|---|---|
| Licensed developers | 20 000 | Licence plan |
| Daily active developers | 12 000 | Pilot telemetry |
| Completion requests per active developer per day (after client debounce) | 400 | Pilot telemetry |
| Working window | 8 h | Main time zone |
| Peak to average ratio | 3 | Pilot telemetry |
| Completion prompt tokens / output tokens | 2 000 / 40 | Context budget |
| Agent tasks per active developer per day | 3 | Pilot telemetry |
| Model turns per agent task | 25 | Pilot telemetry |
| Mean input tokens per agent turn | 30 000 | Context grows during a task |
| Mean output tokens per agent turn | 800 | Pilot telemetry |
| Sandbox time per task | 10 min | Pilot telemetry |
| Lines of code across repositories | 400 million | Source host statistics |

### 3.2 Completion traffic

Requests per day: 12 000 x 400 = 4.8 million. Spread over the 8-hour
window (28 800 s) that is about 167 requests/s on average and about 500
requests/s at peak (ratio 3). Prompt tokens per day: 4.8 million x 2 000 =
9.6 billion.

**Consequence:** completion needs a dedicated small-model pool with prefix
caching and request cancellation; about half of the requests are cancelled
by the next keystroke, so the server must abort generation on client
disconnect or it pays for tokens nobody sees.

### 3.3 Agent traffic

Tasks per day: 12 000 x 3 = 36 000. Per task: 25 turns x 30 000 = 750 000
input tokens and 25 x 800 = 20 000 output tokens. Per day: 36 000 x 750 000
= 27 billion input tokens and 36 000 x 20 000 = 720 million output tokens.

**Consequence:** agent mode is under 1% of requests (36 000 against 4.8
million) yet produces almost three times the input tokens of completion.
Cost control is a context design problem: keep the prompt prefix stable so
it is cached, and compact the history before it grows past a set size.

### 3.4 Cost of agent tokens

With *Example* list prices of USD 3 per million input tokens, USD 0.30 per
million cached input tokens and USD 15 per million output tokens:

| Scenario | Input cost per day | Output cost per day | Total per day |
|---|---|---|---|
| No caching | 27 000 x 3 = USD 81 000 | 720 x 15 = USD 10 800 | USD 91 800 |
| 80% of input served from cache | 27 000 x (0.2 x 3 + 0.8 x 0.30) = USD 22 680 | USD 10 800 | USD 33 480 |

Per active developer that is about USD 7.65 per day without caching and
about USD 2.79 with it. **Consequence:** the context engine (7.2) must
append to the conversation rather than rewrite earlier turns, because any
change to the prefix invalidates the cache. Budgets per task and per day are
enforced at the gateway.

### 3.5 Index and sandboxes

| Item | Calculation | Result | Consequence |
|---|---|---|---|
| Code chunks | 400 million lines / 40 lines per chunk | 10 million chunks | Fits one sharded vector store |
| Embeddings, 1 024 dimensions, float32 | 10 million x 1 024 x 4 B | about 41 GB | Quantize to int8 (about 10 GB); shard by repository |
| Lexical and trigram index | About the size of the source, 400 million x 40 B | about 16 GB | Memory-resident lexical index per shard |
| Re-index load | *Example:* 50 000 commits/day x 20 changed files | 1 million files/day, about 12/s | Incremental indexing by file hash, never full rebuilds |
| Sandbox minutes | 36 000 tasks x 10 min | 360 000 min/day | Bill and budget sandboxes per task, not per developer |
| Concurrent sandboxes | 360 000 / 480 min, then x 2 for peaks | about 750 average, about 1 500 at peak | Warm pool of snapshot-restored microVMs |
| Sandbox resources at peak | 1 500 x 2 vCPU and 4 GB | 3 000 vCPU, 6 TB memory | Dedicated autoscaled node group |

### 3.6 Summary

| Quantity | Estimate | Design consequence |
|---|---|---|
| Completion peak | About 500 requests/s, 2 000-token prompts | Small model, prefix cache, cancellation, regional pools |
| Agent input tokens | About 27 billion per day | Prompt caching and history compaction are the main cost levers |
| Agent cost | About USD 33 000 per day with caching (*Example*) | Per-task and per-developer budgets at the gateway |
| Index | About 10 million chunks, tens of GB | Shard by repository; incremental updates |
| Sandboxes | About 1 500 concurrent at peak | Snapshot-based microVM pool, start under 2 s |

## 4. API design

<!-- Guidance: Define the contract first. Completion and agent APIs have different shapes: one is a short streaming call, the other a long-running task with events. -->

### 4.1 Endpoints

| Method and path | Purpose |
|---|---|
| `POST /v1/completions` | Inline completion; streamed; cancelled when the client closes the stream |
| `POST /v1/chat` | Codebase question with citations to files and lines |
| `POST /v1/tasks` | Start an agent task (repository, branch, instruction, budget) |
| `GET /v1/tasks/{id}/events` | Server-sent events: plan, tool call, approval request, diff, result |
| `POST /v1/tasks/{id}/approvals/{approval_id}` | Approve or reject a pending action |
| `POST /v1/tasks/{id}/cancel` | Stop the task and tear down the sandbox |
| `GET /v1/tasks/{id}/diff` | Current proposed diff |
| `POST /v1/search` | Code search (lexical, symbol, semantic) within permitted repositories |
| `GET /v1/usage` | Tokens, sandbox minutes and cost per user or team |

### 4.2 Request example

An agent task start is shown below. The idempotency key makes a retried
start return the existing task instead of creating a second sandbox.
Completion requests differ: the client sends what it knows cheaply (cursor,
prefix, suffix, open file, recent edits) with a request ID, the server adds
repository context, and the request ID lets the server cancel generation
when a newer request arrives from the same editor.

```http
POST /v1/tasks
Authorization: Bearer [user token]
Idempotency-Key: [uuid]
Content-Type: application/json

{"repository": "[org/repo]", "base_ref": "main", "instruction": "Fix the failing test in billing/invoice_test.py", "budget": {"max_turns": 40, "max_cost_usd": 5}, "approval_policy": "default"}
```

### 4.3 Errors

| Status | Problem type | Meaning |
|---|---|---|
| 403 | `repository-forbidden` | User lacks read access to the repository in the source host |
| 409 | `approval-pending` | Task is waiting for a human decision |
| 422 | `secret-detected` | Context contained a secret that policy forbids sending |
| 429 | `budget-exceeded` | Task, user or team budget reached; `Retry-After` for rate limits |
| 503 | `sandbox-capacity` | No sandbox available within the start deadline |

Errors use RFC 9457 problem details.

## 5. Data model

<!-- Guidance: Choose storage by access pattern. List each query, the key that serves it and why the partition key was chosen. -->

### 5.1 Entities

Index entities are derived from the source host and can be rebuilt; task,
tool call and approval records are the audit trail and must be retained.

```mermaid
erDiagram
  USER ||--o{ TASK : starts
  REPOSITORY ||--o{ FILE_VERSION : contains
  FILE_VERSION ||--o{ CHUNK : "split into"
  CHUNK ||--o{ SYMBOL : defines
  TASK ||--o{ STEP : has
  STEP ||--o{ TOOL_CALL : makes
  TOOL_CALL ||--o| APPROVAL : "may need"
  TASK ||--o| DIFF : proposes
  REPOSITORY {
    string repo_id PK
    string acl_source
    string default_branch
  }
  FILE_VERSION {
    string repo_id PK
    string path PK
    string content_hash
  }
  CHUNK {
    string chunk_id PK
    string repo_id FK
    string language
    int start_line
  }
  TASK {
    string task_id PK
    string user_id FK
    string status
    float cost_usd
  }
  TOOL_CALL {
    string call_id PK
    string tool_name
    string risk_class
    string outcome
  }
```

### 5.2 Access patterns

| Query | Served by | Partition key and rationale |
|---|---|---|
| Lexical or regex search in repositories the user can read | Trigram index per shard, filtered by repository ID list | `repo_id`: permissions are per repository, so a filter on the key prunes shards |
| Semantic search for a natural-language question | Vector index with repository filter, then rerank | `repo_id`: same pruning; avoids cross-repository leakage by construction |
| Go to definition and find references across files | Symbol graph table keyed by fully qualified name | `repo_id` plus symbol name: graph walks stay inside one repository |
| Has this file changed since indexing? | `FILE_VERSION` keyed by repository and path, compared by hash | `repo_id`: the indexer works one repository at a time |
| Stream events of a running task | Event log keyed by task ID, ordered by sequence | `task_id`: one writer, one reader, strict order |
| Pending approvals for a user | Secondary index on approver and status | `user_id`: the IDE polls only its own queue |
| Usage per team per day | Aggregated usage table in the warehouse | `team_id` plus date: reporting only, never on the request path |

### 5.3 Storage choices

| Data | Store | Notes |
|---|---|---|
| Repository metadata, ACL cache | Relational database | ACL cache refreshed from the source host on webhook and every 15 min |
| Lexical and symbol index | Search engine shards in memory | Rebuilt per file on change |
| Embeddings | Vector store, int8 quantized | One embedding model version per index; re-embed on model change |
| Task events and transcripts | Append-only log, 90-day retention (*Example*) | Needed for audit and for review of the diff |
| Sandbox images and snapshots | Object storage plus node-local cache | Snapshot per base image and repository warm-up |
| Usage records | Event stream to the warehouse | Never on the request path |

## 6. High-level design

<!-- Guidance: One diagram with every component on the request paths and where state lives. Name components by responsibility. -->

```mermaid
flowchart LR
  ide([IDE extension]) --> gw[Assistant gateway]
  cli([CLI agent]) --> gw
  gw --> pol[Identity, repo ACL, budgets, secret scan]
  pol --> ctx[Context engine]
  ctx --> idx[(Lexical, symbol and vector index)]
  pol --> cpool[Completion model pool]
  pol --> orch[Agent orchestrator]
  orch --> apool[Agent model pool]
  orch --> sbx[Sandbox manager]
  sbx --> vm[[MicroVM sandboxes]]
  orch --> mcp[MCP client hub]
  mcp --> srv[(Allowlisted MCP servers)]
  orch --> appr[Approval service]
  appr --> ide
  orch --> scm[Source host: branches and pull requests]
  scm --> ixr[Indexer]
  ixr --> idx
  gw --> tel[[Telemetry and usage]]
```

| Component | Responsibility | Scales by |
|---|---|---|
| Assistant gateway | Authentication, routing, streaming, cancellation, metering | Horizontally |
| Policy layer | Repository permissions, budgets, secret scanning on outbound context | Horizontally, cached ACLs |
| Context engine | Retrieve and rank snippets, assemble prompt within a token budget | Horizontally |
| Index and indexer | Lexical, symbol and vector search; incremental updates from webhooks | Shards per repository group |
| Completion pool | Small model, low latency, prefix cache | GPUs, per region |
| Agent orchestrator | Plan, act, observe loop; budgets; event log | Horizontally, tasks are independent |
| Sandbox manager | Create, snapshot, destroy microVMs; egress policy | Node group size |
| MCP client hub | Connections to allowlisted MCP servers, tool policy enforcement | Horizontally |
| Approval service | Pending decisions, notification, audit | Horizontally |

## 7. Detailed design

<!-- Guidance: Deep dives on the hard problems. Each one names the options, the choice and the reason. Pick two to four; do not spread thinly. -->

### 7.1 Completion request flow

```mermaid
sequenceDiagram
  participant E as IDE
  participant G as Gateway
  participant C as Context engine
  participant M as Completion model
  E->>G: completion (prefix, suffix, file, recent edits)
  G->>G: auth, ACL, secret scan
  G->>C: assemble context (budget 2 000 tokens)
  C-->>G: ranked snippets
  G->>M: fill-in-the-middle prompt
  M-->>E: streamed suggestion
  E->>G: next keystroke cancels request ID
  G->>M: abort generation
```

The client debounces keystrokes, sends prefix and suffix around the cursor
and a list of recently edited and open files. The server uses a
fill-in-the-middle prompt format so the model sees code after the cursor.
Suggestions stop at a syntactic boundary (end of block or line) decided by
the client's parser, which keeps them short and cheap.

### 7.2 Deep dive 1: repository indexing and context assembly

The quality of both completion and agent work depends more on what enters
the context window than on the model. The context engine has a fixed token
budget and must fill it with the most useful snippets.

| Option | How it works | Strengths | Weaknesses |
|---|---|---|---|
| A. Open files only | Use the current file and open tabs | No index, no latency | Misses definitions in closed files; poor for agents |
| B. Embedding search | Chunk code, embed, nearest neighbours to the cursor or question | Finds conceptually related code | Weak on exact identifiers; chunk boundaries cut functions |
| C. Lexical and symbol graph | Trigram search plus a graph of definitions and references | Exact, explainable, cheap | Misses natural-language intent |
| D. Hybrid with reranking | Run B and C, fuse ranks, rerank with a small model, pack by budget | Best recall on both identifiers and intent | More components to run and tune |

**Choice:** D for chat and agent context; for completion, C plus open files
and recently edited files, with B only when the prefix is a comment or a
docstring. **Reason:** completion has a 500 ms budget and benefits most
from exact definitions of the symbols near the cursor; agents and chat ask
intent questions where semantic search helps.

Context assembly rules:

| Rule | Detail |
|---|---|
| CTX-01 *(Example)* | Chunk on syntax boundaries (function, class) using a parser, not fixed line counts |
| CTX-02 *(Example)* | Always include definitions of symbols referenced within 30 lines of the cursor |
| CTX-03 *(Example)* | Pack by priority until the token budget is spent; truncate the lowest-ranked snippet, never the cursor region |
| CTX-04 *(Example)* | Exclude files matched by the deny-list (`.env`, key files, `secrets/`) and generated or vendored code |
| CTX-05 *(Example)* | Agent prompts: stable system prompt, tools and skills index first; task history appended; compaction replaces old tool output with summaries only at checkpoints, so the cached prefix survives |

### 7.3 Deep dive 2: agentic edit loop and sandboxed execution

The agent alternates model turns with tool calls (read file, search, edit,
run command) until the tests pass, the budget ends or a human stops it.

```mermaid
stateDiagram-v2
  [*] --> Planning
  Planning --> Acting: plan accepted
  Acting --> Observing: tool result
  Observing --> Acting: continue
  Observing --> AwaitingApproval: risky action
  AwaitingApproval --> Acting: approved
  AwaitingApproval --> Stopped: rejected
  Observing --> Proposing: tests pass
  Observing --> Stopped: budget exhausted
  Proposing --> [*]: diff handed to reviewer
  Stopped --> [*]
```

| Option for where commands run | Strengths | Weaknesses |
|---|---|---|
| A. Developer laptop, with prompts for each command | Real environment, no infrastructure | Agent shares the developer's credentials, network and files; prompt fatigue leads to blanket approval |
| B. Containers on shared hosts | Cheap, fast start | Shared kernel; a container escape reaches other tenants |
| C. MicroVMs per task, restored from snapshots | Strong isolation, fast start from snapshot, disposable | Image and snapshot pipeline to maintain |

**Choice:** C by default, A only for repositories on an approved list with
read-only credentials. **Reason:** the agent runs code that it or the
repository wrote, including test fixtures and build scripts that could be
hostile; a disposable VM with no secrets and an egress allowlist makes a
successful injection mostly harmless.

Sandbox rules:

| Rule | Detail |
|---|---|
| SBX-01 *(Example)* | Fresh microVM per task from a snapshot of the base image plus a warmed dependency cache |
| SBX-02 *(Example)* | Repository mounted as a clone at the base commit; no source host credentials inside |
| SBX-03 *(Example)* | Egress allowlist: package mirrors and the internal artefact proxy only; everything else blocked and logged |
| SBX-04 *(Example)* | CPU, memory, disk and wall-clock limits per command and per task |
| SBX-05 *(Example)* | Destroyed on task end; logs and the diff exported, nothing else |

Budgets end runaway loops: maximum turns, maximum cost, maximum identical
failing command repeats (for example, three) and a wall-clock limit.

### 7.4 Deep dive 3: tool integration with MCP and Agent Skills

The Model Context Protocol (revision 2026-07-28) connects hosts (here, the
orchestrator and IDE), clients (one per server connection) and servers that
expose tools, resources and prompts over JSON-RPC 2.0. The specification
asks hosts to obtain explicit user consent before invoking tools and to
treat tool descriptions and annotations as untrusted unless they come from
a trusted server. Long-running operations can use the optional Tasks
extension where both sides support it.

| Option for tool governance | Strengths | Weaknesses |
|---|---|---|
| A. Users add any MCP server | Flexible | Unknown code with the user's tokens; tool descriptions can carry injected instructions |
| B. Platform allowlist with per-tool risk classes | Reviewed servers, consistent policy | Slower to add tools |
| C. No external tools | Smallest attack surface | Agents cannot reach issue trackers or documentation |

**Choice:** B. **Reason:** it keeps the benefit of tools while letting the
policy layer, not the model, decide what needs approval.

| Risk class | Examples | Policy |
|---|---|---|
| Read-only inside sandbox | Read file, search, list tests | Auto-approved |
| Write inside sandbox | Edit file, run tests, install from mirror | Auto-approved within budget |
| Read outside sandbox | Issue tracker read, documentation search over MCP | Auto-approved for allowlisted servers; logged |
| Write outside sandbox | Comment on an issue, create a branch, open a pull request | Human approval each time |
| Irreversible or privileged | Push to protected branch, change CI secrets, deploy | Not available to the agent |

Agent Skills package team procedures (for example, "how to add a database
migration here") as a folder with a `SKILL.md` file whose YAML front matter
holds a required `name` and `description`. Only names and descriptions are
loaded at start; the body and any `scripts/`, `references/` or `assets/`
files load when the skill is relevant, which keeps the stable prompt prefix
small. Skills come from reviewed repositories only, because a skill's
scripts run in the sandbox and its text enters the prompt. The optional
`allowed-tools` field is experimental in the specification, so the policy
layer, not the skill, remains the source of truth for permissions.
Detailed permission design belongs in Agent Tool Permission Design Template
(`ai-data/agent-tool-permission-design`); server contracts in MCP Server
Specification Template (`ai-data/mcp-server-spec`).

### 7.5 Review and approval loop

- The agent ends with a proposed diff, a summary of what it changed and why,
  and the test results from the sandbox.
- The developer reviews the diff in the IDE, can ask for changes (which
  continue the same task) and approves. Only then does the orchestrator
  push a branch and open a pull request through the source host API with
  the developer's identity.
- Pull requests carry a trailer such as `Assisted-by: [assistant name]` and
  a link to the task transcript, so reviewers know what to look at and
  audits can find AI-assisted changes.
- Normal branch protection, code owners and CI apply unchanged; the agent
  gets no bypass. Oversight patterns follow Human Oversight Design Template
  (`ai-data/human-oversight-design`).

### 7.6 How real systems do it

> **How real systems do it** *(paraphrased from Yang et al., "SWE-agent:
> Agent-Computer Interfaces Enable Automated Software Engineering",
> arXiv:2405.15793, 2024)*. The SWE-agent authors found that the interface
> between the model and the computer matters as much as the model. Instead
> of a raw shell, they gave the agent a small set of purpose-built commands
> for viewing a window of a file, searching, and editing a line range, with
> concise feedback after each action and a syntax check that rejects broken
> edits before they land. These interface choices raised the share of
> resolved issues on the SWE-bench benchmark compared with a plain shell.
> Design lesson for this document: invest in the tool surface (7.4) and
> in compact, structured tool output, not only in a larger model.

## 8. Scalability and performance

<!-- Guidance: Show how the design meets the numbers in section 3 and where it breaks first. -->

| Concern | Approach | Limit before redesign |
|---|---|---|
| Completion latency at peak | Regional completion pools, prefix cache, cancellation | GPU capacity per region |
| Agent token growth within a task | Append-only history, compaction at checkpoints, tool output truncation | Model context window |
| Index freshness | Webhook-driven incremental indexing by file hash | Indexer throughput on monorepo mass changes |
| Sandbox start time | Snapshot restore, warm pool sized from the hourly pattern | Node group quota |
| Large monorepos | Sparse index by directory ownership, path filters in queries | Single shard memory |

Completion latency budget (target p95 500 ms, *Example*):

| Step | Budget |
|---|---|
| Client to gateway (regional) | 40 ms |
| Auth, ACL cache, secret scan | 15 ms |
| Context assembly (index lookups in parallel) | 60 ms |
| Model time to first token | 200 ms |
| Stream first line to client | 40 ms |
| Headroom | 145 ms |

## 9. Reliability, security and operations

<!-- Guidance: Name failure modes and threats with detection and mitigation. Every mitigation should be testable. -->

### 9.1 Failure-mode table

| Component | Failure | Detection | Blast radius | Mitigation | RTO |
|---|---|---|---|---|---|
| Completion pool | Region overloaded | p95 latency, queue depth | Completions slow in one region | Shed to neighbour region; client hides late suggestions | 5 min |
| Agent model provider | Outage or rate limit | Error rate, 429s | Agent tasks pause | Fallback model for opted-in teams; tasks resume from event log | 30 min |
| Index | Stale after missed webhooks | Index lag metric per repository | Wrong or missing context | Periodic reconciliation by commit hash | 1 h |
| ACL cache | Stale permission grants | Drift check against source host | User sees snippets from a repository they lost access to | Short TTL, revoke webhook, fail closed on cache miss | 15 min |
| Sandbox manager | Node group exhausted | Pending sandbox count | New tasks wait | Queue with deadline, scale-out, priority for interactive tasks | 15 min |
| MCP server | Unavailable or misbehaving | Tool error rate | Tool missing in tasks | Circuit breaker; task continues without the tool and says so | Per server SLO |
| Orchestrator | Crash mid-task | Heartbeat | Running tasks on that instance | Resume from event log; sandboxes are idempotently reattached or rebuilt; actions outside the sandbox (branch, comment, pull request) carry an idempotency key of task ID and step, so a resumed step never repeats them | 10 min |

### 9.2 Security

Threat modelling follows Threat Model Template (`security/threat-model`)
and Agentic AI Threat Model Template (`ai-data/agentic-threat-model`). The
mapping below uses OWASP Top 10 for LLM Applications 2026 IDs.

| ID | Threat | OWASP 2026 | Control |
|---|---|---|---|
| SEC-01 | Secrets in source sent to the model | LLM02:2026 Sensitive Information Disclosure | Secret scanner on outbound context; deny-list paths; block and report (`secret-detected`) |
| SEC-02 | Instructions hidden in a README, issue, test fixture, dependency, command output or MCP tool result tell the agent to exfiltrate or run commands | LLM01:2026 Prompt Injection | Repository content and every tool result marked as untrusted data, never as instructions; egress allowlist; no secrets in sandbox; approval for any action outside the sandbox |
| SEC-03 | Agent takes actions beyond the task | LLM03:2026 Excessive Agency | Risk classes in 7.4; no privileged tools; budgets |
| SEC-04 | Agent adds a dependency that does not exist or is typo-squatted | LLM04:2026 Supply Chain | Package installs only through the internal proxy; new dependencies flagged in the diff; software composition analysis in CI |
| SEC-05 | System prompt, tool schemas or skill text reveal internal endpoints | LLM08:2026 Hidden Context Exposure | Treat hidden context as discoverable; keep credentials and internal hostnames out of it |
| SEC-06 | Generated code renders untrusted output (for example, model output rendered as HTML in the IDE panel) | LLM10:2026 Improper Output Handling | IDE panels render Markdown with sanitisation; no automatic link fetching |
| SEC-07 | Cross-repository leakage through the index | LLM02:2026 | Repository filter in every query (5.2); fail closed on ACL cache miss |
| [SEC-nn] | [Threat] | [OWASP ID] | [Control] |

Licence and provenance:

- Enable a filter that suppresses completions matching public code above a
  length threshold, or show the matching source and its licence when the
  organisation allows matches.
- Record AI assistance in commit trailers and keep task transcripts for
  the retention period, so provenance questions can be answered later.
- New third-party code and dependencies go through the normal open source
  review; see Supply Chain Security Plan Template
  (`security/supply-chain-security`).

### 9.3 Observability

Instrument with LLM and Agent Observability Specification Template
(`ai-data/llm-observability-spec`): one trace per task, spans per model
turn, tool call and sandbox command.

| Signal | Metric | Alert threshold |
|---|---|---|
| Completion latency | p95 by region and language | Above 500 ms for 10 min |
| Completion acceptance | Accepted suggestions over shown | Drop of 20% week over week |
| Agent success | Tasks ending in an approved diff over tasks started | Drop of 15% after a model change |
| Agent cost | USD per task, p95 | Above budget for a team |
| Approval wait | Time from request to decision | Informational; tune risk classes |
| Sandbox egress blocks | Blocked connections per task | Spike may indicate injection attempts |
| Secret detections | Count per repository | Any new repository with detections |

### 9.4 Cost estimate

| Item | Unit cost | Quantity | Monthly cost |
|---|---|---|---|
| Agent model tokens | USD per million tokens (3.4) | [Tokens] | [USD] |
| Completion GPUs | [USD per GPU-hour] | [GPU-hours] | [USD] |
| Sandboxes | [USD per vCPU-hour] | [vCPU-hours] | [USD] |
| Index and storage | [USD per node-month] | [Nodes] | [USD] |
| **Total** | | | **[USD]** |

## 10. Trade-offs and alternatives

<!-- Guidance: List options for each major decision, what each costs and why you chose one. Include the 10x / 0.1x row and answer it honestly. -->

| Decision | Options | Chosen | Why | What we give up |
|---|---|---|---|---|
| Completion model | Large hosted model; small self-hosted | Small self-hosted | Latency and token volume | Some quality on long completions |
| Context retrieval | Embeddings only; lexical only; hybrid | Hybrid with reranking | Identifiers and intent both matter | Two index types to operate |
| Execution | Laptop; containers; microVMs | MicroVMs by default | Isolation against hostile repository content | Image pipeline and cost |
| Tool access | Open; allowlist; none | Allowlist with risk classes | Usefulness with control | Slower tool onboarding |
| Merge authority | Agent merges; human approves | Human approves every diff | Accountability and review | Throughput of trivial fixes |
| What changes at 10x (200 000 developers) | | Regional cells with own index shards, sandbox pools and completion pools; per-tenant budgets enforced in each cell | Single global orchestrator and index would become the bottleneck | More operational surface |
| What changes at 0.1x (2 000 developers) | | Hosted completion API instead of self-hosted GPUs; containers with gVisor-style isolation instead of a microVM fleet | GPU and microVM fixed costs dominate at small scale | Less control over latency and data path |
| [Decision] | [Options] | [Chosen] | [Why] | [Cost] |

## 11. Delivery plan and risks

<!-- Guidance: How the design reaches production safely, and what could stop it. -->

### 11.1 Milestones

| Milestone | Scope | Exit criterion | Target date |
|---|---|---|---|
| M1 | Gateway, policy layer, completion path, lexical index | Pilot of 200 developers meets NFR-01 | YYYY-MM-DD |
| M2 | Chat with citations, hybrid index | Answer citation accuracy checked on a labelled set | YYYY-MM-DD |
| M3 | Agent tasks in microVM sandboxes, approval loop | Penetration test of sandbox passes NFR-04 | YYYY-MM-DD |
| M4 | MCP allowlist and Agent Skills | Three reviewed servers and five skills live | YYYY-MM-DD |
| M5 | Organisation-wide rollout | Budgets, chargeback and audit reports live | YYYY-MM-DD |

### 11.2 Rollout and rollback

[*Example:* Roll out by team with a feature flag in the IDE extension. Model
changes go to 5% of users, compared on acceptance rate and agent success;
automatic rollback when either drops past its threshold. The previous
extension version stays installable for 30 days.]

### 11.3 Risks

| ID | Risk | Likelihood | Impact | Mitigation | Owner |
|---|---|---|---|---|---|
| R-01 | *Example:* Prompt injection leads to data exfiltration | Medium | High | Sandbox without secrets, egress allowlist, approvals | [@name] |
| R-02 | *Example:* Token cost exceeds budget | Medium | Medium | Caching, compaction, budgets, model routing | [@name] |
| R-03 | *Example:* Developers approve diffs without reading | High | Medium | Small diffs, test evidence in the summary, code owner review unchanged | [@name] |
| R-04 | *Example:* Licence claim over generated code | Low | High | Public code filter, provenance records, legal review | [@name] |
| [R-nn] | [Risk] | [L / M / H] | [L / M / H] | [Mitigation] | [@name] |

## 12. Appendices

<!-- Guidance: Traceability, open questions, glossary and references. -->

### 12.1 Traceability

| Requirement | Design element | Verification |
|---|---|---|
| FR-03 | 7.3 agent loop and sandbox | Internal task benchmark |
| FR-04 | 7.5 approval loop | Audit of pull requests with assistance trailer |
| FR-05 | 5.2 repository filter, 9.2 SEC-07 | Cross-repository leakage test |
| FR-06 | 7.4 MCP allowlist and risk classes | Tool policy unit tests |
| NFR-01 | 7.1, 8 latency budget | Client telemetry at peak |
| NFR-04 | 7.3 SBX rules | Penetration test |
| [ID] | [Element] | [Test] |

### 12.2 Open questions

| ID | Question | Owner | Due |
|---|---|---|---|
| OQ-01 | *Example:* Allow local mode for repositories that need hardware access? | [@name] | YYYY-MM-DD |
| OQ-02 | *Example:* Retain transcripts longer for regulated codebases? | [@name] | YYYY-MM-DD |
| [OQ-nn] | [Question] | [@name] | YYYY-MM-DD |

### 12.3 Glossary

| Term | Meaning |
|---|---|
| Fill-in-the-middle | Prompt format giving the model code before and after the cursor |
| Agent loop | Repeated cycle of model turn, tool call and observation |
| MCP | Model Context Protocol: hosts, clients and servers exchanging tools, resources and prompts over JSON-RPC 2.0 |
| Agent Skill | Folder with a `SKILL.md` file (name, description, instructions) and optional scripts and references, loaded on demand |
| MicroVM | Lightweight virtual machine with its own kernel, used for strong isolation with fast start |
| Prefix cache | Reuse of computed attention state or billed tokens for an unchanged prompt prefix |
| Acceptance rate | Share of shown completions the developer keeps |

### 12.4 References

| Reference | Use |
|---|---|
| Model Context Protocol specification, revision 2026-07-28, https://modelcontextprotocol.io/specification/2026-07-28 | Roles, features, consent and tool safety principles in 7.4 |
| Agent Skills specification, https://agentskills.io/specification | `SKILL.md` format and progressive disclosure in 7.4 |
| OWASP Top 10 for LLM Applications 2026, OWASP GenAI Security Project, https://genai.owasp.org/ | Threat mapping in 9.2 |
| Yang et al., "SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering", arXiv:2405.15793 (2024) | Box in 7.6 |
| Jimenez et al., "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?", ICLR 2024 (arXiv:2310.06770) | Benchmark style for the internal task suite |
| RFC 9457, Problem Details for HTTP APIs | Error format in 4.3 |
| LLM Inference Serving System Design (`system-design/design-llm-inference-serving`) | Model pools behind this design |
| AI Agent Specification Template (`ai-data/ai-agent-spec`) | Agent behaviour specification |
| Guardrails Specification Template (`ai-data/guardrails-spec`) | Input and output guardrails |
| RAG System Design Template (`ai-data/rag-system-design`) | Retrieval patterns for chat |

## Standard and licence note

Based on the Model Context Protocol specification (revision 2026-07-28),
the Agent Skills specification and the OWASP Top 10 for LLM Applications
2026 by the OWASP GenAI Security Project (https://genai.owasp.org/),
licensed under Creative Commons Attribution-ShareAlike 4.0
(https://creativecommons.org/licenses/by-sa/4.0/). The OWASP IDs and short
names are used for the mapping in 9.2; all descriptions and controls are
original text. Because that mapping adapts the OWASP structure, the adapted
structure is shared under CC BY-SA 4.0 and any further adaptation must
carry the same licence. The research paper in 7.6 is paraphrased, not
copied. All numbers are illustrative examples, not benchmarks. Check the
current official versions of each specification and list before use.
Apart from the adapted OWASP structure, the original text is under the
template licence. Template licence: MIT.