System Design: [AI coding assistant name]

Reference design for an AI coding assistant used by an engineering organisation from the IDE and the command line. It covers repository indexing and code search, context assembly, the split between inline completion and agentic edit loops, sandboxed tool execution and tests, the review and approval loop, tool integration through the Model Context Protocol (MCP, revision 2026-07-28) and Agent Skills (agentskills.io specification), latency and cost, and the security problems specific to code: secrets, prompt injection from repository content, and licence and provenance of generated code. Threats are mapped to the OWASP Top 10 for LLM Applications 2026. It follows the 12-section skeleton of the System Design Document template and is pre-filled with a worked design to adapt. Replace every [bracketed] value and every row marked Example; all numbers are illustrative examples, not benchmarks.

Document control

Approvals

Revision history

Contents

1. Overview
2. Requirements
3. Capacity estimation
4. API design
5. Data model
6. High-level design
7. Detailed design
8. Scalability and performance
9. Reliability, security and operations
10. Trade-offs and alternatives
11. Delivery plan and risks
12. Appendices

1. Overview

1.1 Problem statement

[Example: Developers use three unmanaged AI tools with personal accounts. Source code leaves the company without a data processing agreement, nobody can tell which commits were AI-assisted, and one tool executed a shell command suggested by a README in a cloned third-party repository. Teams want completions inside the IDE and an agent that can fix a failing test, run the suite and open a pull request, under company controls.]

1.2 Proposed solution

[Example: One managed assistant with two clients (IDE extension and CLI) and one backend. Inline completion goes through a low-latency path to a small self-hosted model with a context window built from the open file, neighbouring files and the repository index. Agent tasks run a plan, act, observe loop with a larger model; every file edit and command runs in an isolated remote sandbox with a copy of the repository, no production credentials and an egress allowlist. The agent proposes a diff; a human approves it in the IDE or as a pull request. External tools are attached only through allowlisted MCP servers, and team procedures are packaged as Agent Skills. All requests pass a gateway that enforces identity, repository permissions, secret scanning and cost budgets.]

1.3 Decisions requested from reviewers

1.4 45-minute interview path

2. Requirements

2.1 Core functional requirements

2.2 Non-functional requirements

2.3 Below the line (out of scope)

• [Example: model training or fine-tuning on internal code (separate project with its own data review)]
• [Example: autonomous merging to protected branches; the agent never merges]
• [Example: agents that run on a schedule without a user who started them]
• [Example: code review bots on every pull request (consumer of this platform, designed separately)]
• [Item deferred, with the release that will take it]

3. Capacity estimation

3.1 Assumptions

All values are Example planning inputs.

3.2 Completion traffic

Requests per day: 12 000 x 400 = 4.8 million. Spread over the 8-hour window (28 800 s) that is about 167 requests/s on average and about 500 requests/s at peak (ratio 3). Prompt tokens per day: 4.8 million x 2 000 = 9.6 billion.

Consequence: completion needs a dedicated small-model pool with prefix caching and request cancellation; about half of the requests are cancelled by the next keystroke, so the server must abort generation on client disconnect or it pays for tokens nobody sees.

3.3 Agent traffic

Tasks per day: 12 000 x 3 = 36 000. Per task: 25 turns x 30 000 = 750 000 input tokens and 25 x 800 = 20 000 output tokens. Per day: 36 000 x 750 000 = 27 billion input tokens and 36 000 x 20 000 = 720 million output tokens.

Consequence: agent mode is under 1% of requests (36 000 against 4.8 million) yet produces almost three times the input tokens of completion. Cost control is a context design problem: keep the prompt prefix stable so it is cached, and compact the history before it grows past a set size.

3.4 Cost of agent tokens

With Example list prices of USD 3 per million input tokens, USD 0.30 per million cached input tokens and USD 15 per million output tokens:

Per active developer that is about USD 7.65 per day without caching and about USD 2.79 with it. Consequence: the context engine (7.2) must append to the conversation rather than rewrite earlier turns, because any change to the prefix invalidates the cache. Budgets per task and per day are enforced at the gateway.

3.5 Index and sandboxes

3.6 Summary

4. API design

4.1 Endpoints

4.2 Request example

An agent task start is shown below. The idempotency key makes a retried start return the existing task instead of creating a second sandbox. Completion requests differ: the client sends what it knows cheaply (cursor, prefix, suffix, open file, recent edits) with a request ID, the server adds repository context, and the request ID lets the server cancel generation when a newer request arrives from the same editor.

POST /v1/tasks
Authorization: Bearer [user token]
Idempotency-Key: [uuid]
Content-Type: application/json

{"repository": "[org/repo]", "base_ref": "main", "instruction": "Fix the failing test in billing/invoice_test.py", "budget": {"max_turns": 40, "max_cost_usd": 5}, "approval_policy": "default"}

4.3 Errors

Errors use RFC 9457 problem details.

5. Data model

5.1 Entities

Index entities are derived from the source host and can be rebuilt; task, tool call and approval records are the audit trail and must be retained.

5.2 Access patterns

5.3 Storage choices

6. High-level design

7. Detailed design

7.1 Completion request flow

The client debounces keystrokes, sends prefix and suffix around the cursor and a list of recently edited and open files. The server uses a fill-in-the-middle prompt format so the model sees code after the cursor. Suggestions stop at a syntactic boundary (end of block or line) decided by the client's parser, which keeps them short and cheap.

7.2 Deep dive 1: repository indexing and context assembly

The quality of both completion and agent work depends more on what enters the context window than on the model. The context engine has a fixed token budget and must fill it with the most useful snippets.

Choice: D for chat and agent context; for completion, C plus open files and recently edited files, with B only when the prefix is a comment or a docstring. Reason: completion has a 500 ms budget and benefits most from exact definitions of the symbols near the cursor; agents and chat ask intent questions where semantic search helps.

Context assembly rules:

7.3 Deep dive 2: agentic edit loop and sandboxed execution

The agent alternates model turns with tool calls (read file, search, edit, run command) until the tests pass, the budget ends or a human stops it.

Choice: C by default, A only for repositories on an approved list with read-only credentials. Reason: the agent runs code that it or the repository wrote, including test fixtures and build scripts that could be hostile; a disposable VM with no secrets and an egress allowlist makes a successful injection mostly harmless.

Sandbox rules:

Budgets end runaway loops: maximum turns, maximum cost, maximum identical failing command repeats (for example, three) and a wall-clock limit.

7.4 Deep dive 3: tool integration with MCP and Agent Skills

The Model Context Protocol (revision 2026-07-28) connects hosts (here, the orchestrator and IDE), clients (one per server connection) and servers that expose tools, resources and prompts over JSON-RPC 2.0. The specification asks hosts to obtain explicit user consent before invoking tools and to treat tool descriptions and annotations as untrusted unless they come from a trusted server. Long-running operations can use the optional Tasks extension where both sides support it.

Choice: B. Reason: it keeps the benefit of tools while letting the policy layer, not the model, decide what needs approval.

Agent Skills package team procedures (for example, "how to add a database migration here") as a folder with a SKILL.md file whose YAML front matter holds a required name and description. Only names and descriptions are loaded at start; the body and any scripts/, references/ or assets/ files load when the skill is relevant, which keeps the stable prompt prefix small. Skills come from reviewed repositories only, because a skill's scripts run in the sandbox and its text enters the prompt. The optional allowed-tools field is experimental in the specification, so the policy layer, not the skill, remains the source of truth for permissions. Detailed permission design belongs in Agent Tool Permission Design Template (ai-data/agent-tool-permission-design); server contracts in MCP Server Specification Template (ai-data/mcp-server-spec).

7.5 Review and approval loop

• The agent ends with a proposed diff, a summary of what it changed and why, and the test results from the sandbox.
• The developer reviews the diff in the IDE, can ask for changes (which continue the same task) and approves. Only then does the orchestrator push a branch and open a pull request through the source host API with the developer's identity.
• Pull requests carry a trailer such as Assisted-by: [assistant name] and a link to the task transcript, so reviewers know what to look at and audits can find AI-assisted changes.
• Normal branch protection, code owners and CI apply unchanged; the agent gets no bypass. Oversight patterns follow Human Oversight Design Template (ai-data/human-oversight-design).

7.6 How real systems do it

How real systems do it (paraphrased from Yang et al., "SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering", arXiv:2405.15793, 2024). The SWE-agent authors found that the interface between the model and the computer matters as much as the model. Instead of a raw shell, they gave the agent a small set of purpose-built commands for viewing a window of a file, searching, and editing a line range, with concise feedback after each action and a syntax check that rejects broken edits before they land. These interface choices raised the share of resolved issues on the SWE-bench benchmark compared with a plain shell. Design lesson for this document: invest in the tool surface (7.4) and in compact, structured tool output, not only in a larger model.

8. Scalability and performance

Completion latency budget (target p95 500 ms, Example):

9. Reliability, security and operations

9.1 Failure-mode table

9.2 Security

Threat modelling follows Threat Model Template (security/threat-model) and Agentic AI Threat Model Template (ai-data/agentic-threat-model). The mapping below uses OWASP Top 10 for LLM Applications 2026 IDs.

Licence and provenance:

• Enable a filter that suppresses completions matching public code above a length threshold, or show the matching source and its licence when the organisation allows matches.
• Record AI assistance in commit trailers and keep task transcripts for the retention period, so provenance questions can be answered later.
• New third-party code and dependencies go through the normal open source review; see Supply Chain Security Plan Template (security/supply-chain-security).

9.3 Observability

Instrument with LLM and Agent Observability Specification Template (ai-data/llm-observability-spec): one trace per task, spans per model turn, tool call and sandbox command.

9.4 Cost estimate

10. Trade-offs and alternatives

11. Delivery plan and risks

11.1 Milestones

11.2 Rollout and rollback

[Example: Roll out by team with a feature flag in the IDE extension. Model changes go to 5% of users, compared on acceptance rate and agent success; automatic rollback when either drops past its threshold. The previous extension version stays installable for 30 days.]

11.3 Risks

12. Appendices

12.1 Traceability

12.2 Open questions

12.3 Glossary

12.4 References

Standard and licence note

Based on the Model Context Protocol specification (revision 2026-07-28), the Agent Skills specification and the OWASP Top 10 for LLM Applications 2026 by the OWASP GenAI Security Project (https://genai.owasp.org/), licensed under Creative Commons Attribution-ShareAlike 4.0 (https://creativecommons.org/licenses/by-sa/4.0/). The OWASP IDs and short names are used for the mapping in 9.2; all descriptions and controls are original text. Because that mapping adapts the OWASP structure, the adapted structure is shared under CC BY-SA 4.0 and any further adaptation must carry the same licence. The research paper in 7.6 is paraphrased, not copied. All numbers are illustrative examples, not benchmarks. Check the current official versions of each specification and list before use. Apart from the adapted OWASP structure, the original text is under the template licence. Template licence: MIT.