# System Design: [Chat assistant name]

> Reference design for the application layer of a consumer or workplace chat
> assistant in the style of the well-known general-purpose AI chat products:
> token streaming over server-sent events or WebSocket, the conversation
> store, context window management and summarisation, file uploads, tool use,
> moderation, rate limits and abuse prevention, opt-in memory and regional
> data residency. Model serving is out of scope and covered by the LLM
> Inference Serving and Vector Search Platform Design Template
> (`system-design/design-llm-inference-serving`); provider routing by the LLM
> Gateway System Design Template (`system-design/design-llm-gateway`). It
> follows the 12-section structure of the System Design Document Template
> (`system-design/system-design-document`) and is pre-filled with a worked
> design to adapt. Replace every `[bracketed]` value and every row marked
> *Example*; all numbers are illustrative, not benchmarks. Where it mentions
> laws, this is a documentation format, not legal advice.

<!--
Guidance: How to use this template
- Start from user behaviour: daily active users, messages per user, answer
  length and how long conversations get (3.1). Context length per turn, not
  user count, sets most of the model cost.
- Separate generation from delivery: the answer keeps generating when the
  user's connection drops, and the client reconnects to the same stream (7.2).
- Decide what you store, where and for how long before you design memory or
  search; residency and deletion rules shape the data model (7.5).
- Confirm the current legal texts that apply to your markets with counsel;
  the references here are pointers, not advice.
-->

## Document control

<!-- Guidance: A design is a proposal until approved. Record the markets and data regions the approval covers. -->

| Field | Value |
|---|---|
| Document ID | [SDD-CAS-001] |
| System / feature | [Chat assistant application layer] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Implemented / Superseded] |
| Owner | [@owner] |
| Approvers / reviewers | [@reviewer-1], [@reviewer-2] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal] |
| Markets and data regions | [EU, US, other] |
| Model access | [LLM gateway aliases used] |
| Related | [LLM gateway design], [Guardrails spec], [DPIA], [Observability spec] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Tech lead, assistant | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| Staff / principal engineer | [@name] | [Decision] | YYYY-MM-DD |
| Privacy / data protection officer | [@name] | [Decision] | YYYY-MM-DD |
| Trust and safety | [@name] | [Decision] | YYYY-MM-DD |
| SRE / operations | [@name] | [Decision] | YYYY-MM-DD |

## Revision history

<!-- Guidance: Record design changes, not wording changes. -->

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft for design review |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Made streams resumable after disconnect |
| 0.3 | YYYY-MM-DD | [@name] | *Example:* Added EU data region and per-region conversation stores |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved |

## Contents

<!-- Guidance: Keep aligned with the headings. -->

1. [Overview](#1-overview)
2. [Requirements](#2-requirements)
3. [Capacity estimation](#3-capacity-estimation)
4. [API design](#4-api-design)
5. [Data model](#5-data-model)
6. [High-level design](#6-high-level-design)
7. [Detailed design](#7-detailed-design)
8. [Scalability and performance](#8-scalability-and-performance)
9. [Reliability, security and operations](#9-reliability-security-and-operations)
10. [Trade-offs and alternatives](#10-trade-offs-and-alternatives)
11. [Delivery plan and risks](#11-delivery-plan-and-risks)
12. [Appendices](#12-appendices)

## 1. Overview

<!-- Guidance: One page a busy reviewer can read alone: problem, solution in a paragraph, decisions requested. -->

### 1.1 Problem statement

[*Example:* The current assistant is a prototype: each message opens a
WebSocket, the server calls the model and forwards tokens. When a phone
switches networks the answer is lost and the user is charged a message
against their limit. Conversations are sent whole to the model, so long
threads hit the context limit and fail. Uploaded files sit in one bucket in
one region. Business customers in the EU require their data to stay in the
EU, and the product has no way to honour deletion requests across logs and
caches.]

### 1.2 Proposed solution

[*Example:* A regional chat service that owns conversations and the turn
loop. Each user turn becomes a generation job; the generator streams tokens
into a short-lived per-turn stream buffer, and clients read it over
server-sent events, reconnecting with the last event ID. A context builder
assembles each prompt within a token budget: system instructions, a rolling
summary of older turns, the most recent turns verbatim, retrieved memory
(only if the user turned memory on) and file excerpts. Moderation runs on
input and on streamed output. Rate limits apply per user, per plan and per
abuse signal. Every user is homed in one data region; conversations, files,
memory and logs stay in that region.]

### 1.3 Decisions requested from reviewers

| ID | Question | Recommended option | Section |
|---|---|---|---|
| Q-01 | *Example:* Streaming transport? | SSE over HTTP/2 with resumable per-turn buffers; WebSocket only for voice | 7.2 |
| Q-02 | *Example:* Long conversations? | Rolling summary plus recent turns within a token budget | 7.3 |
| Q-03 | *Example:* Memory? | Off by default; user-visible and editable; per-region store | 7.5 |
| Q-04 | *Example:* Output moderation? | Classify every 200 tokens (*Example*) and at the end; stop the stream on a block | 7.4 |
| Q-05 | *Example:* Residency? | Home region per user or workspace; no cross-region replication of content | 7.5 |
| [Q-nn] | [Question] | [Option] | [Section] |

### 1.4 45-minute interview path

<!-- Guidance: Use this order for an interview or a time-boxed review. Pick one deep dive. -->

| Minutes | Topic | What to show |
|---|---|---|
| 0 to 5 | Scope | Application layer only; model serving below the line |
| 5 to 10 | Requirements | Core list (2.1), time to first token and availability (2.2), below the line (2.3) |
| 10 to 15 | Estimates | Concurrent streams, tokens per day, storage (3.2 to 3.5) with consequences |
| 15 to 20 | API | Send a message, stream events, resume (4) |
| 20 to 28 | Data and high-level design | Access patterns (5.2), diagram (6) |
| 28 to 40 | One deep dive | Resumable streaming (7.2) or context management (7.3); moderation and abuse (7.4) for trust and safety interviews |
| 40 to 45 | Wrap-up | Failure modes (9.1), 10x row (10) |

## 2. Requirements

<!-- Guidance: Split core requirements from what is explicitly below the line. Give every quality a number. -->

### 2.1 Core functional requirements

| ID | Requirement | Priority |
|---|---|---|
| FR-01 | *Example:* Send a message and receive the answer as a token stream | Must |
| FR-02 | *Example:* Resume an interrupted stream without regenerating the answer | Must |
| FR-03 | *Example:* List, open, rename, search and delete conversations | Must |
| FR-04 | *Example:* Regenerate and edit a previous message, keeping branches | Should |
| FR-05 | *Example:* Upload files (documents, images) and ask about them | Must |
| FR-06 | *Example:* Tool use: web search, code execution, and workspace connectors | Should |
| FR-07 | *Example:* Moderation of input and output with user-facing explanations | Must |
| FR-08 | *Example:* Rate limits per plan with clear messages and reset times | Must |
| FR-09 | *Example:* Opt-in memory that the user can view, edit and clear | Should |
| FR-10 | *Example:* Data export and deletion per user and per workspace | Must |
| FR-11 | *Example:* Tell users they are interacting with an AI system where that is not obvious | Must |
| [FR-nn] | [Requirement] | [Must / Should / Could] |

### 2.2 Non-functional requirements

| ID | Quality | Target | Measured as |
|---|---|---|---|
| NFR-01 | Time to first token | *Example:* p95 at or below 1.5 s from send to first token on screen, excluding tool calls | Client timing |
| NFR-02 | Stream continuity | *Example:* 99.9% of streams complete or resume after a disconnect | Client telemetry |
| NFR-03 | Availability | *Example:* 99.9% monthly for send and stream | SLO on turn completion |
| NFR-04 | Conversation load | *Example:* p95 at or below 300 ms to open a conversation of 200 messages | Server timing |
| NFR-05 | Deletion | *Example:* User deletion completed across stores, caches and logs within 30 days | Deletion audit |
| NFR-06 | Residency | *Example:* No user content stored or processed outside the home region, except with explicit transfer basis | Data-flow audit |
| NFR-07 | Moderation latency | *Example:* Input check p95 at or below 150 ms | Span timing |
| [NFR-nn] | [Quality] | [Target] | [Measurement] |

### 2.3 Below the line (out of scope)

- [*Example:* Model serving, batching and GPUs (see LLM Inference Serving and Vector Search Platform Design Template (`system-design/design-llm-inference-serving`))]
- [*Example:* Provider routing, quotas towards providers and redaction (see LLM Gateway System Design Template (`system-design/design-llm-gateway`))]
- [*Example:* Running user or model code (see Agent Code Execution Sandbox System Design Template (`system-design/design-agent-sandbox-execution`))]
- [*Example:* Long-running autonomous agents (see Agent Orchestration Platform System Design Template (`system-design/design-agent-orchestration-platform`))]
- [*Example:* Billing and subscription management]
- [Item deferred, with the release that will take it]

## 3. Capacity estimation

<!-- Guidance: Every number must end in a consequence. Show inputs, formulas and results. -->

### 3.1 Assumptions

All values are *Example* planning inputs.

| Input | Value | Source |
|---|---|---|
| Daily active users | 5 million | Growth plan |
| Messages per user per day | 6 | Beta cohort |
| Peak-to-average ratio | 3 | Hourly pattern |
| Mean answer length | 400 tokens | Beta cohort |
| Mean stream duration | 12 s | Beta cohort |
| Mean prompt assembled per turn | 6,000 tokens (system 1,500; history 3,500; memory and files 1,000) | Context builder logs |
| Stored size per exchange (message and answer) | 3 KB | Schema |
| Messages with a file | 3%, mean 2 MB | Beta cohort |
| Streams per edge node | 2,500 | Load test |
| Internal cost, input | USD 0.50 per million tokens | Gateway metering |
| Internal cost, output | USD 2 per million tokens | Gateway metering |

### 3.2 Traffic and streams

| Quantity | Calculation | Result | Consequence |
|---|---|---|---|
| Messages per day | 5 million x 6 | 30 million | |
| Average messages per second | 30,000,000 / 86,400 | about 347 | |
| Peak messages per second | 347 x 3 | about 1,042 | Turn admission, moderation and rate-limit checks must handle about 1,050 per second |
| Peak concurrent streams | 1,042 x 12 s | about 12,500 | Edge sized for open streams, not requests |
| Edge nodes | 12,500 / 2,500 = 5 at full load; 9 across 3 zones | 9 nodes | Losing one zone leaves 6 nodes (15,000 streams), about 83% load |

### 3.3 Tokens and model cost

| Quantity | Calculation | Result | Consequence |
|---|---|---|---|
| Input tokens per day | 30 million x 6,000 | 180 billion | |
| Input cost per day | 180,000 x USD 0.50 | USD 90,000 | Input dominates; context size is the first cost lever |
| Output tokens per day | 30 million x 400 | 12 billion | |
| Output cost per day | 12,000 x USD 2 | USD 24,000 | |
| Saving from summarising history (3,500 to 2,000 tokens) | 1,500 x 30 million = 45 billion tokens | USD 22,500 per day | Summarisation pays for itself if it costs well under that; run it every 10 turns, not every turn |

Keep the system prompt and tool definitions at the start of the prompt and
unchanged between turns, so provider or engine prefix caching can reuse them.

### 3.4 Storage

| Quantity | Calculation | Result | Consequence |
|---|---|---|---|
| Conversation data per day | 30 million x 3 KB | 90 GB | |
| Conversation data per year | 90 GB x 365 | about 32.9 TB | Partition by user ID; move conversations untouched for 90 days to a cheaper tier |
| Files per day | 30 million x 3% | 900,000 | |
| File bytes per day | 900,000 x 2 MB | 1.8 TB | Uploads go straight to object storage with pre-signed URLs, never through the chat service |
| File bytes, 90-day retention | 1.8 TB x 90 | 162 TB | File retention shorter than conversation retention, with clear notice to users |

### 3.5 Moderation and summarisation calls

| Quantity | Calculation | Result | Consequence |
|---|---|---|---|
| Input moderation per day | 30 million | 30 million | Fast classifier, not a large model |
| Output moderation checks per day | 30 million x (400 / 200) | 60 million | Batched per stream chunk; budget in 8 |
| Summarisation calls per day | One per 10 turns: 30 million / 10 | 3 million | Asynchronous, off the turn path |

### 3.6 Summary

| Quantity | Estimate | Design consequence |
|---|---|---|
| Peak streams | about 12,500 | 9 edge nodes across 3 zones; scale on open streams |
| Input tokens | 180 billion per day | Token budget per turn; rolling summary; stable prefix |
| Conversations | about 33 TB per year | User-partitioned store with tiering |
| Files | 1.8 TB per day | Direct-to-storage upload, async processing, 90-day retention |
| Moderation | about 90 million checks per day | Dedicated classifier pool |

## 4. API design

<!-- Guidance: Define the contract: resources, streaming format, resumption, errors and idempotency. -->

### 4.1 Endpoints

| Method and path | Purpose |
|---|---|
| `POST /v1/conversations` | Create a conversation |
| `GET /v1/conversations?cursor=` | List conversations, newest first |
| `GET /v1/conversations/{id}/messages?cursor=` | Page through messages, including branches |
| `POST /v1/conversations/{id}/turns` | Send a user message; returns `turn_id` and stream URL; `Idempotency-Key` required |
| `GET /v1/turns/{turn_id}/stream` | SSE stream of the answer; honours `Last-Event-ID` |
| `POST /v1/turns/{turn_id}/stop` | Stop generation |
| `POST /v1/uploads` | Get a pre-signed upload URL and a file ID |
| `GET /v1/memory`, `DELETE /v1/memory/{item}` | View and remove memory items |
| `POST /v1/export`, `DELETE /v1/account/data` | Data export and deletion |

### 4.2 Stream format

The stream uses `text/event-stream` (WHATWG HTML Living Standard section 9.2).
Each event has an `id` equal to its sequence number in the turn, so a client
that reconnects with `Last-Event-ID: 41` receives events from 42 onward.

```text
id: 1
event: turn.started
data: {"turn_id": "t_9f2", "model_alias": "chat-default"}

id: 2
event: token
data: {"text": "Here is"}

id: 3
event: tool.call
data: {"tool": "web_search", "status": "running"}

id: 58
event: turn.completed
data: {"finish_reason": "stop", "output_tokens": 412}
```

### 4.3 Errors

| Status | Problem type | Meaning |
|---|---|---|
| 400 | `message-too-long` | Message above the plan's input limit |
| 409 | `turn-in-progress` | Another turn is generating in this conversation |
| 413 | `file-too-large` | Upload above the plan's limit |
| 422 | `content-blocked` | Input blocked by moderation; category given |
| 429 | `rate-limited` | Plan or abuse limit reached; reset time given |
| 451 | `region-unavailable` | Feature not offered in the user's region |

Errors use RFC 9457 problem details. A retried `POST /turns` with the same
`Idempotency-Key` returns the existing turn, so a flaky network does not
create two answers or count twice against the limit.

## 5. Data model

<!-- Guidance: Choose storage by access pattern. List each query, the key that serves it and why. -->

### 5.1 Entities

A conversation is a tree of messages (edits and regenerations create
branches); the client shows one path through it.

```mermaid
erDiagram
  USER ||--o{ CONVERSATION : owns
  CONVERSATION ||--o{ MESSAGE : contains
  MESSAGE ||--o{ MESSAGE : "parent of"
  MESSAGE ||--o{ ATTACHMENT : references
  CONVERSATION ||--o| SUMMARY : "summarised by"
  USER ||--o{ MEMORY_ITEM : "has (opt-in)"
  USER {
    string user_id PK
    string home_region
    string plan
    bool memory_enabled
  }
  CONVERSATION {
    string conversation_id PK
    string user_id FK
    string title
    datetime updated_at
  }
  MESSAGE {
    string message_id PK
    string conversation_id FK
    string parent_id
    string role
    string content
    int tokens
  }
  SUMMARY {
    string conversation_id PK
    string covers_until_message
    string text
  }
  MEMORY_ITEM {
    string item_id PK
    string user_id FK
    string text
    string source_message
  }
  ATTACHMENT {
    string file_id PK
    string object_key
    string status
  }
```

### 5.2 Access patterns

| ID | Query | Key or index that serves it | Partition key and rationale |
|---|---|---|---|
| AP-01 *(Example)* | List a user's conversations, newest first | `(user_id, updated_at desc)` | `user_id`: one user's data together; also the unit of deletion and residency |
| AP-02 *(Example)* | Load the last 50 messages of a conversation | `(conversation_id, created_at desc)` | Clustered under the user partition |
| AP-03 *(Example)* | Build the context for a turn | Summary by `conversation_id` plus messages after `covers_until_message` | Two reads, both on the same partition |
| AP-04 *(Example)* | Resume a stream | Turn stream buffer by `turn_id`, entries after `Last-Event-ID` | In-memory store with TTL of 10 min |
| AP-05 *(Example)* | Search a user's conversations | Per-region search index filtered by `user_id` | Index never shared across regions |
| AP-06 *(Example)* | Retrieve relevant memory | Vector index per region, filter `user_id` | Only if `memory_enabled` |
| AP-07 *(Example)* | Delete everything for a user | Partition drop by `user_id` plus deletion jobs for index, files, logs | Deletion follows the partition key |

### 5.3 Storage choices

| Data | Store | Notes |
|---|---|---|
| Users, plans, settings | Relational database per region | Small |
| Conversations and messages | Wide-column or document store per region, partitioned by user | Tiering after 90 days |
| Turn stream buffers | In-memory store per region, TTL 10 min | Lost buffers fall back to the stored message |
| Files | Object storage per region, encrypted | Extracted text stored next to the file |
| Search and memory indexes | Per-region search and vector indexes | Rebuildable from the source stores |
| Abuse signals | Per-region store with short retention | Feeds rate limits |

## 6. High-level design

<!-- Guidance: One diagram with every component on the turn path and where state lives. -->

```mermaid
flowchart LR
  client([Web and mobile clients]) --> edge[Edge: TLS, auth, region routing]
  edge --> chat[Chat service: turns, conversations]
  chat --> rl[Rate limits and abuse signals]
  chat --> mod[Moderation service]
  chat --> ctx[Context builder]
  ctx --> conv[(Conversation store)]
  ctx --> mem[(Memory index, opt-in)]
  ctx --> files[(Files and extracted text)]
  chat --> gen[Generator workers]
  gen --> gw[LLM gateway]
  gen --> tools[Tool runner]
  gen --> buf[(Turn stream buffers)]
  buf --> edge
  gen --> conv
  client --> up[Pre-signed upload]
  up --> files
  files --> proc[File processor: scan, extract]
  conv --> sum[Summariser jobs]
```

| Component | Responsibility | Scales by |
|---|---|---|
| Edge | Auth, home-region routing, SSE delivery from buffers | Open streams |
| Chat service | Turn admission, idempotency, conversation CRUD | Requests per second |
| Context builder | Token-budgeted prompt assembly | Turns per second |
| Generator workers | Call the gateway, run tools, write tokens to the buffer, store the answer | Concurrent generations |
| Moderation service | Input and output classifiers | Checks per second |
| File processor | Malware scan, type checks, text extraction, chunking | Uploads per second |
| Summariser | Rolling summaries off the turn path | Background queue |

## 7. Detailed design

<!-- Guidance: Go deep on the hard problems; each deep dive lists options, the choice and the reason. -->

### 7.1 Turn flow

The send request returns as soon as the turn is admitted; the answer arrives
on a separate stream that the client can reopen at any time.

```mermaid
sequenceDiagram
  participant C as Client
  participant E as Edge
  participant S as Chat service
  participant G as Generator
  participant B as Stream buffer
  participant L as LLM gateway
  C->>S: POST /turns (Idempotency-Key)
  S->>S: rate limit, input moderation, store user message
  S->>G: enqueue turn
  S-->>C: turn_id, stream URL
  C->>E: GET /stream
  G->>G: build context within token budget
  G->>L: chat request (stream)
  loop tokens
    L-->>G: token chunk
    G->>B: append event (seq)
    B-->>E: event
    E-->>C: SSE event
  end
  G->>S: store answer, usage
  Note over C,E: on disconnect, client reconnects with Last-Event-ID
```

### 7.2 Deep dive 1: resumable token streaming

| Option | Strengths | Weaknesses |
|---|---|---|
| A. Generation tied to the client connection | Simple | A dropped connection loses the answer and its cost |
| B. SSE over HTTP/2, generation decoupled, per-turn buffer, resume by `Last-Event-ID` | Works through proxies and CDNs; built-in reconnection semantics; one-way fits the use case | Needs a buffer store; client sends input over separate requests |
| C. WebSocket for everything | Two-way; good for voice and live collaboration | Harder through some proxies; resumption must be designed from scratch; sticky connections |

**Choice:** B for text chat; C only for voice sessions.

**Reason:** the answer is a one-way stream and the main failure is mobile
disconnects. SSE carries an event ID per event, and browsers resend the last
one on reconnection, so resumption needs only a buffer keyed by turn and
sequence. Stopping generation is a separate `POST /stop`.

Rules:

- The buffer keeps a turn's events for 10 minutes (*Example*) after
  completion; later reconnects read the stored message instead.
- If the user does not reconnect, generation still completes and the answer
  is stored; it counts against the plan once.
- Send a comment line every 15 s (*Example*) to keep idle proxies from
  closing the stream during tool calls.

### 7.3 Deep dive 2: context window management

| Option | Strengths | Weaknesses |
|---|---|---|
| A. Send the whole conversation | Exact | Fails past the context limit; cost grows with every turn |
| B. Last N turns only | Cheap, predictable | Forgets early facts the user still refers to |
| C. Rolling summary of older turns plus recent turns verbatim, within a token budget | Bounded cost; keeps key facts | Summaries can drop or distort details |
| D. Retrieval over the conversation history | Finds old details on demand | Retrieval misses; extra latency |

**Choice:** C by default, with D added for conversations longer than the
summary horizon.

**Reason:** input tokens dominate cost (3.3), and the context limit is a hard
failure. A budget allocator assigns each part a maximum and fills in order of
priority. The *Example* budgets below total 4,500 input tokens: the 6,000 of
3.1 minus the 1,500 that summarisation saves in 3.3 (history 3,500 down to
2,000 as summary plus recent turns; memory and files stay at 1,000).

| Part | *Example* budget | Rule |
|---|---|---|
| System prompt and tool definitions | 1,500 tokens | Fixed, first, unchanged between turns (prefix caching) |
| Rolling summary | 600 tokens | Regenerated every 10 turns by the summariser |
| Recent turns | 1,400 tokens | Newest first until the budget is used |
| Memory items | 300 tokens | Only if memory is on; most relevant first |
| File excerpts | 700 tokens | Chunks retrieved for this question |
| Reserved for the answer | `max_tokens` | Never squeezed by the parts above |

The summariser keeps names, numbers, decisions and open questions, and marks
which message it covers up to, so it can be rebuilt after an edit.

> **How real systems do it: MemGPT virtual context management (paraphrased)**
>
> The MemGPT paper treats the model's context window like the main memory
> of an operating system. A small, fixed working context holds the system
> instructions and the most relevant facts; older conversation and documents
> sit in external storage, and the model is given functions to search that
> storage and to move information in and out of its working context, much
> like paging. The authors evaluate the approach on analysis of documents
> larger than the context window and on multi-session chat where the agent
> must remember earlier sessions. Source: Packer et al., "MemGPT: Towards
> LLMs as Operating Systems", arXiv:2310.08560 (2023, revised 2024). This box
> summarises the idea; read the paper for details.

### 7.4 Deep dive 3: moderation, rate limits and abuse

| Option | Strengths | Weaknesses |
|---|---|---|
| A. Moderate input only | Cheap, adds no output latency | Misses harmful outputs from benign-looking prompts |
| B. Input plus output checked every N tokens while streaming, with a final check | Catches problems early; stops the stream | Adds cost; must retract text already shown |
| C. Output checked only after completion | Simple | User sees the content before the block |

**Choice:** B. Input is checked before the turn is queued; output is checked
every 200 tokens (*Example*) and at the end; a block stops generation,
replaces the shown text with a notice and records the category.

Rate limits are layered:

| Layer | *Example* rule | Purpose |
|---|---|---|
| Plan | Free: 40 messages per 3 h; paid: higher | Cost control and fairness |
| Burst | 5 turns per 10 s per user | Scripts and accidental loops |
| Abuse score | Lower limits for new accounts, failed payment, repeated blocks | Abuse at scale |
| Account creation | Proof of work or challenge after signals | Sign-up fraud |
| Tool use | Separate limits for expensive tools | Cost per tool |

Limits use the counters described in Rate Limiter System Design Template
(`system-design/design-rate-limiter`). Moderation categories and thresholds
belong in the Guardrails and Safety Filter Specification Template
(`ai-data/guardrails-spec`).

### 7.5 Deep dive 4: memory opt-in and regional data residency

| Option | Strengths | Weaknesses |
|---|---|---|
| A. One global store, replicated | Simple operations | Content leaves the user's region |
| B. Home region per user or workspace; all content stores regional; only routing metadata global | Residency by design; deletion by partition | Cross-region features (shared links across regions) need explicit design |
| C. Encrypt globally replicated data with regional keys | Some residency arguments | Data still stored abroad; key management complexity |

**Choice:** B. The global directory maps user ID to home region and holds no
content. Model calls go to gateway routes approved for that region.

**Reason:** NFR-06 and deletion (NFR-05) are both easiest when the partition
key, the region and the deletion unit are the same thing: the user or
workspace.

Memory rules:

- Off by default; when on, the assistant shows what it saved and the user
  can edit or delete each item, or clear all.
- Memory items keep a link to the source message; deleting the conversation
  offers to delete derived memory.
- Memory is never used to train models unless the user separately agreed.
- Text that came from files, web results or tool output is never saved to
  memory without the user confirming it, so an injected instruction cannot
  persist into later conversations.

Legal pointers (this is a documentation format, not legal advice; check the
consolidated texts): under the EU GDPR (Regulation (EU) 2016/679), Chapter V
(Articles 44 to 50) governs transfers of personal data to third countries,
Article 17 covers erasure and Article 25 covers data protection by design and
by default. Under the EU AI Act (Regulation (EU) 2024/1689, as amended),
Article 50(1) requires providers to design systems that interact directly with
people so that people are informed they are dealing with an AI system unless
that is obvious; check the consolidated text for application dates. Record
the analysis with the Data Protection Impact Assessment (DPIA) Template (`compliance/dpia`).

### 7.6 File uploads and tool use

- Uploads go directly to regional object storage with a pre-signed URL; the
  file processor scans for malware, checks the declared type against the
  content, extracts text and chunks it. The turn waits until the file is
  `ready` or fails with a clear message.
- File content is untrusted input to the model: instructions inside files
  are treated as data, and the guardrail input checks run on extracted text.
- Tools run through a tool runner with per-plan limits; code execution goes
  to the sandbox service described in Agent Code Execution Sandbox System
  Design Template (`system-design/design-agent-sandbox-execution`).
- Tool progress is streamed as `tool.call` events so users see why the answer
  is waiting.

## 8. Scalability and performance

<!-- Guidance: Show how the design meets section 3 and where it breaks first. -->

| Concern | Approach | Limit before redesign |
|---|---|---|
| Open streams | Edge reads from buffers; scale on open streams | Buffer store throughput per region |
| Hot conversations | Partition by user; conversations are per user | Shared workspace conversations with many readers |
| Context building | Two partition-local reads plus optional retrieval | Retrieval latency for long histories |
| Moderation | Dedicated classifier pool, batched output checks | Classifier throughput at peak |
| Model capacity | Gateway quotas per region | Provider capacity in a region |

Time to first token budget (NFR-01, p95 1.5 s):

| Step | Budget |
|---|---|
| Admission, rate limit, idempotency | 30 ms |
| Input moderation | 150 ms |
| Context building | 120 ms |
| Gateway and model time to first token | 1,000 ms |
| Buffer and edge delivery | 50 ms |
| Headroom | 150 ms |

## 9. Reliability, security and operations

<!-- Guidance: Name failure modes, threats and the signals operators watch. -->

### 9.1 Failure modes

| ID | Component | Failure | Detection | Blast radius | Mitigation | RTO |
|---|---|---|---|---|---|---|
| FM-01 | Edge node | Crash | Health checks | Streams on that node | Clients reconnect to another node and resume from the buffer | Under 10 s |
| FM-02 | Stream buffer | Data lost | Missing sequence on resume | In-flight turns | Fall back to the stored message when complete; otherwise offer regenerate without charge | 1 min |
| FM-03 | Generator | Crash mid-answer | Turn heartbeat timeout | Turns on that worker | Mark turn failed, no charge, offer regenerate | 30 s |
| FM-04 | LLM gateway or provider | Outage | Error rate | All turns in region | Gateway fallback chain; degraded-mode banner | Gateway RTO |
| FM-05 | Moderation pool | Down | Error rate | All turns | Fail closed for input on free plans; fall back to a smaller local classifier | 5 min |
| FM-06 | Conversation store | Partition unavailable | Read errors | Users on that partition | Replicas within region; read-only mode | 15 min |
| [FM-nn] | [Component] | [Failure] | [Signal] | [Scope] | [Mitigation] | [Time] |

### 9.2 Security and privacy

| ID | Threat | Control |
|---|---|---|
| SEC-01 | Prompt injection through files, web results, connector or other tool output | Treat all of it as untrusted data, never as instructions; no memory writes from it without user confirmation; guardrail checks; tool permissions per plan |
| SEC-02 | Cross-user data exposure | User ID in every key and index filter; tests for authorisation on every endpoint |
| SEC-03 | Hidden instructions or system prompt revealed | Keep secrets out of prompts; output checks for canary strings |
| SEC-04 | Abuse at scale (spam, scraping, fraud) | Layered limits (7.4), challenges, abuse scoring |
| SEC-05 | Malicious uploads | Malware scan, type checks, no execution of uploads outside the sandbox |
| SEC-06 | Insider access to conversations | Access only through audited support tooling with user consent |
| [SEC-nn] | [Threat] | [Control] |

Map the application threats to the OWASP Top 10 for LLM Applications (2026
edition) with the Threat Model Template (`security/threat-model`) and the
Privacy Threat Model Template (`security/privacy-threat-model`).

### 9.3 Observability

| Signal | Metric | Alert threshold |
|---|---|---|
| Time to first token | p95 by region and model alias | Above 1.5 s for 10 min |
| Stream completion | Completed or resumed over started | Below 99.9% |
| Resume rate | Reconnects per 100 turns | Sudden rise (network or edge issue) |
| Moderation blocks | Rate by category | Spike after a model change |
| Rate-limit hits | Per plan | Rise after a pricing change |
| Deletion backlog | Pending deletion jobs by age | Any older than 25 days |

Model call spans follow the LLM and Agent Observability Specification
Template (`ai-data/llm-observability-spec`); do not record message content on
spans by default.

### 9.4 Cost estimate

| Item | Unit cost | Quantity | Monthly cost |
|---|---|---|---|
| Model input tokens | USD 0.50 per million (*Example*) | 180 billion per day x 30 | about USD 2.7 million |
| Model output tokens | USD 2 per million (*Example*) | 12 billion per day x 30 | about USD 720,000 |
| Edge and chat services | [USD per node-month] | [Nodes] | [USD] |
| Conversation and file storage | [USD per TB-month] | about 33 TB per year plus 162 TB of files | [USD] |
| Moderation pool | [USD per node-month] | [Nodes] | [USD] |
| **Total** | | | **[USD]** |

## 10. Trade-offs and alternatives

<!-- Guidance: For each major decision list options, the choice, why and what you give up; add what changes at 10x and 0.1x. -->

| Decision | Options | Chosen | Why | What we give up |
|---|---|---|---|---|
| Streaming | SSE; WebSocket | SSE with resumable buffers | One-way stream, standard reconnection | Separate requests for input |
| Context | Full; last N; summary; retrieval | Summary plus recent, retrieval for long threads | Bounded cost and latency | Summary errors |
| Output moderation | None; streaming; after completion | Streaming checks | Stop harm early | Cost and occasional retraction |
| Residency | Global; regional; encrypted global | Regional by home region | Residency and deletion by design | Cross-region features |
| Memory | Always on; opt-in | Opt-in, visible, editable | User trust and privacy | Personalisation for users who do not opt in |
| What changes at 10x (50 million daily users, about 125,000 peak streams) | | Edge per region in more locations; buffer store sharded by turn ID; summarisation and retrieval mandatory for all long conversations; per-region model capacity contracts | Input tokens (about 1.8 trillion per day) and open streams set the limits | Higher fixed cost and more regions to operate |
| What changes at 0.1x (500,000 daily users, about 1,250 peak streams) | | One region per market; managed SSE edge; last-N context may be enough; moderation through the provider's endpoint | Fixed cost of own buffers and classifier pool is not justified | Less control over moderation and stream resumption |
| [Decision] | [Options] | [Chosen] | [Why] | [Cost] |

## 11. Delivery plan and risks

<!-- Guidance: How the design reaches production safely, and what could stop it. -->

### 11.1 Milestones

| Milestone | Scope | Exit criterion | Target date |
|---|---|---|---|
| M1 | Turn loop, SSE with resumable buffers, conversation store | NFR-01 and NFR-02 in load test with simulated disconnects | YYYY-MM-DD |
| M2 | Context builder and summariser | Long-conversation evaluation shows no loss of key facts above the agreed rate | YYYY-MM-DD |
| M3 | Moderation and layered rate limits | Red-team run passes; abuse simulation stays within limits | YYYY-MM-DD |
| M4 | Files and tools | Malware and type checks on a test corpus | YYYY-MM-DD |
| M5 | EU region, memory opt-in, export and deletion | Residency and deletion audits pass (NFR-05, NFR-06) | YYYY-MM-DD |

### 11.2 Rollout and rollback

[*Example:* New stack behind a flag for 1%, 10%, 50% and 100% of users; the
prototype path stays for 2 weeks. Model alias changes roll out by region with
evaluation gates and instant rollback.]

### 11.3 Risks

| ID | Risk | Likelihood | Impact | Mitigation | Owner |
|---|---|---|---|---|---|
| R-01 | *Example:* Summaries lose facts users rely on | Medium | Medium | Evaluation set of long conversations; retrieval fallback | [@name] |
| R-02 | *Example:* Regional model capacity insufficient | Medium | High | Contracts per region; graceful queueing | [@name] |
| R-03 | *Example:* Deletion misses derived data (indexes, logs) | Medium | High | Deletion inventory per store; audits | [@name] |
| R-04 | *Example:* Abuse waves on free plan | High | Medium | Abuse scoring, challenges, rapid limit changes | [@name] |
| [R-nn] | [Risk] | [L / M / H] | [L / M / H] | [Mitigation] | [@name] |

## 12. Appendices

<!-- Guidance: Traceability, open questions, glossary and references. -->

### 12.1 Traceability

| Requirement | Design element | Verification |
|---|---|---|
| FR-01, FR-02, NFR-02 | 7.2 resumable streaming | Disconnect simulation |
| NFR-01 | 8 latency budget | Client timing |
| FR-07 | 7.4 moderation | Red-team and category tests |
| FR-08 | 7.4 layered limits | Abuse simulation |
| FR-09, FR-10, NFR-05, NFR-06 | 7.5 memory and residency | Deletion and data-flow audits |
| [ID] | [Element] | [Test] |

### 12.2 Open questions

| ID | Question | Owner | Due |
|---|---|---|---|
| OQ-01 | *Example:* Do shared conversation links work across regions? | [@name] | YYYY-MM-DD |
| OQ-02 | *Example:* Retention for files versus conversations for business plans? | [@name] | YYYY-MM-DD |
| [OQ-nn] | [Question] | [@name] | YYYY-MM-DD |

### 12.3 Glossary

| Term | Meaning |
|---|---|
| Turn | One user message and the assistant's answer to it |
| Stream buffer | Short-lived store of a turn's events, used for resumption |
| Rolling summary | Periodically regenerated summary of older turns |
| Token budget | Maximum tokens assigned to each part of the prompt |
| Home region | Region where a user's or workspace's content is stored and processed |
| Memory | Facts the assistant keeps across conversations, with user consent |

### 12.4 References

| Reference | Use |
|---|---|
| WHATWG HTML Living Standard, section 9.2 Server-sent events, https://html.spec.whatwg.org/multipage/server-sent-events.html | Streaming in 4.2 and 7.2 |
| RFC 6455, The WebSocket Protocol (2011) | Voice sessions in 7.2 |
| RFC 9457, Problem Details for HTTP APIs (2023) | Errors in 4.3 |
| Packer et al., "MemGPT: Towards LLMs as Operating Systems", arXiv:2310.08560 | Box in 7.3 (paraphrased) |
| Regulation (EU) 2016/679 (GDPR), Chapter V, Articles 17 and 25, https://eur-lex.europa.eu/eli/reg/2016/679/oj | Residency and deletion pointers in 7.5 |
| Regulation (EU) 2024/1689 (AI Act), Article 50, consolidated text on EUR-Lex | AI interaction notice (FR-11) |
| OWASP Top 10 for LLM Applications, 2026 edition, https://genai.owasp.org/ (CC BY-SA 4.0) | Threat mapping in 9.2 |
| LLM Gateway System Design Template (`system-design/design-llm-gateway`) | Model access |
| Chat System Design Template (`system-design/design-chat-system`) | Messaging patterns for human-to-human chat |
| Guardrails and Safety Filter Specification Template (`ai-data/guardrails-spec`) | Moderation policy |

## Standard and licence note

Based on the WHATWG HTML Living Standard (server-sent events), RFC 6455, RFC
9457 and public EU law (Regulation (EU) 2016/679 and Regulation (EU)
2024/1689), cited by article and paraphrased; this is a documentation format,
not legal advice. OWASP list names are attributed to the OWASP GenAI Security
Project under CC BY-SA 4.0, and any adapted OWASP material you redistribute
must carry the same licence under its ShareAlike term. The research in 7.3 is
paraphrased and cited as inspiration only. The text is original; check the
current official version of each source, and the consolidated legal texts,
before approval. Template licence: MIT.