System Design: [Chat assistant name]

Reference design for the application layer of a consumer or workplace chat assistant in the style of the well-known general-purpose AI chat products: token streaming over server-sent events or WebSocket, the conversation store, context window management and summarisation, file uploads, tool use, moderation, rate limits and abuse prevention, opt-in memory and regional data residency. Model serving is out of scope and covered by the LLM Inference Serving and Vector Search Platform Design Template (system-design/design-llm-inference-serving); provider routing by the LLM Gateway System Design Template (system-design/design-llm-gateway). It follows the 12-section structure of the System Design Document Template (system-design/system-design-document) and is pre-filled with a worked design to adapt. Replace every [bracketed] value and every row marked Example; all numbers are illustrative, not benchmarks. Where it mentions laws, this is a documentation format, not legal advice.

Document control

Approvals

Revision history

Contents

1. Overview
2. Requirements
3. Capacity estimation
4. API design
5. Data model
6. High-level design
7. Detailed design
8. Scalability and performance
9. Reliability, security and operations
10. Trade-offs and alternatives
11. Delivery plan and risks
12. Appendices

1. Overview

1.1 Problem statement

[Example: The current assistant is a prototype: each message opens a WebSocket, the server calls the model and forwards tokens. When a phone switches networks the answer is lost and the user is charged a message against their limit. Conversations are sent whole to the model, so long threads hit the context limit and fail. Uploaded files sit in one bucket in one region. Business customers in the EU require their data to stay in the EU, and the product has no way to honour deletion requests across logs and caches.]

1.2 Proposed solution

[Example: A regional chat service that owns conversations and the turn loop. Each user turn becomes a generation job; the generator streams tokens into a short-lived per-turn stream buffer, and clients read it over server-sent events, reconnecting with the last event ID. A context builder assembles each prompt within a token budget: system instructions, a rolling summary of older turns, the most recent turns verbatim, retrieved memory (only if the user turned memory on) and file excerpts. Moderation runs on input and on streamed output. Rate limits apply per user, per plan and per abuse signal. Every user is homed in one data region; conversations, files, memory and logs stay in that region.]

1.3 Decisions requested from reviewers

1.4 45-minute interview path

2. Requirements

2.1 Core functional requirements

2.2 Non-functional requirements

2.3 Below the line (out of scope)

• [Example: Model serving, batching and GPUs (see LLM Inference Serving and Vector Search Platform Design Template (system-design/design-llm-inference-serving))]
• [Example: Provider routing, quotas towards providers and redaction (see LLM Gateway System Design Template (system-design/design-llm-gateway))]
• [Example: Running user or model code (see Agent Code Execution Sandbox System Design Template (system-design/design-agent-sandbox-execution))]
• [Example: Long-running autonomous agents (see Agent Orchestration Platform System Design Template (system-design/design-agent-orchestration-platform))]
• [Example: Billing and subscription management]
• [Item deferred, with the release that will take it]

3. Capacity estimation

3.1 Assumptions

All values are Example planning inputs.

3.2 Traffic and streams

3.3 Tokens and model cost

Keep the system prompt and tool definitions at the start of the prompt and unchanged between turns, so provider or engine prefix caching can reuse them.

3.4 Storage

3.5 Moderation and summarisation calls

3.6 Summary

4. API design

4.1 Endpoints

4.2 Stream format

The stream uses text/event-stream (WHATWG HTML Living Standard section 9.2). Each event has an id equal to its sequence number in the turn, so a client that reconnects with Last-Event-ID: 41 receives events from 42 onward.

id: 1
event: turn.started
data: {"turn_id": "t_9f2", "model_alias": "chat-default"}

id: 2
event: token
data: {"text": "Here is"}

id: 3
event: tool.call
data: {"tool": "web_search", "status": "running"}

id: 58
event: turn.completed
data: {"finish_reason": "stop", "output_tokens": 412}

4.3 Errors

Errors use RFC 9457 problem details. A retried POST /turns with the same Idempotency-Key returns the existing turn, so a flaky network does not create two answers or count twice against the limit.

5. Data model

5.1 Entities

A conversation is a tree of messages (edits and regenerations create branches); the client shows one path through it.

5.2 Access patterns

5.3 Storage choices

6. High-level design

7. Detailed design

7.1 Turn flow

The send request returns as soon as the turn is admitted; the answer arrives on a separate stream that the client can reopen at any time.

7.2 Deep dive 1: resumable token streaming

Choice: B for text chat; C only for voice sessions.

Reason: the answer is a one-way stream and the main failure is mobile disconnects. SSE carries an event ID per event, and browsers resend the last one on reconnection, so resumption needs only a buffer keyed by turn and sequence. Stopping generation is a separate POST /stop.

Rules:

• The buffer keeps a turn's events for 10 minutes (Example) after completion; later reconnects read the stored message instead.
• If the user does not reconnect, generation still completes and the answer is stored; it counts against the plan once.
• Send a comment line every 15 s (Example) to keep idle proxies from closing the stream during tool calls.

7.3 Deep dive 2: context window management

Choice: C by default, with D added for conversations longer than the summary horizon.

Reason: input tokens dominate cost (3.3), and the context limit is a hard failure. A budget allocator assigns each part a maximum and fills in order of priority. The Example budgets below total 4,500 input tokens: the 6,000 of 3.1 minus the 1,500 that summarisation saves in 3.3 (history 3,500 down to 2,000 as summary plus recent turns; memory and files stay at 1,000).

The summariser keeps names, numbers, decisions and open questions, and marks which message it covers up to, so it can be rebuilt after an edit.

How real systems do it: MemGPT virtual context management (paraphrased)

The MemGPT paper treats the model's context window like the main memory of an operating system. A small, fixed working context holds the system instructions and the most relevant facts; older conversation and documents sit in external storage, and the model is given functions to search that storage and to move information in and out of its working context, much like paging. The authors evaluate the approach on analysis of documents larger than the context window and on multi-session chat where the agent must remember earlier sessions. Source: Packer et al., "MemGPT: Towards LLMs as Operating Systems", arXiv:2310.08560 (2023, revised 2024). This box summarises the idea; read the paper for details.

7.4 Deep dive 3: moderation, rate limits and abuse

Choice: B. Input is checked before the turn is queued; output is checked every 200 tokens (Example) and at the end; a block stops generation, replaces the shown text with a notice and records the category.

Rate limits are layered:

Limits use the counters described in Rate Limiter System Design Template (system-design/design-rate-limiter). Moderation categories and thresholds belong in the Guardrails and Safety Filter Specification Template (ai-data/guardrails-spec).

7.5 Deep dive 4: memory opt-in and regional data residency

Choice: B. The global directory maps user ID to home region and holds no content. Model calls go to gateway routes approved for that region.

Reason: NFR-06 and deletion (NFR-05) are both easiest when the partition key, the region and the deletion unit are the same thing: the user or workspace.

Memory rules:

• Off by default; when on, the assistant shows what it saved and the user can edit or delete each item, or clear all.
• Memory items keep a link to the source message; deleting the conversation offers to delete derived memory.
• Memory is never used to train models unless the user separately agreed.
• Text that came from files, web results or tool output is never saved to memory without the user confirming it, so an injected instruction cannot persist into later conversations.

Legal pointers (this is a documentation format, not legal advice; check the consolidated texts): under the EU GDPR (Regulation (EU) 2016/679), Chapter V (Articles 44 to 50) governs transfers of personal data to third countries, Article 17 covers erasure and Article 25 covers data protection by design and by default. Under the EU AI Act (Regulation (EU) 2024/1689, as amended), Article 50(1) requires providers to design systems that interact directly with people so that people are informed they are dealing with an AI system unless that is obvious; check the consolidated text for application dates. Record the analysis with the Data Protection Impact Assessment (DPIA) Template (compliance/dpia).

7.6 File uploads and tool use

• Uploads go directly to regional object storage with a pre-signed URL; the file processor scans for malware, checks the declared type against the content, extracts text and chunks it. The turn waits until the file is ready or fails with a clear message.
• File content is untrusted input to the model: instructions inside files are treated as data, and the guardrail input checks run on extracted text.
• Tools run through a tool runner with per-plan limits; code execution goes to the sandbox service described in Agent Code Execution Sandbox System Design Template (system-design/design-agent-sandbox-execution).
• Tool progress is streamed as tool.call events so users see why the answer is waiting.

8. Scalability and performance

Time to first token budget (NFR-01, p95 1.5 s):

9. Reliability, security and operations

9.1 Failure modes

9.2 Security and privacy

Map the application threats to the OWASP Top 10 for LLM Applications (2026 edition) with the Threat Model Template (security/threat-model) and the Privacy Threat Model Template (security/privacy-threat-model).

9.3 Observability

Model call spans follow the LLM and Agent Observability Specification Template (ai-data/llm-observability-spec); do not record message content on spans by default.

9.4 Cost estimate

10. Trade-offs and alternatives

11. Delivery plan and risks

11.1 Milestones

11.2 Rollout and rollback

[Example: New stack behind a flag for 1%, 10%, 50% and 100% of users; the prototype path stays for 2 weeks. Model alias changes roll out by region with evaluation gates and instant rollback.]

11.3 Risks

12. Appendices

12.1 Traceability

12.2 Open questions

12.3 Glossary

12.4 References

Standard and licence note

Based on the WHATWG HTML Living Standard (server-sent events), RFC 6455, RFC 9457 and public EU law (Regulation (EU) 2016/679 and Regulation (EU) 2024/1689), cited by article and paraphrased; this is a documentation format, not legal advice. OWASP list names are attributed to the OWASP GenAI Security Project under CC BY-SA 4.0, and any adapted OWASP material you redistribute must carry the same licence under its ShareAlike term. The research in 7.3 is paraphrased and cited as inspiration only. The text is original; check the current official version of each source, and the consolidated legal texts, before approval. Template licence: MIT.