# System Design: [Chat system name]

> Reference design for a real-time chat system with one-to-one and group
> conversations, multi-device delivery, presence, receipts, offline sync and
> push notifications. It follows the structure of the System Design Document
> template and is pre-filled with a worked design so you can adapt it rather
> than start from a blank page. Every value marked *Example* is an assumption:
> replace it with your own numbers and re-run the formulas.

<!--
Guidance: How to use this template
- Start with section 2 and 3.1: change the user counts, message rates and
  retention, then recompute every formula in section 3 before touching the
  diagrams. Gateway count and storage size follow directly from them.
- Keep the ordering rule (per-conversation sequence numbers) unless you have
  a strong reason; most chat bugs come from ordering by client clocks.
- Section 7.6 (end-to-end encryption) changes search, moderation and backup.
  Decide it early and record the decision in section 10.
- Delete reference content that does not apply, but keep the failure-mode
  and trade-off tables filled; reviewers check them first.
-->

## Document control

<!-- Guidance: A design is a proposal until it is approved; record who decided and when. -->

| Field | Value |
|---|---|
| Document ID | [SDD-CHAT-001] |
| System / feature | [Chat platform: 1:1 and group messaging] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Implemented / Superseded] |
| Author(s) | [@author] |
| Reviewers | [@reviewer-1], [@reviewer-2] |
| Last updated | YYYY-MM-DD |
| Related | [PRD], [Threat model], [ADR: message store], [ADR: E2EE] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Tech lead | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| Staff / principal engineer | [@name] | [Decision] | YYYY-MM-DD |
| SRE / operations | [@name] | [Decision] | YYYY-MM-DD |
| Security and privacy | [@name] | [Decision] | YYYY-MM-DD |
| Mobile client lead | [@name] | [Decision] | YYYY-MM-DD |

## Revision history

<!-- Guidance: Note what changed in the design, not the wording. -->

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft for design review |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Replaced client timestamps with server-assigned per-conversation sequence numbers |
| 0.3 | YYYY-MM-DD | [@name] | *Example:* Added large-group fan-out on read above 1 000 members |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved |

## Contents

<!-- Guidance: Keep aligned with the headings. -->

1. [Overview](#1-overview)
2. [Requirements](#2-requirements)
3. [Capacity estimation](#3-capacity-estimation)
4. [API design](#4-api-design)
5. [Data model](#5-data-model)
6. [High-level design](#6-high-level-design)
7. [Detailed design](#7-detailed-design)
8. [Scalability and performance](#8-scalability-and-performance)
9. [Reliability, security and operations](#9-reliability-security-and-operations)
10. [Trade-offs and alternatives](#10-trade-offs-and-alternatives)
11. [Delivery plan and risks](#11-delivery-plan-and-risks)
12. [Appendices](#12-appendices)

## 1. Overview

<!-- Guidance: One page that a busy reviewer can read alone: problem, proposed solution in a paragraph, and the decisions that need their attention. -->

### 1.1 Problem statement

[What problem exists today. *Example:* Users coordinate through email and a
third-party messenger. Messages are lost between devices, there is no shared
history for teams, and we cannot meet data-residency commitments. We need a
first-party chat service that works on web, iOS and Android, including when a
device has been offline for days.]

### 1.2 Proposed solution

[One paragraph. *Example:* Clients hold one persistent WebSocket connection to
a stateless gateway tier. A chat service assigns every message a monotonically
increasing sequence number within its conversation, persists it to a
wide-column store partitioned by conversation, and fans it out to the
recipients' devices through a routing layer that knows which gateway holds each
device's connection. Devices that are offline receive a push notification and
catch up by asking for everything after the last sequence number they have
seen. Presence and typing indicators are ephemeral and never touch durable
storage.]

### 1.3 Decisions requested from reviewers

| ID | Question | Recommended option | Section |
|---|---|---|---|
| Q-01 | *Example:* How are messages ordered? | Server-assigned sequence per conversation | 7.2 |
| Q-02 | *Example:* Fan-out on write for all groups? | Write fan-out up to 1 000 members, read fan-out above | 7.3, 10 |
| Q-03 | *Example:* Is end-to-end encryption in scope for v1? | 1:1 chats in v1, groups in v2 | 7.6 |
| Q-04 | *Example:* Which message store? | Wide-column store, partition by conversation and time bucket | 5.2, 10 |
| [Q-nn] | [Question] | [Option] | [Section] |

## 2. Requirements

<!-- Guidance: Functional requirements describe behaviour; non-functional requirements describe qualities with numbers. Mark what is out of scope so nobody designs for it. -->

### 2.1 Functional requirements

| ID | Requirement | Priority |
|---|---|---|
| FR-01 | *Example:* Send and receive text messages in 1:1 conversations | Must |
| FR-02 | *Example:* Group conversations up to 1 000 members; broadcast channels up to 100 000 members | Must |
| FR-03 | *Example:* Deliver every message to all of a user's devices (up to 5) | Must |
| FR-04 | *Example:* Show delivered and read receipts per message (per member in groups up to 100) | Must |
| FR-05 | *Example:* Show online / last-seen presence and typing indicators | Should |
| FR-06 | *Example:* Sync history after a device has been offline, in order, without gaps | Must |
| FR-07 | *Example:* Push notification to offline devices within 5 s | Must |
| FR-08 | *Example:* Attachments (images, files up to 100 MB) via object storage | Should |
| FR-09 | *Example:* Edit and delete own messages within 24 h | Could |
| [FR-nn] | [Requirement] | [Must / Should / Could] |

### 2.2 Non-functional requirements

| ID | Quality | Target | Measured as |
|---|---|---|---|
| NFR-01 | Latency | *Example:* Send-to-receive p99 at or below 300 ms when both parties are online in the same region | Client-reported timing, sampled |
| NFR-02 | Availability | *Example:* 99.95% monthly for send and sync | Synthetic send/receive probes |
| NFR-03 | Durability | *Example:* An acknowledged message is never lost | Replication RF=3, restore drills |
| NFR-04 | Ordering | *Example:* All members see a conversation in the same order | Sequence-gap detector on clients |
| NFR-05 | Scale | *Example:* 15 million concurrent connections at peak | Gateway connection gauge |
| NFR-06 | Privacy | *Example:* Message content encrypted in transit and at rest; E2EE for 1:1 | Security review |
| [NFR-nn] | [Quality] | [Target] | [Measurement] |

### 2.3 Out of scope

- [*Example:* voice and video calls (separate real-time media design)]
- [*Example:* full-text server-side search of message content]
- [*Example:* federation with external chat networks]
- [Item deferred, with the release that will take it]

## 3. Capacity estimation

<!-- Guidance: Back-of-the-envelope numbers decide the shape of the design. Show inputs, formulas and results; round generously and say which way you rounded. -->

### 3.1 Assumptions

| Input | Symbol | Value | Source |
|---|---|---|---|
| Daily active users | $U$ | *Example:* 50 million | Growth plan |
| Messages sent per user per day | $m$ | *Example:* 40 | Comparable products |
| Average stored message (payload + metadata) | $s$ | *Example:* 300 bytes | Schema in 5.2 |
| Average delivered copies per message (recipients x devices) | $d$ | *Example:* 3 | Mix of 1:1 and small groups |
| Peak concurrent users as share of DAU | $c$ | *Example:* 30% | Traffic history |
| Connections per gateway node | $g$ | *Example:* 100 000 | Load test of gateway |
| Peak to average factor | $k$ | *Example:* 3 | Evening peak |
| Retention | $Y$ | *Example:* 2 years hot, then cold | Product decision |

### 3.2 Throughput

Messages per day and average send rate:

$$
M_{day} = U \times m = 5 \times 10^{7} \times 40 = 2 \times 10^{9} \ \text{messages/day}
$$

$$
QPS_{send} = \frac{2 \times 10^{9}}{86\,400} \approx 23\,000 \ \text{messages/s}, \qquad
QPS_{peak} = k \cdot QPS_{send} \approx 70\,000 \ \text{messages/s}
$$

Deliveries (one per recipient device) at peak:

$$
D_{peak} = d \times QPS_{peak} \approx 3 \times 70\,000 = 210\,000 \ \text{deliveries/s}
$$

### 3.3 Connections and gateways

$$
C_{peak} = c \times U = 0.3 \times 5 \times 10^{7} = 1.5 \times 10^{7} \ \text{connections}
$$

$$
N_{gw} = \frac{C_{peak}}{g} \times 1.5 = \frac{1.5 \times 10^{7}}{10^{5}} \times 1.5 = 225 \ \text{gateway nodes}
$$

The factor 1.5 keeps headroom for losing one availability zone of three
without reconnect storms overloading the survivors.

### 3.4 Storage

$$
S_{day} = M_{day} \times s = 2 \times 10^{9} \times 300 \ \text{B} = 600 \ \text{GB/day}
$$

$$
S_{2y} = 600 \ \text{GB} \times 730 \approx 438 \ \text{TB}, \qquad \text{with RF} = 3: \approx 1.3 \ \text{PB raw}
$$

Attachments are stored separately in object storage and are usually larger
than the message store; estimate them from your media mix.

### 3.5 Bandwidth

$$
B_{out} \approx D_{peak} \times 500 \ \text{B (framed, encrypted)} \approx 105 \ \text{MB/s} \approx 840 \ \text{Mbit/s}
$$

Heartbeats every 30 s add $1.5 \times 10^{7} / 30 = 500\,000$ small frames per
second, terminated at the gateway and never forwarded.

### 3.6 Summary

| Quantity | Estimate | Design consequence |
|---|---|---|
| Send rate | ~23 000/s average, ~70 000/s peak | Partitioned chat service and log |
| Deliveries | ~210 000/s peak | Routing layer must be in memory, not a database lookup per message |
| Connections | 15 million peak | ~225 gateway nodes across 3 zones |
| Storage | ~600 GB/day, ~1.3 PB raw for 2 years | Wide-column store with time buckets; tiering after 2 years |
| [Quantity] | [Estimate] | [Consequence] |

## 4. API design

<!-- Guidance: Define the contract first: resources, methods, request and response shapes, errors, idempotency, pagination, rate limits and versioning. -->

### 4.1 Endpoints and frames

Real-time traffic uses WebSocket frames; history, conversation management and
attachments use HTTPS.

| Channel | Operation | Purpose | Idempotent |
|---|---|---|---|
| WebSocket | `send` | Send a message; carries a client-generated `clientMsgId` | Yes, deduplicated by `clientMsgId` |
| WebSocket | `msg` (server to client) | Deliver a message with `conversationId` and `seq` | Client deduplicates by `seq` |
| WebSocket | `ack` | Device confirms receipt up to a sequence number | Yes |
| WebSocket | `read` | User marks a conversation read up to `seq` | Yes, monotonic |
| WebSocket | `typing`, `presence` | Ephemeral signals | Not stored |
| HTTPS GET | `/v1/conversations/{id}/messages?after={seq}&limit=200` | Catch-up sync and history | Yes |
| HTTPS POST | `/v1/conversations` | Create a 1:1 or group conversation | With `Idempotency-Key` |
| HTTPS POST | `/v1/attachments` | Get a pre-signed upload URL | With `Idempotency-Key` |

### 4.2 Frame examples

A send frame from the client, the acknowledgement to the sender, and the
delivery frame each recipient device receives:

```json
{"type": "send", "clientMsgId": "01J9ZK7Q4V3M2N8R5T6Y", "conversationId": "c_8f2a", "body": {"text": "Running 5 minutes late"}}
```

```json
{"type": "sent", "clientMsgId": "01J9ZK7Q4V3M2N8R5T6Y", "conversationId": "c_8f2a", "seq": 18342, "serverTs": "2026-03-14T18:02:11.412Z"}
```

```json
{"type": "msg", "conversationId": "c_8f2a", "seq": 18342, "sender": "u_19", "body": {"text": "Running 5 minutes late"}, "serverTs": "2026-03-14T18:02:11.412Z"}
```

### 4.3 Errors, limits and versioning

| Code | Meaning | Client action |
|---|---|---|
| `not_member` | Sender is not in the conversation | Remove conversation locally |
| `rate_limited` | Too many sends; includes `retryAfterMs` | Back off and retry |
| `too_large` | Body over 64 KB | Use an attachment |
| `resync_required` | Server cannot replay the requested range from cache | Call the history endpoint |

- Rate limit: [*Example:* 20 messages per second per device, 600 per minute per user].
- Versioning: protocol version is negotiated in the WebSocket subprotocol
  header (`chat.v1`); the server supports the current and previous versions.

## 5. Data model

<!-- Guidance: Choose storage by access pattern, not by habit. Show entities, keys, partitioning and the queries each store must answer. -->

### 5.1 Entities

The core entities are conversations, their members, messages and per-device
cursors that record how far each device has synced.

```mermaid
erDiagram
  USER {
    string user_id PK
    string display_name
    datetime created_at
  }
  DEVICE {
    string device_id PK
    string user_id FK
    string push_token
    string platform
  }
  CONVERSATION {
    string conversation_id PK
    string kind
    bigint last_seq
    int member_count
  }
  MEMBERSHIP {
    string conversation_id PK
    string user_id PK
    string role
    bigint read_seq
    bigint joined_seq
  }
  MESSAGE {
    string conversation_id PK
    int bucket PK
    bigint seq PK
    string sender_id
    string client_msg_id
    blob body
    datetime server_ts
  }
  DEVICE_CURSOR {
    string device_id PK
    string conversation_id PK
    bigint delivered_seq
  }
  USER ||--o{ DEVICE : owns
  USER ||--o{ MEMBERSHIP : has
  CONVERSATION ||--o{ MEMBERSHIP : includes
  CONVERSATION ||--o{ MESSAGE : contains
  DEVICE ||--o{ DEVICE_CURSOR : tracks
```

### 5.2 Storage choices

| Data | Store | Partition key | Main queries | Why |
|---|---|---|---|---|
| Messages | *Example:* Wide-column store (Cassandra / ScyllaDB class) | (`conversation_id`, `bucket`), clustered by `seq` desc | Latest N; range after `seq` | Append-heavy, range reads within one partition |
| Conversations, memberships | *Example:* Relational database, sharded by `conversation_id` | `conversation_id` | Members of a conversation; `last_seq` counter | Transactional membership changes |
| User inbox index | *Example:* Wide-column store | `user_id`, clustered by last activity | Conversation list with unread counts | One read renders the inbox |
| Device cursors | *Example:* Key-value store | `device_id` | Delivered position per conversation | Small, very frequent writes |
| Presence, routing | *Example:* In-memory store with TTL | `user_id`, `device_id` | Which gateway holds this device | Ephemeral; lost data rebuilds on reconnect |
| Attachments | Object storage + CDN | Content hash | Get by URL | Large blobs out of the message path |

`bucket` is the number of whole 10-day periods since the conversation was
created. It caps partition size for very active groups (target below 100 MB per
partition).

## 6. High-level design

<!-- Guidance: One diagram that shows every component on the request paths and where state lives. Name components by responsibility. -->

Clients connect to gateways through a layer-4 load balancer; everything behind
the gateways is stateless except the stores and the log.

```mermaid
flowchart LR
  client([Mobile / web client]) --> lb[L4 load balancer]
  lb --> gw[WebSocket gateways]
  gw --> chat[Chat service]
  chat --> seqdb[(Conversation DB: last_seq)]
  chat --> msgdb[(Message store)]
  chat --> log[[Message log, partitioned by conversation]]
  log --> fan[Fan-out workers]
  fan --> route[(Routing and presence cache)]
  fan --> gw
  fan --> push[Push service]
  push --> apns[APNs / FCM]
  gw --> presence[Presence service]
  presence --> route
  client --> api[HTTPS API: history, attachments]
  api --> msgdb
  api --> obj[(Object storage + CDN)]
```

| Component | Responsibility | Scales by |
|---|---|---|
| WebSocket gateway | Terminate TLS and WebSocket, authenticate, heartbeats, hold device sessions | Connections per node |
| Chat service | Validate membership, deduplicate, assign `seq`, persist, publish to log | Stateless replicas, partitioned by conversation |
| Message log | Ordered, durable hand-off from write path to fan-out | Partitions (by `conversation_id`) |
| Fan-out workers | Expand recipients and devices, route to gateways or push | Consumer group size |
| Presence service | Track online state and last seen; publish changes to subscribers | Sharded by `user_id` |
| Push service | Batch and send notifications through platform providers | Stateless replicas |
| [Component] | [Responsibility] | [Scaling unit] |

## 7. Detailed design

<!-- Guidance: Go deep on the parts that are new, risky or contested. Show the sequence of calls and the failure behaviour of each step. -->

### 7.1 Send path (critical path)

The sender gets its `sent` acknowledgement as soon as the message is durable;
delivery to recipients happens asynchronously from the log.

```mermaid
sequenceDiagram
  participant A as Sender device
  participant G1 as Gateway (sender)
  participant C as Chat service
  participant D as Conversation DB
  participant S as Message store
  participant L as Message log
  participant F as Fan-out worker
  participant G2 as Gateway (recipient)
  participant B as Recipient device
  A->>G1: send(clientMsgId, conversationId, body)
  G1->>C: forward with authenticated userId
  C->>D: check membership, increment last_seq
  D-->>C: seq = 18342
  C->>S: write message (conversationId, bucket, seq)
  C->>L: publish (key = conversationId)
  C-->>G1: sent(seq)
  G1-->>A: sent(seq)
  L-->>F: message 18342
  F->>F: expand members and devices
  alt device online
    F->>G2: deliver to device session
    G2-->>B: msg(seq 18342)
    B-->>G2: ack(18342)
  else device offline
    F->>F: enqueue push notification
  end
```

Steps that fail before the store write return an error and the client retries
with the same `clientMsgId`. If the service crashes after the store write but
before replying, the retry finds the existing `clientMsgId` in a short-lived
deduplication index (24 h TTL) and returns the original `seq`.

### 7.2 Ordering and consistency model

- **Order is per conversation.** The chat service increments
  `conversation.last_seq` with a single-row atomic update in the shard that owns
  the conversation. Every member sees messages in `seq` order, regardless of
  client clocks or which gateway delivered them.
- **Gaps are detectable.** A client that receives `seq` 18344 after 18342 knows
  18343 is missing and requests `after=18342`. This makes delivery
  at-least-once on the wire and exactly-once in the client view.
- **Read-your-writes for the sender**: the sender shows the message
  immediately (optimistic, keyed by `clientMsgId`) and replaces it with the
  server copy when `sent` arrives.
- **Eventual consistency elsewhere**: inbox order, unread counts, presence and
  receipts may lag by seconds. They are derived views and can be rebuilt from
  the message store and memberships.
- Hot conversations (for example a broadcast channel) serialize on one counter
  row. At *Example:* 2 000 sends per second per row this is sufficient; above it,
  batch sequence allocation in the chat service leader for that conversation.

### 7.3 Fan-out

For groups up to *Example:* 1 000 members, fan-out workers push each message
to every member's online devices (fan-out on write). For broadcast channels
above that size, the worker sends only a lightweight "channel has new
messages up to `seq`" signal to online subscribers, and clients pull the
content (fan-out on read). This bounds the worst case: one post to a 100 000
member channel produces 100 000 small signals instead of 300 000 full payloads,
and offline members cost nothing until they open the channel.

### 7.4 Delivery and read receipts

| Receipt | Produced when | Stored as | Shown to |
|---|---|---|---|
| Sent | Message is durable | `seq` returned to sender | Sender |
| Delivered | Any recipient device acks the `seq` | `device_cursor.delivered_seq` (max) | Sender, aggregated per member |
| Read | Recipient opens conversation up to `seq` | `membership.read_seq` (monotonic max) | Sender; per member in groups up to 100 members, count only above that |

Receipts are watermarks, not per-message rows: "read up to 18342" covers every
earlier message. This keeps receipt writes proportional to activity, not to
message count.

### 7.5 Offline sync and push

On reconnect, the device sends its `delivered_seq` per conversation (or a
single opaque sync token). The server returns conversations whose `last_seq`
is higher, and the client pages through `GET .../messages?after=` for each.
Push notifications are hints only: the payload carries the conversation and
sender name (or nothing, for E2EE), never the authoritative message, so a lost
or duplicated push never corrupts history. Push sends are collapsed per
conversation (one notification with a count) and respect the user's mute
settings.

### 7.6 End-to-end encryption note

For 1:1 chats in v1, clients use an established double-ratchet protocol with
per-device identity keys published to a key directory. The server then stores
and routes ciphertext only. Consequences that must be accepted explicitly:
server-side search, link previews and content moderation are not possible on
encrypted content; multi-device requires encrypting to every device key; and
history on a new device needs a client-side backup or device-to-device
transfer. Group E2EE (sender keys or the MLS protocol) is a v2 decision.

## 8. Scalability and performance

<!-- Guidance: Show how the design meets the numbers in section 3 and where it will break first. -->

| Concern | Approach | Limit before redesign |
|---|---|---|
| Connections | Horizontal gateways; connection-count based load balancing | *Example:* ~150 000 connections per node before GC pauses rise |
| Send throughput | Chat service partitioned by `conversation_id`; message log partitions | *Example:* ~300 000 sends/s with 256 log partitions |
| Hot conversation | Single counter row; batch allocation above 2 000/s | One conversation at ~20 000 sends/s |
| Reconnect storm | Jittered reconnect (1-30 s) and token-bucket admission at gateways | Full-zone failover of 5 million clients |
| Storage growth | Time-bucketed partitions; move buckets older than 2 years to cold storage | [Limit] |

Latency budget for send-to-receive, both users online (target p99 300 ms):

| Step | Budget |
|---|---|
| Client to gateway (mobile network, one way) | 80 ms |
| Gateway to chat service, membership check and `seq` | 15 ms |
| Message store write (quorum) | 20 ms |
| Log publish and fan-out consume | 30 ms |
| Routing lookup and gateway delivery | 15 ms |
| Gateway to recipient (one way) | 80 ms |
| Headroom | 60 ms |

## 9. Reliability, security and operations

<!-- Guidance: Name the failure modes, the threats and the signals operators watch. Each mitigation should be testable. -->

### 9.1 Failure modes

| Failure | Impact | Detection | Mitigation |
|---|---|---|---|
| Gateway node dies | ~100 000 devices disconnect | Connection gauge drop | Clients reconnect with jitter to other nodes; routing entries expire by TTL |
| Routing cache stale | Delivery sent to wrong gateway and dropped | Delivery ack rate | Client gap detection triggers sync; routing entries refreshed on every heartbeat |
| Message log lag | Delivery delayed, sends still succeed | Consumer lag seconds | Autoscale fan-out workers; alert at 10 s lag |
| Message store write timeout | Send fails, client retries | Write error rate | Idempotent retry via `clientMsgId`; quorum across 3 zones |
| Push provider outage | Offline users not notified | Provider error rate | Queue and retry with backoff; messages still sync on open |
| Region outage | Users homed in region cannot send | Synthetic probes | *Example:* Conversations homed in one region; warm standby promoted in under 15 min (RPO under 1 min) |
| [Failure] | [Impact] | [Signal] | [Mitigation] |

### 9.2 Security and abuse

| Threat | Control |
|---|---|
| Session hijacking | Short-lived access tokens on WebSocket upgrade; re-authenticate on token expiry |
| Sending to conversations the user left | Membership checked on every send and every history read |
| Spam and abuse | Per-user and per-device rate limits; new-account limits; user reporting (with E2EE, reported messages are forwarded by the reporter's client) |
| Metadata exposure | Minimise retained metadata; delete push tokens on logout; encrypt at rest |
| Malicious attachments | Content-type allowlist, size limits, malware scan before the download URL is issued |
| [Threat] | [Control] |

### 9.3 Observability

| Signal | Metric or log | Alert threshold |
|---|---|---|
| Send latency | p50, p99 from `send` to `sent` | p99 above 150 ms for 10 min |
| End-to-end delivery | Sampled send-to-ack latency | p99 above 300 ms for 10 min |
| Undelivered backlog | Messages older than 60 s without delivered receipt to any online device | Above 0.1% |
| Fan-out lag | Log consumer lag | Above 10 s |
| Connections | Per node and per zone | Node above 130 000 |
| Client gap rate | Sync requests triggered by gaps | Above 1% of sessions |

### 9.4 Cost estimate

| Item | Unit cost | Quantity | Monthly cost |
|---|---|---|---|
| Gateway nodes | [$/node-month] | *Example:* 225 | [$] |
| Chat and fan-out services | [$/node-month] | [n] | [$] |
| Message store | [$/TB-month] | *Example:* ~55 TB raw added per month | [$] |
| Message log | [$/partition-month] | *Example:* 256 partitions | [$] |
| Push provider and SMS fallback | [$/1 000 sends] | [n] | [$] |
| **Total** | | | **[$]** |

## 10. Trade-offs and alternatives

<!-- Guidance: For each major decision list the options, what each costs and why you chose one. Reviewers will propose the alternatives anyway; answer them here. -->

| Decision | Options | Chosen | Why | What we give up |
|---|---|---|---|---|
| Transport | Long polling; Server-Sent Events plus POST; WebSocket | WebSocket | Bidirectional, low overhead per frame, mature on all platforms | Stateful gateways and reconnect handling |
| Ordering | Client timestamps; global sequence; per-conversation sequence | Per-conversation sequence | Total order where users notice it, no global bottleneck | No order across conversations |
| Message store | Relational; document store; wide-column | Wide-column | Linear write scaling, efficient range reads per partition | Limited secondary indexes and transactions |
| Fan-out | Always on write; always on read; hybrid | Hybrid at 1 000 members | Low latency for normal groups, bounded cost for huge channels | Two code paths |
| Receipts | Per-message rows; watermarks | Watermarks | Writes scale with activity, not messages | Cannot show "read" on one older message independently |
| Encryption | TLS only; E2EE for 1:1; E2EE for all | E2EE for 1:1 in v1 | Strong privacy for most sensitive chats with manageable scope | No server search or moderation on 1:1 content |
| [Decision] | [Options] | [Chosen] | [Why] | [Cost] |

## 11. Delivery plan and risks

<!-- Guidance: How the design reaches production safely, and what could stop it. -->

### 11.1 Milestones

| Milestone | Scope | Exit criterion | Target date |
|---|---|---|---|
| M1 | 1:1 chat, single region, web client | Load test at 2x peak sends, zero lost messages | YYYY-MM-DD |
| M2 | Groups, receipts, mobile push | Gap rate below 0.1% for 14 days in beta | YYYY-MM-DD |
| M3 | Offline sync at scale, attachments | 7-day offline device syncs in under 10 s | YYYY-MM-DD |
| M4 | E2EE for 1:1 | External cryptography review passed | YYYY-MM-DD |

### 11.2 Rollout and rollback

[*Example:* Ship behind a per-user flag. Start with internal staff, then 1%,
10%, 50% and 100% of users, holding each step for at least 48 h while watching
send latency, gap rate and undelivered backlog. Rollback disables the flag;
messages already written stay readable through the history API.]

### 11.3 Risks

| ID | Risk | Likelihood | Impact | Mitigation | Owner |
|---|---|---|---|---|---|
| R-01 | *Example:* Reconnect storm after a gateway deploy overloads auth | Medium | High | Rolling deploys with connection draining; jittered reconnect | [@name] |
| R-02 | *Example:* Hot broadcast channel saturates one shard | Medium | Medium | Hybrid fan-out; batch sequence allocation | [@name] |
| R-03 | *Example:* Push provider throttles bursts | Medium | Medium | Collapse keys, per-app send rate control | [@name] |
| [R-nn] | [Risk] | [L / M / H] | [L / M / H] | [Mitigation] | [@name] |

## 12. Appendices

<!-- Guidance: Supporting detail, glossary and open questions. -->

### 12.1 Open questions

| ID | Question | Owner | Due |
|---|---|---|---|
| OQ-01 | *Example:* Do we keep history forever for paid workspaces? | [@name] | YYYY-MM-DD |
| OQ-02 | *Example:* MLS or sender keys for group E2EE? | [@name] | YYYY-MM-DD |
| [OQ-nn] | [Question] | [@name] | YYYY-MM-DD |

### 12.2 Glossary

| Term | Meaning |
|---|---|
| `seq` | Server-assigned, gap-free, increasing number of a message within its conversation |
| Fan-out | Copying or signalling one message to every recipient device |
| Watermark | "Everything up to this `seq`" marker used for receipts and sync |
| Presence | Online or last-seen state of a user |
| E2EE | End-to-end encryption: only endpoints hold the keys |
| [Term] | [Meaning] |

### 12.3 References

| Reference | Use |
|---|---|
| RFC 6455 The WebSocket Protocol | Transport in 4.1 |
| RFC 9420 The Messaging Layer Security (MLS) Protocol | Candidate for group E2EE |
| Apple Push Notification service and Firebase Cloud Messaging documentation | Push integration in 7.5 |
| [Internal ADR or document] | [Use] |

## Standard and licence note

Based on common industry practice for real-time messaging system design and on
the protocols cited in 12.3. This is a reference design to adapt: the numbers
are illustrative assumptions, not measurements. Original text written for this
collection. Template licence: MIT.