# System Design: [Video conferencing service name]

> Reference design for a multiparty video conferencing service built on
> WebRTC: browser and native clients, a signaling service, selective
> forwarding units (SFUs) with simulcast, bandwidth estimation and layer
> selection, cascading across regions for large meetings, recording and
> optional end-to-end encryption. It builds on the W3C Recommendation
> "WebRTC: Real-Time Communication in Browsers" (13 March 2025), RTP
> (RFC 3550), RTP topologies (RFC 7667), simulcast (RFC 8853), congestion
> control feedback (RFC 8888) and SFrame (RFC 9605). It follows the structure
> of the System Design Document template, adds a 45-minute interview path,
> access-pattern and failure-mode tables and a cited "How real systems do it"
> box, and is pre-filled with a worked design to adapt. Replace every
> `[bracketed]` value and every row marked *Example*; values marked *Example*
> are assumptions, not benchmarks.

<!--
Guidance: How to use this template
- Decide the topology first (mesh, MCU or SFU, section 7.1). Almost every
  later number depends on it, because it decides who pays for bandwidth:
  the sender, the server CPU or the server network.
- Fill the bitrate ladder in 3.1 with the layers your clients really send;
  egress bandwidth, not CPU, sizes most SFU fleets.
- Keep media and signaling separate in every diagram. Reviewers lose track
  quickly when a WebSocket and an RTP stream share one arrow.
- Test a meeting on a lossy, low-bandwidth link before launch; the layer
  selection logic in 7.3 is where users feel the design.
-->

## Document control

<!-- Guidance: A design is a proposal until it is approved; record who decided and when. -->

| Field | Value |
|---|---|
| Document ID | [SDD-VC-001] |
| System / feature | [Real-time meetings: audio, video, screen share] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Implemented / Superseded] |
| Owner | [@owner] |
| Author(s) | [@author] |
| Reviewers | [@reviewer-1], [@reviewer-2] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal] |
| Difficulty | Advanced |
| WebRTC baseline | [W3C Recommendation, 13 March 2025] |
| Related | [Live streaming design], [Notification service], [ADR: SFU versus MCU] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Tech lead | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| Staff / principal engineer | [@name] | [Decision] | YYYY-MM-DD |
| Media / real-time engineering | [@name] | [Decision] | YYYY-MM-DD |
| SRE / network operations | [@name] | [Decision] | YYYY-MM-DD |
| Security and privacy | [@name] | [Decision] | YYYY-MM-DD |

## Revision history

<!-- Guidance: Note what changed in the design, not the wording. -->

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft for design review |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Replaced live composite recording with raw track recording after cost review |
| 0.3 | YYYY-MM-DD | [@name] | *Example:* Added SFU cascading for meetings that span regions |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved |

## Contents

<!-- Guidance: Keep aligned with the headings. -->

1. [Overview](#1-overview)
2. [Requirements](#2-requirements)
3. [Capacity estimation](#3-capacity-estimation)
4. [API design](#4-api-design)
5. [Data model](#5-data-model)
6. [High-level design](#6-high-level-design)
7. [Detailed design](#7-detailed-design)
8. [Scalability and performance](#8-scalability-and-performance)
9. [Reliability, security and operations](#9-reliability-security-and-operations)
10. [Trade-offs and alternatives](#10-trade-offs-and-alternatives)
11. [Delivery plan and risks](#11-delivery-plan-and-risks)
12. [Appendices](#12-appendices)

## 1. Overview

<!-- Guidance: One page that a busy reviewer can read alone: problem, proposed solution in a paragraph, and the decisions that need their attention. -->

### 1.1 Problem statement

[*Example:* The current product runs small calls as a peer-to-peer mesh. Each
participant uploads one copy of its video to every other participant, so a
six-person call needs about 7.7 Mbps of upload per person (section 3.2), and
calls above four people freeze on home connections. Customers also ask for
recording, meetings of several hundred people and an end-to-end encrypted
mode for sensitive calls.]

### 1.2 Proposed solution

[*Example:* Clients join through a signaling service over a WebSocket and
send media to the nearest SFU over WebRTC (ICE, DTLS-SRTP). Each sender
publishes three simulcast layers of its camera. The SFU forwards packets
without decoding them and picks, for every receiver, the layer that fits the
receiver's estimated bandwidth and screen layout. Meetings whose participants
are in several regions use one SFU per region, linked by a cascade that
carries each layer once between regions. Recording writes the raw forwarded
tracks to object storage and composes a playable file after the meeting. An
optional end-to-end encrypted mode wraps every frame with SFrame keys that
the server never sees.]

### 1.3 Decisions requested from reviewers

| ID | Question | Recommended option | Section |
|---|---|---|---|
| Q-01 | *Example:* Mesh, MCU or SFU? | SFU for every meeting, including two-person calls, so one code path serves all sizes | 7.1, 10 |
| Q-02 | *Example:* How many simulcast layers? | Three layers (180p, 360p, 720p) for camera, one high layer for screen share | 7.2 |
| Q-03 | *Example:* Live composite recording or raw track recording? | Raw tracks during the meeting, composite afterwards | 7.5 |
| Q-04 | *Example:* Is end-to-end encryption on by default? | Opt-in per meeting; it disables server recording and transcription | 7.6 |
| [Q-nn] | [Question] | [Option] | [Section] |

### 1.4 45-minute interview path

Use this order when the document serves an interview or a short design
review. The written sections below are the long form of each step.

| Minutes | Cover | Output on the whiteboard |
|---|---|---|
| 0 to 5 | Scope: meeting size, audio and video, screen share, recording; below-the-line items (2.3) | Three to five core requirements, latency target |
| 5 to 10 | Estimates: upload per sender, download per receiver, SFU egress, server count (section 3) | "Egress, not CPU, sizes the fleet" |
| 10 to 15 | Interfaces: join API, signaling messages, what travels over RTP (section 4) | Join call plus offer, answer and candidate messages |
| 15 to 25 | High-level design: clients, signaling, SFU, TURN, recording (section 6) | One diagram with media and signaling paths drawn apart |
| 25 to 40 | One deep dive: simulcast and layer selection with bandwidth estimation (default, 7.2 and 7.3), or cascading for large meetings (7.4), or recording (7.5) | Options, choice, reason |
| 40 to 45 | Failures (SFU crash, packet loss, region loss) and the 10x row in section 10 | Two failure rows |

Start the default deep dive with the naive answer, a mesh where every peer
sends to every other peer, and show that upload grows with meeting size. Then
introduce the MCU, which fixes upload but decodes and re-encodes every stream,
and finally the SFU, which only forwards packets but needs the sender to offer
several qualities so that each receiver can get one that fits.

## 2. Requirements

<!-- Guidance: Split core requirements from below-the-line items so the design stays focused. Non-functional targets need numbers and a way to measure them. -->

### 2.1 Core functional requirements

| ID | Requirement | Priority |
|---|---|---|
| FR-01 | *Example:* Create a meeting, share a link, join from a browser or native app | Must |
| FR-02 | *Example:* Two-way audio and video for up to 100 interactive participants | Must |
| FR-03 | *Example:* Screen sharing by one presenter at a time | Must |
| FR-04 | *Example:* Active speaker view and gallery view with up to 25 tiles | Must |
| FR-05 | *Example:* Host controls: mute others, remove participant, lock meeting, waiting room | Must |
| FR-06 | *Example:* Cloud recording available within 30 minutes after the meeting ends | Should |
| FR-07 | *Example:* Optional end-to-end encrypted meetings | Should |
| FR-08 | *Example:* View-only attendees up to 1 000 per meeting | Could |
| [FR-nn] | [Requirement] | [Must / Should / Could] |

### 2.2 Non-functional requirements

| ID | Quality | Target | Measured as |
|---|---|---|---|
| NFR-01 | Media latency | *Example:* Glass-to-glass p95 at or below 250 ms within one region, 400 ms across regions | Synthetic clients with timestamped frames |
| NFR-02 | Join time | *Example:* p95 from click to first audio at or below 3 s | Client telemetry |
| NFR-03 | Availability | *Example:* 99.95% of meeting-minutes without a media interruption longer than 5 s | Client reconnect events |
| NFR-04 | Quality under loss | *Example:* Audio remains intelligible at 10% random packet loss | Network emulation test |
| NFR-05 | Scale | *Example:* 2 million concurrent participants at peak | Load test with synthetic publishers |
| NFR-06 | Privacy | *Example:* Media encrypted in transit on every hop; E2EE mode keeps media keys off servers | Security review, packet capture |
| [NFR-nn] | [Quality] | [Target] | [Measurement] |

### 2.3 Below the line (out of scope)

Name these explicitly so reviewers and interviewers do not pull the design
toward them.

- [*Example:* broadcast to unlimited viewers through a CDN (see the live streaming design)]
- [*Example:* telephone dial-in through a PSTN gateway]
- [*Example:* live captions and translation (separate speech pipeline consuming audio tracks)]
- [*Example:* calendar integration and scheduling user interface]
- [Item deferred, with the release that will take it]

## 3. Capacity estimation

<!-- Guidance: Back-of-the-envelope numbers decide the shape of the design. Show inputs, formulas and results; round generously and say which way you rounded. -->

### 3.1 Assumptions

| Input | Symbol | Value | Source |
|---|---|---|---|
| Peak concurrent participants | $P$ | *Example:* 2 000 000 | Usage forecast |
| Average interactive meeting size | $m$ | *Example:* 6 | Product analytics |
| Simulcast camera layers (180p, 360p, 720p) | $b_l, b_m, b_h$ | *Example:* 0.15, 0.5, 1.5 Mbps | Encoder settings |
| Audio bitrate incl. packet overhead | $a$ | *Example:* 0.04 Mbps | Opus at about 32 kbps plus headers |
| Share of participants with camera on | $c$ | *Example:* 0.7 | Product analytics |
| Receive mix per participant | | *Example:* 1 large tile at 720p, 3 small tiles at 180p, 3 loudest audio streams | Default layout |
| Usable egress per SFU server | $E$ | *Example:* 6 Gbps (10 Gbps NIC at 60%) | Load test |
| Average meeting length | $d$ | *Example:* 45 min | Product analytics |

### 3.2 Why not a mesh

In a mesh each sender uploads its high layer and audio to every other
participant:

$$
U_{mesh} = (m - 1)(b_h + a) = 5 \times (1.5 + 0.04) = 7.7 \ \text{Mbps per sender}
$$

Many home and mobile uplinks cannot sustain this, so a server in the media
path is required for meetings above three or four people.

### 3.3 Upload and download per participant with an SFU

$$
U = c\,(b_l + b_m + b_h) + a = 0.7 \times 2.15 + 0.04 = 1.545 \ \text{Mbps}
$$

$$
D = b_h + 3\,b_l + 3a = 1.5 + 0.45 + 0.12 = 2.07 \ \text{Mbps}
$$

The sender uploads three layers once; the receive mix stays constant as the
meeting grows, because the layout caps the number of tiles.

### 3.4 SFU fleet

$$
I = P \times U = 2 \times 10^{6} \times 1.545 \ \text{Mbps} = 3.09 \ \text{Tbps ingress}
$$

$$
O = P \times D = 2 \times 10^{6} \times 2.07 \ \text{Mbps} = 4.14 \ \text{Tbps egress}
$$

$$
N = \frac{O}{E} = \frac{4\,140 \ \text{Gbps}}{6 \ \text{Gbps}} = 690 \ \text{servers}, \qquad 690 \times 1.25 \approx 863
$$

Ingress alone would need 515 servers, so egress is the binding constraint.
With 25% headroom for failover and uneven placement the fleet is about 870
SFU servers across all regions. With an average packet of about 1 100 bytes,
egress is about 470 million packets per second, about 680 000 per server
before headroom, so
the forwarding path must avoid per-packet allocation and system calls where
possible (batching sends, kernel bypass or large socket buffers).

### 3.5 Signaling and recording

- Joins: $P / d = 2 \times 10^{6} / 2\,700 \ \text{s} \approx 741$ joins/s
  on average at peak, about 7 400 joins/s in the minute after the hour when
  meetings start together. At about 10 signaling messages per join that is
  about 74 000 messages/s, small for a horizontally scaled WebSocket tier.
- Recording: if 5% of 4 million daily meetings are recorded, live composite
  recording at 2.064 Mbps (720p plus 64 kbps audio) writes about 139 TB per
  day (12.5 PB for 90 days) and needs about 16 700 compositors running at
  peak, about 33 000 vCPU at 2 vCPU each. Raw track recording of the 360p
  layers ($6 \times (0.7 \times 0.5 + 0.04) = 2.34$ Mbps per meeting) writes
  about 158 TB per day (14.2 PB for 90 days) but needs no live decoding.

### 3.6 Summary

| Quantity | Estimate | Design consequence |
|---|---|---|
| Mesh upload at 6 people | 7.7 Mbps per sender | Server in the media path from day one |
| Upload per participant | 1.545 Mbps | Simulcast with three layers is affordable for senders |
| SFU egress | 4.14 Tbps | Fleet sized by NIC egress, about 870 servers with headroom |
| Packets | About 680 000 per server per second | Forwarding path optimised for packet rate |
| Joins | About 7 400/s at the top of the hour | Stateless signaling tier, meeting state in a shared store |
| Recording | 139 to 158 TB per day | Raw tracks now, composite later; lifecycle rules on storage |

## 4. API design

<!-- Guidance: Define the contract first: resources, methods, request and response shapes, errors, idempotency, pagination, rate limits and versioning. Use RFC 9457 problem details for errors. -->

### 4.1 Interfaces

| Interface | Operation | Purpose |
|---|---|---|
| REST | `POST /v1/meetings` | Create a meeting; returns ID, join link, settings |
| REST | `POST /v1/meetings/{id}/join-tokens` | Issue a short-lived signed join token for one participant and role |
| WebSocket | `wss://signal.example.com/v1?token=` | Signaling channel for one participant session |
| WebRTC | ICE, DTLS-SRTP, RTP and RTCP to the assigned SFU | Media and media feedback |
| REST | `GET /v1/meetings/{id}/recordings` | List recordings and their processing state |
| Webhook | `meeting.ended`, `recording.ready` | Notify integrations |

### 4.2 Join flow

The join token is a signed, short-lived credential (for example a JWT with a
60 s expiry) that names the meeting, the participant and the role. The
signaling service validates it, picks an SFU (7.4) and returns ICE servers,
including TURN credentials that expire with the session.

```json
{"type": "joined", "meetingId": "mtg_7Qx2", "participantId": "p_19", "role": "host", "sfu": {"region": "eu-west", "node": "sfu-eu-w-114"}, "iceServers": [{"urls": ["turns:turn.eu-west.example.com:443?transport=tcp"], "username": "1730300000:p_19", "credential": "[short-lived]"}]}
```

### 4.3 Signaling messages

| Message | Direction | Content |
|---|---|---|
| `offer` / `answer` | Both | SDP per JSEP (RFC 9429), including the simulcast `rid` list |
| `candidate` | Both | Trickled ICE candidates (RFC 8445) |
| `layout` | Client to server | Which participants the client shows and at what size |
| `track-published` / `track-removed` | Server to client | Track list changes for the meeting |
| `active-speaker` | Server to client | Current dominant speaker from audio levels |
| `host-action` | Both | Mute, remove, lock, admit from waiting room |
| `e2ee-key-epoch` | Server to client | Epoch announcement only; key material never passes through the server in E2EE mode |

The `layout` message matters for capacity: a client that shows only the
active speaker should not receive 24 thumbnails it will never draw.

## 5. Data model

<!-- Guidance: Choose storage by access pattern, not by habit. Show entities, keys, partitioning and the queries each store must answer. -->

### 5.1 Entities

Meetings and recordings are durable records; sessions, tracks and SFU
assignments are live state that exists only while a meeting runs.

```mermaid
erDiagram
  MEETING {
    string meeting_id PK
    string host_user_id
    string region_hint
    string state
    bool e2ee
    datetime starts_at
  }
  SESSION {
    string session_id PK
    string meeting_id FK
    string participant_id
    string role
    string sfu_node
    datetime joined_at
    datetime left_at
  }
  TRACK {
    string track_id PK
    string session_id FK
    string kind
    string rids
    string state
  }
  SFU_NODE {
    string node_id PK
    string region
    float egress_load
    int sessions
    datetime last_heartbeat
  }
  RECORDING {
    string recording_id PK
    string meeting_id FK
    string state
    string object_prefix
    datetime ready_at
  }
  MEETING ||--o{ SESSION : has
  SESSION ||--o{ TRACK : publishes
  SFU_NODE ||--o{ SESSION : hosts
  MEETING ||--o{ RECORDING : produces
```

### 5.2 Storage choices

| Data | Store | Key | Why |
|---|---|---|---|
| Meetings, recordings metadata | *Example:* Relational database, replicated | `meeting_id` | Low write rate, needs transactions for host settings |
| Live meeting state (sessions, tracks, SFU assignment) | *Example:* In-memory store with replication, per region | `meeting:{meeting_id}` | Read on every signaling event; lost state can be rebuilt from clients |
| SFU load registry | *Example:* Service discovery with TTL keys | `sfu:{region}:{node_id}` | Heartbeats every 2 s, expire on node loss |
| Recorded tracks | Object storage | `rec/{meeting_id}/{track_id}/{segment}` | Large sequential writes, lifecycle rules |
| Quality telemetry | Event pipeline to analytics store | Time, meeting | High volume, never on the media path |

### 5.3 Access patterns

| Access pattern | Frequency | Key or index that serves it | Partition key rationale |
|---|---|---|---|
| Find the SFU that hosts a meeting in this region | Every join (about 7 400/s at peak) | `meeting:{meeting_id}` hash, field per region | One meeting is the unit of placement, so all its state lives under one key |
| Pick the least-loaded SFU for a new meeting | Every first join in a region | Sorted set of nodes by egress load per region | Per-region set; joins never scan other regions |
| Broadcast a track change to all participants | Every publish or unpublish | Meeting channel on the signaling tier keyed by `meeting_id` | Signaling nodes subscribe to the meetings they serve |
| List recordings for a meeting | Rare | `meeting_id` index in the relational store | Joins with meeting ownership for access checks |
| Compose a recording after the meeting | Once per recording | Object prefix `rec/{meeting_id}/` | One prefix per meeting lets a worker list every track in one call |
| Analyse call quality by network and client | Batch | Telemetry partitioned by day and region | Never touches live state |

## 6. High-level design

<!-- Guidance: One diagram that shows every component on the request paths and where state lives. Name components by responsibility. -->

Media and signaling take different paths. Media goes from client to SFU over
UDP when possible, through TURN when a firewall blocks UDP.

```mermaid
flowchart LR
  subgraph clients["Clients"]
    a[Browser client]
    b[Native client]
  end
  a -->|"HTTPS: join token"| api[Meeting API]
  a -->|WebSocket signaling| sig[Signaling service]
  b -->|WebSocket signaling| sig
  sig --> state[(Live meeting state)]
  sig --> disc[(SFU registry)]
  a <-->|"SRTP media"| sfu1[SFU region A]
  b <-->|"SRTP via TURN"| turn[TURN relay]
  turn <--> sfu1
  sfu1 <-->|"cascade, one copy per layer"| sfu2[SFU region B]
  sfu1 --> rec[Recorder sink]
  rec --> obj[(Object storage)]
  obj --> comp[Composer jobs]
  api --> db[(Meetings DB)]
  sfu1 --> tel[[Quality telemetry]]
```

| Component | Responsibility | Scales by |
|---|---|---|
| Meeting API | Meetings, join tokens, recordings, webhooks | Stateless instances |
| Signaling service | WebSocket sessions, SDP exchange, layout, host actions, SFU assignment | Connections per node; stateless apart from sockets |
| SFU | Terminate DTLS-SRTP, forward RTP, select layers, generate RTCP feedback | Egress bandwidth and packet rate (section 3.4) |
| TURN relay | Relay media for clients that cannot reach the SFU directly | Relayed bandwidth (about 10% of sessions, *Example*) |
| Recorder sink | Write forwarded packets of recorded meetings into segments | Recorded meetings |
| Composer jobs | Decode raw tracks and produce a playable file after the meeting | Batch workers on spare capacity |
| SFU registry | Load and health of each SFU, heartbeats | Small, per region |

## 7. Detailed design

<!-- Guidance: Go deep on the parts that are new, risky or contested. Show the sequence of calls and the failure behaviour of each step. -->

The deep dives in this section, each with options, choice and reason:

| Deep dive | Options considered | Choice | Reason | Section |
|---|---|---|---|---|
| Topology | Mesh, MCU, SFU | SFU | Constant upload for senders, no server transcoding | 7.1 |
| Quality adaptation | Single stream with server transcoding, simulcast, scalable video coding (SVC) | Simulcast, SVC where every client supports it | Works with every WebRTC client today | 7.2, 7.3 |
| Large and multi-region meetings | One SFU per meeting, cascade of SFUs | Cascade by region | Each participant stays close to its SFU | 7.4 |
| Recording | Live composite bot, raw track capture | Raw tracks, composite after the meeting | No live decoding, cheaper at peak | 7.5 |

### 7.1 Topology: mesh, MCU or SFU

RFC 7667 describes the RTP topologies; the three relevant here are:

| Topology | What the server does | Sender upload | Server cost | Fits |
|---|---|---|---|---|
| Mesh (point to point between every pair) | Nothing in the media path | Grows with meeting size | None | Two or three people |
| MCU (mixer) | Decodes every stream, mixes one composite per layout, re-encodes | One stream | High CPU per meeting, added latency of a decode and encode | Legacy room systems, very weak receivers |
| SFU (selective forwarding middlebox) | Forwards packets and selects which stream each receiver gets | One stream per layer | Bandwidth and packet rate, little CPU | Interactive meetings of any size |

The SFU wins because it adds almost no latency (no decode), keeps CPU cost
low and lets every receiver choose its own layout. Its weakness, that the
sender must offer several qualities, is solved by simulcast.

### 7.2 Simulcast

Each sender encodes its camera at several resolutions at once and sends them
as separate RTP streams, identified by a restriction identifier (`rid`)
negotiated in SDP as RFC 8853 describes. In the W3C WebRTC API the sender
sets `sendEncodings` on `addTransceiver`, one entry per layer, with
`rid` and `scaleResolutionDownBy`.

```js
pc.addTransceiver(cameraTrack, {
  direction: "sendonly",
  sendEncodings: [
    { rid: "l", scaleResolutionDownBy: 4, maxBitrate: 150000 },
    { rid: "m", scaleResolutionDownBy: 2, maxBitrate: 500000 },
    { rid: "h", scaleResolutionDownBy: 1, maxBitrate: 1500000 }
  ]
});
```

The SFU tells a sender to pause a layer nobody is receiving (for example the
720p layer when nobody has that participant in the large tile), which saves
the sender's uplink and CPU. Screen share uses one high-resolution, low frame
rate stream instead, because text must stay sharp.

### 7.3 Bandwidth estimation and layer selection

Two estimates drive adaptation:

- **Sender side**: the sender's congestion controller estimates how much it
  can upload and sets the encoder bitrates. Browsers use delay-based and
  loss-based control (the Google Congestion Control approach, described in
  the expired draft-ietf-rmcat-gcc-02) driven by transport-wide feedback.
  RFC 8888 standardises an RTCP feedback format for congestion control.
- **Receiver side at the SFU**: the SFU sends packets to each receiver and
  estimates that receiver's downlink from feedback about arrival times and
  loss. It then picks layers.

The layer selection loop runs per receiver every few hundred milliseconds:

1. Start from the receiver's layout: one large tile, up to $k$ small tiles,
   the three loudest audio streams.
2. Audio first: audio is never dropped for video.
3. Give each small tile its lowest layer.
4. Give the large tile the highest layer that still fits the estimate.
5. If even the lowest layers do not fit, pause the small tiles and keep the
   large tile at its lowest layer; tell the client so it can show avatars.

Worked example with the ladder from 3.1 (*Example*):

| Receiver estimate | Large tile | Small tiles | Audio | Total | Result |
|---|---|---|---|---|---|
| 2.5 Mbps | 720p (1.5) | 3 x 180p (0.45) | 3 x 0.04 (0.12) | 2.07 Mbps | Full layout |
| 1.2 Mbps | 360p (0.5) | 3 x 180p (0.45) | 0.12 | 1.07 Mbps | Large tile steps down |
| 0.5 Mbps | 180p (0.15) | paused | 0.12 | 0.27 Mbps | Small tiles show avatars |

A layer switch must happen on a key frame of the new layer, otherwise the
receiver cannot decode. The SFU requests a key frame from the sender (RTCP
picture loss indication) and switches when it arrives, and it rewrites
sequence numbers and timestamps so the receiver sees one continuous stream.
To avoid oscillation, step up only after the estimate has stayed above the
next layer's bitrate plus 15% for a few seconds; step down at once.

Packet loss is handled per hop: the SFU retransmits lost packets to a
receiver from its own short buffer (NACK), so a lossy receiver does not
cause retransmissions from the sender to everyone. Audio relies on Opus
forward error correction instead, because a retransmission arrives too late
for speech.

```mermaid
sequenceDiagram
  participant S as Sender
  participant F as SFU
  participant R as Receiver
  S->>F: RTP layers l, m, h
  F->>R: RTP layer h for the large tile
  R-->>F: transport feedback shows rising delay
  F->>F: estimate drops to 1.2 Mbps
  F->>S: PLI for layer m
  S->>F: key frame on layer m
  F->>R: switch large tile to layer m
  F->>S: pause layer h if no receiver needs it
```

### 7.4 Placement and cascading

- **Placement**: the first participant in a region creates the meeting's
  presence on the least-loaded SFU of that region (by egress, not session
  count). Later participants in the same region join the same node, so one
  meeting stays on one SFU per region while it fits.
- **Cascade**: when participants are in several regions, each region's SFU
  forwards to the other region's SFU only the layers that remote receivers
  need, once per layer and not once per receiver. A 40-person meeting split
  across two regions crosses the long-haul link with a few streams instead
  of dozens.
- **Large meetings**: a meeting above the capacity of one SFU spreads its
  receivers over several SFUs in the same region connected as a tree. View-only
  attendees beyond about 1 000 move to the live streaming path, which trades
  seconds of latency for CDN scale.

### 7.5 Recording

Two options were compared for FR-06:

| Option | How it works | Cost at peak | Quality | Chosen |
|---|---|---|---|---|
| Live composite bot | A headless client joins, renders the layout, encodes one file | About 33 000 vCPU (3.5) | What attendees saw | No |
| Raw track capture | The SFU copies forwarded packets of each track into segments; a composer decodes after the meeting | Storage writes only during the meeting, batch CPU afterwards | Can render several layouts later | Yes |

Raw capture stores the 360p layer of each camera, the screen share and audio.
Composition runs on batch capacity within the 30-minute target in FR-06.
Participants are told that recording is on (visual indicator and an audible
notice) before any packet is captured.

### 7.6 Security and end-to-end encryption

- **Transport**: WebRTC requires DTLS-SRTP, so every media hop is encrypted.
  The SFU terminates SRTP, so in standard mode the operator can access media
  on the server (needed for recording and transcription).
- **E2EE mode**: clients encrypt each encoded frame with SFrame (RFC 9605)
  before packetisation. Keys come from a group key agreement between
  clients, for example MLS (RFC 9420); the server only relays opaque
  handshake messages and sees epoch numbers. The SFU still routes packets
  and selects layers because RTP headers stay readable. Server recording,
  transcription and dial-in are disabled in this mode, and the UI shows a
  verification code participants can compare.
- **Access**: join tokens are short-lived and bound to a role; a waiting room
  and meeting lock prevent uninvited joins; TURN credentials expire with the
  session.

### 7.7 How real systems do it

> **How real systems do it** (paraphrased, not copied). Discord's engineering
> post "How Discord Handles Two and Half Million Concurrent Voice Users using
> WebRTC" (Jozsef Vass, 10 September 2018) explains that all calls are routed
> through Discord's own servers rather than peer to peer, because peer-to-peer
> cost rises with participants and routing through servers hides user IP
> addresses. A home-grown C++ SFU forwards audio and video and uses receiver
> reports to tell senders how much bandwidth they have. Native clients use a
> customised WebRTC media engine with a much smaller signaling exchange than
> full SDP, while browsers use the standard API. Signaling services assign a
> voice server from those registered in a service discovery system, and when
> a voice server fails its health check clients reconnect to a new one. The
> post reports more than 850 voice servers in 13 regions. Read the original
> in 12.3; this box keeps only the general idea.

## 8. Scalability and performance

<!-- Guidance: Show how the design meets the numbers in section 3 and where it will break first. -->

| Concern | Approach | Limit before redesign |
|---|---|---|
| SFU egress | Place by egress load, pause unused layers, layout-driven subscriptions | *Example:* 6 Gbps per server at 60% NIC |
| Packet rate | Batched sends, preallocated buffers, one forwarding thread per core | *Example:* About 1 million packets/s per server in load test |
| Signaling | Stateless WebSocket nodes, meeting state in a regional store | [Connections per node] |
| Join storms at the top of the hour | Pre-warmed SFU capacity, join token issuance spread across API nodes | *Example:* 7 400 joins/s |
| Large meetings | SFU tree inside a region, cascade between regions, view-only to streaming | *Example:* 1 000 interactive receivers |

Latency budget within one region (target p95 250 ms glass to glass):

| Step | Budget |
|---|---|
| Capture and encode | 40 ms |
| Sender to SFU network | 30 ms |
| SFU forwarding | 2 ms |
| SFU to receiver network | 30 ms |
| Jitter buffer | 60 ms |
| Decode and render | 30 ms |
| Headroom | 58 ms |

## 9. Reliability, security and operations

<!-- Guidance: Name the failure modes with their blast radius and recovery time, the threats, and the signals operators watch. Each mitigation should be testable. -->

### 9.1 Failure modes

| Component | Failure | Detection | Blast radius | Mitigation | RTO |
|---|---|---|---|---|---|
| SFU node | Process crash or host loss | Missed heartbeats, client ICE failure | Meetings on that node in one region | Clients reconnect through signaling to a new node; signaling replays track list | *Example:* 5 s |
| SFU region | Region outage | Registry empty, synthetic calls fail | Participants in that region | Signaling assigns the next nearest region; higher latency accepted | *Example:* 30 s |
| Signaling node | Crash | WebSocket close | Sessions on that node lose control messages, media continues | Client reconnects with session ID and resumes | *Example:* 3 s |
| TURN relay | Overload or outage | Allocation failures | Clients behind restrictive firewalls | Multiple TURN pools, TLS on 443 fallback | *Example:* 60 s |
| Network | Burst loss on a receiver link | Receiver loss and delay metrics | One receiver | NACK from SFU buffer, layer step-down, Opus FEC | Immediate |
| Recorder sink | Object storage write errors | Segment upload failures | Recordings of affected meetings | Local spool for 15 minutes, then alert host that recording is incomplete | *Example:* 15 min |
| [Component] | [Failure] | [Signal] | [Scope] | [Mitigation] | [Time] |

### 9.2 Security and abuse

| Threat | Control |
|---|---|
| Uninvited joins ("meeting bombing") | Join tokens, waiting room, lock, host removal, rate limits on join attempts |
| Media interception | DTLS-SRTP on every hop; E2EE mode with SFrame for sensitive meetings |
| IP address exposure between participants | All media through SFU or TURN, never peer to peer |
| TURN abuse as an open relay | Short-lived credentials bound to a session, permissions only toward SFU addresses |
| Recording without notice | Server enforces visible and audible notice before capture starts |
| [Threat] | [Control] |

### 9.3 Observability

| Signal | Metric or log | Alert threshold |
|---|---|---|
| Media quality | Freeze rate, receiver loss, round-trip time per region | Freeze rate doubles vs. 7-day baseline |
| Join success | Joins with first audio within 3 s | Below 99% for 10 min |
| SFU load | Egress Gbps, packets/s, CPU per node | Egress above 70% of NIC |
| Reconnects | Client reconnects per 1 000 meeting-minutes | Doubling vs. baseline |
| TURN share | Sessions relayed through TURN | Above 20% (firewall or ICE regression) |

### 9.4 Cost estimate

| Item | Unit cost | Quantity | Monthly cost |
|---|---|---|---|
| SFU servers | [$/server-month] | *Example:* About 870 | [$] |
| Egress bandwidth | [$/GB] | *Example:* 4.14 Tbps at peak, measure 95th percentile | [$] |
| TURN relays | [$/server-month] | [Count] | [$] |
| Recording storage | [$/GB-month] | *Example:* About 14 PB at 90-day retention | [$] |
| **Total** | | | **[$]** |

## 10. Trade-offs and alternatives

<!-- Guidance: For each major decision list the options, what each costs and why you chose one. Reviewers will propose the alternatives anyway; answer them here. -->

| Decision | Options | Chosen | Why | What we give up |
|---|---|---|---|---|
| Topology | Mesh; MCU; SFU | SFU for all sizes | Constant sender upload, no transcoding, per-receiver layouts | Server bandwidth bill; receivers decode several streams |
| Adaptation | Simulcast; SVC; server transcoding | Simulcast now, SVC where supported | Works with all WebRTC clients | Sender uploads about 43% more than its top layer alone |
| Two-person calls | Peer to peer; SFU | SFU | One code path, no IP exposure | Small added latency and server cost |
| Recording | Live composite; raw tracks | Raw tracks | No live decoding at peak | Recording ready minutes after the meeting, not instantly |
| E2EE | Always on; opt-in; none | Opt-in | Keeps recording and transcription for most meetings | Two modes to test and explain |
| What changes at 10x / 0.1x | 10x (20 million concurrent): about 41 Tbps egress, 6 900 servers at 10 Gbps or about 2 760 at 25 Gbps and 60%, so move to 25 or 100 Gbps NICs, push regions closer to users and negotiate peering. 0.1x (200 000 concurrent): about 69 SFU servers, one or two regions, and a managed TURN service | Review at each doubling of peak participants | Egress and peering cost grow first | [Note] |
| [Decision] | [Options] | [Chosen] | [Why] | [Cost] |

## 11. Delivery plan and risks

<!-- Guidance: How the design reaches production safely, and what could stop it. -->

### 11.1 Milestones

| Milestone | Scope | Exit criterion | Target date |
|---|---|---|---|
| M1 | Signaling, single-region SFU, audio and one video layer | 25-person internal meetings for 2 weeks | YYYY-MM-DD |
| M2 | Simulcast, layer selection, NACK and key frame handling | Freeze rate below target on emulated 1.2 Mbps links | YYYY-MM-DD |
| M3 | Multi-region cascade, TURN pools | Cross-region p95 latency at or below 400 ms | YYYY-MM-DD |
| M4 | Raw track recording and composer | Recordings ready within 30 min for 99% of meetings | YYYY-MM-DD |
| M5 | E2EE mode with SFrame | External security review closed | YYYY-MM-DD |

### 11.2 Rollout and rollback

[*Example:* New SFU releases go to 1% of new meetings in one region, then
10%, then all, with automatic rollback when the freeze rate or reconnect
rate rises by more than 20% against the previous release. Meetings already
running stay on their node until they end; nodes are drained, never killed.]

### 11.3 Risks

| ID | Risk | Likelihood | Impact | Mitigation | Owner |
|---|---|---|---|---|---|
| R-01 | *Example:* Browser changes break simulcast negotiation | Medium | High | Run browser beta channels in the synthetic call suite | [@name] |
| R-02 | *Example:* Egress cost exceeds plan | Medium | High | Layout-driven subscriptions; pause unused layers; regional peering | [@name] |
| R-03 | *Example:* E2EE key handling flaw | Low | High | Use a reviewed MLS library; external audit before launch | [@name] |
| [R-nn] | [Risk] | [L / M / H] | [L / M / H] | [Mitigation] | [@name] |

## 12. Appendices

<!-- Guidance: Supporting detail, glossary and open questions. -->

### 12.1 Open questions

| ID | Question | Owner | Due |
|---|---|---|---|
| OQ-01 | *Example:* Which codecs do we require (VP8, VP9, AV1, H.264) for each client platform? | [@name] | YYYY-MM-DD |
| OQ-02 | *Example:* Do recordings need a separate legal notice per jurisdiction where participants join? | [@name] | YYYY-MM-DD |
| [OQ-nn] | [Question] | [@name] | YYYY-MM-DD |

### 12.2 Glossary

| Term | Meaning |
|---|---|
| SFU | Selective forwarding unit: a server that forwards RTP packets and chooses which streams each receiver gets, without decoding |
| MCU | Multipoint control unit: a server that decodes, mixes and re-encodes streams into one composite |
| Simulcast | Sending the same source at several qualities as separate streams |
| SVC | Scalable video coding: one stream with layers a server can drop |
| ICE, STUN, TURN | NAT traversal: candidate gathering and checks, address discovery, media relay |
| PLI, NACK | RTCP messages asking for a key frame and for retransmission of lost packets |
| SFrame | Frame-level encryption for end-to-end protected media (RFC 9605) |
| [Term] | [Meaning] |

### 12.3 References

| Reference | Use |
|---|---|
| W3C, ["WebRTC: Real-Time Communication in Browsers"](https://www.w3.org/TR/webrtc/), W3C Recommendation, 13 March 2025 | Browser API, `sendEncodings`, simulcast |
| [RFC 3550](https://www.rfc-editor.org/rfc/rfc3550) RTP: A Transport Protocol for Real-Time Applications (July 2003) | RTP and RTCP |
| [RFC 7667](https://www.rfc-editor.org/rfc/rfc7667) RTP Topologies (November 2015) | Mesh, mixer and selective forwarding middlebox, 7.1 |
| [RFC 8853](https://www.rfc-editor.org/rfc/rfc8853) Using Simulcast in SDP and RTP Sessions (January 2021) | 7.2 |
| [RFC 8888](https://www.rfc-editor.org/rfc/rfc8888) RTCP Feedback for Congestion Control (January 2021) | 7.3 |
| [draft-ietf-rmcat-gcc-02](https://datatracker.ietf.org/doc/draft-ietf-rmcat-gcc/) Google Congestion Control (Internet-Draft of July 2016, expired January 2017) | Background for 7.3 |
| [RFC 9429](https://www.rfc-editor.org/rfc/rfc9429) JSEP (April 2024), [RFC 8445](https://www.rfc-editor.org/rfc/rfc8445) ICE, [RFC 8656](https://www.rfc-editor.org/rfc/rfc8656) TURN | Signaling and NAT traversal |
| [RFC 9605](https://www.rfc-editor.org/rfc/rfc9605) SFrame (August 2024), [RFC 9420](https://www.rfc-editor.org/rfc/rfc9420) MLS (July 2023) | 7.6 |
| Vass, ["How Discord Handles Two and Half Million Concurrent Voice Users using WebRTC"](https://discord.com/blog/how-discord-handles-two-and-half-million-concurrent-voice-users-using-webrtc), Discord blog, 10 September 2018 | 7.7 |
| Live Streaming System Design (`system-design/design-live-streaming`) | View-only attendees beyond the interactive limit |
| Chat System Design (`system-design/design-chat-system`) | In-meeting chat and presence |
| Notification Service Design (`system-design/design-notification-service`) | Meeting reminders and recording-ready notices |
| Back-of-Envelope Estimation and Capacity Model (`system-design/back-of-envelope-estimation`) | Method for section 3 |

## Standard and licence note

Based on the W3C WebRTC Recommendation (13 March 2025) and the IETF RFCs
cited in 12.3, which are public specifications. The Discord engineering post
is paraphrased in 7.7 only; no text, figures or code were copied apart from
the short reported counts attributed to it, and all other numbers here are
illustrative examples. WebRTC specifications and browser support change;
check the current W3C and IETF pages before implementing. Original text
written for this collection. Template licence: MIT.