# System Design: [Product name] recommendation system

> Reference design for a large-scale recommendation system that ranks
> items (videos, products, articles or posts) for each user. It covers the
> multi-stage funnel of candidate generation, pre-ranking, ranking and
> re-ranking; embeddings and approximate nearest neighbour (ANN) retrieval;
> feature freshness; the training pipeline; online serving within a latency
> budget; exploration; A/B testing; and feedback loops and bias. It follows
> the structure of the System Design Document template and is pre-filled
> with a worked design to adapt. Replace every `[bracketed]` value and every
> row marked *Example*; values marked *Example* are assumptions, not
> benchmarks. This is an engineering format; recommender transparency and
> profiling rules depend on your jurisdiction and are not legal advice.

<!--
Guidance: How to use this template
- Name the objective before the model: what the system optimises (watch
  time, purchases, long-term retention, satisfaction surveys) and which
  guardrails it must not break. Most recommendation failures are objective
  failures, not modelling failures.
- Size the funnel: corpus, candidates per source, items ranked, items
  shown. Each stage exists because the next one is too expensive to run
  on everything.
- Treat logged impressions as the most valuable data you have. Without
  them, you cannot train on what users actually saw, measure position bias
  or run clean experiments.
- Use ML System Design Document (`ai-data/ml-system-design`) for the model
  development record; this template covers the serving system around it.
-->

## Document control

<!-- Guidance: A design is a proposal until it is approved; record who decided and when, including the owner of the optimisation objective. -->

| Field | Value |
|---|---|
| Document ID | [SDD-REC-001] |
| System / feature | [Home feed and related-items recommendations] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Implemented / Superseded] |
| Owner | [@owner] |
| Author(s) | [@author] |
| Approvers / reviewers | [@reviewer-1], [@reviewer-2] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal] |
| Objective owner | [@product-owner] |
| Surfaces in scope | [Home feed / related items / notifications / email] |
| Related | [ML system design], [Model monitoring plan], [Data contract], [A/B test plan], [DPIA] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Tech lead | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| ML lead | [@name] | [Decision] | YYYY-MM-DD |
| Product owner (objective) | [@name] | [Decision] | YYYY-MM-DD |
| Trust and safety | [@name] | [Decision] | YYYY-MM-DD |
| Privacy | [@name] | [Decision] | YYYY-MM-DD |
| SRE / operations | [@name] | [Decision] | YYYY-MM-DD |

## Revision history

<!-- Guidance: Note what changed in the design, not the wording. -->

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft for design review |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Added two-tower retrieval alongside co-visitation |
| 0.3 | YYYY-MM-DD | [@name] | *Example:* Added exploration slots and position-bias correction |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved |

## Contents

<!-- Guidance: Keep aligned with the headings. -->

1. [Overview](#1-overview)
2. [Requirements](#2-requirements)
3. [Capacity estimation](#3-capacity-estimation)
4. [API design](#4-api-design)
5. [Data model](#5-data-model)
6. [High-level design](#6-high-level-design)
7. [Detailed design](#7-detailed-design)
8. [Scalability and performance](#8-scalability-and-performance)
9. [Reliability, security and operations](#9-reliability-security-and-operations)
10. [Trade-offs and alternatives](#10-trade-offs-and-alternatives)
11. [Delivery plan and risks](#11-delivery-plan-and-risks)
12. [Appendices](#12-appendices)

## 1. Overview

<!-- Guidance: One page that a busy reviewer can read alone: problem, proposed solution, decisions needed and the interview path. -->

### 1.1 Problem statement

[*Example:* Our video app has 100 million daily users and a catalogue of
50 million items, with 400 000 new uploads a day. The home feed is ranked
by a popularity score per category, so new creators get almost no
exposure and returning users see the same items for days. Product wants
longer sessions without growing clickbait complaints. A nightly batch
writes a fixed list per user, so the feed ignores what the user watched
five minutes ago.]

### 1.2 Proposed solution

[*Example:* A request-time funnel: several candidate sources (two-tower
embedding retrieval through an ANN index, co-visitation, subscriptions,
fresh uploads, trending) return about 1 500 candidates; a light pre-ranker
cuts them to 300; a deep ranking model scores those with user, item and
context features predicting several engagement and satisfaction targets; a
re-ranker applies diversity, freshness, policy filters and exploration slots
and returns a page of 20. User features update from a streaming pipeline
within a minute. Every impression is logged with position and model
versions; training is daily for ranking and weekly for retrieval; every
change ships through an A/B test with guardrails.]

### 1.3 Decisions requested from reviewers

| ID | Question | Recommended option | Section |
|---|---|---|---|
| Q-01 | *Example:* Primary objective? | Weighted blend of predicted watch time and satisfaction, guardrailed by complaint rate | 2.2, 7.3 |
| Q-02 | *Example:* Retrieval method? | Two-tower embeddings with HNSW index plus heuristic sources | 7.1 |
| Q-03 | *Example:* User feature freshness? | Streaming session features under 60 s; long-term features daily | 7.2 |
| Q-04 | *Example:* Exploration? | 2 of 20 slots for uncertainty-driven exploration of new items | 7.4 |
| Q-05 | *Example:* Experiment unit? | User, with a 1% long-term holdout | 7.5 |
| [Q-nn] | [Question] | [Option] | [Section] |

### 1.4 45-minute interview path

<!-- Guidance: Interviewers expect the funnel, a retrieval deep dive and a ranking deep dive. Show you know where latency and data go. -->

| Minute | Cover | Notes |
|---|---|---|
| 0 to 5 | Scope: surface, item type, objective, cold start | Ask what "good" means: clicks, watch time, satisfaction |
| 5 to 10 | Core requirements; below the line (search, ads auction, creator tools) | |
| 10 to 15 | Estimates with consequences (3.5) | Funnel sizes and index memory |
| 15 to 20 | API and data model with access patterns | Impression log is a first-class entity |
| 20 to 30 | High-level funnel, then deep dive 1: candidate generation and ANN (7.1) | Two-tower, negatives, index choice |
| 30 to 40 | Deep dive 2: ranking objective and features (7.3) or feedback loops (7.6) | Pick 7.6 if asked about bias |
| 40 to 45 | A/B testing, failure modes, 10x and 0.1x | Close on monitoring and guardrails |

## 2. Requirements

<!-- Guidance: Split the core from what is explicitly below the line, and attach a number to every quality, including guardrails. -->

### 2.1 Core functional requirements

| ID | Requirement | Priority |
|---|---|---|
| FR-01 | *Example:* Return a personalised page of 20 items for the home feed | Must |
| FR-02 | *Example:* Return related items for an item page | Must |
| FR-03 | *Example:* Reflect actions in the current session (watch, skip, like, hide) within one minute | Must |
| FR-04 | *Example:* Recommend for new users with little history (cold start) | Must |
| FR-05 | *Example:* Give new items exposure within hours of upload | Must |
| FR-06 | *Example:* Apply policy filters (removed, age-restricted, region-blocked items) before display | Must |
| FR-07 | *Example:* Let users remove items and signal "not interested", and respect it | Must |
| FR-08 | *Example:* Offer a non-personalised feed option | Should |
| [FR-nn] | [Requirement] | [Must / Should / Could] |

### 2.2 Non-functional requirements and objective

| ID | Quality | Target | Measured as |
|---|---|---|---|
| NFR-01 | Latency | *Example:* p99 at or below 200 ms server time for a page | Service timing |
| NFR-02 | Availability | *Example:* 99.95%; degraded feed allowed | SLO |
| NFR-03 | Session freshness | *Example:* Session actions affect the next page within 60 s | Event-to-feature lag |
| NFR-04 | Item freshness | *Example:* New items eligible within 1 h of passing review | Index update lag |
| NFR-05 | Primary metric | *Example:* Satisfied watch time per user per day | A/B test |
| NFR-06 | Guardrails | *Example:* Complaint and "not interested" rates, creator diversity, no increase above agreed bounds | A/B test |
| NFR-07 | Policy safety | *Example:* Zero removed items served after removal plus 5 min | Audit of impression log |
| [NFR-nn] | [Quality] | [Target] | [Measurement] |

### 2.3 Below the line (out of scope)

- [*Example:* search ranking (shares features, separate design)]
- [*Example:* ads selection and auction (blended by a separate mixer)]
- [*Example:* content moderation classifiers (consumed as item flags)]
- [*Example:* notification send-time optimisation (see Notification Service System Design (`system-design/design-notification-service`))]
- [*Example:* the model development record (see ML System Design Document (`ai-data/ml-system-design`))]
- [Item deferred, with the release that will take it]

## 3. Capacity estimation

<!-- Guidance: Every number must end in a consequence for the design. -->

### 3.1 Assumptions

| Input | Value | Source |
|---|---|---|
| Daily active users | *Example:* 100 million | Analytics |
| Recommendation requests per user per day | *Example:* 20 | Analytics |
| Peak to average ratio | *Example:* 3 | Evening peak |
| Corpus size | *Example:* 50 million eligible items | Catalogue |
| Embedding dimension | *Example:* 128, float32 | Model spec |
| Candidates per request after retrieval | *Example:* 1 500 | Funnel design |
| Items scored by the ranker | *Example:* 300 | Funnel design |
| Logged events per user per day | *Example:* 200 (impressions, plays, skips, likes) | Client logs |

### 3.2 Serving throughput

| Quantity | Calculation | Result | Consequence |
|---|---|---|---|
| Requests per day | 100 million x 20 | 2 billion | |
| Average requests/s | 2 billion / 86 400 | about 23 000/s | |
| Peak requests/s | 23 000 x 3 | about 70 000/s | Stateless serving tier in several regions |
| Ranker item scores/s at peak | 70 000 x 300 | 21 million/s | Batch inference on accelerators; a pre-ranker is required to keep 300, not 1 500 |
| Pre-ranker scores/s at peak | 70 000 x 1 500 | 105 million/s | Pre-ranker must be a cheap dot product or small model |

### 3.3 Index and storage

| Quantity | Calculation | Result | Consequence |
|---|---|---|---|
| Raw item embeddings | 50 million x 128 x 4 B | 25.6 GB | Fits in memory of a large node; replicate per region; quantise to cut cost |
| Quantised (8-bit) | 50 million x 128 x 1 B | 6.4 GB | Many replicas affordable for 70 000 queries/s |
| HNSW graph links | 50 million x *Example* 32 links x 4 B | 6.4 GB | Index about 13 GB per replica with quantised vectors; still fits one node |
| Events per day | 100 million x 200 | 20 billion | Streaming ingestion; partitioned event log |
| Events per second (average) | 20 billion / 86 400 | about 231 000/s | Feature pipeline partitioned by user ID |
| Event bytes per day | 20 billion x *Example* 200 B | 4 TB/day | Sample impressions without engagement for training; keep all positives |

### 3.4 Training

| Quantity | Calculation | Result | Consequence |
|---|---|---|---|
| Ranking training window | *Example:* 30 days of impressions | about 120 TB raw | Train on downsampled negatives with weights; daily incremental training |
| Retrieval retrain | *Example:* weekly, full | | Embedding versions must be consistent between towers and index (7.1) |

### 3.5 Summary

| Quantity | Estimate | Design consequence |
|---|---|---|
| Peak requests | 70 000/s | Regional stateless serving, cached fallbacks |
| Funnel | 50 M to 1 500 to 300 to 20 | Four stages; latency budget per stage (8) |
| Index memory | about 13 GB (quantised vectors plus graph links) | Replicated in-memory ANN |
| Events | 231 000/s | Streaming features, impression logging as a product requirement |

## 4. API design

<!-- Guidance: Define the contract: context in, ranked items with tracking IDs out, and how clients report impressions. -->

### 4.1 Endpoints

| Method and path | Purpose | Idempotent |
|---|---|---|
| `GET /v1/feed?surface=home&cursor=` | Next page of recommendations | Yes (read) |
| `GET /v1/items/{id}/related` | Related items | Yes (read) |
| `POST /v1/events` | Batch of client events (impression, play, skip, like, hide) | Yes, keyed by event ID |
| `POST /v1/feedback/not-interested` | Explicit negative feedback | Yes |
| `GET /v1/feed/why/{impression_id}` | Plain reason for a recommendation | Yes (read) |

### 4.2 Request and response examples

```http
GET /v1/feed?surface=home&cursor=c_19a
Authorization: Bearer [token]
X-Client-Context: {"device": "tv", "locale": "en-GB", "network": "wifi"}
```

```http
HTTP/1.1 200 OK
Content-Type: application/json

{"request_id": "req_7b3f", "items": [{"item_id": "it_501", "impression_id": "imp_7b3f_01", "position": 1, "source": "two_tower"}, {"item_id": "it_88a", "impression_id": "imp_7b3f_02", "position": 2, "source": "explore"}], "next_cursor": "c_19b"}
```

- The client sends an impression event only when an item was actually
  visible (*Example:* 50% of the tile on screen for 1 s), carrying the
  `impression_id`. Requests do not count as impressions.
- `request_id` ties events to the model versions and features used, which
  the server logs at serving time.

## 5. Data model

<!-- Guidance: Show entities, stores, and the access-pattern table. The impression log is as important as the item table. -->

### 5.1 Entities

Users and items have features and embeddings; requests produce impressions;
impressions produce engagements. Training joins all four.

```mermaid
erDiagram
  USER ||--o{ REQUEST : makes
  REQUEST ||--|{ IMPRESSION : shows
  IMPRESSION }o--|| ITEM : of
  IMPRESSION ||--o{ ENGAGEMENT : yields
  ITEM ||--|| ITEM_EMBEDDING : has
  USER ||--o{ USER_FEATURE : has
  USER {
    string user_id PK
    string region
    datetime created_at
  }
  ITEM {
    string item_id PK
    string creator_id
    string policy_status
    datetime published_at
  }
  ITEM_EMBEDDING {
    string item_id PK
    string model_version PK
    string vector
  }
  REQUEST {
    string request_id PK
    string user_id FK
    string surface
    string model_versions
    datetime served_at
  }
  IMPRESSION {
    string impression_id PK
    string request_id FK
    string item_id FK
    int position
    string source
    float propensity
  }
  ENGAGEMENT {
    string impression_id PK
    string type PK
    float value
  }
  USER_FEATURE {
    string user_id PK
    string name PK
    string value
    datetime updated_at
  }
```

### 5.2 Storage choices

| Data | Store | Key | Notes |
|---|---|---|---|
| Item metadata and policy flags | Document store plus cache | `item_id` | Policy flag changes pushed to serving within minutes |
| Item embeddings and ANN index | In-memory ANN index per region, versioned | `model_version` | Built from embedding table; swapped atomically |
| User features (long-term) | Key-value store | `user_id` | Daily batch |
| User session features | In-memory store | `user_id` | Streaming, TTL 24 h |
| Request and impression log | Event log, then data lake | date, `user_id` | Server-side features logged with request |
| Engagements | Event log, then data lake | date, `impression_id` | Joined to impressions for labels |

### 5.3 Access patterns

| ID | Query | Served by | Partition key rationale |
|---|---|---|---|
| AP-01 | *Example:* Top K nearest items to a user embedding | ANN index lookup | Whole index replicated; no partitioning by user |
| AP-02 | *Example:* User long-term and session features | Key-value multi-get by `user_id` | One user per request; partition by `user_id` |
| AP-03 | *Example:* Item features for 1 500 candidates | Batched multi-get on item cache | Hot items cached in serving memory |
| AP-04 | *Example:* Items co-watched with recent items | Precomputed co-visitation table by `item_id` | Daily job; lookup by seed item |
| AP-05 | *Example:* Build training rows with features as served | Join impressions, logged features and engagements by `request_id` and `impression_id` | Avoids recomputing features offline |
| AP-06 | *Example:* Check items already shown recently | Session store set of item IDs with TTL | Dedupe across pages |

## 6. High-level design

<!-- Guidance: One diagram that separates the online funnel from the offline training loop. -->

```mermaid
flowchart LR
  client([App]) --> api[Feed API]
  api --> orch[Recommendation orchestrator]
  orch --> uf[(User features)]
  orch --> cg1[Two-tower retrieval]
  cg1 --> ann[(ANN index)]
  orch --> cg2[Co-visitation]
  orch --> cg3[Subscriptions and fresh items]
  orch --> cg4[Trending]
  orch --> pre[Pre-ranker]
  pre --> rank[Ranking model server]
  rank --> rerank[Re-ranker: policy, diversity, exploration]
  rerank --> api
  api --> log[[Request and impression log]]
  client --> events[[Client events]]
  events --> stream[Streaming features]
  stream --> uf
  log --> lake[(Data lake)]
  events --> lake
  lake --> trainr[Ranking training daily]
  lake --> traint[Retrieval training weekly]
  traint --> embed[Item embedding job]
  embed --> ann
  trainr --> registry[(Model registry)]
  registry --> rank
```

| Component | Responsibility | Scales by |
|---|---|---|
| Orchestrator | Fan out to sources, merge, call rankers, enforce budgets | Horizontally |
| Candidate sources | Return candidates with source labels and scores | Per source |
| ANN index | Nearest neighbour retrieval | Replicas |
| Pre-ranker | Cheap scoring of 1 500 candidates | In process |
| Ranking model server | Deep model on 300 candidates, batched | Accelerators |
| Re-ranker | Policy filter, diversity, freshness, exploration, dedupe | In process |
| Streaming features | Session features from events | Event log partitions |

## 7. Detailed design

<!-- Guidance: Two to four deep dives on the hard problems, each with options, choice and reason. -->

### 7.1 Deep dive: candidate generation and ANN retrieval

| Option | Strength | Weakness |
|---|---|---|
| Heuristic sources only (co-visitation, subscriptions, trending) | Simple, explainable | Weak on long-tail and new users |
| Matrix factorisation | Strong baseline for collaborative signals | Cannot use rich context easily |
| Two-tower model (user tower, item tower) | Uses features on both sides; item embeddings precomputed; user embedding at request time | Needs negatives sampling and index operations |
| Generative or sequence retrieval | Captures order of actions | Costlier to serve; younger tooling |

**Choice:** two-tower retrieval plus heuristic sources, each with a quota.
**Reason:** two-tower gives personalised recall across the long tail;
heuristic sources guarantee freshness, subscriptions and robustness when the
model or index fails.

Index options:

| Index | Recall vs speed | Memory | Updates |
|---|---|---|---|
| Brute force | Exact | Full vectors | Trivial |
| HNSW graph | High recall, low latency | Vectors plus graph links | Incremental inserts supported |
| IVF with product quantisation | Good recall with tuning | Small | Rebuild clusters periodically |

**Choice:** HNSW over 8-bit quantised vectors, rebuilt weekly with the
retrieval model and updated incrementally hourly for new items. **Reason:**
low latency at high recall, and incremental inserts meet NFR-04.

- The user tower and item embeddings must come from the same model
  version; the orchestrator pins a version per request and the index is
  swapped atomically.
- Train with in-batch and sampled negatives, correcting for item popularity
  in the sampling; otherwise popular items appear too often as negatives
  and end up under-recommended.
- New items with no engagement get embeddings from content features
  (title, tags, creator), so they are retrievable on day one.

> **How real systems do it** *(paraphrased from a public write-up; not copied)*
>
> The YouTube recommendations paper by Covington, Adams and Sargin
> describes a two-stage system. A candidate generation network uses a
> user's watch and search history plus context to narrow a corpus of
> millions of videos to a few hundred, framed as a very large
> classification over videos whose learned embeddings can be served with
> nearest neighbour search. A separate ranking network then scores those
> candidates with many more features, and is trained to predict expected
> watch time rather than clicks, so that misleading thumbnails are not
> rewarded. The authors also feed the age of each training example to the
> model so it can learn that users prefer fresh content, and they note that
> offline metrics did not always predict live A/B results. This template
> keeps the same split and the watch-time-style objective, and adds
> heuristic sources, a pre-ranker and exploration.
>
> Source: Paul Covington, Jay Adams, Emre Sargin, "Deep Neural Networks
> for YouTube Recommendations", Proceedings of the 10th ACM Conference on
> Recommender Systems (RecSys 2016), pp. 191 to 198,
> https://doi.org/10.1145/2959100.2959190

### 7.2 Deep dive: feature freshness

| Feature group | Example | Freshness target | Pipeline |
|---|---|---|---|
| Request context | Device, time of day, locale | Real time | In request |
| Session | Last 20 items watched, skips in this session | Under 60 s | Streaming into session store |
| Short-term user | Category affinity over 7 days | Hourly | Streaming aggregates |
| Long-term user | Embedding of 90-day history | Daily | Batch |
| Item counters | Plays and completion rate in the last hour | Minutes | Streaming |
| Item static | Duration, language, creator | On change | Change events |

**Options:** recompute everything in batch; streaming for all; tiered by
value. **Choice:** tiered. **Reason:** session features carry most of the
"react to what I just did" value (FR-03), while long-term features change
slowly and are cheaper in batch. Log every feature value used at serving
time with the request, so training reads what serving saw (AP-05) and
train-serve skew is avoided.

### 7.3 Ranking, objective and re-ranking

- The ranker is a multi-task model predicting several targets per item:
  *Example:* probability of play, expected watch time, probability of like,
  probability of "not interested". The final score is a weighted blend
  owned by product (Q-01), versioned like code.
- Position is a training feature (set to a constant at serving) so the
  model learns item quality separately from position bias.
- The re-ranker enforces: policy filters (NFR-07), dedupe against recent
  impressions, at most *Example* 2 items per creator per page, a minimum
  share of fresh items, and exploration slots (7.4).

| Objective option | Risk |
|---|---|
| Clicks | Rewards clickbait thumbnails |
| Watch time | Rewards long items and binge patterns |
| Blend with satisfaction survey model | Needs survey data and calibration |
| Long-term retention | Slow, noisy signal; used as experiment metric |

**Choice:** blend of watch time and predicted satisfaction, judged in
experiments by long-term retention and guardrails. **Reason:** short-term
proxies drift from user value; anchoring the experiments on retention keeps
the proxy honest.

### 7.4 Exploration and cold start

| Option | How it works | Use |
|---|---|---|
| Epsilon-greedy | Random item in a slot with small probability | Simple baseline; wastes impressions |
| Upper confidence or Thompson sampling | Prefer items with high uncertainty of high value | New items and new users |
| Fixed fresh-item quota | Guarantee exposure for new uploads | Creator fairness |

**Choice:** 2 of 20 slots use Thompson sampling over new and
under-exposed items, plus a fresh-item quota in candidate generation.
**Reason:** new items need impressions to learn about them, and logging the
exploration probability (`propensity`) gives unbiased data for evaluation.
New users get a short onboarding of topic choices and a popularity prior by
region, updated after each action through session features.

### 7.5 A/B testing and evaluation

- Offline: recall at K for retrieval on held-out next actions; normalised
  ranking metrics and calibration for the ranker; counterfactual estimates
  from logged propensities for re-ranking changes.
- Online: user-level randomisation; primary metric NFR-05; guardrails
  NFR-06; latency and error rate. Use A/B Test Plan (`product/ab-test-plan`)
  to fix hypothesis, sample size and duration before launch.
- Keep a long-term holdout (*Example:* 1% of users on a frozen baseline for
  a quarter) to measure cumulative effects that weekly tests miss.
- Interference: recommendations change what creators upload and what
  others see; for creator-side metrics consider cluster or time-split
  designs.

### 7.6 Feedback loops and bias

| Bias or loop | Effect | Mitigation |
|---|---|---|
| Position bias | Top items get more engagement regardless of quality | Position as training feature; randomised swaps in a small slice |
| Exposure bias | Items never shown never get positive labels | Exploration slots; propensity-weighted training |
| Popularity feedback | Popular items get more popular | Popularity-corrected negative sampling; creator diversity constraint |
| Filter bubble | Narrowing topics over time | Diversity in re-ranking; topic breadth guardrail |
| Harmful amplification | Borderline content optimised for engagement | Trust and safety scores as re-ranking penalties; policy review of objective |

Monitor these with Model Monitoring Plan (`ai-data/model-monitoring-plan`),
including distribution of impressions across creators and topics, not only
model accuracy.

## 8. Scalability and performance

<!-- Guidance: Show how the design meets the numbers in section 3 and where it will break first. -->

Latency budget for a feed page (p99 200 ms):

| Stage | Budget |
|---|---|
| Fetch user and session features | 15 ms |
| Candidate sources in parallel (ANN about 10 ms) | 30 ms |
| Merge and item feature fetch | 20 ms |
| Pre-ranker on 1 500 | 15 ms |
| Ranking model on 300 (batched) | 50 ms |
| Re-ranking and policy filters | 10 ms |
| Network and serialisation | 20 ms |
| Headroom | 40 ms |

| Concern | Approach | Limit before redesign |
|---|---|---|
| Ranking compute | Batch on accelerators; distil model if needed | *Example:* cost per 1 000 requests |
| ANN throughput | More replicas; quantisation | Memory per replica |
| Event ingestion | Partitioned log by `user_id` | Partition count |
| Hot items | Item features cached in serving memory | Cache size |
| Multi-region | Index and models replicated per region | Index build and distribution time |

## 9. Reliability, security and operations

<!-- Guidance: Name the failure modes, the threats and the signals operators watch. A degraded feed is better than an error. -->

### 9.1 Failure modes

| ID | Component | Failure | Detection | Blast radius | Mitigation | RTO |
|---|---|---|---|---|---|---|
| FM-01 | Ranking model server | Down or slow | Timeouts | All personalised pages | Fall back to pre-ranker order | *Example:* 5 min |
| FM-02 | ANN index | Corrupt or version mismatch | Recall canary queries; version check | Two-tower source | Drop source; heuristic sources continue; roll back index | *Example:* 15 min |
| FM-03 | Feature store | Unavailable | Error rate | Personalisation quality | Default features; cached last page per user | *Example:* 10 min |
| FM-04 | Streaming features | Lag | Event-to-feature lag | Session freshness | Alert at 5 min; scale consumers | *Example:* 10 min |
| FM-05 | Bad model release | Metric drop | Guardrail alarms in canary | Canary share | Automatic rollback to previous version | *Example:* 5 min |
| FM-06 | Policy flag propagation | Removed item still served | Impression audit | Users who see the item | Policy filter reads item flags from a fast path; block list pushed directly | *Example:* 5 min |
| [FM-nn] | [Component] | [Failure] | [Signal] | [Scope] | [Mitigation] | [RTO] |

### 9.2 Security, abuse and privacy

| ID | Threat or duty | Control |
|---|---|---|
| SEC-01 | Engagement fraud (fake views to game ranking) | Bot filtering before features and labels; anomaly detection per item |
| SEC-02 | Data poisoning through coordinated accounts | Weight labels by account trust; cap influence per account |
| SEC-03 | Sensitive inference from history | Exclude sensitive categories from targeting features; DPIA (`compliance/dpia`) |
| SEC-04 | Transparency and opt-out duties for recommender systems where they apply | "Why this" endpoint, non-personalised option (FR-08); confirm obligations with counsel; this is a format, not legal advice |
| [SEC-nn] | [Threat] | [Control] |

### 9.3 Observability

| Signal | Metric | Alert threshold |
|---|---|---|
| Latency | Page p99 | Over 200 ms for 5 min |
| Fallback | Share of pages from fallback path | Above 1% |
| Source mix | Share of impressions per candidate source | Change above 30% day over day |
| Engagement | Play rate and watch time per impression | Drop above 5% hour over hour |
| Diversity | Distinct creators per 100 impressions | Below guardrail |
| Policy | Impressions of removed items | Any after grace window |

## 10. Trade-offs and alternatives

<!-- Guidance: List options, the cost of each choice, and what changes at 10x and 0.1x. -->

| Decision | Options | Chosen | Why | What we give up |
|---|---|---|---|---|
| Serving mode | Precompute lists nightly; request time | Request time | Session freshness (FR-03) | Higher serving cost |
| Retrieval | Heuristics; MF; two-tower | Two-tower plus heuristics | Recall and robustness | Index operations |
| Index | HNSW; IVF-PQ | HNSW quantised | Latency and incremental inserts | More memory than IVF-PQ |
| Objective | Clicks; watch time; blend | Blend with satisfaction | Avoid clickbait and binge proxies | Survey data pipeline |
| Exploration | None; epsilon; Thompson | Thompson in 2 slots | Learns new items, logs propensity | Some short-term engagement |
| **At 10x (700 000 requests/s, 500 M items)** | Bigger index; sharded index | Shard ANN by item partitions with scatter-gather; cheaper distilled ranker | Index memory and ranking cost dominate | More tail latency from fan-out |
| **At 0.1x (7 000 requests/s, 5 M items)** | Full funnel; simpler | Drop pre-ranker; brute-force or managed vector search; single ranker | Fewer stages to run | Less headroom to grow |
| [Decision] | [Options] | [Chosen] | [Why] | [Cost] |

## 11. Delivery plan and risks

<!-- Guidance: How the design reaches production safely, and what could stop it. -->

### 11.1 Milestones

| Milestone | Scope | Exit criterion | Target date |
|---|---|---|---|
| M1 | Impression logging with request features | 99% of impressions joinable to requests | YYYY-MM-DD |
| M2 | Request-time funnel with heuristic sources and ranker | A/B win on primary metric, guardrails flat | YYYY-MM-DD |
| M3 | Two-tower retrieval and ANN index | Recall at K above baseline; latency within budget | YYYY-MM-DD |
| M4 | Streaming session features | Lag under 60 s at peak | YYYY-MM-DD |
| M5 | Exploration and bias corrections | Long-term holdout readout | YYYY-MM-DD |

### 11.2 Risks

| ID | Risk | Likelihood | Impact | Mitigation | Owner |
|---|---|---|---|---|---|
| R-01 | *Example:* Objective weights tuned for short-term metrics hurt retention | Medium | High | Long-term holdout; quarterly objective review | [@name] |
| R-02 | *Example:* Client impression events unreliable on some devices | High | Medium | Client SDK tests; server-side fallback signals; Data Contract (`ai-data/data-contract`) | [@name] |
| R-03 | *Example:* Accelerator capacity shortfall at peak | Medium | Medium | Distilled ranker; reduce ranked items under load | [@name] |
| [R-nn] | [Risk] | [L / M / H] | [L / M / H] | [Mitigation] | [@name] |

## 12. Appendices

<!-- Guidance: Supporting detail, traceability, glossary and references. -->

### 12.1 Traceability

| Requirement | Design element | Verification |
|---|---|---|
| FR-03, NFR-03 | 7.2 session features | Lag dashboard; end-to-end test |
| FR-05, NFR-04 | 7.1 incremental index, 7.4 quota | Time to first impression for new items |
| NFR-01 | 8 latency budget | Load test at peak |
| NFR-05, NFR-06 | 7.3 objective, 7.5 experiments | A/B readouts |
| NFR-07 | 7.3 policy filter, FM-06 | Impression audit |
| [ID] | [Element] | [Test] |

### 12.2 Open questions

| ID | Question | Owner | Due |
|---|---|---|---|
| OQ-01 | *Example:* Do we personalise for signed-out users from device history? | [@name] | YYYY-MM-DD |
| OQ-02 | *Example:* How often should objective weights change, and who approves? | [@name] | YYYY-MM-DD |
| [OQ-nn] | [Question] | [@name] | YYYY-MM-DD |

### 12.3 Glossary

| Term | Meaning |
|---|---|
| Candidate generation | Retrieval of a few hundred to a few thousand items from the full corpus |
| Two-tower model | Model with separate user and item networks whose embeddings are compared by dot product |
| ANN | Approximate nearest neighbour search |
| HNSW | Hierarchical navigable small world graph index |
| Pre-ranker | Cheap model that trims candidates before the expensive ranker |
| Propensity | Probability that the system showed an item at a position, logged for unbiased evaluation |
| Position bias | Higher engagement caused by position, not quality |

### 12.4 References

| Reference | Use |
|---|---|
| Covington, Adams, Sargin, "Deep Neural Networks for YouTube Recommendations", RecSys 2016, doi:10.1145/2959100.2959190 | Two-stage funnel and objective (7.1); paraphrased |
| ML System Design Document (`ai-data/ml-system-design`) | Model development record |
| Model Monitoring Plan (`ai-data/model-monitoring-plan`) | Monitoring and drift |
| Data Contract (`ai-data/data-contract`) | Event and feature contracts |
| A/B Test Plan (`product/ab-test-plan`) | Experiment design |
| News Feed System Design (`system-design/design-news-feed`) | Feed assembly and fan-out |
| Video Streaming System Design (`system-design/design-video-streaming`) | Item delivery after recommendation |

## Standard and licence note

Based on common industry practice for large-scale recommender systems and
on the public references in 12.4. Papers and company write-ups were used as
inspiration only; the text, diagrams and numbers are original and
illustrative. Rules on recommender transparency, profiling and minors
differ by jurisdiction and change; check the current official texts. This
is an engineering format, not legal advice. Template licence: MIT.