# ML System Design: [System or feature name]

> Complete design document for a production machine learning system:
> problem framing, success metrics, data and labelling, features, model
> choice, training, offline and online evaluation, serving, budgets,
> monitoring, retraining, fairness and safety, failure modes and rollout.
> Replace every `[bracketed]` value and every row marked *Example*. Worked
> examples use a product recommendation feature for an online shop.

<!--
Guidance: How to use this template
- Start with section 1. Most failed ML projects solved a problem nobody
  needed solved, or optimised a model metric that did not move the
  business metric.
- Always compare against a simple baseline (rules, popularity, last
  value). If the model cannot beat it clearly, ship the baseline.
- Treat data and labels as the main design risk. Sections 2 and 3 usually
  deserve more review time than model architecture.
- Write down how the system fails and how you will know (sections 10 and
  12) before writing training code.
-->

## Document control

<!-- Guidance: Record ownership and the decision trail. An ML design is a proposal until reviewed by engineering, data science, product and responsible AI. -->

| Field | Value |
|---|---|
| Document ID | [MLSD-SYSTEM-001] |
| System / feature | [Name] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Implemented / Superseded] |
| Author(s) | [@author] |
| Reviewers | [@ml-reviewer], [@platform-reviewer], [@product-owner] |
| Classification | [Internal / Confidential] |
| Last updated | YYYY-MM-DD |
| Related | [PRD], [Model card], [Datasheet], [AI risk assessment], [Data pipeline spec] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| ML lead | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| Product owner | [@name] | [Decision] | YYYY-MM-DD |
| Platform / SRE | [@name] | [Decision] | YYYY-MM-DD |
| Responsible AI / privacy | [@name] | [Decision] | YYYY-MM-DD |

## Revision history

<!-- Guidance: Note design changes, not wording changes. -->

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Replaced real-time features with hourly batch after latency review |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved for build |

## Contents

<!-- Guidance: Keep aligned with the headings below. -->

1. [Problem framing and success metrics](#1-problem-framing-and-success-metrics)
2. [Data sources and labelling](#2-data-sources-and-labelling)
3. [Feature pipeline](#3-feature-pipeline)
4. [Model selection and baselines](#4-model-selection-and-baselines)
5. [Training pipeline](#5-training-pipeline)
6. [Offline evaluation](#6-offline-evaluation)
7. [Online evaluation](#7-online-evaluation)
8. [Serving architecture](#8-serving-architecture)
9. [Latency and cost budgets](#9-latency-and-cost-budgets)
10. [Monitoring](#10-monitoring)
11. [Retraining](#11-retraining)
12. [Fairness, safety and privacy](#12-fairness-safety-and-privacy)
13. [Failure modes](#13-failure-modes)
14. [Rollout plan](#14-rollout-plan)
15. [Open questions and references](#15-open-questions-and-references)

## 1. Problem framing and success metrics

<!-- Guidance: State the user problem, why ML is needed, the ML task formulation, and the metrics at business and model level. Show how the model metric is expected to move the business metric. -->

### 1.1 Problem statement

[Who has the problem and what it costs today. Example: "Shoppers on
product pages see 'related items' chosen by a static category rule;
fewer than 2% click them, and catalogue items outside the top 500 are
almost never discovered."]

### 1.2 Why machine learning

[Why rules or heuristics are not enough, and what non-ML alternative was
considered. Example: "A co-purchase rule covers only 30% of products;
personalised ranking is needed for the long tail."]

### 1.3 ML task formulation

| Aspect | Decision |
|---|---|
| Task type | *Example:* Ranking: order 200 candidate products for a (user, page) pair |
| Prediction target | *Example:* Probability the user adds the item to cart within the session |
| Unit of prediction | *Example:* (session, anchor product, candidate product) |
| Prediction time | *Example:* Online, at page render |
| Action taken on output | *Example:* Show top 8 candidates in the carousel |

### 1.4 Success metrics

| Level | Metric | Current | Target | Guardrail |
|---|---|---|---|---|
| Business | *Example:* Add-to-cart rate from carousel | 1.8% | 3.0% | Overall conversion must not drop |
| Business | *Example:* Revenue per session | [value] | +1.5% | Return rate must not rise more than 0.2 pt |
| Model (offline) | *Example:* NDCG@8 | 0.21 (rule) | at least 0.30 | Calibration error at most 0.05 |
| Model (offline) | *Example:* Recall@50 of candidate generation | n/a | at least 0.85 | |
| System | *Example:* p99 inference latency | n/a | at most 60 ms | Page render p99 unchanged |

For classification-style components, use the usual definitions:

$$
\text{Precision} = \frac{TP}{TP + FP}, \qquad
\text{Recall} = \frac{TP}{TP + FN}, \qquad
F_1 = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}
$$

For ranking, normalised discounted cumulative gain at cut-off $k$:

$$
\text{DCG@}k = \sum_{i=1}^{k} \frac{2^{rel_i} - 1}{\log_2(i + 1)}, \qquad
\text{NDCG@}k = \frac{\text{DCG@}k}{\text{IDCG@}k}
$$

## 2. Data sources and labelling

<!-- Guidance: List every source, how labels are produced, the label delay, and the leakage risks. Link the datasheet for each dataset. -->

### 2.1 Data sources

| ID | Source | Content | Volume | Freshness | Owner | Contains personal data |
|---|---|---|---|---|---|---|
| SRC-01 | *Example:* Clickstream events | Views, clicks, add-to-cart | 40 M events/day | Streaming, under 1 min | Web platform | Yes, pseudonymous user ID |
| SRC-02 | *Example:* Product catalogue | Title, category, price, stock | 1.2 M products | Hourly | Catalogue team | No |
| SRC-03 | *Example:* Orders | Purchases and returns | 300 k/day | Daily | Commerce team | Yes |
| [SRC-nn] | [Source] | [Content] | [Volume] | [Freshness] | [Owner] | [Yes / No] |

### 2.2 Labels

[How labels are defined and produced, label delay, noise and class
balance. Example: "Positive = add-to-cart within the same session after
an impression; negatives sampled from impressions without action at 1:20.
Labels available within 1 hour. Position bias corrected with inverse
propensity weights from a 1% randomised slot."]

### 2.3 Leakage and bias risks

| Risk | Example | Prevention |
|---|---|---|
| Target leakage | *Example:* Feature uses cart contents after the impression | Point-in-time feature joins |
| Feedback loop | *Example:* Model only learns from items it already shows | 5% exploration traffic |
| Selection bias | *Example:* Logged-in users over-represented | Weighting; separate evaluation for guests |
| [Risk] | [Example] | [Prevention] |

## 3. Feature pipeline

<!-- Guidance: Describe features, how they are computed in training and serving, and how you guarantee the two match. Training-serving skew is the most common silent failure. -->

| Feature group | Examples | Computed | Store | Freshness | Owner |
|---|---|---|---|---|---|
| User history | *Example:* Last 50 viewed categories, 30-day spend band | Streaming aggregates | Online feature store | 5 min | Recs team |
| Product | *Example:* Price band, category embedding, stock flag | Batch | Online and offline store | Hourly | Recs team |
| Context | *Example:* Device, time of day, page type | At request | None | Real time | Web platform |
| Cross | *Example:* Co-view count of anchor and candidate | Daily batch | Offline and online store | Daily | Recs team |
| [Group] | [Examples] | [How] | [Store] | [Freshness] | [Owner] |

Skew controls: [Example: one feature definition used for both offline and
online computation; daily job compares feature distributions between
training snapshots and logged serving features].

## 4. Model selection and baselines

<!-- Guidance: Compare candidates against baselines on the same data and metric. Prefer the simplest model that meets the target. -->

| Option | Description | NDCG@8 (validation) | p99 latency | Complexity | Decision |
|---|---|---|---|---|---|
| Baseline A | *Example:* Static category rule (current) | 0.21 | 2 ms | Low | Fallback |
| Baseline B | *Example:* Popularity within category, last 7 days | 0.24 | 2 ms | Low | Comparison |
| Candidate 1 | *Example:* Gradient-boosted ranker on features in 3 | 0.31 | 18 ms | Medium | Chosen |
| Candidate 2 | *Example:* Two-tower neural retrieval plus transformer re-ranker | 0.33 | 75 ms | High | Rejected: latency, ops cost |
| [Option] | [Description] | [value] | [ms] | [L / M / H] | [Decision] |

Rationale: [Why the chosen model, and what would make you revisit.]

## 5. Training pipeline

<!-- Guidance: Make training reproducible: pinned data snapshots, code version, configuration, seeds, and a model registry entry for every run. -->

| Step | Detail |
|---|---|
| Data snapshot | *Example:* 28-day window, point-in-time join, snapshot ID recorded |
| Splits | *Example:* Time-based: train days 1-24, validation 25-26, test 27-28 |
| Hyperparameter search | *Example:* 40 trials Bayesian search on validation NDCG@8 |
| Compute | *Example:* 1 CPU node, 64 vCPU, about 40 minutes per run |
| Artefacts | Model file, feature list, metrics, data snapshot ID, code commit |
| Registry | [Model registry and naming convention] |
| Reproducibility | *Example:* Same snapshot and seed reproduce NDCG within 0.002 |

## 6. Offline evaluation

<!-- Guidance: Define the test set, metrics, slices and the acceptance bar before training. Report uncertainty and per-slice results. -->

| Check | Method | Acceptance criterion |
|---|---|---|
| Overall quality | NDCG@8 on test days | at least 0.30 and at least 0.05 above Baseline B |
| Slices | *Example:* New users, guests, mobile, long-tail products | No slice more than 10% below its baseline |
| Calibration | Expected calibration error | at most 0.05 |
| Robustness | *Example:* Products with missing price or image | No errors; graceful score |
| Uncertainty | Bootstrap 95% CI over sessions | Lower bound above target |
| [Check] | [Method] | [Criterion] |

## 7. Online evaluation

<!-- Guidance: Plan the experiment before launch: unit of randomisation, primary metric, guardrails, sample size and duration. -->

| Item | Plan |
|---|---|
| Design | *Example:* A/B test, 50/50 split, randomised by user ID |
| Primary metric | *Example:* Add-to-cart rate from carousel |
| Guardrail metrics | *Example:* Overall conversion, page latency p99, return rate, error rate |
| Minimum detectable effect | *Example:* +0.3 pt absolute on 1.8% baseline |
| Duration | *Example:* At least 14 days to cover two weekly cycles |
| Stop rule | *Example:* Stop early if any guardrail degrades significantly at the 99% level |

Approximate sample size per arm for a two-proportion test with baseline
rate $p$, absolute effect $\delta$, significance $\alpha$ and power
$1 - \beta$:

$$
n \approx \frac{2\,(z_{1-\alpha/2} + z_{1-\beta})^{2}\; p(1-p)}{\delta^{2}}
= \frac{2\,(1.96 + 0.84)^{2} \times 0.018 \times 0.982}{0.003^{2}} \approx 30\,800
$$

## 8. Serving architecture

<!-- Guidance: Show the request path, where features come from, where the model runs, and the fallback when any part fails. -->

The request path retrieves candidates, fetches features, scores with the
model and falls back to the rule-based list on any timeout.

```mermaid
flowchart LR
  web["Web page request"] --> api["Recommendation API"]
  api --> cand["Candidate generator (co-view, category)"]
  api --> fs[("Online feature store")]
  api --> model["Ranker service"]
  model --> reg[("Model registry")]
  api --> fb["Fallback: static rule list"]
  api --> log[["Impression and feature log"]]
  log --> lake[("Data lake")]
  lake --> train["Training pipeline"]
  train --> reg
```

| Component | Responsibility | Scaling | Timeout |
|---|---|---|---|
| Recommendation API | Orchestrate, enforce budget, fall back | Stateless replicas | 80 ms overall |
| Candidate generator | Return 200 candidates | Cached per anchor product | 20 ms |
| Online feature store | Serve user and product features | Managed, partitioned | 10 ms |
| Ranker service | Score candidates | CPU replicas, autoscaled on QPS | 25 ms |
| [Component] | [Responsibility] | [Scaling] | [ms] |

## 9. Latency and cost budgets

<!-- Guidance: Give budgets per step and a cost estimate per 1 000 predictions. Budgets are design constraints, not hopes. -->

| Step | p99 budget |
|---|---|
| Candidate generation | 20 ms |
| Feature fetch | 10 ms |
| Scoring 200 candidates | 25 ms |
| Network and serialisation | 5 ms |
| **Total** | **60 ms** (headroom 20 ms to the 80 ms timeout) |

Serving cost per 1 000 requests, with instance cost $c$ per hour and
sustained throughput $q$ requests per second per instance:

$$
C_{1000} = \frac{1000 \cdot c}{3600 \cdot q}
= \frac{1000 \times 0.40}{3600 \times 150} \approx 0.00074 \ \text{currency units}
$$

| Item | Estimate per month |
|---|---|
| Serving compute | [value] |
| Feature store | [value] |
| Training compute | [value] |
| Logging and storage | [value] |
| **Total** | **[value]** |

## 10. Monitoring

<!-- Guidance: Monitor the system, the data, the model and the business outcome. Every signal needs an owner and a threshold. -->

| Layer | Signal | Threshold | Action | Owner |
|---|---|---|---|---|
| System | p99 latency, error rate, fallback rate | p99 above 60 ms, fallback above 2% for 10 min | Page on-call | [@platform] |
| Data quality | Null rate, out-of-range values per feature | Any feature null rate above 5% | Alert, block retraining | [@data] |
| Data drift | Population stability index per top-20 feature | PSI above 0.2 | Investigate within 1 day | [@ml] |
| Prediction drift | Score distribution shift | KS statistic above 0.1 | Investigate | [@ml] |
| Model quality | Daily NDCG@8 on logged outcomes | Drop above 10% vs 28-day mean | Consider rollback | [@ml] |
| Business | Carousel add-to-cart rate | Drop above 15% day over day | Review with product | [@product] |

Population stability index for one feature with bins $i$, expected share
$e_i$ and actual share $a_i$:

$$
PSI = \sum_{i} (a_i - e_i) \ln \frac{a_i}{e_i}
$$

## 11. Retraining

<!-- Guidance: State when and how the model is retrained, how a new model is validated, and who approves promotion. -->

| Aspect | Policy |
|---|---|
| Schedule | *Example:* Weekly, Sunday 02:00 UTC |
| Triggers | *Example:* PSI above 0.2 on 3 or more top features; quality drop above 10% |
| Validation gate | *Example:* Offline criteria in 6 plus shadow scoring for 24 h |
| Promotion | *Example:* Automatic if gate passes and change in NDCG within +/-5%; manual approval otherwise |
| Rollback | *Example:* Previous registry version re-pinned in under 5 minutes |

## 12. Fairness, safety and privacy

<!-- Guidance: Identify who could be harmed or treated unequally and how you measure it. Link to the AI risk assessment for the full register. -->

| Concern | Measure | Mitigation |
|---|---|---|
| Seller exposure fairness | *Example:* Share of impressions for small sellers vs catalogue share | Exposure floor in re-ranking |
| Sensitive inferences | *Example:* Recommendations revealing health or pregnancy purchases | Sensitive categories excluded from history features |
| Personal data | *Example:* Use of browsing history | Consent check; opt-out disables personalisation |
| Harmful content | *Example:* Age-restricted products to minors | Policy filter after ranking |
| [Concern] | [Measure] | [Mitigation] |

## 13. Failure modes

<!-- Guidance: List how the system can fail, how you detect it and what users see. Include ML-specific failures, not only infrastructure ones. -->

| ID | Failure | Impact | Detection | Mitigation |
|---|---|---|---|---|
| FM-01 | Feature store unavailable | No personalisation | Fallback rate alert | Serve static rule list |
| FM-02 | Silent training-serving skew | Quality drop without errors | Feature parity job | Block promotion on parity failure |
| FM-03 | Popularity feedback loop | Long tail never shown | Catalogue coverage metric | Exploration traffic |
| FM-04 | Bad model promoted | Business metric drop | Quality and business alerts | Automatic rollback |
| [FM-nn] | [Failure] | [Impact] | [Signal] | [Mitigation] |

## 14. Rollout plan

<!-- Guidance: Move from shadow to full traffic in steps, with exit criteria at each step and an exact rollback procedure. -->

| Stage | Traffic | Duration | Exit criterion |
|---|---|---|---|
| Shadow | 0% shown, 100% scored | 3 days | Latency and parity within budget |
| Canary | 1% | 2 days | No guardrail degradation |
| Experiment | 50% | 14 days | Primary metric significant, guardrails healthy |
| General availability | 100% | n/a | Approval from product and ML lead |

Rollback: [Feature flag name, who can flip it, expected time to effect].

## 15. Open questions and references

<!-- Guidance: Track unresolved decisions with owners, and cite documents relied on. -->

| ID | Question | Owner | Due |
|---|---|---|---|
| OQ-01 | *Example:* Do guests get personalisation from session-only history? | [@name] | YYYY-MM-DD |
| [OQ-nn] | [Question] | [@name] | YYYY-MM-DD |

References: [PRD], [Model card], [Datasheets], [Experiment platform guide].

## Standard and licence note

Based on common industry practice for machine learning system design
documents. Original text written for this collection; example numbers are
illustrative. Template licence: MIT.