ML System Design Document Template

Problem framing, metrics, data, features, baselines, training, offline and A/B evaluation, serving, monitoring, rollout

AI 및 데이터전체system design math업데이트됨 2026. 10. 20.

사용법: Use before building a production machine learning feature. Fill problem framing, success metrics and data sources first, and always compare against a simple baseline. The main pitfall is optimising a model metric that does not move the business metric.

미리보기

ML System Design: [System or feature name]

Complete design document for a production machine learning system: problem framing, success metrics, data and labelling, features, model choice, training, offline and online evaluation, serving, budgets, monitoring, retraining, fairness and safety, failure modes and rollout. Replace every [bracketed] value and every row marked Example. Worked examples use a product recommendation feature for an online shop.

Document control

Field	Value
Document ID	[MLSD-SYSTEM-001]
System / feature	[Name]
Version	[0.1]
Status	[Draft / In review / Approved / Implemented / Superseded]
Author(s)	[@author]
Reviewers	[@ml-reviewer], [@platform-reviewer], [@product-owner]
Classification	[Internal / Confidential]
Last updated	YYYY-MM-DD
Related	[PRD], [Model card], [Datasheet], [AI risk assessment], [Data pipeline spec]

Approvals

Role	Name	Decision	Date
ML lead	[@name]	[Approved / Changes requested]	YYYY-MM-DD
Product owner	[@name]	[Decision]	YYYY-MM-DD
Platform / SRE	[@name]	[Decision]	YYYY-MM-DD
Responsible AI / privacy	[@name]	[Decision]	YYYY-MM-DD

Revision history

Version	Date	Author	Change summary
0.1	YYYY-MM-DD	[@name]	First draft
0.2	YYYY-MM-DD	[@name]	Example: Replaced real-time features with hourly batch after latency review
1.0	YYYY-MM-DD	[@name]	Example: Approved for build

Problem framing and success metrics
Data sources and labelling
Feature pipeline
Model selection and baselines
Training pipeline
Offline evaluation
Online evaluation
Serving architecture
Latency and cost budgets
Monitoring
Retraining
Fairness, safety and privacy
Failure modes
Rollout plan
Open questions and references

1. Problem framing and success metrics

1.1 Problem statement

[Who has the problem and what it costs today. Example: "Shoppers on product pages see 'related items' chosen by a static category rule; fewer than 2% click them, and catalogue items outside the top 500 are almost never discovered."]

1.2 Why machine learning

[Why rules or heuristics are not enough, and what non-ML alternative was considered. Example: "A co-purchase rule covers only 30% of products; personalised ranking is needed for the long tail."]

1.3 ML task formulation

Aspect	Decision
Task type	Example: Ranking: order 200 candidate products for a (user, page) pair
Prediction target	Example: Probability the user adds the item to cart within the session
Unit of prediction	Example: (session, anchor product, candidate product)
Prediction time	Example: Online, at page render
Action taken on output	Example: Show top 8 candidates in the carousel

1.4 Success metrics

Level	Metric	Current	Target	Guardrail
Business	Example: Add-to-cart rate from carousel	1.8%	3.0%	Overall conversion must not drop
Business	Example: Revenue per session	[value]	+1.5%	Return rate must not rise more than 0.2 pt
Model (offline)	Example: NDCG@8	0.21 (rule)	at least 0.30	Calibration error at most 0.05
Model (offline)	Example: Recall@50 of candidate generation	n/a	at least 0.85
System	Example: p99 inference latency	n/a	at most 60 ms	Page render p99 unchanged

For classification-style components, use the usual definitions:

\text{Precision} = \frac{TP}{TP + FP}, \qquad \text{Recall} = \frac{TP}{TP + FN}, \qquad F_1 = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}

For ranking, normalised discounted cumulative gain at cut-off $k$ :

\text{DCG@}k = \sum_{i=1}^{k} \frac{2^{rel_i} - 1}{\log_2(i + 1)}, \qquad \text{NDCG@}k = \frac{\text{DCG@}k}{\text{IDCG@}k}

2. Data sources and labelling

2.1 Data sources

ID	Source	Content	Volume	Freshness	Owner	Contains personal data
SRC-01	Example: Clickstream events	Views, clicks, add-to-cart	40 M events/day	Streaming, under 1 min	Web platform	Yes, pseudonymous user ID
SRC-02	Example: Product catalogue	Title, category, price, stock	1.2 M products	Hourly	Catalogue team	No
SRC-03	Example: Orders	Purchases and returns	300 k/day	Daily	Commerce team	Yes
[SRC-nn]	[Source]	[Content]	[Volume]	[Freshness]	[Owner]	[Yes / No]

2.2 Labels

[How labels are defined and produced, label delay, noise and class balance. Example: "Positive = add-to-cart within the same session after an impression; negatives sampled from impressions without action at 1:20. Labels available within 1 hour. Position bias corrected with inverse propensity weights from a 1% randomised slot."]

2.3 Leakage and bias risks

Risk	Example	Prevention
Target leakage	Example: Feature uses cart contents after the impression	Point-in-time feature joins
Feedback loop	Example: Model only learns from items it already shows	5% exploration traffic
Selection bias	Example: Logged-in users over-represented	Weighting; separate evaluation for guests
[Risk]	[Example]	[Prevention]

3. Feature pipeline

Feature group	Examples	Computed	Store	Freshness	Owner
User history	Example: Last 50 viewed categories, 30-day spend band	Streaming aggregates	Online feature store	5 min	Recs team
Product	Example: Price band, category embedding, stock flag	Batch	Online and offline store	Hourly	Recs team
Context	Example: Device, time of day, page type	At request	None	Real time	Web platform
Cross	Example: Co-view count of anchor and candidate	Daily batch	Offline and online store	Daily	Recs team
[Group]	[Examples]	[How]	[Store]	[Freshness]	[Owner]

Skew controls: [Example: one feature definition used for both offline and online computation; daily job compares feature distributions between training snapshots and logged serving features].

4. Model selection and baselines

Option	Description	NDCG@8 (validation)	p99 latency	Complexity	Decision
Baseline A	Example: Static category rule (current)	0.21	2 ms	Low	Fallback
Baseline B	Example: Popularity within category, last 7 days	0.24	2 ms	Low	Comparison
Candidate 1	Example: Gradient-boosted ranker on features in 3	0.31	18 ms	Medium	Chosen
Candidate 2	Example: Two-tower neural retrieval plus transformer re-ranker	0.33	75 ms	High	Rejected: latency, ops cost
[Option]	[Description]	[value]	[ms]	[L / M / H]	[Decision]

Rationale: [Why the chosen model, and what would make you revisit.]

5. Training pipeline

Step	Detail
Data snapshot	Example: 28-day window, point-in-time join, snapshot ID recorded
Splits	Example: Time-based: train days 1-24, validation 25-26, test 27-28
Hyperparameter search	Example: 40 trials Bayesian search on validation NDCG@8
Compute	Example: 1 CPU node, 64 vCPU, about 40 minutes per run
Artefacts	Model file, feature list, metrics, data snapshot ID, code commit
Registry	[Model registry and naming convention]
Reproducibility	Example: Same snapshot and seed reproduce NDCG within 0.002

6. Offline evaluation

Check	Method	Acceptance criterion
Overall quality	NDCG@8 on test days	at least 0.30 and at least 0.05 above Baseline B
Slices	Example: New users, guests, mobile, long-tail products	No slice more than 10% below its baseline
Calibration	Expected calibration error	at most 0.05
Robustness	Example: Products with missing price or image	No errors; graceful score
Uncertainty	Bootstrap 95% CI over sessions	Lower bound above target
[Check]	[Method]	[Criterion]

7. Online evaluation

Item	Plan
Design	Example: A/B test, 50/50 split, randomised by user ID
Primary metric	Example: Add-to-cart rate from carousel
Guardrail metrics	Example: Overall conversion, page latency p99, return rate, error rate
Minimum detectable effect	Example: +0.3 pt absolute on 1.8% baseline
Duration	Example: At least 14 days to cover two weekly cycles
Stop rule	Example: Stop early if any guardrail degrades significantly at the 99% level

Approximate sample size per arm for a two-proportion test with baseline rate $p$ , absolute effect $\delta$ , significance $\alpha$ and power $1 - \beta$ :

n \approx \frac{2\,(z_{1-\alpha/2} + z_{1-\beta})^{2}\; p(1-p)}{\delta^{2}} = \frac{2\,(1.96 + 0.84)^{2} \times 0.018 \times 0.982}{0.003^{2}} \approx 30\,800

8. Serving architecture

The request path retrieves candidates, fetches features, scores with the model and falls back to the rule-based list on any timeout.

Component	Responsibility	Scaling	Timeout
Recommendation API	Orchestrate, enforce budget, fall back	Stateless replicas	80 ms overall
Candidate generator	Return 200 candidates	Cached per anchor product	20 ms
Online feature store	Serve user and product features	Managed, partitioned	10 ms
Ranker service	Score candidates	CPU replicas, autoscaled on QPS	25 ms
[Component]	[Responsibility]	[Scaling]	[ms]

9. Latency and cost budgets

Step	p99 budget
Candidate generation	20 ms
Feature fetch	10 ms
Scoring 200 candidates	25 ms
Network and serialisation	5 ms
Total	60 ms (headroom 20 ms to the 80 ms timeout)

Serving cost per 1 000 requests, with instance cost $c$ per hour and sustained throughput $q$ requests per second per instance:

C_{1000} = \frac{1000 \cdot c}{3600 \cdot q} = \frac{1000 \times 0.40}{3600 \times 150} \approx 0.00074 \ \text{currency units}

Item	Estimate per month
Serving compute	[value]
Feature store	[value]
Training compute	[value]
Logging and storage	[value]
Total	[value]

10. Monitoring

Layer	Signal	Threshold	Action	Owner
System	p99 latency, error rate, fallback rate	p99 above 60 ms, fallback above 2% for 10 min	Page on-call	[@platform]
Data quality	Null rate, out-of-range values per feature	Any feature null rate above 5%	Alert, block retraining	[@data]
Data drift	Population stability index per top-20 feature	PSI above 0.2	Investigate within 1 day	[@ml]
Prediction drift	Score distribution shift	KS statistic above 0.1	Investigate	[@ml]
Model quality	Daily NDCG@8 on logged outcomes	Drop above 10% vs 28-day mean	Consider rollback	[@ml]
Business	Carousel add-to-cart rate	Drop above 15% day over day	Review with product	[@product]

Population stability index for one feature with bins $i$ , expected share $e_i$ and actual share $a_i$ :

PSI = \sum_{i} (a_i - e_i) \ln \frac{a_i}{e_i}

11. Retraining

Aspect	Policy
Schedule	Example: Weekly, Sunday 02:00 UTC
Triggers	Example: PSI above 0.2 on 3 or more top features; quality drop above 10%
Validation gate	Example: Offline criteria in 6 plus shadow scoring for 24 h
Promotion	Example: Automatic if gate passes and change in NDCG within +/-5%; manual approval otherwise
Rollback	Example: Previous registry version re-pinned in under 5 minutes

12. Fairness, safety and privacy

Concern	Measure	Mitigation
Seller exposure fairness	Example: Share of impressions for small sellers vs catalogue share	Exposure floor in re-ranking
Sensitive inferences	Example: Recommendations revealing health or pregnancy purchases	Sensitive categories excluded from history features
Personal data	Example: Use of browsing history	Consent check; opt-out disables personalisation
Harmful content	Example: Age-restricted products to minors	Policy filter after ranking
[Concern]	[Measure]	[Mitigation]

13. Failure modes

ID	Failure	Impact	Detection	Mitigation
FM-01	Feature store unavailable	No personalisation	Fallback rate alert	Serve static rule list
FM-02	Silent training-serving skew	Quality drop without errors	Feature parity job	Block promotion on parity failure
FM-03	Popularity feedback loop	Long tail never shown	Catalogue coverage metric	Exploration traffic
FM-04	Bad model promoted	Business metric drop	Quality and business alerts	Automatic rollback
[FM-nn]	[Failure]	[Impact]	[Signal]	[Mitigation]

14. Rollout plan

Stage	Traffic	Duration	Exit criterion
Shadow	0% shown, 100% scored	3 days	Latency and parity within budget
Canary	1%	2 days	No guardrail degradation
Experiment	50%	14 days	Primary metric significant, guardrails healthy
General availability	100%	n/a	Approval from product and ML lead

Rollback: [Feature flag name, who can flip it, expected time to effect].

15. Open questions and references

ID	Question	Owner	Due
OQ-01	Example: Do guests get personalisation from session-only history?	[@name]	YYYY-MM-DD
[OQ-nn]	[Question]	[@name]	YYYY-MM-DD

References: [PRD], [Model card], [Datasheets], [Experiment platform guide].

Standard and licence note

Based on common industry practice for machine learning system design documents. Original text written for this collection; example numbers are illustrative. Template licence: MIT.