# System Design: [Feature store name]

> Reference design for a machine learning feature store: one registry of
> feature definitions, batch and streaming pipelines that compute features
> once, an offline store with full history for training, an online store
> with the latest values for low-latency serving, and point-in-time correct
> joins that build training sets without leaking future data. It covers the
> registry, materialisation, freshness, training and serving consistency,
> backfills and feature monitoring. This is an original design that draws on
> common practice and public write-ups (Uber's Michelangelo post of 2017 and
> the Feast documentation on point-in-time joins); no standard governs
> feature stores. It follows the structure of the System Design Document
> template, adds a 45-minute interview path, access-pattern and failure-mode
> tables and a cited "How real systems do it" box, and is pre-filled with a
> worked design to adapt. Replace every `[bracketed]` value and every row
> marked *Example*; values marked *Example* are assumptions, not benchmarks.

<!--
Guidance: How to use this template
- Start with the consumers: which models read features online, at what rate
  and latency, and which teams build training sets. A feature store with no
  online consumer is a data catalog plus a join library; say so if that is
  your case.
- Fill the worked point-in-time example in 7.2 with one of your own
  features. Reviewers understand leakage faster from a five-row table than
  from a paragraph.
- Decide where transformations run (7.3) before choosing tools; it decides
  whether training and serving can drift apart.
- Pair this design with the ML system design and model monitoring templates
  listed in 12.3.
-->

## Document control

<!-- Guidance: A design is a proposal until it is approved; record who decided and when. -->

| Field | Value |
|---|---|
| Document ID | [SDD-FS-001] |
| System / feature | [Shared feature platform for ranking and risk models] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Implemented / Superseded] |
| Owner | [@owner] |
| Author(s) | [@author] |
| Reviewers | [@reviewer-1], [@reviewer-2] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal] |
| Difficulty | Advanced |
| Main consumers | [Ranking service, fraud scoring, demand forecasting] |
| Related | [ML system design], [Model monitoring plan], [ADR: online store choice] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Tech lead | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| ML platform lead | [@name] | [Decision] | YYYY-MM-DD |
| Data platform lead | [@name] | [Decision] | YYYY-MM-DD |
| SRE / operations | [@name] | [Decision] | YYYY-MM-DD |
| Privacy and data protection | [@name] | [Decision] | YYYY-MM-DD |

## Revision history

<!-- Guidance: Note what changed in the design, not the wording. -->

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft for design review |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Added in-process cache for merchant features after fan-out estimate |
| 0.3 | YYYY-MM-DD | [@name] | *Example:* Training sets filter on created timestamp to exclude late corrections |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved |

## Contents

<!-- Guidance: Keep aligned with the headings. -->

1. [Overview](#1-overview)
2. [Requirements](#2-requirements)
3. [Capacity estimation](#3-capacity-estimation)
4. [API design](#4-api-design)
5. [Data model](#5-data-model)
6. [High-level design](#6-high-level-design)
7. [Detailed design](#7-detailed-design)
8. [Scalability and performance](#8-scalability-and-performance)
9. [Reliability, security and operations](#9-reliability-security-and-operations)
10. [Trade-offs and alternatives](#10-trade-offs-and-alternatives)
11. [Delivery plan and risks](#11-delivery-plan-and-risks)
12. [Appendices](#12-appendices)

## 1. Overview

<!-- Guidance: One page that a busy reviewer can read alone: problem, proposed solution in a paragraph, and the decisions that need their attention. -->

### 1.1 Problem statement

[*Example:* Each of nine model teams computes its own user and merchant
features: once in SQL for training and again in service code for serving.
The two versions drift; last quarter a ranking model lost a measured share
of its offline lift in production because a "last 7 days" count was computed
in UTC offline and in local time online. Training sets are built with plain
joins on the latest value, so some features leak information from after the
label time, and offline metrics look better than production.]

### 1.2 Proposed solution

[*Example:* Teams declare features once in a registry: entity, source,
transformation, owner, freshness target and time-to-live. Batch jobs and
streaming jobs compute the features and write every value with its event
time to an offline store (columnar tables) and the latest value per entity to
an online key-value store. Models fetch online features through one serving
API with a multi-get per request. Training sets are built by a point-in-time
join that, for each labelled event, picks the latest feature value at or
before the event time within the time-to-live. The serving path logs the
feature vector it returned, so training can later use exactly what the model
saw.]

### 1.3 Decisions requested from reviewers

| ID | Question | Recommended option | Section |
|---|---|---|---|
| Q-01 | *Example:* Build, adopt open source, or buy? | Adopt an open-source framework for the registry and SDK, run our own stores | 10 |
| Q-02 | *Example:* Online store technology? | In-memory key-value cluster, 4 shards with 3 replicas | 3.2, 5.2 |
| Q-03 | *Example:* Where do transformations run? | In the shared pipelines, not in model code; on-demand transforms only for request data | 7.3 |
| Q-04 | *Example:* How are training sets built? | Point-in-time join with TTL, filtered on created timestamp | 7.2 |
| [Q-nn] | [Question] | [Option] | [Section] |

### 1.4 45-minute interview path

Use this order when the document serves an interview or a short design
review. The written sections below are the long form of each step.

| Minutes | Cover | Output on the whiteboard |
|---|---|---|
| 0 to 5 | Scope: online serving, offline training, freshness, who defines features; below-the-line items (2.3) | Two consumers, three requirements |
| 5 to 10 | Estimates: online store size, lookups per second, offline history (section 3) | "Fan-out per request, not request rate, drives the online store" |
| 10 to 15 | Interfaces: feature definition, `get_online_features`, `get_historical_features` (section 4) | Three calls |
| 15 to 25 | High-level design: registry, batch and stream pipelines, offline and online stores (section 6) | One diagram |
| 25 to 40 | One deep dive: point-in-time joins and leakage (default, 7.2), or training and serving consistency (7.3), or streaming freshness (7.4) | Options, choice, reason |
| 40 to 45 | Failures (stale online values, pipeline lag), the 10x row in section 10 | Two failure rows |

Start the default deep dive with the naive answer: join each training label
to the current feature table on the entity key. Show with the table in 7.2
how that copies the future into the past, then build the as-of join.

## 2. Requirements

<!-- Guidance: Split core requirements from below-the-line items so the design stays focused. Non-functional targets need numbers and a way to measure them. -->

### 2.1 Core functional requirements

| ID | Requirement | Priority |
|---|---|---|
| FR-01 | *Example:* Declare features once with entity, source, transformation, owner, TTL and freshness target | Must |
| FR-02 | *Example:* Serve the latest values for a list of entities and features in one call | Must |
| FR-03 | *Example:* Build training sets with point-in-time correct joins | Must |
| FR-04 | *Example:* Batch features (daily) and streaming features (seconds) | Must |
| FR-05 | *Example:* Backfill a new feature over 2 years of history | Must |
| FR-06 | *Example:* Search and reuse features across teams, with lineage to sources | Should |
| FR-07 | *Example:* On-demand features computed from request data at serving time | Should |
| FR-08 | *Example:* Log served feature vectors for training and monitoring | Should |
| [FR-nn] | [Requirement] | [Must / Should / Could] |

### 2.2 Non-functional requirements

| ID | Quality | Target | Measured as |
|---|---|---|---|
| NFR-01 | Online latency | *Example:* p99 at or below 10 ms for one request of up to 51 entities | Serving API timing |
| NFR-02 | Freshness | *Example:* Streaming features p99 at or below 60 s from event; batch features by 06:00 UTC | Event time vs. write time |
| NFR-03 | Availability | *Example:* 99.95% for the online API | Successful responses |
| NFR-04 | Correctness | *Example:* Zero future values in training sets | Automated leakage test (7.2) |
| NFR-05 | Consistency | *Example:* Online and offline values for the same entity and time differ in under 0.1% of samples | Daily skew job (7.3) |
| [NFR-nn] | [Quality] | [Target] | [Measurement] |

### 2.3 Below the line (out of scope)

Name these explicitly so reviewers and interviewers do not pull the design
toward them.

- [*Example:* model training, registry of models and model serving (see the ML system design)]
- [*Example:* embedding similarity search (see the vector database design)]
- [*Example:* raw data ingestion from source systems (owned by the data platform)]
- [*Example:* label generation and annotation]
- [Item deferred, with the release that will take it]

## 3. Capacity estimation

<!-- Guidance: Back-of-the-envelope numbers decide the shape of the design. Show inputs, formulas and results; round generously and say which way you rounded. -->

### 3.1 Assumptions

| Input | Symbol | Value | Source |
|---|---|---|---|
| User entities | $U$ | *Example:* 50 million | User table |
| Merchant entities | $M$ | *Example:* 2 million | Merchant table |
| Online bytes per user (300 features x 8 B + 200 B key and overhead) | $s_u$ | *Example:* 2 600 B | Feature registry |
| Online bytes per merchant (100 features x 8 B + 200 B) | $s_m$ | *Example:* 1 000 B | Feature registry |
| Peak predictions per second | $q$ | *Example:* 20 000 | Ranking service metrics |
| Entities per prediction (1 user + 50 candidate merchants) | $k$ | *Example:* 51 | Ranking design |
| History kept offline | | *Example:* 2 years of daily user snapshots | Retention policy |
| Streaming events feeding features | | *Example:* 200 000 events/s at peak | Event bus metrics |
| Online store write rate for bulk loads | | *Example:* 50 000 rows/s | Load test |

### 3.2 Online store

$$
S_{online} = U s_u + M s_m = 5 \times 10^{7} \times 2\,600 + 2 \times 10^{6} \times 1\,000 \ \text{B} = 132 \ \text{GB}
$$

With three replicas that is 396 GB. On nodes with 64 GB of memory filled to
70%, about 9 nodes are the minimum; choose 4 shards with 3 replicas (12
nodes, about 33 GB per shard) for headroom and even resharding.

### 3.3 Online lookups

$$
K = q \times k = 20\,000 \times 51 = 1.02 \times 10^{6} \ \text{entity lookups/s}
$$

The fan-out, not the request rate, dominates. Merchant features are only
2 GB ($2 \times 10^{6} \times 1\,000$ B) and change daily, so each serving
host caches all of them in process; the online store then serves only the
user lookups, 20 000 multi-gets per second. This is the main reason the
online store stays small.

### 3.4 Offline store and pipelines

- History: $5 \times 10^{7} \times 730 = 36.5$ billion user rows over two
  years, about 94.9 TB uncompressed at 2 600 B, about 19 TB at a 5:1
  columnar compression ratio (*Example*, measure yours).
- Daily materialisation: loading 50 million user rows at 50 000 rows/s takes
  $5 \times 10^{7} / 5 \times 10^{4} = 1\,000$ s, about 17 minutes, which
  fits the 06:00 deadline with room for one retry.
- Streaming: 200 000 events/s update windowed aggregates; writes to the
  online store are coalesced per entity per second, so the write rate is
  bounded by active entities, not events.

### 3.5 Summary

| Quantity | Estimate | Design consequence |
|---|---|---|
| Online data | 132 GB (396 GB with replicas) | In-memory store, 4 shards x 3 replicas |
| Lookups | 1.02 million entities/s | Multi-get per request; cache merchant features in process |
| Remote reads | 20 000 multi-gets/s | Small online cluster |
| Offline history | About 19 TB compressed | Partition by date, cluster by entity for as-of joins |
| Daily load | About 17 min | Bulk load with versioned swap |

## 4. API design

<!-- Guidance: Define the contract first: resources, methods, request and response shapes, errors, idempotency, pagination, rate limits and versioning. Use RFC 9457 problem details for errors. -->

### 4.1 Feature definition

Features are declared as code, reviewed in pull requests and applied to the
registry by CI:

```json
{"featureView": "user_order_stats", "version": 3, "entity": "user_id", "source": {"type": "batch", "table": "warehouse.orders_daily", "timestampField": "event_ts", "createdField": "created_ts"}, "features": [{"name": "orders_7d", "type": "int32"}, {"name": "spend_30d", "type": "float64"}], "ttl": "2d", "freshness": "daily by 06:00 UTC", "owner": "team-growth", "online": true, "tags": {"pii": "none"}}
```

### 4.2 Serving and training calls

| Interface | Operation | Purpose |
|---|---|---|
| gRPC | `GetOnlineFeatures(entities[], feature_refs[]) -> rows` | Latest values for up to 100 entities, with per-value age |
| SDK | `get_historical_features(entity_df, feature_refs, filter_by_created=True)` | Point-in-time join for a training set |
| REST | `GET /v1/features?search=` | Discovery with owner, lineage, freshness and usage |
| REST | `POST /v1/backfills` | Request a backfill for a feature view and time range |
| Event | `feature_vector_served` | Logged vector per prediction (FR-08) |

```json
{"entities": [{"user_id": "u_42"}], "features": ["user_order_stats:orders_7d", "user_order_stats:spend_30d", "user_rt:clicks_10m"], "results": [{"user_id": "u_42", "values": [5, 182.4, 7], "ageSeconds": [52200, 52200, 14], "status": ["PRESENT", "PRESENT", "PRESENT"]}]}
```

Each value carries its age and a status (`PRESENT`, `EXPIRED`, `MISSING`) so
models can apply a defined default instead of silently reading zero.

## 5. Data model

<!-- Guidance: Choose storage by access pattern, not by habit. Show entities, keys, partitioning and the queries each store must answer. -->

### 5.1 Entities

The registry holds definitions; the offline store holds every value with two
timestamps; the online store holds only the latest value per entity.

```mermaid
erDiagram
  ENTITY {
    string entity_name PK
    string join_key
  }
  FEATURE_VIEW {
    string view_name PK
    int version PK
    string entity_name FK
    string source
    string ttl
    string owner
  }
  FEATURE {
    string view_name FK
    string feature_name PK
    string dtype
  }
  OFFLINE_ROW {
    string entity_key
    datetime event_ts
    datetime created_ts
    string view_name FK
  }
  ONLINE_ROW {
    string entity_key PK
    string view_name PK
    datetime event_ts
  }
  ENTITY ||--o{ FEATURE_VIEW : "keyed by"
  FEATURE_VIEW ||--|{ FEATURE : contains
  FEATURE_VIEW ||--o{ OFFLINE_ROW : "history in"
  FEATURE_VIEW ||--o{ ONLINE_ROW : "latest in"
```

### 5.2 Storage choices

| Data | Store | Key | Why |
|---|---|---|---|
| Registry | *Example:* Relational database behind the registry service | `view_name`, `version` | Small, versioned, audited |
| Offline values | *Example:* Columnar tables in the lakehouse | Partitioned by date of `event_ts`, clustered by entity key | Scans and as-of joins over years |
| Online values | *Example:* In-memory key-value cluster | `{entity_key}:{view_name}` hash with one field per feature | One multi-get per request, sub-millisecond |
| Served vectors | Event log to the offline store | Prediction ID, time | Training on served values, skew checks |

### 5.3 Access patterns

| Access pattern | Frequency | Key or index that serves it | Partition key rationale |
|---|---|---|---|
| Latest features of one user for a prediction | 20 000/s | `{user_id}:{view}` hash, multi-get across views | Entity key spreads load evenly; views of one entity fetched together |
| Latest features of 50 merchants | 1 million entity reads/s, served from process cache | Local map refreshed after each daily load | Small and slow-changing, so no remote call |
| As-of join for a training set | Batch, hundreds of millions of label rows | Offline tables partitioned by date, clustered by entity | Join scans only the dates within label time minus TTL |
| Backfill a new feature | Occasional | Source tables by date | Runs as normal batch over historical partitions |
| Discover features | Interactive | Registry search index on name, tags, owner | Small catalog |
| Compare served and offline values | Daily | Served-vector log joined to offline rows by entity and time | Same as-of join logic |

## 6. High-level design

<!-- Guidance: One diagram that shows every component on the request paths and where state lives. Name components by responsibility. -->

```mermaid
flowchart LR
  dev[Feature definitions in Git] --> ci[CI apply]
  ci --> reg[(Registry)]
  src[(Warehouse tables)] --> batch[Batch pipelines]
  bus[[Event bus]] --> stream[Streaming pipelines]
  reg --> batch
  reg --> stream
  batch --> off[(Offline store)]
  stream --> off
  batch -->|"materialise latest"| on[(Online store)]
  stream -->|"latest within seconds"| on
  svc[Model serving] -->|GetOnlineFeatures| api[Feature serving API]
  api --> on
  api -->|"served vectors"| bus
  train[Training jobs] -->|"get_historical_features"| join[Point-in-time join engine]
  join --> off
  mon[Feature monitoring] --> off
  mon --> on
```

| Component | Responsibility | Scales by |
|---|---|---|
| Registry | Definitions, versions, owners, lineage, access rules | Small |
| Batch pipelines | Compute daily and hourly features from warehouse tables | Warehouse compute |
| Streaming pipelines | Windowed aggregates from events, written within seconds | Event partitions |
| Offline store | All values with event and created timestamps | Storage and scan compute |
| Online store | Latest value per entity and view | Memory, shards |
| Feature serving API | Multi-get, defaults, ages, logging, access control | Stateless instances |
| Point-in-time join engine | Training sets from label rows and feature history | Batch compute |
| Feature monitoring | Freshness, null rates, distribution drift, skew | Batch jobs and metrics |

## 7. Detailed design

<!-- Guidance: Go deep on the parts that are new, risky or contested. Show the sequence of calls and the failure behaviour of each step. -->

The deep dives in this section, each with options, choice and reason:

| Deep dive | Options considered | Choice | Reason | Section |
|---|---|---|---|---|
| Training set correctness | Join on latest value, daily snapshots, point-in-time join | Point-in-time join with TTL and created-time filter | No leakage, no snapshot explosion | 7.2 |
| Training and serving consistency | Separate code paths, shared definitions, logging served values | Shared definitions plus served-value logging | One computation, measurable skew | 7.3 |
| Streaming freshness | Micro-batch, streaming aggregates, compute at request time | Streaming aggregates with sliding windows | Seconds of freshness without request-time load | 7.4 |
| Online loading | In-place upserts, versioned table swap | Versioned swap for batch, upserts for streams | Consistent daily snapshots, no half-loaded state | 7.5 |

### 7.1 Feature lifecycle

```mermaid
stateDiagram-v2
  [*] --> Draft
  Draft --> Backfilling: definition merged
  Backfilling --> Validating: history written
  Validating --> Published: checks pass
  Validating --> Draft: checks fail
  Published --> Deprecated: owner retires it
  Deprecated --> Deleted: no reads for 90 days
  Deleted --> [*]
```

A feature becomes `Published` only after backfill and validation (null rate,
value ranges, offline and online agreement on a sample). `Deprecated`
features keep serving but warn every consumer; deletion waits until reads
stop.

### 7.2 Point-in-time joins

For a label event of entity $e$ at time $t$, the joined value of a feature is
the one with the latest event time $t_i$ at or before $t$, provided it is not
older than the TTL $\tau$:

$$
v(e, t) = v_{j} \quad \text{where} \quad t_{j} = \max \{ t_i \le t \ : \ t - t_i \le \tau \}
$$

If no value qualifies the result is null (status `EXPIRED` or `MISSING`), and
the model sees the same default it sees online.

Worked example for feature `orders_7d` of user `u_42`, TTL 2 days
(*Example*):

| Feature row | Event time | Created time | Value |
|---|---|---|---|
| FV-1 | 2026-03-01 00:00 | 2026-03-01 02:10 | 3 |
| FV-2 | 2026-03-02 00:00 | 2026-03-02 02:05 | 5 |
| FV-3 | 2026-03-03 00:00 | 2026-03-03 02:20 | 6 |
| FV-4 (late correction of FV-2) | 2026-03-02 00:00 | 2026-03-04 09:00 | 4 |

| Label event | Naive join (latest value) | Point-in-time join | With created-time filter |
|---|---|---|---|
| LE-1 at 2026-03-02 14:30 | 6 (leaks FV-3, from the future) | 4 (FV-4 replaces FV-2 for the same event time) | 5 (FV-4 was created after the label time) |
| LE-2 at 2026-03-06 10:00 | 6 | Null: FV-3 is 3 days 10 hours old, beyond the 2-day TTL | Null |

The created-time filter reproduces what the online store held at the time of
the label, which is what the model will see in production. Without it,
corrections written later improve offline metrics but not live behaviour.
The Feast documentation describes the same backward scan bounded by TTL and
an optional created-timestamp filter.

Implementation: sort label rows and feature rows by entity and time and run
an as-of merge per entity partition; read only feature partitions from
$t_{min} - \tau$ to $t_{max}$. Add an automated leakage test to CI: a
synthetic feature equal to its own event time must never exceed the label
time in any built training set (NFR-04).

### 7.3 Training and serving consistency

Skew has three usual causes: different code (SQL offline, service code
online), different data (late events, time zones, defaults) and different
timing (online value older than assumed). Controls:

- Transformations live in the feature definition and run in the shared
  pipelines; models read finished features, never raw tables.
- On-demand features (computed from request data such as cart size) use one
  function packaged for both the serving API and the join engine.
- The serving API logs the vector it returned; the daily skew job compares
  logged values with the point-in-time join for the same entities and times
  and alerts above 0.1% mismatch (NFR-05).
- Defaults for missing values are defined once in the registry.

### 7.4 Streaming features and freshness

Streaming pipelines maintain sliding-window aggregates (for example clicks
in the last 10 minutes) keyed by entity, with event-time windows and a
watermark that accepts events up to 30 s late (*Example*). Each update is
written to the online store with its event time and appended to the offline
store, so training sees the same values. Writes are coalesced per entity per
second. Freshness is measured as now minus the latest event time per view and
exported as a metric (NFR-02).

### 7.5 Online materialisation

- **Batch views**: the daily job writes into a new versioned key space, checks
  row counts and sample values, then switches the serving API to the new
  version in one step. Rollback switches back. Merchant caches reload on the
  switch event.
- **Streaming views**: upserts with a monotonic check on event time, so an
  out-of-order write never replaces a newer value.
- **TTL**: online values carry an expiry equal to the view TTL; expired values
  return `EXPIRED`, never a stale number.

### 7.6 How real systems do it

> **How real systems do it** (paraphrased, not copied). Uber's post "Meet
> Michelangelo: Uber's Machine Learning Platform" (Mike Del Balso and Jeremy
> Hermann, 5 September 2017) describes a shared feature store where teams
> publish curated features with an owner, description and SLA, and models
> refer to features by name. Offline features are computed by scheduled Spark
> and Hive jobs over the HDFS data lake and used for training and batch
> prediction. For online prediction, features are precomputed and kept in
> Cassandra: batch features are bulk loaded every few hours or daily, and
> near-real-time features are computed by streaming jobs from Kafka, written
> to Cassandra and logged back to HDFS so training uses the same data. The
> post reports about 10 000 shared features at the time. The open-source
> Feast documentation describes point-in-time joins as a backward scan from
> each entity row's timestamp bounded by a TTL. Read the originals in 12.3;
> this box keeps only the general idea.

## 8. Scalability and performance

<!-- Guidance: Show how the design meets the numbers in section 3 and where it will break first. -->

| Concern | Approach | Limit before redesign |
|---|---|---|
| Online fan-out | Multi-get per request; process cache for small entity sets | *Example:* 100 entities per call |
| Online memory | Shards added by resharding; compact encoding (no feature names in values) | *Example:* About 64 GB per node |
| As-of joins | Partition pruning by date and TTL, clustering by entity | Label sets of several billion rows per job |
| Streaming state | Keyed state per entity, partitioned by entity | Event partitions and state size |
| Registry growth | Search index, ownership, deprecation | Thousands of feature views |

Online latency budget (target p99 10 ms):

| Step | Budget |
|---|---|
| Request parsing and authorisation | 0.5 ms |
| Merchant features from process cache | 0.5 ms |
| User multi-get to online store | 3 ms |
| On-demand transforms | 2 ms |
| Response and logging (asynchronous) | 1 ms |
| Headroom | 3 ms |

## 9. Reliability, security and operations

<!-- Guidance: Name the failure modes with their blast radius and recovery time, the threats, and the signals operators watch. Each mitigation should be testable. -->

### 9.1 Failure modes

| Component | Failure | Detection | Blast radius | Mitigation | RTO |
|---|---|---|---|---|---|
| Daily batch job | Late or failed | Freshness metric past deadline | Every model using the view | Keep serving the previous version within TTL; page owner; models see ages | *Example:* 2 h |
| Streaming pipeline | Lag or stop | Consumer lag, freshness above 60 s | Real-time views | Values expire by TTL and return `EXPIRED`; models fall back to batch features | *Example:* 15 min |
| Online shard | Node loss | Health checks | One quarter of users on one replica set | Replica promotion | *Example:* 30 s |
| Online store | Full outage | Error rate | All online models | Serving API returns defaults with `MISSING`; models degrade to a fallback model | *Example:* 30 min |
| Bad feature values | Bug in a definition | Distribution drift and null-rate monitors | Models using the feature | Roll back the view version; block the definition | *Example:* 1 h |
| Leakage | Join bug | CI leakage test | Every training set built since | Rebuild affected training sets; flag models trained on them | *Example:* 1 day |
| [Component] | [Failure] | [Signal] | [Scope] | [Mitigation] | [Time] |

### 9.2 Security and privacy

| Threat | Control |
|---|---|
| Personal data reused for a purpose it was not collected for | Registry tags for personal data and purpose; access approval per consumer |
| Over-broad online access | Per-service allow lists of feature views in the serving API |
| Deletion requests not reaching features | Entity deletion propagated to offline and online stores and to served-vector logs |
| Sensitive attributes used as proxies | Fairness review of feature lists for regulated uses |
| [Threat] | [Control] |

### 9.3 Observability

| Signal | Metric or log | Alert threshold |
|---|---|---|
| Freshness | Age of latest value per view | Above its target |
| Online latency | p99 of `GetOnlineFeatures` | Above 10 ms for 10 min |
| Missing and expired values | Share by view | Doubling vs. 7-day baseline |
| Drift | Distribution distance per feature, daily | Above the threshold set per feature |
| Skew | Served vs. offline mismatch rate | Above 0.1% |

### 9.4 Cost estimate

| Item | Unit cost | Quantity | Monthly cost |
|---|---|---|---|
| Online store nodes | [$/node-month] | *Example:* 12 | [$] |
| Offline storage | [$/TB-month] | *Example:* About 19 TB plus served-vector logs | [$] |
| Batch and streaming compute | [$/hour] | [Hours] | [$] |
| **Total** | | | **[$]** |

## 10. Trade-offs and alternatives

<!-- Guidance: For each major decision list the options, what each costs and why you chose one. Reviewers will propose the alternatives anyway; answer them here. -->

| Decision | Options | Chosen | Why | What we give up |
|---|---|---|---|---|
| Platform | Build; open-source framework; managed service | [Choice] | [Reason] | [Cost] |
| Training data | Point-in-time join; daily snapshots of all features; served-value logs only | Point-in-time join plus served logs | Correct history for new features and exact served values for retraining | Join compute cost |
| Transformations | In pipelines; in model code; in the store at read time | In pipelines, on-demand only for request data | One definition, measurable skew | Less freedom per team |
| Online store | Disk-based wide-column store; in-memory key-value | In-memory key-value | 132 GB fits memory; lowest latency | Higher cost per GB |
| Merchant features | Online store; process cache | Process cache | Removes 1 million lookups/s | Cache reload logic |
| What changes at 10x / 0.1x | 10x (200 000 predictions/s, 10.2 million entity lookups/s): merchant cache still removes most lookups, but user multi-gets reach 200 000/s, so add shards and replicas; at 10x users the online store passes 1 TB and a disk-backed store becomes cheaper. 0.1x (2 000 predictions/s): one small replicated node, or serve features from the warehouse with a cache | Review at each doubling of predictions or entities | Fan-out and entity count grow first | [Note] |
| [Decision] | [Options] | [Chosen] | [Why] | [Cost] |

## 11. Delivery plan and risks

<!-- Guidance: How the design reaches production safely, and what could stop it. -->

### 11.1 Milestones

| Milestone | Scope | Exit criterion | Target date |
|---|---|---|---|
| M1 | Registry, batch views, offline store, point-in-time join | First team builds a training set with zero leakage in CI | YYYY-MM-DD |
| M2 | Online store and serving API for batch views | Ranking model served at p99 under 10 ms | YYYY-MM-DD |
| M3 | Streaming views | Freshness at or below 60 s p99 | YYYY-MM-DD |
| M4 | Monitoring and skew job | Skew below 0.1% for 30 days | YYYY-MM-DD |

### 11.2 Rollout and rollback

[*Example:* Each model migrates behind a flag. For 14 days the model reads
features from both the old code path and the feature store, logs both and
uses the old one; the switch happens when values agree for 99.9% of
requests. Rollback flips the flag.]

### 11.3 Risks

| ID | Risk | Likelihood | Impact | Mitigation | Owner |
|---|---|---|---|---|---|
| R-01 | *Example:* Teams keep private features outside the store | High | Medium | Make discovery and backfill easier than private pipelines | [@name] |
| R-02 | *Example:* Hidden leakage in legacy training sets | Medium | High | Rebuild key training sets with point-in-time joins; compare metrics | [@name] |
| R-03 | *Example:* Online cost grows with unused features | Medium | Medium | Usage tracking; deprecate unread views | [@name] |
| [R-nn] | [Risk] | [L / M / H] | [L / M / H] | [Mitigation] | [@name] |

## 12. Appendices

<!-- Guidance: Supporting detail, glossary and open questions. -->

### 12.1 Open questions

| ID | Question | Owner | Due |
|---|---|---|---|
| OQ-01 | *Example:* Do embeddings belong in this store or in the vector database? | [@name] | YYYY-MM-DD |
| OQ-02 | *Example:* How long are served-vector logs kept, given data protection limits? | [@name] | YYYY-MM-DD |
| [OQ-nn] | [Question] | [@name] | YYYY-MM-DD |

### 12.2 Glossary

| Term | Meaning |
|---|---|
| Entity | The thing a feature describes, identified by a join key (user, merchant) |
| Feature view | A group of features from one source with one entity, TTL and schedule |
| Event time | When the fact the value describes happened or became true |
| Created time | When the value was written to the store |
| Point-in-time join | As-of join that uses only values known at each label time |
| TTL | Maximum age of a value that may be joined or served |
| Materialisation | Copying the latest values into the online store |
| Training and serving skew | A difference between values used in training and values served in production |
| [Term] | [Meaning] |

### 12.3 References

| Reference | Use |
|---|---|
| Del Balso and Hermann, ["Meet Michelangelo: Uber's Machine Learning Platform"](https://www.uber.com/en-US/blog/michelangelo-machine-learning-platform/), Uber blog, 5 September 2017 | 7.6 |
| Feast documentation, ["Point-in-time joins"](https://docs.feast.dev/getting-started/concepts/point-in-time-joins) | 7.2 semantics, TTL and created-timestamp filter |
| ML System Design (`ai-data/ml-system-design`) | The model side of the platform |
| Model Monitoring Plan (`ai-data/model-monitoring-plan`) | Drift and skew alerts in 9.3 |
| ML Reproducibility Checklist (`ai-data/ml-reproducibility-checklist`) | Versioned training sets |
| Experiment Tracking Report (`ai-data/experiment-tracking-report`) | Recording which feature versions an experiment used |
| Recommendation System Design (`system-design/design-recommendation-system`) | Main online consumer |
| Fraud Detection System Design (`system-design/design-fraud-detection`) | Streaming features consumer |
| Stream Processing Engine Design (`system-design/design-stream-processing-engine`) | Windows and watermarks for 7.4 |
| Data Lineage Specification (`data/data-lineage-specification`) | Lineage from sources to features |

## Standard and licence note

Based on common industry practice for machine learning feature platforms; no
formal standard governs feature stores. The Uber engineering post and the
Feast documentation cited in 12.3 are paraphrased in original words; no
text, figures or code were copied, and all numbers marked *Example* are
illustrative. Product capabilities change; check the current documentation
of any framework you adopt. Original text written for this collection.
Template licence: MIT.