Feature Store System Design Template (Online, Offline, Point-in-Time)

System Design: [Feature store name]

Reference design for a machine learning feature store: one registry of feature definitions, batch and streaming pipelines that compute features once, an offline store with full history for training, an online store with the latest values for low-latency serving, and point-in-time correct joins that build training sets without leaking future data. It covers the registry, materialisation, freshness, training and serving consistency, backfills and feature monitoring. This is an original design that draws on common practice and public write-ups (Uber's Michelangelo post of 2017 and the Feast documentation on point-in-time joins); no standard governs feature stores. It follows the structure of the System Design Document template, adds a 45-minute interview path, access-pattern and failure-mode tables and a cited "How real systems do it" box, and is pre-filled with a worked design to adapt. Replace every [bracketed] value and every row marked Example; values marked Example are assumptions, not benchmarks.

Document control

Approvals

Revision history

Contents

1. Overview
2. Requirements
3. Capacity estimation
4. API design
5. Data model
6. High-level design
7. Detailed design
8. Scalability and performance
9. Reliability, security and operations
10. Trade-offs and alternatives
11. Delivery plan and risks
12. Appendices

1. Overview

1.1 Problem statement

[Example: Each of nine model teams computes its own user and merchant features: once in SQL for training and again in service code for serving. The two versions drift; last quarter a ranking model lost a measured share of its offline lift in production because a "last 7 days" count was computed in UTC offline and in local time online. Training sets are built with plain joins on the latest value, so some features leak information from after the label time, and offline metrics look better than production.]

1.2 Proposed solution

[Example: Teams declare features once in a registry: entity, source, transformation, owner, freshness target and time-to-live. Batch jobs and streaming jobs compute the features and write every value with its event time to an offline store (columnar tables) and the latest value per entity to an online key-value store. Models fetch online features through one serving API with a multi-get per request. Training sets are built by a point-in-time join that, for each labelled event, picks the latest feature value at or before the event time within the time-to-live. The serving path logs the feature vector it returned, so training can later use exactly what the model saw.]

1.3 Decisions requested from reviewers

1.4 45-minute interview path

Use this order when the document serves an interview or a short design review. The written sections below are the long form of each step.

Start the default deep dive with the naive answer: join each training label to the current feature table on the entity key. Show with the table in 7.2 how that copies the future into the past, then build the as-of join.

2. Requirements

2.1 Core functional requirements

2.2 Non-functional requirements

2.3 Below the line (out of scope)

Name these explicitly so reviewers and interviewers do not pull the design toward them.

• [Example: model training, registry of models and model serving (see the ML system design)]
• [Example: embedding similarity search (see the vector database design)]
• [Example: raw data ingestion from source systems (owned by the data platform)]
• [Example: label generation and annotation]
• [Item deferred, with the release that will take it]

3. Capacity estimation

3.1 Assumptions

3.2 Online store

With three replicas that is 396 GB. On nodes with 64 GB of memory filled to 70%, about 9 nodes are the minimum; choose 4 shards with 3 replicas (12 nodes, about 33 GB per shard) for headroom and even resharding.

3.3 Online lookups

The fan-out, not the request rate, dominates. Merchant features are only 2 GB ($2 \times 10^{6} \times 1\,000$ B) and change daily, so each serving host caches all of them in process; the online store then serves only the user lookups, 20 000 multi-gets per second. This is the main reason the online store stays small.

3.4 Offline store and pipelines

• History: $5 \times 10^{7} \times 730 = 36.5$ billion user rows over two years, about 94.9 TB uncompressed at 2 600 B, about 19 TB at a 5:1 columnar compression ratio (Example, measure yours).
• Daily materialisation: loading 50 million user rows at 50 000 rows/s takes $5 \times 10^{7} / 5 \times 10^{4} = 1\,000$ s, about 17 minutes, which fits the 06:00 deadline with room for one retry.
• Streaming: 200 000 events/s update windowed aggregates; writes to the online store are coalesced per entity per second, so the write rate is bounded by active entities, not events.

3.5 Summary

4. API design

4.1 Feature definition

Features are declared as code, reviewed in pull requests and applied to the registry by CI:

{"featureView": "user_order_stats", "version": 3, "entity": "user_id", "source": {"type": "batch", "table": "warehouse.orders_daily", "timestampField": "event_ts", "createdField": "created_ts"}, "features": [{"name": "orders_7d", "type": "int32"}, {"name": "spend_30d", "type": "float64"}], "ttl": "2d", "freshness": "daily by 06:00 UTC", "owner": "team-growth", "online": true, "tags": {"pii": "none"}}

4.2 Serving and training calls

{"entities": [{"user_id": "u_42"}], "features": ["user_order_stats:orders_7d", "user_order_stats:spend_30d", "user_rt:clicks_10m"], "results": [{"user_id": "u_42", "values": [5, 182.4, 7], "ageSeconds": [52200, 52200, 14], "status": ["PRESENT", "PRESENT", "PRESENT"]}]}

Each value carries its age and a status (PRESENT, EXPIRED, MISSING) so models can apply a defined default instead of silently reading zero.

5. Data model

5.1 Entities

The registry holds definitions; the offline store holds every value with two timestamps; the online store holds only the latest value per entity.

5.2 Storage choices

5.3 Access patterns

6. High-level design

7. Detailed design

The deep dives in this section, each with options, choice and reason:

7.1 Feature lifecycle

A feature becomes Published only after backfill and validation (null rate, value ranges, offline and online agreement on a sample). Deprecated features keep serving but warn every consumer; deletion waits until reads stop.

7.2 Point-in-time joins

For a label event of entity $e$ at time $t$, the joined value of a feature is the one with the latest event time $t_i$ at or before $t$, provided it is not older than the TTL $\tau$:

If no value qualifies the result is null (status EXPIRED or MISSING), and the model sees the same default it sees online.

Worked example for feature orders_7d of user u_42, TTL 2 days (Example):

The created-time filter reproduces what the online store held at the time of the label, which is what the model will see in production. Without it, corrections written later improve offline metrics but not live behaviour. The Feast documentation describes the same backward scan bounded by TTL and an optional created-timestamp filter.

Implementation: sort label rows and feature rows by entity and time and run an as-of merge per entity partition; read only feature partitions from $t_{min} - \tau$ to $t_{max}$. Add an automated leakage test to CI: a synthetic feature equal to its own event time must never exceed the label time in any built training set (NFR-04).

7.3 Training and serving consistency

Skew has three usual causes: different code (SQL offline, service code online), different data (late events, time zones, defaults) and different timing (online value older than assumed). Controls:

• Transformations live in the feature definition and run in the shared pipelines; models read finished features, never raw tables.
• On-demand features (computed from request data such as cart size) use one function packaged for both the serving API and the join engine.
• The serving API logs the vector it returned; the daily skew job compares logged values with the point-in-time join for the same entities and times and alerts above 0.1% mismatch (NFR-05).
• Defaults for missing values are defined once in the registry.

7.4 Streaming features and freshness

Streaming pipelines maintain sliding-window aggregates (for example clicks in the last 10 minutes) keyed by entity, with event-time windows and a watermark that accepts events up to 30 s late (Example). Each update is written to the online store with its event time and appended to the offline store, so training sees the same values. Writes are coalesced per entity per second. Freshness is measured as now minus the latest event time per view and exported as a metric (NFR-02).

7.5 Online materialisation

• Batch views: the daily job writes into a new versioned key space, checks row counts and sample values, then switches the serving API to the new version in one step. Rollback switches back. Merchant caches reload on the switch event.
• Streaming views: upserts with a monotonic check on event time, so an out-of-order write never replaces a newer value.
• TTL: online values carry an expiry equal to the view TTL; expired values return EXPIRED, never a stale number.

7.6 How real systems do it

How real systems do it (paraphrased, not copied). Uber's post "Meet Michelangelo: Uber's Machine Learning Platform" (Mike Del Balso and Jeremy Hermann, 5 September 2017) describes a shared feature store where teams publish curated features with an owner, description and SLA, and models refer to features by name. Offline features are computed by scheduled Spark and Hive jobs over the HDFS data lake and used for training and batch prediction. For online prediction, features are precomputed and kept in Cassandra: batch features are bulk loaded every few hours or daily, and near-real-time features are computed by streaming jobs from Kafka, written to Cassandra and logged back to HDFS so training uses the same data. The post reports about 10 000 shared features at the time. The open-source Feast documentation describes point-in-time joins as a backward scan from each entity row's timestamp bounded by a TTL. Read the originals in 12.3; this box keeps only the general idea.

8. Scalability and performance

Online latency budget (target p99 10 ms):

9. Reliability, security and operations

9.1 Failure modes

9.2 Security and privacy

9.3 Observability

9.4 Cost estimate

10. Trade-offs and alternatives

11. Delivery plan and risks

11.1 Milestones

11.2 Rollout and rollback

[Example: Each model migrates behind a flag. For 14 days the model reads features from both the old code path and the feature store, logs both and uses the old one; the switch happens when values agree for 99.9% of requests. Rollback flips the flag.]

11.3 Risks

12. Appendices

12.1 Open questions

12.2 Glossary

12.3 References

Standard and licence note

Based on common industry practice for machine learning feature platforms; no formal standard governs feature stores. The Uber engineering post and the Feast documentation cited in 12.3 are paraphrased in original words; no text, figures or code were copied, and all numbers marked Example are illustrative. Product capabilities change; check the current documentation of any framework you adopt. Original text written for this collection. Template licence: MIT.