Design an ML Feature Store
Design an ML feature store: feature computation, offline/online store, and data lineage.
What is a feature store?
A feature is an input a model uses to make a prediction, such as a customer's recent purchases or the number of failed logins in the last hour. A feature store is the system that defines, computes, documents, and serves those inputs so the values used to train a model mean the same thing as the values it receives in production.
This matters because a model can be accurate in an offline notebook and wrong in production when its live inputs are stale, calculated differently, or taken from the future. The central invariant is simple: a prediction must use a feature definition, event-time rule, and freshness policy compatible with the model that was evaluated.
Meaning
Feature definition
A versioned contract names the entity, transformation, source fields, owner, event-time rule, and allowed missing-value behavior.
Training
Offline history
Historical, point-in-time joins reconstruct only the information that existed when each training example occurred.
Serving
Online materialization
A low-latency store serves the current approved feature bundle, together with its version and freshness metadata.
Start with the prediction, not the database
Assume this platform supports checkout-risk, ranking, and personalization models. Product teams need to reuse features without each team rebuilding pipelines, while model owners need a defensible answer to: what did the model know at the moment it made this decision?
Agree on the requirements that change the design
- Functional requirements
- Register a versioned feature definition and make it discoverable by entity, owner, source, and data classification.
- Build point-in-time training sets, materialize approved features online, and retrieve a named feature bundle for one entity.
- Record lineage from raw event through transform, snapshot, materialization, model release, and serving decision.
- Non-functional requirements
- Keep the online lookup contribution below 10 ms at p99 for the latency-sensitive path.
- Publish a freshness SLA per feature group instead of pretending every feature is real time.
- Preserve 99.99% availability with an explicit, audited fallback policy.
- Scope boundary
- The store owns feature semantics and delivery. It does not replace a warehouse, a general event bus, model training infrastructure, or product authorization policy.
Name the costly failure modes
- A training join leaks information from after the label time, making offline metrics optimistic.
- A transform changes online while the trained model still expects the prior definition.
- The materializer is fast but behind, so a valid lookup returns a stale behavior signal.
- An online timeout silently becomes default values with no version, age, or reason code recorded.
For a short primer on moving events through reliable pipelines, review the data pipelines lesson. This practice assumes you can distinguish an event's occurrence time from the time a system happened to ingest it.
Serving features online and offline
Your model trains on historical features in batch and needs the same features served at inference. How do you store and serve features so training and serving agree?
Constraint: Eliminate train/serve skew (identical feature values at training and inference) while keeping online serving latency low (single-digit ms).