ML Systems Design
Design ML systems: architecture patterns, data flow, model serving, and production ML system design principles.
What is ML systems design?
ML systems design is the work of turning a prediction problem into a dependable product capability. It connects data collection, feature computation, model training, evaluation, deployment, serving, monitoring, and feedback so that a useful decision reaches a user under real constraints.
The model is only one versioned component. A complete design states which decision is being improved, what evidence is available at decision time, how quickly an answer is needed, what happens when ML is unavailable, and how outcomes return as trustworthy learning data. Complete ML Problem Framing first if the decision, target, or cost of mistakes is still unclear.
Begin with the decision, not the model
Before choosing an algorithm or platform, write the contract between the product and the ML system. The contract turns vague goals such as "improve recommendations" into constraints that can shape data, evaluation, and serving.
Product action
Decision
Name who consumes the prediction and what they do with it. A risk score used to block a payment has different failure costs from a score used to order a review queue.
Available now
Evidence
List features that legally and technically exist at prediction time. Historical correlations are not usable if they arrive after the decision.
System boundary
Service promise
Set freshness, latency, throughput, availability, privacy, and cost targets. Use percentiles and peak traffic rather than averages alone.
Safe behavior
Failure policy
Define fallback, abstention, human review, and rollback behavior before deployment. A model timeout must not invent product policy.
Keep four kinds of success separate
- Business outcome: conversion, loss prevented, time saved, retention, or another effect the product values.
- Model quality: precision, recall, ranking quality, calibration, or error by an important slice.
- System health: p95 and p99 latency, availability, throughput, freshness, and cost per prediction.
- Safety and governance: harmful-error rate, fairness constraints, consent, auditability, and human escalation.
A higher offline score is not a successful system if stale features, slow inference, or biased exposure prevent that score from improving the real decision.
Design one connected lifecycle
Training and serving run on different clocks, but they must share definitions and identifiers. The design should make every production prediction reproducible enough to investigate and every observed outcome attributable to the policy that produced it.
1 Decision
Frame and baseline
Specify the consumer, target, error costs, constraints, and a non-ML or simple-model baseline.
2 Data
Build trustworthy evidence
Version sources and feature logic, enforce point-in-time correctness, and preserve lineage from raw event to training example.
3 Model
Train and gate candidates
Compare candidates against the baseline on representative time periods, slices, calibration, robustness, and resource use.
4 Deploy
Release with containment
Use shadow traffic, canaries, fallbacks, immutable artifacts, and automatic rollback conditions to limit exposure.
5 Operate
Observe outcomes and adapt
Join predictions, decisions, exposures, and delayed outcomes; detect drift; then retrain only from validated evidence.
Choose an architecture from freshness and traffic
An inference pattern describes when predictions are computed and what must run on the product's critical path.
Precompute
Scheduled batch
Score many entities on a schedule and store the results. This is simple and efficient when the product accepts predictions that are minutes or hours old.
Compute now
Request-time online
Fetch eligible features and run inference while the user waits. This supports live context but consumes the request latency and availability budget.
Two clocks
Hybrid
Precompute expensive candidates or features, then use fresh context to filter or rerank online. The added path is justified only when both scale and freshness demand it.
Keep training and serving compatible
The core invariant is the deployed model must receive features with the same meaning, units, time boundary, and version it was evaluated with. Reusing names is not enough; purchase_count_30d can silently change when its event filter, timezone, default, or late-data policy changes.
A traceable prediction and learning path
Every arrow crosses a contract boundary. Carry entity, event-time, feature, model, prediction, exposure, and policy identifiers far enough to reproduce the decision.
Evidence
Source events
Immutable events record provenance, event time, consent, and stable entity identifiers.
Transform
Feature definitions
One versioned definition creates point-in-time training values and compatible online values.
Gate
Model registry
An immutable artifact is linked to its feature contract, code, data snapshot, and evaluation evidence.
Serve
Decision service
Bounded feature lookup and inference return a prediction or an explicit fallback before the deadline.
Learn
Outcome join
Prediction, exposure, action, and delayed outcome records produce debiased monitoring and future examples.
Make the runtime contract executable
Validate model and feature compatibility at deployment and at request time. Reject stale or mismatched evidence, return a documented fallback, and emit enough trace metadata to explain the result later.
Release changes without making users the test suite
Offline evaluation cannot reproduce every production dependency, traffic burst, or behavior shift. Promotion therefore needs progressively stronger evidence and bounded exposure.
1 Offline
Replay
Test a time-based holdout, critical slices, calibration, robustness, latency, and cost against the current production baseline.
2 No authority
Shadow
Run the candidate on production inputs without affecting decisions. Compare feature parity, outputs, latency, and resource demand.
3 Bounded authority
Canary
Expose a small, representative cohort with guardrails for quality, latency, errors, safety, and business harm.
4 Decision
Promote or roll back
Increase traffic only when predeclared checks pass. Keep the prior artifact and routing policy ready for immediate restoration.
Never delete the previous model at promotion time. Rollback is a system capability, not a manual reconstruction exercise.
Diagnose the loop, not only the model
Production failures appear in different parts of the lifecycle. A latency spike may require capacity or fallback changes, while gradual concept drift requires fresh labels and retraining evidence. Treating every alert as "retrain the model" can make the incident worse.
Monitor a connected set of signals:
- Input health: schema validity, missing values, feature age, distribution shifts, and slice coverage.
- Prediction health: score distribution, confidence, abstention, fallback use, and model-version mix.
- Service health: p95 and p99 latency, errors, saturation, queue depth, dependency health, and cost.
- Outcome health: delayed quality, calibration, business KPI, harmful-error rate, and performance by critical slice.
- Learning health: label delay, join coverage, exposure logging, retraining-data freshness, and evaluation comparability.
Make ownership and trade-offs explicit
An operable design names who owns the data contract, feature definitions, training pipeline, artifact registry, serving SLO, product fallback, model-quality review, and incident response. A diagram without those boundaries hides the work most likely to fail during change.
Common trade-offs
- Accuracy versus latency and cost: a larger model may improve offline quality but miss the product deadline or make peak traffic unaffordable.
- Freshness versus complexity: streaming features reduce staleness while adding event-time, replay, state, and recovery obligations.
- Automation versus control: automatic retraining shortens adaptation time but needs strong data gates, promotion evidence, and rollback.
- Shared platform versus local ownership: common infrastructure reduces duplication, while product teams still need authority over decision policy and failure behavior.
- Fast iteration versus reproducibility: experimentation should be quick, but every promoted artifact needs immutable code, data, configuration, and evidence.
The strongest design is not the one with the most components. It is the smallest system that satisfies the decision contract, exposes its failure modes, and can change safely.