Uber MLOps Production Pipeline
End-to-end MLOps pipeline: automated training, deployment, monitoring, and continuous delivery for production ML systems.
What is an MLOps production pipeline?
An MLOps production pipeline is the system that turns a trained model into a repeatable, observable production decision. It connects versioned data and feature definitions to training, evaluation, approval, gradual deployment, monitoring, rollback, and retraining.
In plain language: software delivery ships code; MLOps delivery ships a tested decision contract. The contract includes the model weights, but also the data snapshot, feature definitions, preprocessing, thresholds, serving configuration, and evidence used to approve the release.
Core invariant
Never route a request to a model unless its request schema, feature values, preprocessing, artifact, and decision policy are compatible and traceable to one approved release. A fast prediction with the wrong features is not a successful prediction.
This case study designs an enterprise pipeline for 100+ production models, up to 1M predictions per second, 10 TB of source data per day, and a four-hour daily retraining window. For serving foundations, review model serving before treating inference as an isolated model-runtime problem.
The pipeline exists to make production changes reversible
The platform serves fraud, ranking, forecasting, and operations models. These models have different quality metrics, but every team needs the same platform guarantees: an approved artifact can be reproduced, released in a bounded cohort, observed with versioned telemetry, and replaced quickly when its contract no longer holds.
Functional requirements
- Trigger training from approved data arrivals, schedules, or a reviewed manual request.
- Validate raw inputs and feature definitions before they are used for training or online inference.
- Register model artifacts with owners, lineage, evaluation evidence, and approval state.
- Support shadow, canary, and full deployment for real-time and batch consumers.
- Detect data drift, quality regression, bias regressions, and serving failures.
- Roll back to a compatible prior release and create a retraining work item when evidence fails.
Non-functional requirements
- Keep the critical online path below 100 ms P99 and available at 99.9% or better.
- Retain complete release lineage for audits and incident investigation.
- Complete the standard daily training workload within four hours without starving online serving.
- Keep online features fresh enough for their product contract, with an explicit stale or missing state.
- Isolate tenants, models, experiments, and rollout cohorts so one failure cannot consume shared capacity.
Scope boundary
The platform automates evidence collection and safe mechanical actions. It does not silently decide that a new objective, a changed label definition, or a material fairness trade-off is acceptable; those remain accountable human decisions.