MLflow | System Designer

What is MLflow?

MLflow is an open-source platform for recording machine learning work and carrying its evidence from an experiment to a packaged, evaluated, and registered model. Its APIs and UI help teams track runs, store artifacts, package models behind a common interface, compare evaluation results, and manage named registry references.

MLflow does not train a model by itself, schedule every pipeline, or operate a production serving fleet for you. A training framework creates the model; workflow automation runs the jobs; deployment infrastructure routes traffic. MLflow connects those systems through durable metadata and model artifacts.

The core invariant is: a production decision should resolve to an immutable model version backed by traceable data, code, configuration, evaluation, and release evidence. A mutable registry alias is a pointer to that evidence, not evidence by itself.

Tracking

Record the execution

An experiment groups comparable runs. Each run captures one execution's parameters, metrics, tags, dataset inputs, timestamps, and artifacts.

Models

Package the contract

An MLflow Model bundles model data with flavor metadata, an inference signature, an input example, dependencies, and any required code.

Registry

Control the reference

A registered model has immutable numbered versions plus mutable aliases such as candidate or champion that automation can move after policy passes.

Follow the evidence from training to serving

Treat MLflow as an evidence and packaging plane. The useful path is not "train, click Production, and hope." It is a sequence of boundaries with different owners and failure modes.

MLflow model lifecycle

Each boundary adds evidence. The serving platform consumes an immutable package and remains responsible for runtime health.

Observe

Experiment and run

Record one execution, its dataset references, code identity, configuration, metrics, and ordinary artifacts.

Package

Logged MLflow Model

Persist model files, flavors, signature, example payload, dependencies, and a stable model URI.

Decide

Evaluation and registry

Evaluate a fixed candidate, apply release policy, create or copy a model version, and move an alias only when the evidence is complete.

Operate

Deployment controller

Resolve the alias to one immutable version, verify the package, canary it, observe it, and retain a tested rollback version.

Keep these identities distinct

Experiment: a logical grouping for runs that answer one comparable ML task.
Run: one execution with a unique run ID, lifecycle status, metadata, inputs, and artifacts. A repeated job should create a new run rather than overwrite history.
Logged model: a model artifact with model-specific metadata and a model URI. In current MLflow tracking, logged models can be searched separately from runs.
Registered model version: an immutable numbered registry entry backed by a model source.
Alias: a mutable name that points to one registered model version. Moving it is a release action.

Make experiments answer lineage questions

Logging more fields is not the goal. The goal is to answer a concrete question after the training process, notebook state, and temporary compute are gone.

Put comparable executions in one experiment and use stable tags for task, owner, pipeline, branch, and data-contract identity.
Create one run per execution. Use parent and child runs when a search, fold, or distributed job needs a visible grouping.
Log parameters as configuration, metrics as measured outcomes, and tags as searchable operational context. Do not encode all three into a run name.
Log dataset inputs with source, digest, name, schema, target, and context such as training or evaluation. A file path alone is weak identity when its contents can change.
Link immutable code or build identity and capture the resolved environment. A mutable branch name and requirements.txt with open ranges do not reconstruct an execution.
Keep sensitive rows, secrets, tokens, and personal data out of parameters, tags, examples, and unrestricted artifacts.

The lab is a review model, not a claim that every field has equal value. Capture the minimum evidence that answers your workflow's reproducibility, comparison, and audit questions.

Track data, code, metrics, and a packaged model

Run the example with --dry-run first to validate the co-located dataset contract without MLflow dependencies. A real run needs a pinned MLflow and scikit-learn environment and a configured tracking URI.

Package a model as an inference contract

An arbitrary pickle is a serialized object. An MLflow Model is a directory with an MLmodel metadata file and one or more flavors describing how a tool can load or use the model. The common python_function flavor exposes a uniform predict interface; framework flavors can preserve native loading behavior.

Interface

Signature and example

The signature describes input, output, and optional inference-parameter schemas. The input example makes the contract concrete and supports serving-payload validation.

Runtime

Dependencies and code

Captured environment files and intentional code paths help recreate loading and prediction behavior. Inference is useful, but a locked and scanned environment is stronger evidence.

Payload

Artifacts and provenance

Weights, tokenizer files, encoders, lookup data, and other required assets must travel with stable provenance and remain readable by the deployment identity.

Test the package, not only the in-memory model

Load the logged model in a clean environment using its model URI.
Validate a serving-compatible example plus malformed, missing, extra, and boundary inputs against the signature.
Confirm dependency installation, import behavior, artifact checksums, startup time, memory use, and prediction parity.
Scan serialized model files, bundled code, and dependencies as untrusted supply-chain inputs. Loading a model can execute framework or custom Python code.
Keep training-only files out of the serving package and do not embed credentials in model artifacts.

Packaging improves portability; it does not prove behavioral equivalence on every target. Native libraries, CPU instructions, accelerators, locale, and dependency resolution can still change results or prevent loading.

Evaluate a fixed candidate before promotion

Evaluation measures a model; promotion changes shared state. Keep them as separate, retryable pipeline steps so a successful evaluation call cannot accidentally imply production readiness.

1
Identify
Freeze the candidate
Resolve one logged-model or registered-model version and one versioned evaluation dataset. Record both immutable identities.
2
Evaluate
Measure behavior
Use mlflow.models.evaluate for classic classification or regression metrics and artifacts, then add business, slice, robustness, calibration, and safety checks that match the product risk.
3
Gate
Validate policy
Apply absolute floors, regression limits, latency and resource ceilings, contract checks, approvals, and operational readiness outside the training code.
4
Audit
Persist the decision
Store the policy version, evaluation run, candidate version, result, approver or automation identity, timestamp, and failure reason before any alias moves.

Overall accuracy can hide a failing region, account tier, device, or rare class. A production gate should therefore include three layers:

Model behavior: overall metrics, worst important slice, calibration, robustness, and comparison with the current champion on the same data.
Serving behavior: signature compatibility, clean-environment loading, valid and invalid payloads, latency, throughput, memory, and dependency policy.
Operational behavior: canary scope, dashboards, alerts, ownership, rollback, artifact availability, and a failure drill.

Evaluate and validate a classifier candidate

The classic evaluation and GenAI evaluation APIs have different scorers and result types. Choose the system that matches the workload and do not mix their metric contracts in one release rule.

Promote with aliases and an explicit handoff

MLflow model stages are deprecated. Current registry workflows use model-version tags for status evidence and aliases for named references such as candidate or champion. An alias URI has the form models:/<registered-model-name>@<alias>.

Use separate registered model names when development, staging, and production need different permissions or ownership boundaries. Copy a validated version across that boundary, then apply an environment-local alias. Do not treat one alias on one global model as an access-control system.

The deployment controller should resolve champion once, record the returned model name, version, source, and package digest, and deploy that immutable result. Runtime workers that repeatedly poll a mutable alias can silently diverge during a rollout.

Gate a candidate and move champion with a stale-read check

The example evaluates the external policy without MLflow. --apply connects to the registry, checks that champion still matches the version used by the decision, tags the candidate, and moves the alias. In a real release service, make this operation idempotent, authenticated, audited, and serialized per registered model.

Contain deployment and runtime failures

MLflow can serve a model locally and can package containers or integrate with deployment targets, but the production handoff still needs a controller that owns routing and runtime state.

Deployment handoff contract

Resolve an approved alias to an immutable version and verify the expected source, signature, package digest, and policy evidence.
Fetch and load the package with a least-privilege deployment identity in a clean, isolated environment.
Run startup, prediction-contract, dependency, and health smoke tests before adding traffic.
Send a bounded canary or shadow workload and compare candidate and champion quality, errors, latency, saturation, and business guardrails.
Increase traffic in explicit steps. Record the deployed version at every step; do not infer it later from the current alias.
Keep the previous immutable package warm or quickly loadable until the rollback window closes.

Failure containment rules

If the artifact store is unavailable, stop the new deployment and keep already loaded champion replicas serving. Do not evict healthy capacity to retry forever.
If signature rejection, error rate, or a protected slice regresses, route candidate traffic to zero before investigating. Preserve bounded diagnostic samples with sensitive values removed.
If an alias changes concurrently, fail the stale promotion rather than overwrite the newer decision. Refresh evidence against the new champion.
If tracking is unavailable, training may buffer or fail according to policy, but serving an already loaded model should not depend on a live tracking-server request.
If rollback cannot load, halt expansion and restore known-good capacity before attempting another candidate.

Secure and observe the MLflow control plane

A shared tracking deployment has three distinct storage and trust surfaces:

Tracking server: accepts API and UI requests, authenticates callers, authorizes experiments and registry actions, and may proxy artifact traffic.
Backend store: persists run, experiment, registry, and related metadata. Use a supported database, backups, tested restoration, connection limits, and restricted credentials for a team deployment.
Artifact store: holds larger files such as models, plots, and datasets. Decide whether clients access it directly or through the tracking server; this changes where storage credentials and authorization must exist.

Security baseline

Put remote traffic behind TLS and explicit authentication; grant separate read, write, promotion, and administration permissions according to role.
Restrict network reachability. For the default FastAPI server, configure allowed hosts and CORS origins when listening beyond localhost.
Prefer proxied artifact access when clients should not hold broad object-store credentials. If clients write directly, scope and rotate those credentials.
Treat model packages, custom code, and serialized objects as executable supply-chain inputs. Scan before promotion and isolate loading.
Redact secrets and sensitive feature values from parameters, tags, examples, evaluation tables, traces, logs, and failure payloads.
Audit model-version creation, tag changes, alias moves, deletions, permission changes, and administrative actions with actor and timestamp.

Observe both control plane and model behavior

Track API request rate, error rate, latency, authentication failures, database pool pressure, migration state, artifact transfer failures, queueing, and storage growth.
Alert on failed or unusually long runs, missing metrics, incomplete artifacts, evaluation failures, alias churn, and versions whose source artifacts are unreadable.
At serving time, emit the resolved registered-model name and immutable version with latency, errors, resource saturation, schema rejection, data quality, drift, and product-quality signals.
Keep high-cardinality run and model IDs searchable without turning all of them into unbounded time-series labels.

Migrate as a controlled system change

MLflow server, client, database schema, artifact routing, and model package behavior can evolve independently. A package upgrade is therefore a release, not routine image refresh.

Upgrade the tracking service

Inventory server and client versions, backend and artifact stores, authentication plugins, registry usage, and deployment consumers.
Back up metadata and verify artifact references. Rehearse restoration before changing the production database.
Test the target server against pinned training, evaluation, registry, and deployment clients. Newer clients may call endpoints an older server does not expose.
For a database backend, stop writes according to the migration plan and run mlflow db upgrade <backend-store-url>. Official guidance warns that schema migrations can be slow and are not guaranteed to be transactional.
Verify representative experiments, model downloads, registry aliases, permissions, and evaluation workflows before reopening all writers.

Migrate lifecycle conventions

Replace stage-based URIs and transition calls with aliases, version tags, and environment boundaries. Run old and new readers in shadow mode before removing stage assumptions from deployment code.
When changing from direct to proxied artifact access, remember that existing experiments retain the artifact location recorded when they were created. Test old and new experiments instead of assuming one server flag rewrites history.
Rebuild and smoke-test representative model packages when dependency capture, signatures, flavors, or Python versions change. Metadata migration does not prove runtime compatibility.
Preserve immutable version references in deployment records so a registry migration cannot erase what was actually serving.

Use a production release checklist

A model is ready for an alias move only when the release record can answer all of the following:

Identity: Which run, logged model, registered version, code commit, dataset digest, feature contract, and environment produced this candidate?
Package: Does a clean runtime load it, enforce its signature, reproduce expected predictions, and pass dependency and artifact scanning?
Evaluation: Which versioned dataset and policy produced the overall, slice, robustness, safety, latency, and comparison results?
Registry: Which alias will move, what immutable version does it currently point to, who may change it, and how is a concurrent promotion detected?
Handoff: Which deployment controller resolves the alias, verifies the package, records the immutable version, and controls canary expansion?
Observability: Can operators query version-specific quality, input rejection, latency, errors, saturation, artifact health, and alias audit events?
Containment: What stops expansion, what keeps the champion available, how fast is rollback, and when does a human take control?
Migration: Are server, client, database, artifact, and model-package changes tested independently with backup and restoration evidence?

Promotion should fail closed when required evidence is missing. That policy is less convenient than moving an alias from a notebook, but it makes a production decision repeatable and reviewable.

Sources and version notes

MLflow APIs and defaults evolve. These links are current primary documentation used for the lifecycle claims in this lesson; pin an exact MLflow release and verify its API reference and migration notes before adopting the examples.

No quiz questions available

Could not load questions file