Model Serving
Deploy model serving systems: inference optimization, serving architectures, scaling, and production deployment.
What is model serving?
Model serving is the production system that accepts an input, builds compatible features, runs a specific model version, applies decision logic, and returns or records the result. It turns a trained artifact into a reliable product capability.
Serving design starts from the workload: freshness, request volume, latency, batchability, model size, hardware, error cost, and fallback behavior. Optimizing the model kernel alone is insufficient when feature lookup, queueing, transfer, or post-processing dominates the request.
The core invariant is version compatibility: the request schema, feature definitions, preprocessing, model artifact, and post-processing thresholds must describe one tested serving contract.
Allocate an end-to-end latency budget
The following is an example budget for a 100 ms P95 product target, not a universal recommendation.
20 ms
Feature retrieval
Online store, transforms, and validation
50 ms
Queue + inference
Batch wait, transfer, and model execution
15 ms
Decision + network
Post-processing, policy, and serialization
15 ms
Failure headroom
Jitter, retries, and dependency variance
Budget percentiles for the complete path. Adding several individually measured P95 values does not produce the request P95 because slow stages may occur on different requests.
Which serving architecture fits the decision?
Request/response
Online
Use for interactive decisions that need fresh features and bounded latency. Capacity, timeout, fallback, and overload behavior are part of correctness.
Scheduled bulk work
Batch
Use when predictions can be precomputed. Large batches improve throughput and simplify recovery, but outputs are only as fresh as the schedule.
Continuous events
Streaming
Use when each event needs prompt scoring outside a synchronous user request. Ordering, replay, checkpoints, and idempotent outputs matter.
Near the device
Edge
Use for privacy, offline operation, or very low network latency. Artifacts must fit device memory, power, update, and compatibility constraints.
Trace one online prediction
Bounded online inference path
Every dependency consumes the same request deadline. The fallback must avoid calling the failing dependency again.
Admission
Gateway
Authenticates, validates size and schema, rate-limits abuse, and attaches the remaining deadline.
Context
Feature service
Returns point-in-time-compatible online features or an explicit missing/stale status.
Inference
Model runtime
Batches within a bounded wait, executes the pinned artifact, and records model-level latency and errors.
Action
Decision policy
Applies thresholds, business constraints, uncertainty handling, and human-review or fallback rules.
Feedback
Outcome log
Records exposure, prediction, action, and join keys for delayed outcomes without blocking the response.
Optimize the bottleneck that measurement reveals
Amortize compute
Batching
Combine compatible requests to improve accelerator utilization. Bound the wait so throughput gains do not violate tail latency.
Reduce precision
Quantization
Lower weight or activation precision only when the runtime has efficient kernels and the exported model passes quality checks.
Avoid repeat work
Caching
Cache only when inputs, model version, policy, freshness, and privacy permit safe reuse. Include all behavior-changing values in the key.
Match work to capacity
Routing
Send requests by model, hardware, region, or quality tier. Keep overload shedding and fallback deterministic and observable.
Deploy a model as a reversible change
1 Build
Package one contract
Bundle the artifact, runtime, preprocessing, schemas, dependency versions, and metadata required to reproduce inference.
2 Verify
Benchmark representative load
Measure quality, latency percentiles, throughput, memory, cold starts, saturation, and failure behavior on target hardware.
3 Observe
Shadow production inputs
Compare candidate outputs and resource use without changing decisions. Investigate disagreements by important slice.
4 Limit risk
Canary bounded traffic
Expose a representative fraction with automatic rollback on service, model, safety, and business guardrails.
5 Operate
Promote and retain rollback
Scale gradually, keep the previous compatible version warm enough to recover, and monitor delayed outcomes after full rollout.
Design degradation before an incident
- Reject malformed or over-budget requests before expensive model work begins.
- Propagate deadlines and cancel abandoned work so queues do not grow from timed-out requests.
- Shed low-priority traffic when saturation threatens the critical path.
- Fall back to a cached result, heuristic, smaller model, or human review only when that behavior is safe for the decision.
- Make retries bounded and idempotent; a retry storm must not amplify a dependency failure.
- Separate outcome logging from the response so analytics loss does not take inference down.