Model Serving Patterns | System Designer

What are model serving patterns?

Model serving patterns are reusable ways to turn trained model artifacts into dependable prediction services. They define how a request is admitted, queued, batched, routed to a compatible model replica, executed, observed, and either completed or rejected.

In plain language: a model server does more than call predict. It protects scarce compute while keeping the product's latency, quality, isolation, and availability promises.

The core invariant is admit only work the fleet can finish inside an explicit contract. Unbounded queues, blind retries, and cold replicas do not create capacity; they move failure later in the request path.

SLO

Bound the wait

Budget queue, execution, network, and post-processing latency together

Goodput

Count useful work

Measure completions inside the latency and quality contract, not raw throughput

Warm

Make capacity ready

A scheduled process is not serving capacity until weights and caches are usable

Versioned

Attribute every result

Record model, runtime, policy, and feature versions for each response

Start by choosing the request contract

The serving shape follows the product deadline and response semantics. Do not select a framework before deciding whether callers wait online, poll later, consume a stream, or invoke a composed model graph.

Caller waits

Synchronous online

A gateway holds the connection until one prediction finishes. Tight tail latency, cancellation, bounded retries, and load shedding matter more than maximum batch size.

Caller polls or receives an event

Asynchronous job

Durable work can wait longer and form large batches. The contract needs job identity, deduplication, progress, result retention, and explicit expiry.

Partial output

Streaming response

The service emits incremental results. Time to first result, inter-result cadence, backpressure, disconnect propagation, and partial-failure semantics become first-class.

Several stages

Composed inference

Preprocessing, retrieval, models, and policy checks form a graph. Each stage needs its own deadline, ownership, observability, and fallback behavior.

Write these fields before implementation

request and response schema, size limits, model or capability name, and version policy;
latency SLO, deadline propagation rule, priority class, and cancellation owner;
quality floor, abstention or fallback behavior, and any deterministic output checks;
tenant identity, authorization boundary, data-retention policy, and cache namespace;
retry eligibility, idempotency key, and the final error returned under overload.

Trace the serving hot path

Every online request crosses the same control boundaries even when products package them differently. Keep the control plane that deploys versions separate from the data plane that serves requests.

A bounded online prediction

The deadline travels with the request. Each stage either preserves enough budget for the next stage or stops work early.

Validate

Gateway

Authenticate, validate shape and size, attach tenant and request IDs, and reject impossible deadlines.

Protect

Admission queue

Reserve queue, memory, and accelerator capacity by priority before accepting expensive work.

Shape

Router and batcher

Choose a compatible warm replica and combine only requests whose deadlines and tensor shapes can coexist.

Execute

Model worker

Run the pinned model and runtime version, honor cancellation, and return structured outputs.

Verify

Response gate

Apply deterministic checks, record outcome evidence, and return or route to the declared fallback.

Separate the latency budget

Ingress time: authentication, validation, rate limits, and network transit.
Queue time: waiting for a compatible slot; this is the earliest overload signal.
Batch delay: intentional waiting to collect compatible requests.
Execution time: preprocessing, model runtime, transfer, and post-processing.
Response time: policy gates, serialization, streaming, and network delivery.

Balance admission, batching, and deadlines

Batching can increase device efficiency, but every batching window spends latency before execution begins. More replicas add capacity only after model loading and warm-up. Admission policy decides whether excess requests wait, fail early, or use a declared fallback.

Use the lab to change workload shape, batch window, batch size, replica count, and overload policy. The model is intentionally transparent: it is a planning aid to replace with measurements from the chosen model and hardware.

Loading admission model...

Make overload a deterministic decision

Queueing every request feels fair until expired work occupies the accelerator while newer viable requests wait. Admission must compare predicted completion against the caller's remaining budget.

A useful admission sequence

Validate the request and reject unsupported shapes, versions, or sizes before queueing.
Estimate queue, batch, execution, and response time from the matching workload slice.
Reserve memory and concurrency, including model state and per-request runtime buffers.
Admit only if the predicted finish time is before the deadline and the tenant budget allows it.
Otherwise reject, defer to an asynchronous path, or route to an explicitly compatible fallback.

Deadline-aware admission model

NVIDIA Triton's maintained dynamic batcher documentation exposes maximum queue delay, queue size, priority, and timeout controls. Its rate limiter also models shared resources across model instances. The product names are optional; the capacity boundaries are not.

Choose a pattern from workload shape

One model, stable demand

Dedicated replica pool

Each pool owns one version and a narrow workload. Isolation and debugging are simple, but low-volume models can strand expensive memory.

Many models, shared fleet

Multi-model worker

Workers load several artifacts or swap them from a model cache. Utilization improves when routing preserves locality, but cold loads and eviction become tail-latency risks.

Fixed inference graph

Model ensemble

Several models or transforms execute as one declared graph. It simplifies the client contract while coupling stage deadlines, version compatibility, and failure behavior.

Different resource phases

Specialized serving pools

Route preprocessing, embedding, prefill, decode, or policy work to different pools when their hardware and scheduling needs differ. Network transfer and cross-pool failure are the cost.

Match the scheduler to the execution shape

Stateless fixed-shape inference can use dynamic batches formed from requests waiting at the same time.
Stateful sequences need affinity or explicit state transfer; arbitrary rebatching can corrupt sequence state.
Autoregressive generation benefits from iteration-level scheduling because completed sequences can leave and new sequences can join between decode steps.
Heterogeneous requests may need length buckets, padding limits, or separate pools so one large request does not dominate a batch.

The Orca paper introduced iteration-level scheduling and selective batching for transformer serving. Treat those ideas as scheduler semantics to measure, not as a reason to assume every runtime behaves identically.

Route for warmth, balance, and recovery

A least-loaded worker may not have the requested model in memory. A cache-aware route may preserve warmth while concentrating traffic. Autoscaling may request more replicas, but the fleet remains short until scheduling, artifact download, model load, and warm-up finish.

Inject a steady workload, a long-tail model catalog, or a zone loss. Then compare routing policies, autoscaling signals, warm capacity, and fallback behavior.

Loading routing model...

Treat caches as correctness boundaries

Serving systems commonly use several unrelated caches. Name each one because its key, invalidation rule, and security boundary are different.

Files near workers

Artifact cache

Stores immutable model files by digest. Verify integrity before activation and keep the version manifest separate from a mutable path or tag.

Weights in accelerator memory

Resident model cache

Avoids model-load latency. Routing, eviction, pinning, and memory fragmentation determine whether a request reaches a warm worker.

Reusable inputs

Feature or embedding cache

Keys need source version, transformation version, authorization scope, and expiry. Stale features can produce valid-looking but wrong predictions.

Reusable outputs

Prediction cache

Safe only for deterministic or explicitly reusable results. Include model, adapter, decoding parameters, policy version, tenant boundary, and normalized input in the key.

Never let cache reuse hide these changes

model, tokenizer, feature, adapter, or policy version;
tenant, authorization, consent, or data-residency boundary;
stochastic decoding parameters or nondeterministic preprocessing;
input normalization, locale, or response schema;
freshness requirements and deletion obligations.

The Clipper paper demonstrates that batching, caching, and adaptive model selection can be serving-system concerns. Production safety still depends on the key and invalidation contract chosen for the actual product.

Design autoscaling as a delayed control loop

CPU or GPU utilization describes current work, while queue age and reserved model memory often expose pressure earlier. A scaling rule must account for measurement delay, provisioning delay, model-load delay, and scale-down churn.

1
Leading signals
Observe pressure
Measure queue age, queued work units, active concurrency, deadline risk, cache misses, and accelerator memory by model and priority.
2
Contract-aware target
Compute desired capacity
Convert forecast demand into warm replicas or accelerator slots at a measured utilization target with failure headroom.
3
Delayed actuator
Provision and warm
Schedule hardware, fetch an immutable artifact, allocate runtime memory, run warm-up inputs, and pass readiness before routing traffic.
4
Stable scale-down
Drain deliberately
Stop new admission, finish or cancel bounded work, release model state, and preserve enough warm floor for the next burst.

KServe's maintained control-plane API documents queue-depth and KV-cache-aware scaling options for inference workloads. The general lesson is to scale on the constrained work unit, not a convenient host metric that reacts too late.

Separate readiness, liveness, and correctness

Startup: the process has loaded required artifacts and initialized the runtime; it is not yet proof that outputs are correct.
Readiness: the replica can accept new traffic for a specific model and version within its resource envelope.
Liveness: the process can still make progress; transient overload should usually remove readiness before it triggers a restart storm.
Correctness: a canary request, shadow evaluation, or release gate verifies output behavior independently of process health.

The official Kubernetes probe guidance warns that incorrect liveness checks can create cascading failures under load. Keep readiness cheap, model-aware, and able to fail while the process remains alive for diagnosis or recovery.

Build the shutdown path too

Mark the replica unavailable for new work.
Stop admission and propagate cancellation to queued requests.
Drain bounded in-flight work inside the termination budget.
Fail or hand off stateful work according to its contract.
Flush telemetry before releasing accelerator memory and exiting.

Kubernetes documents the endpoint and graceful Pod termination lifecycle. Verify the actual gateway and runtime drain behavior rather than assuming a grace period preserves requests automatically.

Make routing policy executable

This dependency-free example filters incompatible workers, preserves model locality, respects deadlines, and calculates a warm-replica target. Replace its constants with trace-derived service times and real provisioning measurements.

Locality-aware routing and warm-capacity target

Operate by request outcome and resource boundary

Request and queue evidence

arrival rate, accepted, rejected, deferred, fallback, cancelled, and completed requests by reason;
p50, p95, and p99 queue age, batch delay, execution time, and end-to-end latency;
deadline misses and useful goodput by model, version, tenant class, shape bucket, and priority;
retry amplification, duplicate work, disconnects, and abandoned accelerator time.

Worker and model evidence

ready, loading, draining, failed, and warm replicas by model and failure domain;
batch-size distribution, padding waste, scheduler occupancy, and per-replica concurrency;
accelerator utilization, memory allocation, model-cache hits, evictions, cold loads, and load duration;
output validity, task-quality slices, fallback rate, and version-attributed incidents.

Alerts need an action

Page on sustained SLO goodput loss, exhausted admission budget, or failure-domain capacity below the declared minimum.
Open an investigation on rising queue age, cold-load frequency, padding waste, or fallback use before users exhaust their deadline.
Record deployment, model, router, cache, and autoscaler changes on the same timeline as serving signals.
Prefer burn-rate or multi-window alerts over one noisy instantaneous threshold.

The Clockwork paper is a useful primary reference for deadline-aware scheduling and predictable inference. Its specific results are not universal capacity numbers; benchmark the selected model, runtime, hardware, and workload together.

Select a platform by capabilities, not a logo

Evaluate candidate runtimes and control planes against the service contract:

protocol and schema support, streaming, cancellation, deadlines, and idempotency;
supported model formats, hardware, parallelism, quantization, and custom operators;
dynamic or continuous batching, sequence state, priority, and queue policy;
model repositories, immutable version rollout, warm-up, cache control, and multi-model scheduling;
metrics, traces, request attribution, health semantics, and safe administrative APIs;
autoscaling inputs, failure-domain placement, disruption handling, and graceful drain;
security updates, project maintenance, portability, and operator skill.

Run load tests with representative input shapes, bursts, versions, cancellations, cache states, and failures. A single unconstrained throughput number cannot prove the production contract.

Recognize serving failures before saturation

The queue becomes the product: requests wait longer than their deadline while devices remain busy. Reject or defer before expired work consumes capacity.
Batching hides tail latency: average throughput rises while sparse or large requests wait behind preferred shapes. Track queue age by shape and cap batch delay.
Autoscaling reacts after the deadline: new replicas arrive only after the burst ends. Maintain a warm floor, forecast known events, and include model-load time in the control loop.
Retries multiply overload: gateways retry timed-out inference while original work continues. Propagate cancellation, bound retries, and use idempotency where side effects exist.
Readiness lies about warmth: the port opens before weights, runtime memory, or required adapters are usable. Make readiness model-version-aware and exercise a bounded warm-up.
Cache locality defeats balance: cache-aware routing overloads a few workers. Include queue pressure in the route score and replicate hot models deliberately.
Fallback silently changes quality: a cheaper model preserves availability but violates the task floor. Make fallback compatibility explicit and measure its outcomes separately.

Primary references

No quiz questions available

Could not load questions file