Production Deployment

Treat deployment as two connected control loops

A production deployment is not one container launch. The serving loop keeps admitted demand inside measured capacity. The release loop limits how much user traffic a candidate receives until production evidence supports wider exposure.

1. 1

   Version

   Identify the artifact

   Pin the application image, model or provider route, prompt, retrieval index, tool contracts, policy, and runtime configuration.

2. 2

   Serving

   Bound each request

   Apply authentication, deadlines, context and output limits, admission control, retry budgets, and explicit degraded behavior.

3. 3

   Evidence

   Bound the release

   Start with shadow or canary traffic, compare the candidate with a stable control, and stop on predeclared quality, policy, latency, or efficiency gates.

4. 4

   Operations

   Recover deliberately

   Route users to the known-good version, preserve traces and decision evidence, and recover mutable state with its own tested plan.

Write the production contract before choosing infrastructure

A production contract names what can change, what must stay bounded, and which signal triggers each operational action. Infrastructure choices are useful only when they enforce that contract.

Inventory every independently deployable change

• Application artifact: immutable image digest, dependencies, runtime, and API contracts
• Model behavior: provider and model identifier, decoding settings, fallback route, and alias-resolution policy
• Knowledge path: embedding model, retrieval index, chunking, ranking, and freshness watermark
• Control path: system prompt, tool permissions, safety policy, rate limits, and feature flags
• Stateful dependencies: schemas, queues, caches, audit stores, and external side effects

Turn measured demand into an admission envelope

A capacity envelope is the highest request rate the serving path can admit while keeping every declared bottleneck below its planning target. It is not a prediction from CPU alone.

The lab uses three measured inputs: mean service time, average tokens per request, and concurrency that one ready replica can sustain. Change the workload and capacity controls to find whether local serving slots or the upstream token budget constrains admission first.

Read the envelope in a fixed order

1. Calculate offered concurrency. Multiply arrival rate by measured mean service time. This is the average active work implied by Little's Law.
2. Calculate ready slots. Multiply ready replicas by measured concurrency per replica; do not count starting or unready capacity.
3. Calculate token demand. Multiply request rate by average input-plus-output tokens and by 60 seconds.
4. Take the minimum boundary. The safe admission rate is constrained by whichever capacity becomes saturated first.
5. Bound overflow. Queue only within an explicit time and depth budget; reject or degrade excess work before an unbounded queue hides overload.

Build a request path that can degrade deliberately

A degradation path preserves the most important user outcome when a dependency, quota, or latency budget fails. Each boundary owns one deadline and one fallback; nested retries must not outlive the caller.

Define failure behavior before traffic arrives

• Queue full: reject with a retryable response and backoff guidance; do not accept work that cannot meet its deadline.
• Retrieval unavailable: return an explicitly ungrounded or no-answer response only when the product contract permits it.
• Tool timeout: stop under the remaining deadline and avoid retrying a non-idempotent side effect without a stable key.
• Primary model unavailable: route to a tested fallback with known capability differences, or fail closed for unsupported high-risk tasks.
• Telemetry unavailable: preserve the user path only when a durable local buffer or explicit sampling policy prevents silent loss of required audit evidence.

Make orchestration enforce the contract

Container orchestration can replace unhealthy processes and add replicas, but it cannot infer model quality, upstream quotas, or safe fallbacks. Configure it around application-level signals.

Keep the health signals separate

• Startup answers whether initialization has completed. While it fails, Kubernetes does not run liveness or readiness probes.
• Readiness answers whether this replica should receive new traffic. It may fail during overload, warm-up, or a required dependency outage without demanding a restart.
• Liveness answers whether the process is irrecoverably stuck and should restart. It should not fail merely because an optional dependency is slow.
• Rollout progress answers whether the new ReplicaSet is becoming ready before progressDeadlineSeconds.

The example scales on a queue-related external metric because pending work is closer to the serving bottleneck than CPU for many remote-model gateways. It requires an external metrics adapter. Use the metric that load tests show predicts saturation, and keep token or provider quotas in admission control because adding Pods cannot raise those limits.

Promote only from versioned production evidence

A canary release sends a bounded share of representative traffic to a candidate while a stable version remains available as the control and rollback target. Small exposure limits blast radius, but it also gathers evidence more slowly.

Use the lab to compare a healthy candidate with quality, policy, and operational regressions. Change traffic share and observation time to see why more exposure can satisfy a sample gate but cannot repair a hard failure.

Separate hard blockers from operational warnings

• Roll back: a candidate violates a hard task-quality, protected-slice, safety, policy, privacy, or correctness boundary.
• Hold: hard boundaries pass, but sample size, latency, efficiency, instrumentation, or diagnosis is incomplete.
• Expand gradually: every declared gate passes, the stable route remains healthy, and the same abort rules apply to the next traffic step.
• Never average away a critical failure: a higher global task score cannot compensate for a failed policy boundary or high-impact cohort.

Match the rollout pattern to the failure boundary

A rollout pattern controls who sees a candidate, how quickly exposure grows, and how traffic returns to the stable route. It does not replace evidence gates.

Observe the user outcome and the causal path

Production observability connects a user-visible outcome to the exact release path that produced it. Fleet averages without cohort and version identity can hide a severe regression.

Alert on decisions, not isolated charts

• Page when the request path cannot meet its reliability contract or a hard policy boundary fails.
• Pause expansion when evidence volume, latency, efficiency, or instrumentation is incomplete.
• Diagnose by release, cohort, route, dependency, and time window before blaming the model.
• Preserve sampled inputs and outputs only under explicit privacy, access, redaction, and retention controls.