Deployment Strategies
Modern deployment strategies: blue-green, canary, rolling updates, and feature flags explained.
What is a deployment strategy?
A deployment strategy is the plan for moving a running system from one software version to another. It specifies who receives the new version, how the team decides it is healthy, and how traffic returns to the previous version if the change causes harm.
The important idea is not the pattern name. A safe release keeps a known-good path available long enough to observe the new path. That gives the team a bounded blast radius, a stop rule, and a recovery action instead of a hopeful one-way change.
Treat every release as a controlled experiment
The release invariant is: do not increase exposure until the version is observable, compatible with the old version, and reversible at the traffic boundary. A deployment can be fast, but it cannot skip that decision loop.
1 Who can be affected
Set the exposure
Choose the first cohort, the maximum traffic share, and whether a maintenance window is acceptable.
2 What can coexist
Keep versions compatible
Ensure old and new applications, jobs, clients, APIs, and schema reads can run together during the transition.
3 What proves health
Evaluate a gate
Compare error rate, latency, saturation, and one customer outcome against named thresholds over an observation window.
4 What happens next
Promote or recover
Increase traffic only after the gate passes. On a breach, freeze exposure and restore traffic before investigating the code.
Choose from the governing constraint
Start with the constraint that would make a bad release unacceptable. Select one below, then inspect the path and result. The recommendation is a starting point; it still needs compatible data changes and an owned rollback procedure.
The four useful default choices
- Recreate: Stop the old version and start the new one. Use it for development, isolated batch workers, or a planned maintenance window. Its low machinery cost is paid in downtime and a broad blast radius.
- Rolling update: Replace healthy instances in batches while both versions serve traffic. It works when spare capacity is limited, but demands backward-compatible behavior because the fleet is mixed.
- Blue-green: Keep a complete standby environment and switch the router between environments. It makes traffic rollback fast, but doubles much of the runtime footprint and does not undo an unsafe data migration.
- Canary or shadow: Send a small real cohort to the candidate, or mirror requests without changing customer responses. Use them when production behavior is uncertain and the team can measure a meaningful difference.
Move traffic in stages, with a real stop rule
A progressive rollout is not a timer that eventually reaches 100%. Each stage is a new decision: the candidate receives a larger share only after the prior cohort has been healthy for the agreed observation window.
What the lab should make you notice
- Shadow traffic answers a comparison question. The stable version still owns the customer response; the candidate's response, error, and resource use are observed separately. Do not mirror writes unless they are isolated or idempotent.
- A canary answers an outcome question. A small, identifiable cohort sees the candidate. Compare its error rate and latency with the stable cohort, then also check a domain signal such as completed checkout, successful upload, or valid recommendation.
- Rollback begins with routing. Stop the next promotion, route new requests to the stable version, and preserve evidence. A rollback that waits for a code fix leaves more users on a known-bad path.
Make the mixed-version window safe
Traffic rollback only works when the older version can still handle the state created by the newer one. Plan the compatibility window before deploying application code.
Schema and contracts
Expand first
Add nullable columns, additive fields, or new endpoints before any writer depends on them. Old readers must ignore what they do not understand.
Dual compatibility
Migrate while both run
Deploy code that can read old and new forms, backfill in bounded batches, and make retries idempotent. Version background jobs too.
After retirement
Contract last
Remove the old field, behavior, or environment only after no old binaries, delayed jobs, replay consumers, or rollback window need it.
Feature flags solve a different problem from deployment routing. A flag can activate a deployed capability for a cohort, while routing decides which binary serves the request. Use both when needed, but give each one an owner, expiry date, and emergency-off behavior.
A database rollback is often a forward fix
Do not drop or rename a populated column in the same release that starts writing its replacement. If the new writer has already produced data the old code cannot read, route traffic back first and repair forward with a compatible migration or reconciliation job.
Write the gate before the deployment
The policy below turns a vague canary into an executable operating decision. The exact values vary by service; the discipline does not. Every threshold needs an owner, a dashboard, and a response that is safe to execute under pressure.
1%
First exposure
A bounded, identifiable cohort
10 min
Observation window
Long enough for the failure signal
0.5 pp
Error budget delta
Candidate versus stable cohort
1 action
On breach
Freeze and route back
For a payment path, the business gate might be authorization success or duplicate-charge rate. For a search ranking change, it might be empty-result rate or query reformulation. A CPU graph alone cannot tell you whether the release is acceptable.
Compare strategies at the decision boundary
Replace a batch of instances, wait for readiness and health checks, then continue. It is often the default for stateless services because the orchestrator can preserve availability without duplicating the whole environment.
- Use it when: existing capacity can absorb one batch leaving service, and old and new versions share compatible APIs, caches, sessions, and schema behavior.
- Watch for: a bad binary that reaches many instances before the metric reacts, or a mixed fleet that produces inconsistent behavior.
- Recover by: pausing the rollout, restoring the prior replica template, and keeping capacity above the load target while replacement instances become ready.
Operate the release after it starts
The deployment is an operations event, not just a CI job. Before the first traffic shift, name the people and signals that turn an alert into a decision.
- Release owner: owns the promotion, pause, and rollback command. This person should not need to guess which dashboard or flag controls traffic.
- Guardrails: track request success, p95 or p99 latency, saturation, and one domain outcome by version and cohort. Keep the stable baseline visible beside the candidate.
- Recovery data: retain version labels, cohort assignment, request correlation IDs, migration state, and logs long enough to compare and diagnose after traffic returns.
- Dependencies and jobs: decide whether consumers, schedulers, queues, and retries are paused, versioned, or drained. A web rollback does not stop a new background worker from mutating data.
- Practice: rehearse the route-back command and a failed migration in a non-production environment. Measure the time from a breached gate to restored traffic.
The best rollback is boring: one authorized action changes the routing or flag state, verification confirms the stable path is serving traffic, and a separate incident workflow investigates the candidate.
Continue learning
- Load Balancing: Learn how routers distribute a cohort across healthy instances.
- Feature Flags: Separate code deployment from controlled feature activation.
- Database Migrations: Apply expand, migrate, and contract changes without breaking compatible versions.
- Monitoring Metrics: Define the latency, error, saturation, and outcome signals used by release gates.