Istio
Master Istio: service mesh architecture, traffic management, security, and observability.
What is Istio?
Istio is a service mesh: an infrastructure layer that applies traffic, identity, security, and telemetry policy to communication between workloads. Applications keep calling normal service addresses. Istio programs proxies in the request path so teams can change routing and policy without rebuilding every service.
Istio is most useful when many workloads need the same controls and the platform team can operate those controls as a product. It is not free reliability. A mesh adds a control plane, data-plane processing, configuration propagation, resource demand, and failure modes that must be measured.
Core invariant
Istiod distributes desired configuration; data-plane proxies enforce the configuration they have accepted. The control plane does not normally forward application requests.
Who is calling?
Workload identity
Istio identities are based on workload credentials rather than source IP alone. Mutual TLS authenticates both peers and encrypts the connection between participating proxies.
Where and how?
Traffic policy
Routing, timeout, retry, load-balancing, and outlier policies shape a request after the proxy has matched the destination and protocol.
Which workloads?
Policy scope
Namespace, host, subset, selector, and attachment rules decide where configuration applies. Valid YAML can still target the wrong scope.
What happened?
Observed evidence
Proxy metrics, access logs, traces, status, and configuration inspection explain the network path. Application evidence is still required for business correctness.
Review Kubernetes first if Services, namespaces, labels, and declarative resources are unfamiliar.
Choose the data plane before choosing features
Istio supports sidecar and ambient data-plane modes. Both use Istiod as the control plane, but they place request processing differently.
Proxy per workload pod
Sidecar mode
Each participating pod runs an Envoy sidecar. That proxy can enforce Layer 4 and Layer 7 behavior for the workload. Adding a sidecar normally requires pod injection and a restart, and each workload must reserve proxy resources.
Layered shared infrastructure
Ambient mode
A per-node ztunnel provides the secure Layer 4 overlay. Add a waypoint proxy when a workload needs Layer 7 routing, authorization, or telemetry. Workloads can join without putting a proxy in every application pod.
Choose from requirements, supported deployment shapes, feature status, and measured operating cost:
- use the base ambient secure overlay when workload identity, mutual TLS, Layer 4 authorization, and Layer 4 telemetry are sufficient;
- add an ambient waypoint where HTTP, gRPC, or other Layer 7 policy is required;
- use sidecar mode when its workload-local processing or supported deployment shapes fit the platform better;
- verify current support before adopting multi-cluster, VM, extension, or mixed-mode designs because those boundaries change across Istio releases.
The current Istio data-plane comparison documents the supported modes and feature boundaries. Do not copy benchmark numbers from another cluster into a capacity plan; measure your protocols, payloads, policies, telemetry, and resource limits.
Follow one request from configuration to enforcement
1 Platform API
Declare intent
Apply Istio or Kubernetes Gateway API resources with explicit hosts, namespaces, selectors, and policy attachment. Admission validation catches schema errors, not every semantic conflict.
2 Istiod
Compile and distribute
Istiod combines service discovery and policy, then sends proxy-specific configuration over xDS. Distribution is eventually consistent, so proxies may observe a rollout at different moments.
3 Data plane
Match the request
The proxy identifies protocol, destination, and the first matching route. It then applies destination policy, security policy, and any eligible resilience behavior.
4 Operations
Verify the outcome
Test positive and negative traffic, inspect proxy status and effective configuration, and correlate proxy evidence with application outcomes before expanding the rollout.
If Istiod is briefly unavailable, an already-configured proxy can continue serving from its last accepted configuration. New workloads, certificate operations, discovery changes, and policy updates still depend on a healthy control plane.
Bound retries inside one caller deadline
A retry creates another upstream attempt; it does not create more time. Treat the caller's end-to-end deadline as the outer budget and make every downstream timeout and retry fit inside it.
The lab uses a transparent upper-bound model:
- Configured attempts = 1 initial attempt + retry attempts.
- Timeout windows that fit = the smaller of configured attempts and
floor(route timeout / per-try timeout), with at least the initial attempt. - Maximum attempt pressure = incoming requests per second x timeout windows that fit.
This is a stress bound, not a throughput prediction. Actual retries depend on the selected retryOn conditions, response timing, backoff, remaining route deadline, and whether the request is still valid to repeat.
Load the route budget
The lesson-owned timeout, retry, operation, and failure model is loading.
Loading retry model
Preparing the route inputs and explicit arithmetic.
Use retries only for transient failures and operations that are safe to repeat or protected by an idempotency mechanism. A three-attempt policy can turn 1,000 incoming requests per second into as many as 3,000 upstream attempts per second while the dependency is already unhealthy.
Istio's current traffic-management documentation defines route ordering, timeouts, retries, destination rules, and circuit breaking.
Make route precedence and fallback explicit
A VirtualService evaluates HTTP routes in order. Put specific match conditions before the final catch-all route. A destination subset must be defined by a DestinationRule, and its labels must match real workload labels before traffic is sent to it.
The canary example uses fully qualified service hosts, defines stable and canary subsets, sends an internal tester to the canary, and keeps a weighted fallback for everyone else.
Roll out route resources as a state transition:
- deploy the new workload and wait for ready endpoints;
- create the subset and verify that proxies received it;
- add a small weighted route and observe user-facing and proxy metrics;
- increase traffic only while error, latency, saturation, and rollback signals remain healthy;
- retain a catch-all route so unmatched requests never fall into an accidental path.
Istio configuration distribution is eventually consistent. Applying a subset and a route in one command does not guarantee every proxy observes them atomically. The traffic-management best practices describe how to avoid 503 responses during route changes.
Decide when strict mTLS is safe to enforce
PeerAuthentication controls what the receiving workload accepts. In sidecar mode, PERMISSIVE accepts both mutual TLS and plaintext so teams can migrate incrementally; STRICT requires mutual TLS. Istio automatically uses mutual TLS when both participating workloads have mesh proxy capability and policy permits it.
The migration lab makes the cutover consequence visible. The model assumes the destination is enrolled and separates traffic into:
- mesh-capable sources, which reach the destination through Istio mutual TLS; and
- legacy sources, which still reach it as plaintext.
Change the legacy share, receiving policy, and data-plane mode. In ambient mode, require a waypoint when the scenario needs Layer 7 routing or authorization.
Load the workload coverage model
The lesson-owned traffic mix, receiving policy, and data-plane choices are loading.
Loading workload coverage
Preparing the mTLS traffic paths and cutover checks.
Move to STRICT only after workload coverage is complete and both allowed and denied paths have been tested. Mutual TLS proves workload identity and encrypts transport; it does not decide whether an authenticated caller may perform an action. Add AuthorizationPolicy for that decision.
In ambient mode, ztunnel provides mutual TLS for enrolled workloads and DISABLE is not supported. STRICT is useful for rejecting traffic that bypasses the mesh. See the current PeerAuthentication reference and security best practices.
Express resilience and authorization as narrow policy
The resilience example gives idempotent reads bounded retries while write requests get only the shared route timeout. It names the exact retry conditions rather than relying on a default that may not match the application.
The security example requires namespace-wide mutual TLS, denies access by default, and allows only the checkout service account to call the payment endpoint.
Before enforcement:
- run
istioctl analyzeagainst the complete candidate file set and the target cluster; - inspect
istioctl proxy-statusfor stale or rejected proxy configuration; - inspect the effective listener, route, cluster, endpoint, and secret configuration with
istioctl proxy-config; - test one request that should succeed and one that should fail for every policy boundary;
- monitor
pilot_total_xds_rejects, proxy response flags, upstream resets, retry attempts, mTLS identity, and certificate expiry; - keep application metrics and traces because proxy telemetry cannot prove that a payment, order, or state transition is correct.
istioctl analyze can inspect local files, the live cluster, or both. The official diagnostic guide documents the current commands and limits.
Adopt the mesh as a measured platform capability
Adoption
Start from one pain
Choose a bounded service group and one measurable outcome such as identity coverage, safer canaries, or consistent deadline enforcement. Do not begin by enabling every feature.
Governance
Own configuration scope
Define who may create gateways, routes, destination policy, and security policy. Review host ownership, namespace export, selectors, and cross-team attachment.
Operations
Rehearse mesh failure
Test control-plane loss, stale configuration, expired identity, rejected xDS, proxy resource pressure, bad routes, and a dependency that triggers maximum retry pressure.
An Istio rollout is ready to expand when the team can explain the effective request path, estimate retry amplification, prove mTLS and authorization outcomes, inspect the configuration inside a proxy, and roll back policy without guessing.