Service Mesh
Learn service mesh control and data planes, workload identity, bounded resilience, authorization, failure modes, and adoption decisions.
What is a service mesh?
A service mesh is infrastructure that applies networking policy to communication between services. Applications still call ordinary service addresses. Proxies or other traffic-enforcement points handle selected connections so a platform team can apply identity, authorization, routing, resilience, and telemetry consistently.
The mesh matters when the same internal, or east-west, communication rules must work across many workloads and teams. It is not a replacement for application correctness, a Kubernetes Service, an API gateway, or a healthy operating model.
Core invariant
The control plane distributes desired state. The data plane handles application traffic and enforces the configuration it has received. A control plane should not sit synchronously in every application request.
Where and how?
Traffic policy
Route to healthy endpoints, split traffic between versions, and apply bounded timeouts, retries, and connection limits.
Who is calling?
Workload identity
Authenticate participating workloads and encrypt proxy-to-proxy traffic with mutual TLS. Authorization remains a separate decision.
What happened?
Network evidence
Measure request rate, errors, duration, connections, and policy outcomes at the traffic boundary without requiring one telemetry library in every runtime.
Who operates it?
Platform cost
Budget for data-plane CPU, memory, and latency plus control-plane availability, certificate rotation, configuration review, upgrades, and incident response.
Review Kubernetes first if Pods, Services, namespaces, and NetworkPolicy are unfamiliar.
Trace configuration separately from requests
A mesh has two related paths. The configuration path can change while application traffic is running, so updates are not necessarily observed by every enforcement point at the same instant.
1 Platform API
Declare intent
Operators define routing, identity, telemetry, and authorization policy for an explicit service, namespace, route, or workload.
2 Control plane
Distribute state
The control plane combines policy with service discovery and sends each data-plane component the configuration it needs.
3 Data plane
Enforce locally
Traffic enforcement points authenticate, authorize, route, limit, or observe a request without asking the control plane to approve that request synchronously.
4 Operations
Verify outcome
Teams inspect effective configuration, allowed and denied calls, proxy health, and application results before expanding a policy.
Use the topology to compare a healthy request with control-plane loss, an authorization denial, an unmeshed caller, and a failed outbound enforcement point. Select a component to inspect what it owns.
The control-plane outage scenario preserves the last accepted data-plane state in this model. That can keep existing traffic moving, but it does not make new discovery, policy, identity, or configuration changes safe. A failed data-plane component is different: it is already on the request path and can directly affect availability.
The current Istio data-plane documentation describes both per-workload sidecars and a shared ambient design. The Linkerd architecture documents a sidecar data plane and separate control-plane services.
Treat resilience policy as bounded extra work
Traffic controls are useful only when their failure behavior matches the application:
- Timeouts bound how long a caller waits, but every downstream timeout must fit inside the caller's end-to-end deadline.
- Retries create additional upstream attempts. Use them for transient failures and operations that are safe to repeat or protected by application idempotency.
- Circuit breaking and connection limits reject or defer work before an unhealthy dependency consumes every caller, connection, or queue slot.
- Traffic splitting supports canaries and migrations, but a percentage is not a safety check; health gates and rollback still decide whether to continue.
If a route allows two retries, one incoming request can produce at most three upstream attempts when every eligible attempt fails. At 1,000 incoming requests per second, that policy can permit 3,000 attempts per second before accounting for backoff or the outer deadline. Size the dependency for the failure path, not only the healthy average.
The example applies retries only to idempotent reads. Writes keep the same outer timeout but are not repeated by the mesh.
Istio's current traffic-management reference and Linkerd's retry guidance both warn that retry policy changes latency and load. The application must still own fallbacks and business-level recovery.
Separate encryption, authentication, and authorization
Mutual TLS performs two jobs between participating data-plane peers: it encrypts the connection and authenticates both presented workload identities. It does not prove that an authenticated caller may read an invoice, create an order, or administer a service.
Authorization should start from an explicit trust boundary:
- inventory the real callers and destination operations;
- enable identity and observe which calls are actually meshed;
- define least-privilege allow rules for stable workload identities;
- test one call that must succeed and one that must fail;
- enforce default deny only after coverage and recovery procedures are proven.
This sidecar-mode Istio example requires mutual TLS for the payments workload, creates a default-deny authorization boundary, and allows only the checkout service account to call POST /v1/charges.
Policy attachment differs across data-plane modes and products. In Istio ambient mode, Layer 7 policy for a Service is attached through a waypoint and targetRefs; do not copy a sidecar selector without checking the current mode.
See the official Istio AuthorizationPolicy reference and security best practices. Consul expresses service-to-service access through service intentions.
Use proxy telemetry for network behavior, not business truth
The data plane observes connections and request metadata at the network boundary. It can reveal a caller-callee edge even when the application has no shared instrumentation library.
Rate
Demand
Requests or connections per second by source, destination, and route.
Errors
Outcome
Response classes, resets, denials, and other proxy-visible failures.
Duration
Latency
Use p50 with tail percentiles such as p95 and p99; averages hide outliers.
Attempts
Amplification
Compare caller requests with upstream attempts to expose retries.
Proxy evidence cannot prove that a payment was captured once, inventory remained correct, or a response contained the right data. Keep application metrics, traces, and domain events for those claims.
Control telemetry cost deliberately:
- bound high-cardinality labels such as raw paths, user IDs, and untrusted headers;
- sample traces from a documented policy rather than collecting every request forever;
- redact credentials and sensitive payload fields from access logs;
- export long-term evidence to an owned observability system;
- monitor certificate expiry, rejected configuration, stale proxies, and control-plane distribution lag as mesh health signals.
Choose the smallest architecture that solves the requirement
Service count alone is a weak adoption signal. A small regulated system may need workload identity, while a large single-runtime system may already have safe shared libraries. The stronger question is whether a cross-cutting internal traffic requirement is repeated across workloads and whether someone can operate the new failure domain.
Use the decision lab to change the primary requirement and the available operating owner. The comparison shows which controls are covered, which remain gaps, and whether the technically matching option is ready to run.
Load the service-mesh decision model
The lesson-owned requirements and operating-readiness model is loading.
Loading decision model
Interpret a blocked result literally: the requirement may justify a mesh, while the current team cannot safely own one. Narrow the first use case, select a managed option, or build the operating capability before putting the mesh on every request path.
Kubernetes already provides service discovery, load distribution, Gateway API implementations, and L3/L4 NetworkPolicy building blocks. Its networking overview helps separate those responsibilities from the extra Layer 7, identity, and telemetry features a mesh may add.
Adopt a mesh as a measured platform product
Sidecar or ambient
Istio
Istio supports Envoy sidecars and an ambient mode with per-node ztunnel plus optional waypoints for Layer 7 processing. Check the current feature boundary before choosing a mode.
Focused sidecar mesh
Linkerd
Linkerd uses a purpose-built Rust proxy beside meshed workloads and separates outbound routing and resilience from inbound authorization.
Discovery and intentions
Consul service mesh
Consul combines service discovery with mesh gateways, proxy integration, identities, and intentions that authorize source-to-destination communication.
Product selection follows requirements, supported environments, data-plane placement, policy semantics, upgrade strategy, and measured cost. Do not choose from feature counts alone.
Use a staged operating sequence:
- Name one problem. Choose identity coverage, a canary boundary, or consistent telemetry, and define the user-visible success metric.
- Measure a baseline. Record latency distributions, CPU, memory, request rate, error rate, certificate state, and incident workload before enrollment.
- Pilot a bounded path. Enroll a non-critical caller and destination, then test allowed, denied, overloaded, stale-configuration, and control-plane-loss states.
- Prove rollback. Rehearse removing policy, bypassing or repairing a failed data plane, restoring trust material, and returning to the previous traffic path.
- Expand by evidence. Add workloads only while ownership, dashboards, policy review, capacity headroom, and upgrade procedures remain credible.
Benchmark the exact protocols, payloads, telemetry, policies, and deployment mode used in production. Istio publishes a performance methodology and reference results, but another cluster's numbers are not a capacity plan.