HashiCorp Consul | System Designer

What is Consul?

HashiCorp Consul is a distributed control plane for finding, securing, and connecting network services. A workload registers a stable service identity instead of publishing an address directly to every caller. Consul records each instance in a service catalog, tracks health observations, and exposes discovery through DNS, HTTP, and service-mesh proxies.

This matters because service locations change during deploys, scaling, and failures. Without discovery, every caller needs a separate mechanism for finding current addresses. Consul centralizes that control-plane state while application traffic still flows directly or through a data-plane proxy.

The core invariant is catalog state is evidence, not traffic. Registration says an instance exists, a health check says what was last observed, a discovery interface returns a view with a specific freshness contract, and the caller or proxy still owns connection, timeout, retry, and load-balancing behavior.

Identity and location

Service catalog

Stores nodes, service instances, tags, metadata, and health observations. Catalog writes are authoritative control-plane changes replicated by the server cluster.

Read contract

Discovery interface

DNS and HTTP expose catalog results with different consistency, caching, filtering, and failure behavior. The interface is part of the application contract.

Traffic policy

Service mesh

Connect adds workload identity, mutual TLS, proxy-mediated routing, and intentions that authorize new service connections or requests.

Place agents and servers according to their responsibilities

Every Consul process runs as an agent, but agents have two modes:

A client agent runs near workloads. It registers local services, executes local health checks, answers local DNS or HTTP requests, participates in LAN gossip, and forwards authoritative operations to servers.
A server agent does the same agent work and also stores replicated state, participates in Raft consensus, handles catalog and ACL writes, and can lead the datacenter's control plane.
A service workload is not a Consul voter. It talks to the local agent or a configured proxy and remains responsible for its business protocol.
A sidecar or gateway proxy is part of the data plane when service mesh is used. It receives configuration and identity from Consul but forwards application traffic without sending each request through the Consul servers.

One service registration and discovery path

The local agent is the workload-facing boundary; server agents own authoritative datacenter state.

Service process

Payments workload

Registers payments, owns a readiness endpoint, and sends application traffic to selected downstream addresses.

Node boundary

Local client agent

Runs the check, serves local DNS and HTTP, caches supported reads, and forwards catalog changes to a server.

Control plane

Consul server cluster

Commits registration, health, ACL, and configuration state through a Raft quorum.

Data plane

Healthy service instance

Receives application traffic directly or through a service-mesh proxy after discovery and authorization succeed.

Run three or five dedicated server agents for a normal production datacenter. Three voters tolerate one failure; five tolerate two. More voting servers increase quorum size and write coordination, so they are not a generic read-scaling mechanism. Client agents can scale independently because they are outside the Raft peer set.

Treat registration and discovery as separate contracts

Service registration creates a catalog record with a stable name and a changing instance location. A normal discovery cycle is:

1
Desired presence
Register locally
The workload, scheduler, or deployment system registers one service instance with the client agent on its node.
2
Measured state
Observe health
The local agent runs active checks or receives a TTL update, then reports the resulting status to the server cluster.
3
Read contract
Query by identity
A caller asks for payments, optionally restricts results to passing checks, and chooses DNS, HTTP consistency, or agent caching semantics.
4
Application policy
Connect with bounds
The caller or proxy selects an address and applies connection deadlines, retry limits, load balancing, and circuit breaking.

The catalog is not a load balancer by itself. DNS returns records for the resolver or client to choose from. The HTTP API returns structured instances, leaving selection to application code. Service mesh proxies can apply routing configuration in the data plane, but Consul does not make a failed business operation idempotent.

Design health checks around what traffic needs

A Consul check reports passing, warning, or critical. That status is useful only when its probe matches the traffic contract:

Application-aware

HTTP or gRPC readiness

Best when the endpoint can answer whether this instance should receive new work. Keep the probe bounded and avoid making every optional dependency a hard requirement.

Transport-only

TCP reachability

Confirms that a socket can be opened. It is cheap and useful for simple services, but a hung process may still accept connections.

Push-based

TTL heartbeat

Requires a trusted process to renew status before a deadline. Missing renewals become critical, so the updater and its failure boundary must be explicit.

Bound transitions and cleanup

interval and timeout bound each active probe. The user-visible removal delay also includes network scheduling and failures_before_critical.
success_before_passing prevents one transient success from immediately returning a recovering instance to traffic.
deregister_critical_service_after deletes a service that stays critical for the configured duration. Use it as orphan cleanup, not as the first failure detector.
An alias check mirrors another node or service's health. It couples failure domains deliberately and can hide the actual dependency if overused.
Script checks execute local commands and expand the agent's attack surface. Prefer protocol checks when they can express the same contract.

Service registration with bounded readiness transitions

Health status is not instantaneous truth. During a network partition, agent restart, or server outage, a caller may see a last-observed state. Pair passing-only lookup with short connection deadlines and bounded retries so stale discovery does not become a retry storm.

Keep gossip membership and Raft consensus separate

Consul uses two distributed protocols because they answer different questions:

Who appears reachable?

Serf gossip

The LAN pool shares agent membership and distributes failure suspicion within one datacenter. The WAN pool connects server agents across federated datacenters. Gossip is fast and decentralized, but membership alone cannot commit catalog state.

What state is committed?

Raft consensus

Server agents in one datacenter elect a leader and replicate an ordered log. Catalog, ACL, configuration, and other writes require a majority of the configured voters.

A node can appear alive in consul members while the server peer set has no quorum. Conversely, a transient gossip suspicion can occur while Raft remains healthy. Use consul operator raft list-peers and a controlled read/write probe to establish consensus state instead of inferring it from membership.

2 quorum

3 servers

Tolerates 1 failed voter

3 quorum

5 servers

Tolerates 2 failed voters

0 votes

Client agents

Scale outside the Raft peer set

Need leader

Catalog writes

And a majority acknowledgement

Choose freshness, latency, and outage behavior explicitly

Consul's read modes trade freshness for scalability and availability:

DNS

Consul DNS uses stale reads by default. Any reachable server can answer from its replicated state, which spreads load and can preserve lookup during leader loss. The trade-off is an unbounded stale window. Set non-zero DNS TTLs only after measuring how the application's resolver caches records; many language and operating-system resolvers add their own behavior.

HTTP API

Default reads use the leader-lease path and fit most API lookups.
Consistent reads require the leader to verify authority with a quorum. They add a round trip and should be reserved for decisions that cannot tolerate the default read window.
Stale reads allow a follower to answer. They scale reads and can work without a leader, but the caller must accept potentially old state.
Blocking queries use an index or hash to wait for change. A returned request does not prove the value changed, and clients must handle timeouts and an index that moves backward after recovery.

Agent cache

Supported HTTP endpoints can use ?cached. Simple caches honor Cache-Control freshness directives; stale-if-error can deliberately return old data when servers are unreachable. Background-refresh caches consolidate many local blocking queries into one server watch. Inspect X-Cache, Age, X-Consul-KnownLeader, X-Consul-LastContact, and X-Consul-Effective-Consistency where available.

A tight dns_config.max_stale or discovery_max_stale can worsen an overloaded cluster. Followers that exceed the bound forward cheap stale reads to the leader, increasing the pressure that caused the lag. Prefer explicit client freshness policy, agent caching, caller rate limits, and server telemetry.

Secure the control plane before onboarding services

A production deployment needs encrypted gossip, authenticated RPC and HTTP, and ACLs with a deny-by-default policy. Treat these as separate boundaries:

Gossip encryption protects LAN and WAN gossip messages with a shared key. Rotate the key through the keyring workflow; do not confuse it with TLS identity.
TLS authenticates and encrypts agent RPC and HTTP connections. Verify server hostnames and maintain a certificate rotation runbook.
ACLs authorize control-plane operations. Bootstrap once to obtain a global-management token, secure that token, and create scoped identities for agents, services, DNS readers, automation, and operators.
Agent tokens authorize an agent's internal node operations. Service registration and application queries should use identities scoped to their own resources.
The anonymous token applies to requests without a token. Keep it minimal when unauthenticated DNS must be supported.

Server baseline with explicit encryption and ACL boundaries

The example makes trust boundaries visible but is not a drop-in deployment. Replace discovery, certificates, and secrets through the platform's secret and identity systems. Validate with consul validate, stage certificate rotation, and never commit the bootstrap or agent token into a configuration repository.

Add service mesh only for a clear traffic-policy need

Consul service mesh, historically called Connect, adds an identity-aware data plane. A sidecar or gateway proxy receives certificates, upstream discovery, and traffic configuration from the local Consul boundary.

Service-mesh authorization path

Consul distributes identity and policy; proxies enforce new traffic in the data plane.

Source identity

Payments proxy

Presents a short-lived workload certificate and verifies the destination's trust domain before establishing mutual TLS.

Configuration

Consul control plane

Publishes CA roots, leaf identity, discovery results, and matching intentions through local agent caches and blocking updates.

Destination policy

Ledger proxy

Authenticates the source certificate and enforces the destination's L4 connection or L7 request intention.

Application

Ledger workload

Receives only proxy-authorized traffic, while still enforcing business authorization and request validity.

Allow payments to open new traffic to ledger

Intentions authorize by source and destination service identity. L4 intentions govern new TCP connections; L7 intentions can govern new HTTP requests when the protocol is configured appropriately. Changing an intention does not terminate an existing connection. Cached intention data can continue authorizing during a control-plane disconnect, so revocation expectations must include connection lifetime and cache behavior.

Service mesh adds proxies, certificate authority operations, xDS configuration, gateways, and another debugging surface. Use it when identity-aware encryption, authorization, or traffic policy justifies that cost. Ordinary DNS or HTTP discovery is often enough for a trusted network with simpler requirements.

Federate datacenters without building one global quorum

Each Consul datacenter has its own server cluster and its own Raft quorum. WAN federation connects server agents so Consul can support cross-datacenter queries, service-mesh communication, and replicated global control-plane data.

Traditional WAN federation

Server agents communicate directly across the WAN. This requires routing and firewall exposure between servers and makes the server network boundary part of the cross-datacenter design.

Federation through mesh gateways

Mesh gateways provide an explicit path between datacenters and reduce the need for every remote service or server address to be directly routable. They do not merge two Raft clusters, provide application data replication, or automatically move user traffic during an outage.

Design these behaviors explicitly:

which datacenter is primary for ACL, intention, and service-mesh CA replication;
which tokens authorize replication into secondary datacenters;
whether remote service lookup is permitted and how local-first behavior works;
what happens when the WAN path is partitioned but both local datacenters retain quorum;
how application data and user traffic fail over independently of Consul; and
how the primary datacenter is recovered without overwriting healthy secondary state.

Recover quorum with evidence, not membership guesses

Loss of quorum blocks new Raft log entries and peer-set changes. The safe response depends on whether a majority still exists:

1
Observe
Establish the peer set
Capture raft list-peers, current leader state, server logs, network reachability, Autopilot state, disk health, and the last successful snapshot.
2
Stabilize
Contain the cause
Stop automated restarts and scale actions that churn voters. Remove request overload, repair network reachability, or replace failed infrastructure one node at a time.
3
Decide
Choose recovery
If quorum remains, use the normal peer lifecycle. If it is lost, follow HashiCorp's documented outage-recovery procedure or restore a tested snapshot into a controlled replacement cluster.
4
Validate
Prove service
Verify leader stability, applied-index progress, catalog writes, ACL resolution, discovery freshness, health transitions, federation, and application connectivity.

Take automated Consul snapshots, copy them outside the failure domain, protect them as sensitive artifacts because they include ACL state, and rehearse restoration. A snapshot is not useful until its recovery point, encryption, access policy, and restore time have been measured.

Do not enable bootstrap mode on multiple servers to force progress. Do not delete Raft data or add random replacement peers while the surviving peer set is unknown. Those actions can create divergent state and destroy the best recovery source.

Operate from control-plane and user-path evidence

Monitor Consul as a dependency with separate service-level indicators:

Leader + quorum

Consensus

Election stability, peer health, applied index

Fresh + bounded

Discovery

DNS/API latency, errors, cache age, result count

Truthful state

Health

Transition delay, flapping, critical age, check load

Authorized path

Mesh

xDS freshness, certificate expiry, denied traffic

Correlate at least these signals:

Servers: CPU, memory, disk latency and capacity, RPC rate limits, request latency, Raft commit/apply latency, leader last-contact, elections, snapshots, and Autopilot health.
Agents: LAN membership churn, failed joins, check execution duration, DNS and HTTP latency, cache age, blocked-query count, and server reconnects.
Catalog: registration churn, flapping checks, orphaned critical services, unexpected tags or metadata, and passing-instance count by service and zone.
Security: ACL denials, token expiry or rotation failure, gossip key state, TLS expiry, CA rotation, intention changes, and privileged operator actions.
User path: service connection errors, stale-address failures, retry rate, proxy configuration age, remote-datacenter latency, and time to remove or restore an endpoint.

Inspect membership, consensus, and one cached discovery path

Run synthetic probes for DNS, default HTTP reads, stale or cached reads, a controlled catalog write, and service-mesh connectivity. A single green /v1/status/leader probe does not prove the discovery and data paths are usable.

Roll out Consul with an explicit rollback path

1
Foundation
Secure servers first
Deploy three or five dedicated servers, TLS, gossip encryption, ACL bootstrap, snapshots, telemetry, and tested quorum recovery before registering production services.
2
Compatibility
Canary local discovery
Register one non-critical service, compare Consul DNS and HTTP answers with the current registry, and measure resolver caching, health transitions, and stale-address behavior.
3
Containment
Move callers in slices
Route a bounded caller cohort to Consul, hold on discovery errors or retry growth, and keep the former endpoint current until peak traffic and a server failure pass.
4
Policy
Add mesh selectively
Onboard one service edge, establish deny-by-default intentions, verify certificate and proxy failure behavior, then expand by dependency rather than by namespace.

Rollback must restore a known service address or prior resolver path, drain or refresh client caches, and prevent connection retries from amplifying the event. During a registry migration, dual registration creates two sources of truth; define which system is authoritative and continuously compare membership and health before cutting over.

Know when Consul is the wrong trade-off

Consul is a strong fit when heterogeneous runtimes need one service catalog, VM-centric DNS discovery, cross-platform service identity, or service-mesh policy beyond one orchestrator. It may be unnecessary when:

one Kubernetes cluster's native services and network policy already satisfy the discovery and authorization contract;
a cloud load balancer or managed registry provides the required health, routing, and regional behavior with less operational ownership;
the team cannot staff quorum recovery, ACL and certificate lifecycle, upgrades, snapshots, and proxy debugging; or
the use case is application configuration distribution rather than service networking, and a purpose-built configuration system has a clearer ownership model.

The service catalog, KV store, and mesh share one operational control plane. Adding features increases coupling to that plane. Start with the smallest Consul capability that closes a real requirement and add mesh, federation, or dynamic configuration only after the previous boundary is observable and recoverable.

Read the current primary documentation

Consul architecture distinguishes the control plane, data plane, persistence, and network coordinates.
Consul agents defines client and server responsibilities and the recommended server count.
Gossip and consensus explain membership, the Raft peer set, quorum, and failure tolerance.
DNS discovery, consistency modes, agent caching, and blocking queries define the read contracts and response evidence.
Health check configuration defines supported checks, thresholds, TTLs, and critical-service cleanup.
ACL bootstrap and ACL tokens define the initial management boundary and scoped agent or service identities.
Consul service mesh and service intentions define mTLS identity, proxy caching, and L4 or L7 authorization behavior.
WAN federation and disaster recovery define multi-datacenter and snapshot-recovery boundaries.
Server resource requirements explains leader-path pressure, Raft timing, stale-read scaling, and request limits.

Check the exact documentation and release notes for the Consul version and deployment runtime you operate. Defaults, supported integrations, and Enterprise-only features change across releases.

Test the production decisions

The assessment checks agent roles, gossip versus Raft, health semantics, read consistency, quorum arithmetic, service-mesh authorization, ACL handling, and overload response.

No quiz questions available

Could not load questions file