Circuit Breakers
Design circuit breakers with explicit failure classification, rolling-window trip policy, bounded recovery probes, and observable fallback behavior.
What is a circuit breaker?
A circuit breaker is a stateful policy around a remote call. It observes recent outcomes, temporarily blocks calls when failure or slowness crosses a boundary, and admits a limited set of probes before restoring normal traffic.
The core invariant is: when a dependency is unlikely to succeed, the caller must stop spending shared capacity on repeated doomed attempts. The breaker contains the failure; it does not repair the dependency or make a fallback correct.
Measure normal calls
Closed
Calls reach the dependency while the breaker records failures, timeouts, and slow responses in a bounded window.
Fail fast
Open
Calls do not reach the unhealthy dependency. The caller returns a controlled error or an explicitly designed degraded response.
Probe recovery
Half-open
Only a small number of representative calls pass. Their outcomes decide whether normal traffic returns or the circuit opens again.
Put the breaker at the dependency boundary
The breaker belongs around a specific remote operation or failure domain. It needs to see the same timeout, status, and exception semantics that the caller experiences.
Protected synchronous request path
The breaker changes whether the remote call is attempted; the caller still owns the user-visible result.
Owns the budget
Caller
Starts with an end-to-end latency budget and decides whether a degraded response is acceptable for this operation.
Bounds attempts
Timeout and retry policy
Limits each wait and retries only classified transient failures with backoff, jitter, and an aggregate retry budget.
Contains repeated failure
Circuit breaker
Aggregates recent outcomes, fails fast while open, and controls the number of recovery probes.
Independent failure domain
Remote dependency
May be healthy, slow, overloaded, throttling, partially available, or completely unreachable.
Create separate breakers when dependency instances, shards, regions, endpoints, or operations can fail independently. One global breaker can block healthy work because an unrelated path is failing.
Open from evidence, not from one alarming request
A production breaker usually needs both a measurement window and a minimum call count. The minimum prevents a tiny sample from producing a mathematically correct but operationally bad decision. Slow calls deserve their own boundary because a dependency can return successful responses while exhausting caller threads or connections.
Use the lab to compare a healthy dependency, a slow brownout, a hard outage, and a low-volume fluke. Then change the rates and window to see which signal opens the circuit.
Decide which outcomes count
- Count connection failures, timeouts, dependency
5xxresponses, and workload-specific throttling only when the protected contract treats them as dependency failures. - Ignore business rejections such as an invalid payment method or a missing required field. Retrying or opening a breaker cannot make those requests valid.
- Treat slow successful calls separately from hard failures.
- Record breaker rejections separately from dependency failures. An open breaker is a caller policy state, not fresh evidence that the dependency failed again.
Resilience4j's CircuitBreaker documentation describes count- and time-based sliding windows, a minimum number of calls, separate failure and slow-call rates, an open wait, and a bounded number of half-open calls. These are configurable policies, not universal defaults.
Implement one explicit state machine
The state transition must be concurrency-safe: one observation opens the breaker once, later failures must not keep extending the open timer accidentally, and only the configured number of half-open probes may pass.
The example separates three concerns:
allowdecides whether a call may reach the dependency.recordclassifies an actual dependency outcome.- the policy defines window size, evidence minimum, trip boundaries, open duration, and probe count.
Use a maintained library in production when one exists for the runtime. Confirm its exception predicates, concurrency model, metrics, and retry integration instead of assuming the defaults match the workload.
Recovery needs new evidence
Time passing is permission to test, not proof that the dependency recovered. Half-open exists to prevent a recovering service from receiving the same traffic flood that contributed to its failure.
Choose a dependency condition, probe count, and recovery contract. The lab evaluates failures, slow calls, and latency together before deciding whether to close.
Make probes representative and bounded
- Exercise a safe operation with the same network path and dependency capacity as real traffic; a shallow health endpoint can produce false confidence.
- Keep probe concurrency small enough that a partial recovery is not overwhelmed.
- Require enough evidence for the risk of the operation. Two successful reads may not justify restoring a high-volume payment write path.
- Reopen immediately when the recovery contract fails, then wait before another probe cohort.
- Watch the first restored traffic for a fresh breach instead of treating
closedas permanent health.
Microsoft's Circuit Breaker pattern defines closed, open, and half-open behavior and emphasizes that half-open limits trial requests during recovery.
Compose four controls without duplicating work
Each resilience control protects a different boundary. Ordering and ownership matter.
1 Timeout
Bound one attempt
Stop waiting before the caller's end-to-end latency budget is exhausted.
2 Retry
Retry only transient faults
Use a small attempt limit, backoff, jitter, idempotency, and one aggregate retry budget.
3 Breaker
Stop repeated doomed calls
Open after enough recent evidence and make retry logic stop when the breaker rejects.
4 Bulkhead
Reserve independent capacity
Keep one dependency, tenant, or operation from consuming every worker or connection.
Do not stack invisible retry policies in an SDK, service mesh, application client, and job runner. Three retries at two layers can create nine dependency attempts. Azure's transient-fault guidance recommends finite retries, per-attempt timeouts, backoff with jitter, and an aggregate retry budget.
A fallback is a product contract
Failing fast protects the system only if the caller handles the result deliberately.
Safe fallback candidates
- Return a last-known-good catalog entry with its freshness visible.
- Omit a noncritical recommendation panel while keeping checkout available.
- Queue an idempotent, durable command when delayed completion is acceptable.
- Return an explicit
503 Service Unavailablewith bounded retry guidance.
Dangerous fallback candidates
- Report a payment as accepted when authorization never completed.
- Return stale authorization, inventory, identity, or quota data as current.
- Call another dependency chain that has the same failure cause or tighter capacity.
- Hide degraded data from callers, telemetry, and customer support.
A circuit breaker changes availability and latency; it cannot preserve correctness by itself. Name the freshness, durability, and feature loss of every degraded path.
Operate the policy with evidence
Monitor enough context to tell a contained incident from a badly tuned breaker:
- state and every state transition, labeled by breaker and dependency;
- permitted, failed, slow, rejected, and ignored calls;
- fallback volume and fallback errors;
- window volume, failure rate, slow-call rate, and dependency latency;
- time spent open, probe outcomes, recovery attempts, and state flapping;
- caller saturation, queueing, end-to-end success, and user-visible latency.
Alert on a sustained open state, widespread fallback use, or repeated open/half-open flapping. Do not page on each rejected call: rejection is the intended behavior while the circuit is open.
Test the policy with timeout, connection-reset, throttling, slow-success, partial recovery, and concurrent probe scenarios. The AWS circuit breaker guidance also stresses fail-fast behavior, recovery detection, concurrency, manual override, and observability.