Reliability Engineering Workbench
What makes a service reliable?
A reliable service completes its critical user journey within an agreed SLO despite component failures, maintenance, and recovery. The journey is only as strong as its serial dependencies, shared failure domains, and tested restore path.
Step 1
Model
Shape the critical path, replica success rule, failure-domain placement, correlation assumption, MTBF, repair MTTR, detection and failover time, restore time, replication lag, RTO, RPO, SLO, and planned user impact.
Step 2
Observe
Compare independent and correlation-aware availability, translate the result into downtime and error-budget use, and see which architecture or recovery objective breaks first.
Step 3
Challenge
Start with the healthy plan, then inject common-mode failure, zone loss, slow recovery, maintenance overlap, and a cascading dependency. Use measured incident and restore-test data before treating the result as an SLO commitment.
Design the journey, then break its assumptions
Combine series dependencies, parallel replicas, failure domains, repair behavior, recovery objectives, and an SLO without pretending every failure is independent.
Scenario status
Objectives fit
Challenge the plan
Failure and operating scenarios
Scenarios change the evaluated month, not the assumptions you entered.
3 serial stages, 2 of 3 replicas required
Each stage uses parallel replicas. The complete journey succeeds only when every stage reaches its success rule.
Placement
3 domains · up to 1 replica/domain
User request
Journey begins
Critical stage 1
2-of-3
99.992% modeled stage
Critical stage 2
2-of-3
99.992% modeled stage
Critical stage 3
2-of-3
99.992% modeled stage
User outcome
99.975%
Independent redundancy versus correlated reality
Parallel replicas improve a stage; serial stages multiply their risk. Correlation assigns a share of replica downtime to one shared cause, where adding copies cannot help.
One replica
99.9%
MTBF 2,000 hr / MTTR 2 hr
Independent parallel stage
>99.999%
Probability at least 2 of 3 are healthy
Correlation-aware stage
99.992%
8% shared-failure assumption
Complete journey
99.975%
3 stage probabilities multiplied in series
Model equations
A(replica) = MTBF / (MTBF + MTTR)
A(parallel) = P(at least k of n replicas healthy)
U(stage) ≈ c × U(replica) + (1 − c) × U(parallel)
A(journey) = A(stage) ^ serial stages
The correlation blend is a planning approximation, not a substitute for failure telemetry, load tests, or a dependency-specific Markov model. Rounded outputs communicate model scale rather than contractual precision.
Turn availability into time the team can operate
99.95% monthly SLO
21.6 min
Total monthly error budget
Scenario month impact
12.7 min
59% of the error budget
Annual model downtime
2.6 hr
Steady-state estimate plus planned impact
Budget remaining
8.9 min
Reserve after this scenario
Error-budget consumption
12.7 min / 21.6 min
Includes 2 min planned impact and 0 sec from the selected challenge.
RTO: restore service
24 min / 30 min
2 min detect + 4 min fail over + 18 min restore
RPO: committed-data exposure
2 min / 5 min
Modeled from replication lag under the selected scenario.
Healthy plan
The configured path keeps its modeled error-budget reserve.
Use measured failure-domain correlation and incident data before treating the estimate as a commitment.
RTO
Met
RPO
Met
Quorum
Survives
What the current assumptions require
- 13 of 3 replicas remain; the 2-of-3 success rule can still complete.
- 2Shared failure accounts for 8% of replica unavailability; verify this assumption with incident and placement data.
- 3The 24 min recovery path fits the 30 min RTO.
- 4The 2 min replication exposure fits the 5 min RPO.
Planning rule: derive MTBF, MTTR, correlation, and recovery times from production evidence. Confirm the model with fault injection and restore tests before using it for an SLO commitment.