Reliability & Availability
Understand system reliability and availability: SLA levels, downtime calculations, and strategies for building fault-tolerant systems.
What is Reliability and Availability?
Reliability is the probability that a system performs correctly during a specific period. Its focus is preventing failures from happening in the first place. Availability is the percentage of time a system is operational and accessible. Its focus is minimizing downtime when failures do occur.
These qualities are related but different. A system can be available but unreliable if it responds with incorrect data. It can also be reliable but not highly available if it often goes down but works correctly whenever it is running. System designers therefore have to prevent failures, contain the failures that still happen, and restore service quickly.
99%
3.65 days/year
Basic internal or non-critical service
99.9%
8.76 hours/year
Common production starting target
99.99%
52.6 minutes/year
Requires tested redundancy and fast recovery
99.999%
5.26 minutes/year
Demands exceptional isolation and operations
1 MTBF
Prevent predictable failures
Use sound design, testing, capacity management, and maintenance to increase the time between incidents.
2 Observe
Detect the user impact
Alert on service-level indicators such as successful requests and latency, not only on machine health.
3 Isolate
Contain the blast radius
Apply timeouts, circuit breakers, bulkheads, and load shedding so one dependency cannot consume every shared resource.
4 MTTR
Restore and learn
Fail over or roll back quickly, verify recovery, then remove the conditions that allowed the incident to recur.
Availability Levels and SLAs
An availability target defines how much downtime a service can tolerate. Each additional "nine" sharply reduces the downtime budget and usually increases cost and operational complexity.
The original comparison used 99% as the basic target and 99.999% as the ultra-high target:
Reliability Patterns
Reliability patterns help systems handle failures gracefully instead of allowing one fault to become a full outage.
MTBF, MTTR, and Availability
MTBF: Mean Time Between Failures
MTBF is the average time between system failures. Improving it means preventing failures through:
- Better testing and quality assurance
- Redundant components and systems
- Regular maintenance and updates
MTTR: Mean Time to Recovery
MTTR is the average time needed to restore service after a failure. Reducing it means recovering faster through:
Automated monitoring and alerting
Quick rollback mechanisms
Clear incident-response procedures
Formula:
Availability = MTBF / (MTBF + MTTR)
The practical strategy is to increase MTBF and decrease MTTR.
Reducing MTTR
Real-world Examples
Netflix: 99.99% Availability
- Strategy: Chaos engineering, microservices, and multi-region deployment. The architecture can lose an entire AWS region without service impact, and production components are intentionally broken to test resilience.
Google: 99.99% SLA
- Strategy: Global load balancing, automatic failover, and Site Reliability Engineering practices. SRE teams focus on automation and reducing toil, while error budgets balance reliability against feature velocity.
AWS: 99.99% SLA
- Strategy: Multi-AZ deployment, auto-scaling, and health checks. Each service has specific SLA commitments, and Availability Zones are isolated from one another to prevent cascading failures.
Common Failure Modes
Building Reliable Systems Checklist
Design Phase
- Identify single points of failure.
- Plan for redundancy at each layer.
- Design for graceful degradation.
- Define SLA requirements early.
- Plan the monitoring and alerting strategy.
Implementation Phase
- Implement circuit breakers.
- Add comprehensive logging.
- Set up health checks.
- Create deployment automation.
- Test failure scenarios.
Turn an availability target into an operating budget
An SLA percentage becomes useful only after it is converted into time. The allowed downtime in an evaluation window is:
downtime budget = window duration × (1 - availability target)projected downtime = incident count × average recovery time
Use the lab to compare the promise with an incident plan. Notice how another nine can make an otherwise reasonable recovery time consume the entire budget.
Turn a target into an incident budget
Set the promise, then test whether incident frequency and recovery speed fit inside it.
99.907%
Projected availability for this window
Budget remaining
3.2 min
43 min
Allowed downtime
40 min
Projected downtime
$6,667
Revenue exposure
The operational lever: At 2 incidents, recovery must stay below 22 min per incident to hold this target.
Match redundancy to the failure boundary
Redundancy helps only when the healthy copy sits outside the boundary that failed. Two instances in one zone can survive a process crash, but not a zone outage. Two zones in one region can survive a zone outage, but not a region outage. A shared slow dependency can still affect every copy unless timeouts, circuit breakers, and fallbacks contain it.
Inject failures below and change the deployment shape. The outcomes are illustrative; the important question is whether traffic retains a tested healthy path.
Place redundancy where failure can happen
Inject a fault, choose a deployment boundary, and see whether traffic has somewhere healthy to go.
3. Add recovery controls
Survival map
Can traffic cross the failed boundary?
Zone A
Failed
Zone B
Standby
Not provisioned
Shared provider
Healthy
Traffic retains a usable path
A healthy copy exists outside the failed boundary, and automated routing moves traffic to it.
5%
Users affected
4 min
Time to stabilize
Replica remains available
Data posture
2.15x
Illustrative cost
Outcomes are an illustrative teaching model. Real impact depends on capacity headroom, replication lag, health-check thresholds, and tested recovery procedures.
Learn from representative incidents
These scenarios show how detection, isolation, redundancy, and recovery choices affect business outcomes.
Database Primary Failure
The main database goes down and automatically fails over to a replica.
Outcome: Service is restored quickly with minimal data loss, maintaining 99.995% monthly availability.
Lessons: Automated failover is crucial for database reliability; read replicas can serve traffic during primary failures; sub-minute monitoring prevents longer outages; and regular failover tests verify behavior under pressure.
Cascading Service Failure
An overloaded authentication service causes downstream failures.
Outcome: A major outage caused by missing circuit breakers loses $500K in revenue and damages customer trust.
Lessons: Circuit breakers prevent cascades; bulkheads isolate critical services; rate limiting protects upstream services; and service dependencies should be mapped and monitored.
Regional Datacenter Outage
An entire AWS region goes down, but a multi-region setup keeps the service running.
Outcome: Multi-region architecture limits user impact and maintains 99.99% availability.
Lessons: Multi-region deployment protects against regional failures; DNS failover enables automatic switching; cross-region data replication is essential; and the higher cost is justified only by critical availability requirements.
Memory Leak in Production
Gradual memory consumption causes rolling server crashes.
Outcome: A quick rollback prevents a complete outage, and memory-monitoring alerts are improved.
Lessons: Monitor memory with automatic alerts; use canary deployments to catch issues before full rollout; make rollback fast; and set resource limits so one process cannot crash a server.
DDoS Attack During Peak Hours
A massive botnet targets API endpoints during Black Friday.
Outcome: CDN protection and rate limiting absorb the attack, allowing the business to continue normally.
Lessons: A CDN provides the first line of defense; rate limiting by IP and API key prevents abuse; geo-blocking can reduce the attack surface; and a DDoS mitigation service should be ready.
Configuration Typo in Production
A database connection-string typo causes all new connections to fail.
Outcome: Health-check failures trigger an automated rollback and limit customer impact.
Lessons: Validate configuration before deployment; use automated health checks; stage rollouts to limit blast radius; and keep configuration as code for quick rollback.
Third-party Payment Provider Outage
Stripe becomes unavailable during checkout processing.
Outcome: A secondary provider prevents total checkout failure and lets most customers complete their purchases.
Lessons: Multiple providers improve resilience; graceful degradation preserves partial functionality; failed transactions can be queued for retry; and clear communication reduces frustration.
Kubernetes Node Failure
A worker node suffers a hardware failure during a traffic spike.
Outcome: Kubernetes redistributes pods automatically with no customer impact.
Lessons: Container orchestration provides automatic recovery; resource reservations leave capacity for redistribution; node auto-scaling handles capacity changes; and health checks trigger pod restarts.
Cache Stampede After Restart
A Redis restart causes every client to hit the database simultaneously.
Outcome: The database connection pool is exhausted, causing a partial outage until the cache warms.
Lessons: Warm caches deliberately; use jittered TTLs to avoid simultaneous expiration; protect the database with circuit breakers; and preload critical data when appropriate.