Scalability Basics
Master scalability fundamentals: vertical vs horizontal scaling, bottleneck identification, and scaling strategies with real examples.
What is Scalability?
Scalability is a system's ability to maintain or improve performance as workload increases. It means handling growth in users, data, and requests without breaking, slowing down, or becoming prohibitively expensive.
Why Scalability Matters
- The scalability test: A truly scalable system maintains or improves performance as load increases. If doubling users doubles response time or cost, the system is growing linearly rather than scaling well. True scalability aims for sub-linear cost growth and stable or better performance.
Core invariant
scale the resource that a measured workload is exhausting while preserving correctness and operability. User count alone does not justify a distributed architecture; request shape, data growth, and service targets do.
The Scalability Challenge
Your startup launches on Monday with 100 users. By Friday, a celebrity tweet sends you to 100,000 users. Does the system survive or crash? This is the scalability challenge every successful product faces.
What Happens Without Scalability
- Pokemon GO (2016): Crashed on launch day and could not handle viral success.
- HealthCare.gov (2013): Cost $2B and failed under load on day one.
- Knight Capital (2012): Lost $440M in 45 minutes because of poor deployment scaling.
- Common pattern: Slow pages lead to frustrated users, lost revenue, and damaged reputation.
What Good Scalability Enables
- Netflix: Moved from DVDs to 230M+ global streaming users.
- Zoom: Handled a 10x traffic spike during COVID-19 without major issues.
- Discord: Grew from gaming chat to 150M+ users across many communities.
- Common pattern: Stable performance leads to happy users, sustainable growth, and market strength.
Two Approaches to Scaling
When a system reaches capacity, there are two choices: make existing machines more powerful, called vertical scaling, or add more machines, called horizontal scaling. Each choice changes cost, complexity, and maximum capacity.
The original visual comparison used these relative values:
Vertical Scaling (Scale Up)
Vertical scaling adds more power to an existing machine: more CPU, RAM, disk, or network bandwidth.
Advantages
- Simple, with no architecture changes required.
- Low complexity because the same code runs on a larger machine.
- Avoids many distributed-system challenges.
- Easy to implement and maintain.
Disadvantages
Hardware limits prevent infinite growth.
High-end hardware becomes disproportionately expensive.
One machine remains a single point of failure.
Upgrades can require downtime.
Example: Upgrade an AWS EC2 instance from
t3.mediumwith 2 vCPU and 4 GB RAM tom5.4xlargewith 16 vCPU and 64 GB RAM, increasing cost from about $34/month to $560/month.
Horizontal Scaling (Scale Out)
Horizontal scaling adds machines to a resource pool and distributes load across them.
Advantages
- Nearly unlimited scaling potential.
- Linear cost growth as capacity is added.
- Redundant machines improve availability.
- Scaling can happen without downtime.
Disadvantages
Load balancing and state management increase complexity.
Applications often need stateless design and code changes.
Consistency and network latency become distributed-system concerns.
More machines create more operational overhead.
Example: Move from one server to ten servers behind a load balancer. At $34/month per server, capacity grows roughly 10x while unit cost stays constant.
When to Use Which
Choose vertical scaling when:
- The product is in its early stages with hundreds or a few thousand users.
- The application is not designed for distribution.
- A quick capacity gain is needed without code changes.
- Database performance is the current bottleneck.
Choose horizontal scaling when:
- The system is planning for more than 100K users.
- High availability and fault tolerance are required.
- Cost-effective growth at large scale matters.
- The workload is stateless and parallelizable.
Zero to Millions: The Complete Journey
Alex Xu's framework shows how systems evolve from a single user to millions, with each stage introducing specific challenges and architectural solutions.
Stage 1: Single Server (1-1,000 Users)
Architecture
- Web app, database, and cache on one server.
- DNS points to one IP address.
- Users connect directly to the server.
What breaks first
- CPU or memory limits at roughly 500-1,000 users.
- Database connection-pool exhaustion.
- The server is a single point of failure.
Stage 2: Database Separation (1K-10K Users)
Architecture changes
- Separate web and database servers.
- Scale both servers vertically.
- Connect the tiers over the network.
Benefits
- Web and database tiers can scale independently.
- Resource utilization improves.
- The system supports roughly 5-10x more users.
Stage 3: Load Balancer and Cache (10K-100K Users)
New components
- A load balancer distributes traffic across web servers.
- Redis or Memcached provides a cache layer.
- A CDN serves static assets.
- Redis or the database stores sessions.
Performance gains
- An 80-90% cache hit ratio.
- Static assets can load 10x faster.
- The application can scale horizontally.
Stage 4: Database Scaling (100K-1M Users)
Database strategy
- Primary-replica replication.
- Read replicas to scale read traffic.
- Write-through caching for consistency.
- Connection pooling.
Key decisions
- Split read and write traffic, often around a 90/10 ratio.
- Choose SQL or NoSQL according to access patterns.
- Plan explicitly for eventual consistency.
Stage 5: Microservices (1M+ Users)
Service decomposition
- Break the monolith into services.
- Use a service mesh for communication.
- Process asynchronous work with message queues.
- Give each service ownership of its database.
New challenges
- Distributed-system complexity.
- Service discovery and configuration.
- Monitoring and logging across services.
- Data consistency across service boundaries.
Key Takeaways
- Scale incrementally: Do not build for 1M users when the product has 1K.
- The database is usually the bottleneck: Plan the data layer carefully.
- Caching wins: As much as 80% of requests can often be served from cache.
- Stateless design: Enables horizontal scaling.
- Monitor everything: You cannot optimize what you cannot measure.
Real-world Scaling Examples
Netflix
Challenge: 230M+ subscribers, 15,000+ titles, and global streaming.
Solution: More than 700 microservices, AWS auto-scaling, and the Netflix Open Connect CDN.
Challenge: 2B+ users and 100B+ messages per day with 55 engineers.
Solution: Erlang for massive concurrency, a deliberately narrow feature set, and FreeBSD optimization.
Discord
Challenge: Real-time messaging, voice chat, and 150M+ active users.
Solution: Elixir clusters, Rust for performance-critical paths, and global edge nodes.
Identifying Bottlenecks
Before scaling, identify what limits the system's performance.
Load Testing and Metrics
Key Metrics to Monitor
- Response time: How long requests take.
- Throughput: Requests handled per second.
- Error rate: Percentage of failed requests.
- Resource usage: CPU, memory, disk, and network use.
- Concurrent users: Users active at the same time.
Load Testing Types
- Load test: Expected normal traffic.
- Stress test: Traffic beyond normal capacity.
- Spike test: Sudden traffic increases.
- Endurance test: Sustained traffic over an extended period.
- Volume test: Very large amounts of data.
Scaling Timeline
Common Scaling Mistakes
Premature Optimization
Building for 1M users when the product has 100. Start simple and scale when measurements show the need.
Not Measuring
Guessing at bottlenecks instead of using monitoring and profiling tools.
Only Vertical Scaling
Continuing to buy larger servers instead of designing for horizontal scale when the workload requires it.
🧮 Scaling Cost Calculator
Compare the costs of vertical vs horizontal scaling strategies
Inputs
Scaling Comparison
Vertical Scaling
Vertical scaling becomes 31.6x more expensive due to premium hardware costs
Horizontal Scaling
Horizontal scaling requires 10 servers but costs scale linearly
🎯 Real-world Scaling Scenarios
Explore how different companies handle scaling challenges
Metrics
Outcome
Without auto-scaling, servers crashed. Lost $2M in sales during 3-hour outage.
Lessons Learned
- Pre-scale infrastructure before known traffic spikes
- Implement auto-scaling with aggressive scaling policies
- Use CDN and caching to reduce server load
- Load test with 10x expected traffic
ScenariosClick to explore
Context
Black Friday traffic spike: 10x normal load in 1 hour
Metrics
Outcome
Without auto-scaling, servers crashed. Lost $2M in sales during 3-hour outage.
Key Lessons
- •Pre-scale infrastructure before known traffic spikes
- •Implement auto-scaling with aggressive scaling policies
- •Use CDN and caching to reduce server load
- •Load test with 10x expected traffic
1. E-commerce Flash Sale
Context
Black Friday traffic spike: 10x normal load in 1 hour
Metrics
Outcome
Without auto-scaling, servers crashed. Lost $2M in sales during 3-hour outage.
Key Lessons
- •Pre-scale infrastructure before known traffic spikes
- •Implement auto-scaling with aggressive scaling policies
- •Use CDN and caching to reduce server load
- •Load test with 10x expected traffic
2. Social Media Viral Post
Context
Unexpected viral content: 100x traffic in 30 minutes
Metrics
Outcome
Auto-scaling and circuit breakers prevented total failure. Minor degradation for 5 minutes.
Key Lessons
- •Horizontal scaling handles unpredictable spikes better
- •Circuit breakers prevent cascading failures
- •Graceful degradation maintains core functionality
- •Monitor social media for early viral detection
3. Gaming Launch Day
Context
New game release: Sustained high load for 48 hours
Metrics
Outcome
Queue system managed demand. Players waited but stayed engaged. Server costs 3x budget but revenue 5x.
Key Lessons
- •Queue systems can manage demand spikes gracefully
- •Communication with users during delays is crucial
- •Over-provisioning for launches can be profitable
- •Plan for success - prepare for higher than expected demand
4. Streaming Service Super Bowl
Context
Live sports streaming: 50M concurrent viewers during halftime
Metrics
Outcome
Pre-scaled infrastructure to 3x normal capacity. Minor buffering in some regions but overall success.
Key Lessons
- •Pre-event capacity planning is critical for live content
- •Global CDN distribution prevents regional overload
- •Adaptive bitrate streaming maintains quality during congestion
- •Real-time monitoring enables rapid response to issues
5. Banking System Payment Rush
Context
Tax deadline day: 10x increase in money transfers and payments
Metrics
Outcome
Database became bottleneck. Read replicas helped but write contention caused delays. No data loss.
Key Lessons
- •Financial systems require careful database scaling planning
- •Write-heavy workloads need different scaling strategies than read-heavy
- •Queue systems prevent data loss during overload
- •Regulatory compliance limits how aggressively you can scale
6. News Site Breaking Story
Context
Major world event: Traffic from 10K to 2M users in 20 minutes
Metrics
Outcome
Site initially slowed but auto-scaling kicked in. Static content caching saved the day.
Key Lessons
- •News sites need aggressive caching for breaking stories
- •Auto-scaling policies should be tuned for rapid response
- •Static content delivery is crucial for content sites
- •Performance during viral moments directly impacts ad revenue
7. Ride-sharing New Year's Eve
Context
Peak demand night: 50x normal ride requests in major cities
Metrics
Outcome
Surge pricing activated. Matching algorithm scaled but driver supply was the real bottleneck.
Key Lessons
- •Some bottlenecks are business model constraints, not technical
- •Predictable demand spikes should trigger pre-scaling
- •Algorithm efficiency matters more at extreme scale
- •Economic incentives (surge pricing) can balance supply/demand
8. Cloud Storage Backup Sunday
Context
Weekly backup day: 100TB+ of data uploaded simultaneously
Metrics
Outcome
Storage clusters handled load well but network bandwidth became bottleneck in some regions.
Key Lessons
- •Data-intensive applications need network capacity planning
- •Geographic distribution of storage improves performance
- •Retry mechanisms and resumable uploads are essential
- •Predictable batch workloads allow for scheduled scaling