Bottleneck Analysis
Master bottleneck identification: CPU, memory, I/O, and network analysis techniques for system optimization.
What is bottleneck analysis?
A bottleneck is the stage that limits an important system outcome, such as completed requests per second or checkout latency. Bottleneck analysis is the evidence-driven process of finding that limiting stage, changing it safely, and proving that the end-to-end outcome improved.
A busy component is not automatically the bottleneck. A CPU can be 90% utilized while requests are actually waiting for database connections, and a nearly idle application can still be slow because a downstream service is timing out. Start with the user-visible symptom, measure the request path, and test one causal hypothesis before optimizing.
The core invariant is measurable: changing the active constraint should improve the target end-to-end signal under the same workload. If throughput and latency do not move, the change did not address the active bottleneck.
This lesson builds on Latency vs Throughput. Review it first if request rate, percentiles, or queueing are unfamiliar.
Measure the system before changing it
Optimization without a baseline turns an incident into guesswork. Define one user path and one controlled workload window, then collect enough context to compare before and after.
1 Outcome
Name the symptom
State the affected path, population, time window, request rate, error ratio, and p95 or p99 latency. "The service is slow" is not a testable claim.
2 Baseline
Reproduce the workload
Hold request mix, payload size, cache state, concurrency, and dependency behavior as stable as possible. Record the release and configuration.
3 Evidence
Correlate the path
Use traces to divide active work from waiting. Compare queue age, pool waits, CPU run queues, garbage-collection pauses, I/O latency, and dependency spans.
4 Cause
Perturb and verify
Change one suspected constraint in a canary or repeatable test. Rerun the same workload and confirm that the end-to-end outcome moves.
A useful baseline records four kinds of evidence:
- Demand: arrivals per second, concurrent requests, request mix, payload sizes, and burst duration.
- User outcome: completed throughput, errors, timeouts, and latency percentiles for the affected path.
- Waiting and work: queue depth and oldest age, pool acquisition time, active service time, dependency spans, and retry volume.
- Resource context: CPU saturation and throttling, memory and pause time, disk or network wait, open connections, release version, and recent configuration changes.
Do not treat a generic threshold such as 80% CPU as proof. A threshold is an investigation cue. Causal evidence connects a resource or wait state to the affected request and shows that changing it improves the outcome.
Turn service capacity into queue pressure
A worker can complete about 1,000 / average service time in milliseconds requests per second when work is independent. A pool's approximate capacity is:
worker count x per-worker completion rate
Utilization is arrival rate / completion capacity. At 100%, arrivals equal average completions and there is no room for variation. Before 100%, bursts and slower-than-average requests can still queue. As utilization approaches the boundary, expected wait grows nonlinearly because fewer workers are free when a request arrives.
400/s
Measured arrivals
Requests entering one worker pool
20 ms
Average service
About 50 completions/s per worker
10
Workers
About 500 completions/s total
80%
Utilization
400 arrivals / 500 capacity
The average is only a planning input. Measure p95 service time, burst size, and queue age as well, because a stable average can hide a harmful tail.
Make the capacity boundary visible
Change measured demand, concurrency, and average service time. The model shows why queueing rises nonlinearly before a worker pool is completely saturated.
84%
420 arrivals / 500 completions per second
6 ms
50% chance an arrival waits in this model.
+80 req/s
11 workers would provide about 20% planning headroom at this service time.
Worker-pool pressure
The line at 100% is a hard capacity boundary. The earlier bands are investigation cues, not universal alert thresholds.
Watch the tail
Average capacity is available, while bursts or slower requests can still create a visible queue.
The estimate has about 80 req/s of average headroom. Compare this with burst size, p95 service time, and the pool's real queue-age distribution before changing capacity.
This is an Erlang C planning model: arrivals are independent, workers are equivalent, and work waits in one unbounded queue. Real bursts, retries, priorities, downstream limits, and bounded queues require measured distributions and load tests.
When the model crosses 100%, its queue has no steady state: backlog grows by approximately arrivals - completions each second. In a real service, a bounded queue eventually rejects work, consumes a deadline, or pushes pressure into another dependency.
Locate the constraint on the request path
End-to-end latency includes both active service time and waiting between stages. A trace should make those boundaries visible so the team can ask where a request stopped making progress.
A checkout path with bounded resources
Every stage can create work or waiting. The bottleneck is the stage that currently limits the measured checkout outcome.
Admit demand
Client and gateway
Record route, region, payload class, deadline, and rejected work. A burst can exceed downstream capacity even when daily traffic looks normal.
Execute
Application workers
Separate CPU time from scheduler wait, lock contention, garbage-collection pauses, and time blocked on another resource.
Wait for a slot
Connection pool
A bounded pool protects the database, but callers queue when every connection is borrowed. Measure acquisition time and waiters.
Read or write
Database
Measure query execution, lock waits, scanned rows, storage latency, active sessions, and replication behavior separately.
Cross a boundary
Payment dependency
Track connection setup, remote processing, timeouts, retries, and fallback results. Local utilization cannot explain a remote slowdown.
Three distinctions prevent common mistakes:
- Busy versus waiting: high CPU means active compute; high pool wait means requests cannot begin their database work.
- Cause versus consequence: an origin queue may rise because a cache regression multiplied misses. The queue is real, but the cache change initiated the pressure.
- Average versus tail: a healthy average can coexist with repeated long pauses or one slow route. Use percentiles and bounded dimensions such as route, region, and release.
Build a diagnosis from discriminating evidence
The same symptom can support several explanations. Rising latency and flat throughput could come from CPU saturation, a full connection pool, long garbage-collection pauses, storage contention, or a slow dependency. Choose evidence that separates those hypotheses instead of collecting more copies of the same graph.
Turn a plausible guess into a tested constraint
Choose an incident, then collect probes. Each observation changes the evidence score for competing causes, so the diagnosis follows measured facts instead of the loudest graph.
Loading incident evidence...
A strong diagnosis usually combines at least two forms of evidence:
- Correlation: the wait or resource signal moves in the same window and on the same request path as the user symptom.
- Mechanism: a trace, profile, or runtime metric explains how that signal delays or rejects work.
- Perturbation: changing the suspected constraint changes the end-to-end result under a comparable workload.
Correlation narrows the search. Mechanism and perturbation make the causal claim credible.
Recognize signatures without jumping to fixes
Signatures are starting hypotheses. For each one, connect a request-level wait to a resource-level signal and run the smallest safe experiment that can disprove the idea.
CPU
Compute pressure
Look for a sustained run queue, throttling, hot stacks, and active CPU time aligned with the slow route. A profiler identifies which work consumes the cores.
Memory
Runtime pauses
Look for heap growth, allocation rate, garbage-collection duration, swap, or out-of-memory events aligned with latency gaps. High memory usage alone is not a leak.
I/O and network
Blocked operations
Look for storage queue depth, I/O wait, packet loss, retransmits, connection setup, and dependency spans. Separate local disk wait from remote service time.
Pools and locks
Bounded concurrency
Look for acquisition wait, queued callers, lock duration, database sessions, and rejected work. Increasing a pool can overload the dependency it was protecting.
Useful disproof experiments include reducing a representative payload, bypassing optional work for a small cohort, raising one canary's worker limit, restoring a previous cache policy, or replaying the same trace mix against a query index. Keep a rollback condition and observe the downstream consequence.
Expect pressure to amplify and move
Systems often hide a bottleneck until a feedback loop makes it visible. Plan for these failure behaviors before increasing capacity.
- Queues trade rejection for waiting. A deeper queue can absorb a short burst, but it also consumes deadlines, memory, and recovery time. Oldest-item age is often more actionable than depth alone.
- Retries multiply demand. When a slow dependency triggers retries, attempted work can rise while successful throughput stays flat. Bound attempts, add jitter, and retry only safe operations.
- Larger pools move pressure downstream. More application concurrency may reduce local waiting while exhausting database sessions, locks, memory, or provider quotas.
- Caches can shift the constraint. Better hit rate protects an origin, while stale or incorrect policy can break product correctness. Provision and test the miss path.
- One fix reveals the next limit. After a query index removes database work, CPU serialization or a downstream quota may become the active constraint.
Never declare success because one resource graph falls. Success means the target user outcome improves, errors and downstream saturation remain acceptable, and the same controlled test is repeatable.
Fix one constraint and rerun the baseline
Choose the smallest change that addresses the measured mechanism. Optimizing code, adding concurrency, shedding demand, caching work, batching I/O, adding an index, or isolating a dependency solve different causes.
1 Predict
State the hypothesis
Write the expected movement: "If connection-pool waiting is active, reducing acquisition p95 should raise completed checkout throughput above 320/s."
2 Protect
Bound the experiment
Use a canary, fixed load test, feature flag, or small cohort. Set rollback thresholds for errors, downstream saturation, and data correctness.
3 Verify
Compare the outcome
Measure the same route, workload, percentiles, throughput, errors, waits, and downstream signals. Reject the hypothesis if only an internal metric improves.
4 Repeat
Find the new limit
Keep the evidence and rerun the analysis. Capacity work is a control loop because traffic, code, data shape, and dependencies continue to change.
Trade-offs belong in the decision record:
- Adding workers costs compute and can increase pressure on shared dependencies.
- Reducing service time usually improves both latency and capacity, but profiling and code changes take engineering time.
- Shedding low-priority traffic protects critical work, but the product must define the degraded user experience.
- Caching removes repeated work, but freshness, invalidation, warm-up, and miss-path capacity become operating concerns.
Operate an evidence-first performance loop
Record a small, stable signal set before an incident. The example recording rules keep request demand and user outcome beside queue and pool evidence so an operator can move from symptom to cause without changing tools.
For each critical path, keep an operational record with:
- the owner, user-visible objective, representative load test, and known safe capacity envelope;
- request rate, completion rate, error ratio, latency percentiles, queue age, retry volume, and dominant dependency timing;
- bounded breakdowns by route template, region, workload class, and release version;
- recent bottlenecks, the evidence that proved each one, the change made, and the next observed constraint;
- an alert that points to a usable runbook and a mitigation that protects correctness.
Review the baseline after releases, traffic-shape changes, schema migrations, cache-policy changes, and dependency quota updates. A capacity number without its workload assumptions will become misleading.