Performance Metrics
Learn essential performance metrics: response time, error rates, resource utilization, and monitoring strategies.
What are Performance Metrics?
Performance metrics are measurements of how a system behaves while users ask it to do work. They turn a report such as "checkout feels slow" into evidence: how much work arrived, how long it waited, how much completed, what failed, and which bounded resource is under pressure.
They matter because an application can look healthy from one number while users are failing from another. The core rule is to measure the user outcome and the pressure that can prevent it. A low average response time does not cancel a slow tail, a fast error, or a growing queue.
How long?
Latency
Measure the complete user path. Percentiles show typical work and the slower tail separately.
How much?
Traffic
Count arrivals, completed work, and concurrency in units that match the actual product action.
What failed?
Errors
Count failed user outcomes, including timeouts and incorrect results, not only server exceptions.
What is bounded?
Saturation
Watch the resource that can run out: a queue, connection pool, CPU, memory, I/O path, or dependency quota.
Start with the user path, then name the measurements
An SLI is the measurement used to judge an important behavior, such as successful checkout under 350 ms. An SLO is the target for that measurement, such as 99% of successful checkouts completing within 350 ms over a period. A metric is useful only when its unit, scope, and owner are clear.
For a synchronous API, instrument the same request identity across the edge, application, dependencies, and response. For asynchronous work, record enqueue time, dequeue time, completion, retry, and final failure. This lets a team distinguish "the system accepted work" from "the system delivered the promised outcome."
One user action creates several measurable signals
An end-to-end outcome needs a correlated view of demand, waiting, execution, and result.
Traffic
Request arrives
Count attempts and request shape. Arrival rate is demand, not proof of completed work.
Queue
Bounded wait
Measure depth and age. A queue protects a service only when work still exits on time.
Saturation
Service and dependency
Track capacity, pool waits, CPU, I/O, and each dependency's time contribution.
Latency and errors
User outcome
Measure percentiles and successful outcomes at the boundary the user actually sees.
Read a distribution instead of one average
Latency is a distribution. P50 is the middle request and describes the typical experience. P95 shows whether a meaningful minority is slowing down. P99 exposes the tail, where queueing, retries, cache misses, and a contended dependency often appear first.
Use an end-to-end percentile for the promise made to the user, then break it into stage timings to investigate. Keep successful and failed requests visible separately: a failed request can be fast because validation rejects it immediately, while a timeout may disappear from a success-only latency chart.
P50
Typical path
What the middle request experienced
P95
Early pressure
A slower minority needs explanation
P99
Tail promise
Where queueing and shared limits surface
Outcome rate
User result
Success, timeout, rejection, or incorrect result
Lab 1: load turns spare capacity into queueing and tail latency
The constrained stage must finish work faster than it arrives. As demand approaches its service capacity, even a small burst has less room to drain and the tail grows faster than the typical request. Change the load, capacity, and P99 objective, then compare utilization with the latency distribution.
Keep the queue from consuming the tail budget
Change incoming work, replica count, and the P99 objective. This single-gate model exposes why an average can remain calm while queued work makes the tail fragile.
Each replica completes 240 requests per second. Fixed application work is 55 ms; queueing is estimated from the remaining service margin.
Tail budget protected
The tail target has headroom for ordinary variation.
1,440 RPS
6 replicas x 240 RPS
50%
Below 100% is necessary, not enough.
55 ms / 81 ms
Typical versus a slower minority
111 ms
350 ms objective
Read the model as a measurement hypothesis
Inspect arrival rate, completed RPS, queue age, and P50/P95/P99 together. If arrivals rise while completions flatten and queue age grows, add or protect the constrained capacity before extending timeouts. A real service has multiple queues, variable work, and retries, so verify this direction with a representative load test.
The lab is a single-gate teaching model, not a production forecast. In production, test the real request mix, payload sizes, cache state, dependency behavior, and retry policy. The operational invariant remains: rising arrivals plus flat completions plus growing queue age means the constrained path needs protection or capacity.
Failure behavior: measure the symptom that changes the next decision
The four golden signals are a useful starting vocabulary, but they are not a dashboard scavenger hunt. Start with the user-visible symptom, select the measurement that can test its first explanation, then add companion metrics to separate load, failure, and saturation.
Choose evidence that can disprove the first hypothesis
Select a user-visible symptom, then choose the first measurement worth opening. A metric is useful when it changes the next operational decision, not when it is merely available on a dashboard.
Evidence matches the symptom
This is the leading measurement because it tests the failure described. Segment the end-to-end request path by P95/P99 and compare it with the same interval of traffic and dependency timing.
End-to-end P99 latency
The slowest one percent of the user path.
Collect correlated evidence
Trace the slow cohort. A low average does not disprove a tail problem.
Failure behavior under investigation
The average API latency is 110 ms, but checkout complaints rise during the busiest five minutes.
After the first measurement points to a cause, add the companion signals that distinguish load, failure, and saturation. Do not alert from one metric in isolation: a fast error, a full queue, and a slow successful request require different actions.
When an API's P99 rises, compare it with arrivals, completed RPS, queue age, and dependency timing in the same window. When outcomes fail quickly, measure failure rate and deployment version before treating the lower latency as improvement. When an async queue ages, decide whether to add consumers, slow admissions, shed low-priority work, or repair the blocked downstream dependency.
Design controlled pressure instead of a cascading outage
A capacity limit should produce a bounded, explainable behavior rather than unlimited waiting. Use these controls deliberately:
- Deadlines: Give each downstream call a smaller deadline than its caller so there is time to return a fallback or error.
- Bounded queues and pools: Limit retained work, connections, threads, and memory. Track rejection and queue age so admission control is visible.
- Selective retries: Retry only transient, safe work; cap attempts and add jitter. Retrying a saturated dependency multiplies its load.
- Load shedding and priority: Protect checkout, writes, or other critical work by degrading optional enrichment, analytics, or fan-out first.
A longer timeout is not extra capacity. It can reduce immediate errors while more requests wait, consume resources, and make recovery slower. Confirm the constrained stage can complete more work before increasing waiting time.
Operate with baselines, guardrails, and evidence
Set a baseline using a known workload, then compare the same request mix after a release or capacity change. Alert before the SLO is exhausted, for example on sustained utilization above a declared headroom target combined with rising queue age, P95/P99, or failed outcomes.
During an incident, preserve the time window and dimensions that make evidence comparable: route, region, dependency, deployment version, customer cohort, and outcome class. A dashboard should help answer four operational questions:
- What user outcome is degraded, and for whom?
- Did arrivals, completions, or queue age change first?
- Which bounded resource or dependency is now waiting or saturated?
- Which reversible action protects the critical path while the cause is investigated?
After mitigation, repeat the controlled workload and watch for the next constraint. Scaling the application does not fix a full database pool, a locked table, or a quota-bound third-party API.
A practical metric contract
For each critical path, write down the following before the next incident:
- The user action and exact success condition.
- The SLI, percentile or rate, time window, and SLO target.
- The traffic unit and expected peak request shape.
- The queue, pool, dependency, or resource that can become the active constraint.
- The alert threshold, on-call owner, and bounded fallback or shedding behavior.
This contract prevents a dashboard from becoming an unowned collection of gauges. It also creates the evidence needed for Bottleneck Analysis: change one suspected constraint, rerun the same workload, and verify whether the end-to-end result moves.