Key Monitoring Metrics
Essential monitoring metrics: RED method, USE method, golden signals, and SLI/SLO/SLA definitions.
What is a monitoring metric?
A monitoring metric is a number recorded over time: completed checkouts per minute, the share of requests that failed, or the time a customer waited for a response. It turns a changing system into evidence a team can inspect.
The important distinction is purpose. A metric is a measurement; an SLI is the measurement chosen to represent a user-visible promise; an SLO is the target for that indicator. The core invariant is that an urgent alert must mean a user outcome is at risk and that an operator has a concrete next step.
CPU, memory, queue depth, and database connections still matter. They are often diagnostic signals: they help explain why a user-facing SLI changed. They should not automatically become pages.
Start from the promise, then collect evidence
Monitoring becomes useful when its measurements answer one decision. A checkout team needs to know whether customers can complete payment, not merely whether one host is busy. A delivery team needs to know whether an accepted event becomes visible on time, not merely how many workers are running.
1 Promise
Name the user outcome
State the behavior a user can observe, such as completing payment within 30 seconds or seeing an accepted update within five minutes.
2 Population
Define eligible events
Count only requests or events that should participate in the promise. Exclude health checks, test traffic, and cancelled work deliberately.
3 SLI
Choose a measurable indicator
Use a success ratio, latency percentile, freshness delay, or correctness rate. Record the numerator, denominator, window, and owner.
4 Action
Attach a response
Set an objective, page only on sustained risk, and link the alert to the first diagnostic checks and mitigation.
Indicator
SLI
The measured value. Example: successful payment authorizations divided by eligible authorization attempts in a five-minute window.
Objective
SLO
The allowed performance boundary. Example: at least 99.9% of eligible authorizations succeed in a rolling 30-day window.
Response rule
Alert
A policy that asks for attention when evidence indicates the objective is being consumed too quickly to wait for normal review.
Signal selection lab
Choose a customer promise and compare candidate measurements. The best SLI directly represents the promise; a resource measurement can still be valuable as a correlated diagnostic signal.
Measure the promise, not the machinery
Choose a customer promise and a candidate signal. The result identifies whether the signal is an SLI, a helpful diagnostic, or a mismatch.
Direct SLI
A primary SLI must directly represent the user-visible promise.
Checkout service
At least 99.9% of eligible payment attempts succeed.
Selected SLI expression
successful payment authorizations / eligible payment attempts
This measurement directly answers whether users receive the promise. It can anchor at least 99.9% of eligible payment attempts succeed.
Page when sustained error-budget risk is high; break down by route, region, release, and dependency.
Thresholds should distinguish risk from noise
An alert threshold is not simply a number that looks high. It combines a measured SLI, a time window, and the cost of waiting. A one-minute spike can be a deploy blip or a retry burst; a sustained error ratio may consume a monthly budget fast enough to need a page.
For a success-ratio SLI, calculate failure rate as failed eligible events / all eligible events. For a latency SLI, calculate the share of eligible requests below the stated time boundary. Do not substitute an average for a percentile: an average can remain stable while a meaningful tail of users waits too long.
Do not page solely because a host reaches 85% CPU. High utilization can be healthy batch work, while a customer-impacting outage can occur at low CPU. Page from the user-facing signal and use CPU, queue depth, connection waits, and dependency errors to diagnose the cause.
Incident threshold and failure diagnosis lab
Inject an incident, choose the alert sensitivity, and choose how long the condition must persist. The lab separates an actionable page from a dashboard signal and names the next evidence to inspect.
Page on sustained customer risk
Inject a failure, then set a threshold and confirmation window. A useful policy names why it pages and which diagnostic evidence comes next.
4.2%
42 failed events per 1,000 eligible events
12 min
Must persist for 5 min
Page now
Actionable customer risk
Policy explanation
The condition is above 1% and lasts at least 5 minutes, so it threatens the customer promise long enough to page.
Likely failure boundary
The release version is the leading suspect because the error increase begins with the rollout.
Next evidence
Break down failed authorizations by release version and payment-provider response code.
First operator action
Stop promotion and route new traffic to the previous healthy release while preserving incident evidence.
Build a small signal set with clear jobs
Keep the first dashboard compact. Each signal should either describe the user outcome, identify a likely constraint, or prove that telemetry itself is working.
- Outcome signals: successful eligible requests, p95 or p99 latency against a user boundary, freshness lag, and business completion rate. These anchor SLOs and pages.
- Cause signals: dependency timeout ratio, queue age and depth, connection-pool wait time, saturation, retry rate, and cache effectiveness. These reduce diagnosis time after an outcome signal moves.
- Telemetry health: scrape success, missing-series rate, ingest lag, clock skew, and cardinality growth. A dashboard that silently stops receiving data can make a healthy-looking graph dangerous.
For every alert, record the owner, service, runbook link, affected promise, current value, objective, and a query that breaks the signal down by route, region, dependency, or release version. Tags must remain bounded: use route templates such as /orders/:id, never raw user IDs, request IDs, or unbounded error strings as metric labels.
Operate the alert system like a production system
- Test the population: Verify that successful, failed, timed-out, retried, and cancelled events enter the numerator or denominator intentionally. A denominator that disappears during an outage can hide the outage.
- Review alert quality: Track pages that were actionable, duplicates, false positives, and missed incidents. Tighten a threshold only when the changed policy still detects the failure mode you need to catch.
- Use multi-window evidence: A fast window catches sudden failures; a longer window filters brief noise. Both should describe the same user promise and point to the same owner.
- Treat missing data explicitly: Decide whether absent telemetry means unknown, healthy, or failing. For a critical service, missing a primary outcome signal usually deserves an alert about observability itself.
- Rehearse diagnosis: During a game day or controlled rollout, confirm that an alert carries enough labels and links to find the failing route, release, dependency, or capacity boundary quickly.
An SLO does not replace judgment. A planned maintenance window, a known external-provider outage, or a low-volume service can require a different response policy. Keep that policy documented so alert handling does not depend on tribal memory.
Common trade-offs
A lower threshold catches more early symptoms but can wake people for harmless noise. A higher threshold reduces pages but can delay recognition. Prefer a threshold tied to error-budget burn or a clearly unacceptable user impact, then validate it with incident history and controlled failure tests.