Amazon CloudWatch
Learn CloudWatch metric identity, dimension cardinality, structured logs, alarm evaluation, missing-data policy, and incident response.
What is Amazon CloudWatch?
Amazon CloudWatch is AWS's monitoring and observability service. It receives telemetry from AWS resources and applications, stores signals such as metrics and logs, helps teams investigate them, and can turn an alarm state change into a notification or supported automated action.
In plain language, CloudWatch helps answer three questions: Is the system healthy? What changed? What should happen next? Collecting data alone does not answer them. Teams still have to choose stable metric identities, useful log fields, retention, alarm rules, owners, and response paths.
The core invariant is: a signal is operationally useful only when its identity, meaning, and response are explicit. The current CloudWatch overview describes the service's metrics, logs, traces, dashboards, alarms, and application observability capabilities.
Follow one signal from workload to response
Start with a user-visible symptom, not a dashboard widget. A checkout service might need a fast failure-rate metric, detailed failure logs, and a trace that shows which dependency consumed the request time. Those signals answer different questions.
Trend and threshold
Metric
A numeric time series answers how much, how often, or how long. Its bounded dimensions support fast aggregation and alarms.
Event evidence
Log
A structured event records what happened with fields such as operation, status, safe error category, deployment version, and trace identifier.
Request path
Trace
A trace connects work across services so an operator can locate latency, errors, and dependency boundaries for one request path.
From application behavior to an operational response
Each boundary needs an owner and a stable contract. A dashboard is a view of signals, not the source of truth.
Instrument
Workload
AWS services publish built-in telemetry. Application code, an agent, or OpenTelemetry instrumentation can publish the signals the business path needs.
Store and correlate
CloudWatch
Namespaces, metric dimensions, log groups, fields, trace context, and retention decide what can be retrieved and compared later.
Query and alarm
Evaluate
Dashboards and queries support investigation. Alarm rules turn selected evidence into OK, ALARM, or INSUFFICIENT_DATA.
Own the consequence
Respond
A state transition can notify through SNS, invoke Lambda, drive supported scaling or EC2 actions, create operations work, or emit an EventBridge event.
Do not force every detail into every signal. Metrics stay bounded and aggregatable; logs and traces carry request-level evidence. The common keys between them should help an operator move from a symptom to the relevant detail.
Design metric identity before resolution
A traditional CloudWatch metric is identified by its namespace, metric name, and complete dimension set. A data point adds a timestamp, value, and optional unit. CloudWatch treats every unique dimension combination as a separate metric. For custom metrics, it does not automatically aggregate across dimension combinations that were never published.
That makes dimension design a multiplication problem. Four services, three environments, and eight operations can create 4 x 3 x 8 = 96 time series for one metric name. Adding 100,000 request IDs can turn that bounded model into millions of series without improving the service-level question.
Loading the decision model
Preparing the CloudWatch controls and consequences.
Loading CloudWatch model
Loading lesson-owned telemetry data.
Use the lab's totals as a design model, not a price quote. CloudWatch pricing and quotas change; the durable lesson is that unique dimension values multiply metric identities, while a shorter publication interval multiplies data-point volume.
The official metrics concepts reference defines metric identity, dimensions, statistics, periods, and automatic retention rollups. The custom metrics guide documents standard and high-resolution publication and recommends OpenTelemetry for new custom-metric implementations.
Choose the statistic that matches the question
- Use
Sumfor counts accumulated during a period, such as failed operations. - Use
Averageonly when averaging the samples preserves the operational meaning. - Use percentiles for latency when the raw samples support them and tail experience matters more than the mean.
- Keep units consistent for one metric identity. A number without a stable unit is easy to graph and hard to trust.
- Publish zero deliberately when zero and no data mean different things. Silence is a separate state that alarm configuration must interpret.
Keep request detail in logs and traces
Structured logs turn an event into queryable fields. Use stable names and bounded categories for service, environment, operation, status, and error class. Put high-cardinality investigation keys such as a request or trace ID in logs or traces, not in metric dimensions.
CloudWatch Logs keeps log data indefinitely by default unless a log group has a retention policy. The log-group guide documents the current retention behavior. Set retention from debugging, security, and compliance needs rather than accepting that default accidentally. Before ingestion, remove secrets, session tokens, credentials, raw payment data, and personal data that the response workflow does not require.
Explicit
Retention
Assign an owner and lifecycle to every production log group
Stable fields
Structure
Keep query keys predictable across releases and services
Trace context
Correlation
Connect a metric symptom to safe request-level evidence
Narrow first
Query scope
Select the smallest time range and log groups that answer the question
The example emits one valid embedded metric format (EMF) JSON object. Service and Environment are bounded metric dimensions; detailed context remains in the log event. EMF is a supported path for extracting custom metrics from CloudWatch Logs. OpenTelemetry is another supported instrumentation path.
The EMF guide specifically warns that high-cardinality dimensions such as requestId create a custom metric for every unique combination. The Logs Insights guide also notes that queries are charged by uncompressed log data scanned, so constrain the time range and selected log groups before broadening an investigation.
Make alarms encode an operational decision
A metric alarm aggregates each period into one data point, compares that point with a threshold, and evaluates recent periods. In an M-out-of-N alarm, DatapointsToAlarm is M and EvaluationPeriods is N. Requiring several breaching periods can reject a short spike, but a larger window can also delay a response.
Missing data needs an explicit contract:
notBreachingtreats missing periods as good. It can fit a sparse metric that emits only when an error occurs.breachingtreats missing periods as bad. It can fit a continuous heartbeat where silence is a failure.missingcan produceINSUFFICIENT_DATAwhen all evaluated data is absent.ignoreretains the current alarm state, which avoids a transition but can leave a stale state in place.
Loading the decision model
Preparing the CloudWatch controls and consequences.
Loading CloudWatch model
Loading lesson-owned telemetry data.
The lab isolates two decisions: how much breaching evidence is required and what total silence means. The real sliding-window evaluator can retrieve additional older points and uses configured missing values only when needed. The alarm evaluation guide documents that wider evaluation range and the four missing-data policies.
Attach an owned consequence
An alarm without an owner and response is only a colored state. Give each page-worthy alarm:
- a user or business symptom it protects;
- a threshold and M-of-N rule justified by measured behavior;
- an explicit missing-data policy;
- an action channel with a tested destination and least-privilege permissions;
- links to a dashboard, query, or runbook that can confirm the cause;
- recovery evidence, including the transition back to
OK.
This CloudFormation example alarms when a sparse Failures metric breaches in two of five one-minute periods. notBreaching is intentional because the example emits that metric only for failures. A separate heartbeat or request-count alarm should detect a broken telemetry path.
The CloudFormation alarm reference defines DatapointsToAlarm, EvaluationPeriods, TreatMissingData, and the supported alarm properties used by the example.
CloudWatch supports actions on state transitions, including SNS notifications, Lambda invocation, and service-specific actions. It also emits state-change events to EventBridge. Review the current alarm actions matrix because support differs by alarm and action type.
Investigate from symptom to cause
An effective dashboard begins the investigation; it does not finish it. Keep a short path from the alarmed symptom to the evidence needed to decide whether to mitigate, roll back, scale, or continue observing.
1 Symptom
Confirm impact
Check the user-visible rate, latency, or availability signal and the exact periods that changed the alarm state.
2 Scope
Correlate the change
Compare deployment version, Region, service, operation, dependency, and traffic over the same time range.
3 Cause
Inspect evidence
Query structured logs and traces with the bounded dimensions from the metric, then follow safe request identifiers only as far as needed.
4 Recovery
Act and verify
Execute the runbook, watch both the protected symptom and telemetry health, and confirm the alarm returns to
OKfor the expected reason.
For a global or multi-account system, decide where operators investigate and where alarms live. Metrics are regional, and cross-account or cross-Region capabilities have different support boundaries. Do not assume that one dashboard automatically gives every alarm the same scope.
Review the production monitoring contract
Instrumentation
- Start with user-facing service indicators, then add resource metrics that explain pressure and saturation.
- Keep metric dimensions bounded. Store request, customer, and raw error identity in protected logs or traces.
- Monitor the telemetry path itself: failed EMF extraction, agent health, missing heartbeats, and rejected or throttled publication.
- Version field names and dashboards with application changes.
Logs and access
- Set retention explicitly for every production log group and test deletion, archival, and legal requirements.
- Encrypt and restrict telemetry access by account, workload, and operator role.
- Redact before ingestion. Query permissions do not undo sensitive data already written to a log.
- Keep Logs Insights queries narrow, saved, and linked from the relevant alarm or runbook.
Alarms and operations
- Alarm on conditions that require an action now; use dashboards and reports for conditions that only need observation.
- Test
OK,ALARM,INSUFFICIENT_DATA, action delivery, suppression, and recovery during a controlled exercise. - Use composite alarms when a higher-level service condition or suppression rule reduces duplicate notifications without hiding the underlying metric alarms.
- Review thresholds after traffic, architecture, ownership, or response-time changes.
- Track notification delivery and action failures separately from the alarm state.
The design is ready when an on-call engineer can name the protected symptom, metric identity, alarm evidence, missing-data meaning, investigation path, action owner, and proof of recovery.