Monitoring & Logging

Domain 4 of the AWS Certified DevOps Engineer – Professional exam is worth about 15% and it's where the tool-neutral idea of "observability" gets a very specific AWS accent: CloudWatch for metrics, logs, alarms, and dashboards; X-Ray for distributed tracing; CloudTrail for the audit record of who did what; and EventBridge for turning any of the above into an automatic fix instead of a page. This page covers all four in exam-relevant depth, shows how they aggregate across a multi-account estate, and flags the traps AWS likes to hide in this domain's scenario questions.

What Domain 4 actually tests

☺ Like you're 10: AWS groups this domain into three jobs — capture the data, figure out what it means, and let the boring fixes happen automatically.

AWS's own domain framing (paraphrase — verify exact task-statement wording against the current DOP-C02 exam guide before you rely on it for study planning) splits Domain 4 into three verbs: collecting, aggregating, and storing logs and metrics; auditing and analyzing them to detect issues and anomalies; and automating monitoring and event management so known failure modes remediate themselves. The table below maps each verb to the services this page covers and to what "done" looks like on the exam.

If any of "metrics, logs, and traces" or "known-unknowns vs. unknown-unknowns" sounds unfamiliar, back up to Monitoring & observability first — that page covers the tool-neutral shape of this problem: what a metric, a log, and a trace are each good for, and why monitoring alone can't catch a failure mode nobody predicted. Everything on this page is that same framing wearing AWS's specific service names. Read that page for the concepts; read this one for which AWS API call implements each concept and what the exam does with it.

CloudWatch: metrics, alarms, and dashboards

☺ Like you're 10: CloudWatch is the scoreboard — numbers over time, a red line you set yourself, and a wall of screens showing whichever numbers you care about.

Every metric in CloudWatch lives in a namespace (AWS-owned ones like AWS/EC2 or AWS/Lambda, or your own custom namespace) and is identified by its name plus a set of dimensions — key-value pairs like InstanceId or Environment that let you slice the same metric name many ways. AWS services publish their own metrics for free at 1- or 5-minute resolution depending on the service and whether detailed monitoring is enabled; your own application code publishes custom metrics via PutMetricData, at standard 1-minute resolution or, if you ask for it, high-resolution down to 1-second granularity.

The lowest-overhead way to publish a custom metric from something already writing structured logs is the Embedded Metric Format (EMF): you write one JSON log line that is both a normal log entry and, thanks to a special _aws block, a metric CloudWatch extracts automatically — no separate PutMetricData call, no extra API cost:

{
  "_aws": {
    "Timestamp": 1755331200000,
    "CloudWatchMetrics": [{
      "Namespace": "Checkout/Service",
      "Dimensions": [["Environment"]],
      "Metrics": [{ "Name": "OrderLatencyMs", "Unit": "Milliseconds" }]
    }]
  },
  "Environment": "prod",
  "OrderLatencyMs": 812,
  "orderId": "ord_8841"
}

Alarms — thresholds, composites, and anomaly detection

A CloudWatch alarm watches one metric (or a metric-math expression combining several) and sits in one of three states: OK, ALARM, or INSUFFICIENT_DATA — the last one meaning the alarm hasn't received enough data points yet to evaluate, which is a distinct exam-tested state, not the same as OK. Two settings the exam likes to probe: evaluation periods vs. datapoints-to-alarm (an "M out of N" alarm — e.g. 3 breaching datapoints out of the last 5 periods — is more resistant to a single noisy sample than the classic "every period must breach"), and treat-missing-data, which decides whether a gap in data counts as breaching, not-breaching, ignored, or missing:

aws cloudwatch put-metric-alarm \
  --alarm-name checkout-error-rate-high \
  --namespace Checkout/Service \
  --metric-name ErrorRate \
  --statistic Average \
  --period 60 \
  --evaluation-periods 5 \
  --datapoints-to-alarm 3 \
  --threshold 5 \
  --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching \
  --alarm-actions arn:aws:sns:us-east-1:111122223333:page-oncall

Two variants push past a simple threshold. Anomaly detection alarms compare a metric against a machine-learned expected band instead of a fixed number, which suits metrics with a strong daily or weekly cycle (traffic that's naturally low at 3 a.m. shouldn't page anyone). Composite alarms combine several existing alarms with a boolean AlarmRule, which is the standard fix for alarm fatigue when one root cause trips five child alarms at once:

Type: AWS::CloudWatch::CompositeAlarm
Properties:
  AlarmName: checkout-degraded
  AlarmRule: "ALARM(checkout-error-rate-high) AND ALARM(checkout-latency-p99-high)"
  AlarmActions:
    - arn:aws:sns:us-east-1:111122223333:page-oncall

Dashboards and cross-account observability

Dashboards are collections of widgets — line graphs, single-value numbers, alarm status, and Logs Insights query results can all live on the same dashboard, defined as JSON so they can be version-controlled and deployed like any other infrastructure. For an organization running many accounts, CloudWatch cross-account observability (set up through Observability Access Manager, creating a sink in a central monitoring account and a link from each source account) lets one dashboard show metrics, logs, and traces from dozens of accounts without copying the data anywhere — a newer, lighter alternative to physically shipping every account's telemetry into one place.

CloudWatch Logs & Logs Insights

☺ Like you're 10: Logs Insights is a librarian fast enough to search every diary entry from the last week in a couple of seconds, even for a question nobody asked in advance.

Application and system logs land in log groups (one per application or component) made up of log streams (one per source — a container, a Lambda invocation, an instance). The single most commonly forgotten setting is retention: a new log group defaults to never expire, which quietly becomes a real storage bill over months — setting an explicit retention period is a one-line fix and a favorite exam distractor. A metric filter turns a recurring log pattern you already know to watch for into a CloudWatch metric (and from there, an alarm) — "count every line containing ERROR" becomes a number you can graph and page on. A subscription filter streams matching log events in near real time to a Kinesis stream, Kinesis Data Firehose, or a Lambda function, which is the mechanism behind cross-account log aggregation covered below.

Metric filters answer a question you already knew to ask; Logs Insights is for the question you didn't. It's a purpose-built query language that runs directly against a log group's raw events, no indexing pipeline to stand up first:

fields @timestamp, @message
| filter @message like /ERROR/
| stats count(*) as errors by bin(5m)
| sort @timestamp desc
| limit 20

Live Tail streams a log group's events to your terminal or console in real time, closer to tail -f than a query, useful for watching a deploy land. Contributor Insights answers a narrower but frequent exam question — "which top-N callers or resources are dominating this metric or log pattern" — without hand-writing a Logs Insights query every time.

X-Ray: distributed tracing on AWS

☺ Like you're 10: X-Ray is dye injected into one request so you can watch exactly which stop on its journey slowed it down.

A trace in X-Ray is a tree of segments (one per service the request touched) and subsegments (finer-grained work inside a segment — a downstream call, a DB query), stitched together across process boundaries by a trace ID that propagates in the X-Amzn-Trace-Id header. Two ways to attach detail to a segment matter for the exam because they behave differently: annotations are indexed and filterable — you can search "every trace where order_id equals this value" — while metadata is stored but not indexed, meant for larger payloads you want attached to a trace for reading, not for searching.

from aws_xray_sdk.core import xray_recorder, patch_all
patch_all()  # auto-instruments boto3, requests, and other supported libraries

@xray_recorder.capture("process_order")
def process_order(order_id):
    subsegment = xray_recorder.current_subsegment()
    subsegment.put_annotation("order_id", order_id)          # indexed, searchable
    subsegment.put_metadata("payload", {"sku": "A-1841"})    # attached, not indexed

X-Ray doesn't trace every single request by default — a sampling rule controls the volume, and the built-in default reserves 1 request per second plus 5% of any additional requests above that, tunable per service and per URL path with your own rules:

{
  "version": 2,
  "rules": [{
    "description": "checkout — sample more heavily than the default",
    "service_name": "checkout",
    "http_method": "*",
    "url_path": "*",
    "fixed_target": 1,
    "rate": 0.10
  }],
  "default": { "fixed_target": 1, "rate": 0.05 }
}

The service map renders every traced call as a graph, coloring nodes by error rate and latency, so a degraded downstream dependency is visible at a glance before you drill into any single trace. X-Ray Insights goes further and automatically flags an anomalous fault-rate spike without you having to be staring at the map when it happens — the trace-side equivalent of an anomaly-detection alarm. AWS's current recommended instrumentation path is the AWS Distro for OpenTelemetry (ADOT) rather than the legacy X-Ray SDK directly — ADOT speaks the vendor-neutral OpenTelemetry protocol and can still export to X-Ray as the backend, which matters if you also run non-AWS services. For the tool-neutral mechanics of spans, context propagation, and OpenTelemetry itself, see Distributed Tracing & Telemetry — this section is the AWS-specific backend that framing plugs into.

CloudTrail: the audit trail

☺ Like you're 10: CloudTrail is the guard's logbook of every door anyone opened in your account — and unlike your own application logs, it writes itself whether you asked it to or not.

CloudTrail records API activity across your AWS account automatically. Management events — creating a role, launching an instance, changing a security group — are captured by default and free for the first copy. Data events — object-level activity like S3 GetObject/PutObject or Lambda function invocations — are not captured by default; you opt in explicitly per resource, and because they can be high-volume, they cost more and are worth scoping deliberately rather than turning on everywhere. A third category, Insights events, flags unusual API call-volume patterns automatically, the CloudTrail analogue of X-Ray Insights and anomaly-detection alarms.

A Trail's delivered log files are protected by log file integrity validation: CloudTrail periodically writes a signed digest file whose hash chains back through every prior digest, so tampering with or deleting a delivered log file is detectable, not just prevented by S3 permissions. For ad-hoc investigation without standing up Athena or exporting to a SIEM, CloudTrail Lake lets you run SQL directly against an event data store with a much longer configurable retention window than the default trail:

# quick lookup against the always-on 90-day event history
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=DeleteBucket \
  --start-time 2026-08-01T00:00:00Z

-- CloudTrail Lake: SQL against a long-retention event data store
SELECT eventTime, eventName, userIdentity.arn
FROM my_event_data_store
WHERE eventName = 'DeleteBucket'
  AND eventTime > '2026-08-01'

Because CloudTrail is the record of who changed what, it's also load-bearing evidence for Security & Compliance, Domain 6 — audit logging shows up in both domains' scenario questions, and the exam expects you to know it's the same CloudTrail underneath either framing.

Aggregating logs across accounts and services

☺ Like you're 10: Instead of every team keeping their own diary in their own room, everyone's diary gets copied to one central library where security can search all of them at once.

A landing zone built with AWS Control Tower or a hand-rolled Organizations setup typically dedicates one account — commonly named a Log Archive account — as the destination for every other account's logs, so no workload account can tamper with or delete its own audit trail. Two mechanisms move logs there. CloudTrail organization trails deliver directly to a central S3 bucket, no extra plumbing required. CloudWatch Logs, which don't have an organization-wide equivalent, move cross-account via a subscription filter pointed at a Kinesis Data Firehose delivery stream in the destination account — the source account's log group needs a resource policy granting CloudWatch Logs permission to write into that destination, and the receiving account fans the stream out to S3 for archival and often to OpenSearch for search.

For workloads running on EKS or otherwise leaning on the CNCF stack, Amazon Managed Service for Prometheus and Amazon Managed Grafana sit alongside CloudWatch rather than replacing it — metrics scraped the Prometheus way land in a managed, horizontally-scaled Prometheus-compatible store, and Managed Grafana queries both that and CloudWatch as data sources in one dashboard. That's the exact seam between this AWS-native domain and the tool-neutral stack covered in Monitoring & observability and in this course's own Prometheus and Grafana pages — same telemetry pillars, AWS-managed control plane underneath.

Automated remediation: EventBridge rules driving Lambda and SSM Automation

☺ Like you're 10: EventBridge is the wire from any alarm straight to a robot that already knows the fix — most nights, nobody's phone even has to ring.

EventBridge is AWS's event bus: a rule matches an event pattern against everything flowing through an event bus (the default bus carries every AWS service's own events — alarm state changes, EC2 instance state changes, AWS Health events — without you publishing anything yourself) and hands a match to one or more targets. A rule can also fire on a schedule instead of an event, using a rate() or cron() expression, for periodic checks that don't wait on any specific trigger. This is the mechanism that turns "we noticed" into "we fixed it" without a human in the loop for known failure signatures:

{
  "source": ["aws.cloudwatch"],
  "detail-type": ["CloudWatch Alarm State Change"],
  "detail": {
    "alarmName": ["checkout-error-rate-high"],
    "state": { "value": ["ALARM"] }
  }
}

The target of that rule is typically an SSM Automation runbook or a Lambda function. SSM Automation documents describe a sequence of steps — aws:executeAwsApi to call an AWS API directly, aws:invokeLambdaFunction to hand off custom logic, aws:branch for conditional paths — and AWS ships pre-built, AWS-owned documents for common fixes (AWS Support's AWSSupport-TroubleshootManagedInstance is a well-known one) alongside letting you author your own. Because EventBridge target invocations are asynchronous, reliability matters: a target that fails retries automatically — by default up to 24 hours and as many as 185 attempts, both configurable — and after retries are exhausted, an optional dead-letter queue captures the event so a human finds out instead of it silently vanishing:

{
  "Rule": "checkout-alarm-to-automation",
  "Targets": [{
    "Id": "restart-unhealthy-task",
    "Arn": "arn:aws:ssm:us-east-1:111122223333:automation-definition/Checkout-RestartUnhealthyTask:$DEFAULT",
    "RoleArn": "arn:aws:iam::111122223333:role/EventBridge-SSMAutomation",
    "DeadLetterConfig": { "Arn": "arn:aws:sqs:us-east-1:111122223333:remediation-dlq" },
    "RetryPolicy": { "MaximumRetryAttempts": 10, "MaximumEventAgeInSeconds": 3600 }
  }]
}

Notice what this loop can and can't do: it's excellent for failure signatures you've seen before and can describe as a fixed sequence of steps — an unhealthy task that needs restarting, a non-compliant Config rule that needs a known correction. It is not a substitute for the harder problem of a genuinely novel failure, which is exactly why Incident & Event Response, Domain 5, immediately next in this blueprint, is a separate 14%-weighted domain covering what happens once a human is actually on the hook.

Common exam traps in Domain 4

☺ Like you're 10: Most of the wrong answers in this domain sound reasonable — they're just describing a feature that's off by default, or mixing up two tools that look similar but answer different questions.

• CloudTrail vs. CloudWatch Logs. CloudTrail records API activity automatically, whether or not you instrumented anything. CloudWatch Logs records whatever your application chose to write. A scenario about "who deleted this resource" wants CloudTrail; a scenario about "why did this request fail" wants CloudWatch Logs.
• Event history vs. a Trail. Covered above, and worth repeating because it's the domain's single most-tested distinction: 90 days, free, always-on Event history is not durable, alertable, cross-account audit logging. Only a Trail delivers that.
• Management events vs. data events. Management events are on by default; data events (S3 object-level, Lambda invocations) are not, and enabling them everywhere is a cost trap the exam likes to dangle as a wrong answer.
• Metric filters vs. Logs Insights. A metric filter is for a pattern you already know to watch continuously and want alarmed on. Logs Insights is for the question you didn't predict — the same monitoring-vs-observability split from Monitoring & observability, wearing CloudWatch's specific names.
• X-Ray sampling undercounts by design. If a scenario says traces are missing for some requests and nothing looks broken, the answer is usually "adjust the sampling rule," not "X-Ray is malfunctioning" — the default rate deliberately doesn't trace every request.
• Log group retention defaults to forever. A scenario about unexpectedly high CloudWatch Logs storage cost is very often just a missing retention setting, not a data-volume problem.
• Page on symptoms, not causes — still applies here. An alarm on a raw resource metric like CPU percentage is a cause, not a symptom; the DevOps course's own throughline from Monitoring & observability holds inside AWS's tooling exactly as it does everywhere else.