Capstone Part 4 — Observability

This is Part 4 of the six-part capstone running through this whole course: one continuously evolving service, checkout-svc, carried from a gated pipeline in Part 1 through three real AWS environments — each its own Application Load Balancer and Auto Scaling Group — in Part 2. It runs reliably and nobody can see inside any of it. This part gives it eyes — three additions, no more: metrics for the four golden signals, one structured JSON log line per request, and a deliberately short list of alerts. By the end, exactly two Prometheus alerts will be wired to page a human, not the twenty a nervous team is tempted to write. Part 5 opens the moment one of those two actually fires.

What you're building on, and what this part adds

☺ Like you're 10: The service is already built, shipped, and running on real infrastructure — this part just adds the instruments that let you see it working.

This page assumes Part 1, Part 2, and Part 3 are all behind you: a repo with a branch-protected main and a pipeline that lints, tests, and builds an immutable registry.internal/checkout-svc:<SHA> image on every push; three isolated AWS environments — dev, staging, prod — each provisioned from one reusable Terraform module with its own Application Load Balancer, Auto Scaling Group, and security groups (Part 2); and, in prod specifically, a second target group and a checkout-prod-canary Auto Scaling Group behind that same ALB, a weighted listener rule, and one narrow CloudWatch alarm — checkout-prod-canary-5xx — that watches only the canary during a rollout's bake window (Part 3). Part 3 also grew the shared module's output surface with three lines this page reuses directly, most importantly app_security_group_id. This page adds one new thing to that picture — a dedicated observability host — and three changes to checkout-svc itself:

Everything below assumes the tool-neutral concepts from Monitoring & observability — monitoring vs. observability, the three telemetry pillars, the four golden signals, and the three alerting principles — and goes deep on two specific tools already covered on their own pages: Prometheus for metrics, service discovery, and alerting rules, and Grafana for the dashboard on top of them. Read those first if ec2_sd_configs, PromQL, or "dashboards as code" are unfamiliar terms — this page puts them to work on real infrastructure rather than re-explaining them.

Standing up a dedicated observability host

☺ Like you're 10: One small extra machine, running three watchers side by side, wired to find the fleet on its own instead of being handed a list that goes stale.

Nothing in Parts 1–2 required Prometheus or Grafana to exist. Because the app runs on plain EC2 instances behind an ALB — not Kubernetes — there's no Operator to install; instead, one small dedicated instance runs the whole stack via Docker Compose, and Prometheus finds the fleet itself through AWS's own API rather than a file someone has to update every time the ASG scales. Add a new file alongside Part 2's infra/envs/prod/main.tf:

# infra/envs/prod/observability.tf — new file, same directory as Part 2's prod environment
data "aws_vpc" "default" { default = true }
data "aws_subnets" "default" {
  filter { name = "vpc-id", values = [data.aws_vpc.default.id] }
}
data "aws_ami" "al2023" {
  most_recent = true
  owners      = ["amazon"]
  filter { name = "name", values = ["al2023-ami-*-x86_64"] }
}

variable "my_ip" { type = string }   # your own CIDR, e.g. "203.0.113.4/32" — Part 6 replaces this with something narrower

resource "aws_security_group" "observability" {
  name   = "checkout-observability"
  vpc_id = data.aws_vpc.default.id
  ingress { from_port = 3000, to_port = 3000, protocol = "tcp", cidr_blocks = [var.my_ip] }   # Grafana
  ingress { from_port = 9090, to_port = 9090, protocol = "tcp", cidr_blocks = [var.my_ip] }   # Prometheus
  egress  { from_port = 0, to_port = 0, protocol = "-1", cidr_blocks = ["0.0.0.0/0"] }
}

resource "aws_security_group_rule" "app_allow_observability" {
  type                     = "ingress"
  from_port                = 8080
  to_port                  = 8080
  protocol                 = "tcp"
  security_group_id        = module.checkout.app_security_group_id
  source_security_group_id = aws_security_group.observability.id
}

resource "aws_iam_role" "observability" {
  name               = "checkout-observability"
  assume_role_policy = jsonencode({
    Version   = "2012-10-17"
    Statement = [{ Effect = "Allow", Action = "sts:AssumeRole", Principal = { Service = "ec2.amazonaws.com" } }]
  })
}
resource "aws_iam_role_policy" "observability_describe" {
  role   = aws_iam_role.observability.id
  policy = jsonencode({
    Version   = "2012-10-17"
    Statement = [{ Effect = "Allow", Action = ["ec2:DescribeInstances"], Resource = "*" }]
  })
}
resource "aws_iam_instance_profile" "observability" {
  name = "checkout-observability"
  role = aws_iam_role.observability.name
}

resource "aws_instance" "observability" {
  ami                    = data.aws_ami.al2023.id
  instance_type          = "t3.small"
  subnet_id              = data.aws_subnets.default.ids[0]
  vpc_security_group_ids = [aws_security_group.observability.id]
  iam_instance_profile   = aws_iam_instance_profile.observability.name
  user_data              = base64encode("#!/bin/bash\ndnf install -y docker && systemctl enable --now docker\ncurl -SL https://github.com/docker/compose/releases/latest/download/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose && chmod +x /usr/local/bin/docker-compose\n")
  tags = { Name = "checkout-observability" }
}

output "observability_url" { value = "http://${aws_instance.observability.public_ip}:3000" }

The instance's own IAM role, not a static access key, is what lets the Prometheus container inside it call ec2:DescribeInstances — the container picks up those credentials automatically through the EC2 metadata service, so nothing gets pasted into a config file. Copy a small Docker Compose stack onto the instance (via your preferred mechanism — scp, a bootstrapped Git clone, a baked AMI; the delivery mechanism matters less than what's inside these files, which the rest of this page covers) and bring it up:

# infra/envs/prod/observability/docker-compose.yml
services:
  prometheus:
    image: prom/prometheus:v2.55.0
    volumes: ["./prometheus.yml:/etc/prometheus/prometheus.yml", "./alerts:/etc/prometheus/alerts"]
    command: ["--config.file=/etc/prometheus/prometheus.yml", "--web.enable-lifecycle"]
    ports: ["9090:9090"]

  alertmanager:
    image: prom/alertmanager:v0.27.0
    volumes: ["./alertmanager.yml:/etc/alertmanager/alertmanager.yml"]
    ports: ["9093:9093"]

  pushgateway:
    image: prom/pushgateway:v1.10.0
    ports: ["9091:9091"]

  grafana:
    image: grafana/grafana-oss:11.2.0
    environment:
      GF_SECURITY_ADMIN_PASSWORD: "${GRAFANA_ADMIN_PASSWORD}"
    volumes: ["./grafana/provisioning:/etc/grafana/provisioning", "./grafana/dashboards:/var/lib/grafana/dashboards"]
    ports: ["3000:3000"]

cd infra/envs/prod/observability
GRAFANA_ADMIN_PASSWORD='...' docker-compose up -d
docker-compose ps   # all four should show Up

terraform output observability_url
# http://54.x.x.x:3000 — reachable only from your.own.ip/32, on purpose

Instrumenting checkout-svc: the four golden signals as real metrics

☺ Like you're 10: Two counters and a stopwatch, attached to every request, are enough to answer "how busy," "how often it fails," and "how fast" — the three signals the service itself can report.

Of the four golden signals from monitoring & observability, three — traffic, errors, latency — live inside checkout-svc itself and need to be instrumented directly; saturation is mostly a host-level metric that node_exporter gives you for free once it's running alongside the app on every instance, no application code required. Add prom-client and expose one counter and one histogram:

// metrics.js — mounted once, before any route handler
const client = require("prom-client");
client.collectDefaultMetrics({ prefix: "checkout_" });  // process/runtime metrics, no extra code

const httpRequests = new client.Counter({
  name: "http_requests_total",
  help: "Total HTTP requests",
  labelNames: ["route", "method", "status"],
});
const httpDuration = new client.Histogram({
  name: "http_request_duration_seconds",
  help: "HTTP request duration in seconds",
  labelNames: ["route", "method", "status"],
  buckets: [0.05, 0.1, 0.25, 0.5, 1, 2, 5],
});

function metricsMiddleware(req, res, next) {
  const end = httpDuration.startTimer();
  res.on("finish", () => {
    // req.route.path, never req.originalUrl — a raw URL with an ID baked in is unbounded cardinality
    const route = req.route ? req.route.path : "unmatched";
    const labels = { route, method: req.method, status: res.statusCode };
    httpRequests.inc(labels);
    end(labels);
  });
  next();
}

module.exports = { metricsMiddleware, register: client.register };

// index.js — wire it in, and expose the scrape endpoint on the same port /healthz already answers
const { metricsMiddleware, register } = require("./metrics");
app.use(metricsMiddleware);
app.get("/metrics", async (req, res) => {
  res.set("Content-Type", register.contentType);
  res.end(await register.metrics());
});

The label choice in that middleware is deliberate, not incidental — it's the exact cardinality gotcha the Prometheus page warns about, applied to your own service instead of someone else's mistake. route is the Express route pattern with a small, fixed number of distinct values; the raw URL or a customer/order identifier as its own label would each be a brand-new, permanently-indexed time series per request, and a busy fleet would OOM-kill a perfectly healthy Prometheus by the second week. High-cardinality identifiers belong in the structured logs covered further down this page, never in a metric label. Rebuild and redeploy the image the same way Part 2 taught — a new image_tag promoted through terraform apply — and every instance the ASG launches from here on serves /metrics on port 8080 next to /healthz.

Finding the fleet: EC2 service discovery instead of a static target list

☺ Like you're 10: Instead of writing down every machine's address by hand — a list that's wrong the moment the fleet grows or shrinks — Prometheus just asks AWS "who's currently tagged checkout-prod" every time it looks.

A static targets: [...] list, the shape used for the fixed node_exporter hosts on the Prometheus page's own example config, breaks the moment an Auto Scaling Group adds or removes an instance — which checkout-prod does constantly by design, and which checkout-prod-canary does even more dramatically, scaling from zero to several instances and back on every release. ec2_sd_configs is Prometheus's answer: on every scrape cycle it calls the same DescribeInstances API the observability host's IAM role was granted above, and turns the result into a fresh target list automatically, across both fleets:

# infra/envs/prod/observability/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "alerts/*.yml"

alerting:
  alertmanagers:
    - static_configs: [{ targets: ["alertmanager:9093"] }]

scrape_configs:
  - job_name: "checkout-svc"
    ec2_sd_configs:
      - region: us-east-1
        port: 8080
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name]
        regex: checkout-prod(-canary)?
        action: keep
      - source_labels: [__meta_ec2_tag_Name]
        regex: checkout-prod-canary
        target_label: fleet
        replacement: canary
      - source_labels: [__meta_ec2_tag_Name]
        regex: checkout-prod
        target_label: fleet
        replacement: blue
      - source_labels: [__meta_ec2_instance_id]
        target_label: instance_id

  - job_name: "pushgateway"
    static_configs: [{ targets: ["pushgateway:9091"] }]
    honor_labels: true

The relabel_configs block does the real work. The first rule keeps only instances tagged Name: checkout-prod or Name: checkout-prod-canary — the exact two tags Part 2's and Part 3's launch templates already apply at boot — and drops every other EC2 instance in the account before Prometheus ever tries to scrape it. The next two rules stamp a fleet label of blue or canary onto every series, because Prometheus's relabel regexes are fully anchored: checkout-prod-canary only matches that exact tag, and checkout-prod only matches the exact tag "checkout-prod," never as a prefix of the longer one — so each instance gets exactly one fleet label, never both. When checkout-prod-canary is idle at zero instances between releases, it simply contributes nothing to any query below — no special-casing required. Reload the config without restarting the container:

curl -s -X POST http://localhost:9090/-/reload
curl -s 'localhost:9090/api/v1/query?query=up{job="checkout-svc"}' | jq

Proving the scrape and reading the golden signals back

☺ Like you're 10: Ask the same questions the metrics page taught you to ask, but point them at your own fleet this time instead of someone else's example.

With targets up, generate a little traffic against the ALB and run the exact query shapes from the Prometheus page against your own service:

# traffic: requests per second, by route
sum(rate(http_requests_total{job="checkout-svc"}[5m])) by (route)

# errors: error rate as a fraction of total requests
sum(rate(http_requests_total{job="checkout-svc", status=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="checkout-svc"}[5m]))

# latency: p99 across every instance, from the histogram
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket{job="checkout-svc"}[5m])) by (le)
)

Run each one in the Prometheus expression browser at :9090/graph before wiring anything to a dashboard or an alert. environment="prod" should show up as a label on every series, courtesy of the relabel rule above — a cheap way to confirm the right instances are actually the ones answering.

Instrumenting the release itself: a deploy marker via the Pushgateway

☺ Like you're 10: A release runs for a minute and then vanishes, so instead of waiting to be asked for a number, it drops one off on its way out the door.

Prometheus's scrape loop can't catch terraform apply itself — it runs for a minute and exits, gone before the next 15-second scrape gets a chance. This is exactly the one deliberate exception to the pull-only rule covered on the Prometheus page: the Pushgateway, already running in the compose stack above, accepts one final metric right before a short-lived job exits, and Prometheus then scrapes the gateway like any other target. Append this to the exact terraform apply sequence Part 2 taught, once it succeeds:

cd infra/envs/prod
terraform apply tfplan
terraform output checkout_url

printf 'checkout_deploy_info{version="%s",environment="prod"} %s\n' \
  "$(terraform output -raw checkout_url | grep -oE '[a-f0-9]{7}' || echo "$CHECKOUT_IMAGE_TAG")" \
  "$(date +%s)" \
  | curl -sf --data-binary @- \
    "http://$(terraform -chdir=../prod output -raw observability_url | sed 's#http://##;s#:3000##'):9091/metrics/job/checkout-deploy/instance/$CHECKOUT_IMAGE_TAG"

Keying the push by instance/$CHECKOUT_IMAGE_TAG matters: the Pushgateway's warning label from the Prometheus page — that it makes a metric "sticky" until deleted — is a real anti-pattern for anything long-running, but a one-shot deploy marker with a unique instance per release is the pattern's intended, correct use, not a misuse of it. Each release gets its own series instead of overwriting the last one, and Grafana can turn checkout_deploy_info into a vertical annotation line on every dashboard panel — the fastest way to eyeball "did the error-rate spike start before or after the last release." This is also the raw data deployment frequency, the first DORA metric, is built from.

Structured logs: one JSON line per request, correlated by request ID

☺ Like you're 10: Every request gets a unique ticket number stamped on it the moment it arrives, and that same number appears on every log line about that request — so you can pull the whole story by asking for one ticket.

Metrics tell you that something is wrong and roughly how much; they can't tell you which specific request failed or why — that's what the logs pillar from monitoring & observability is for. Structured, not free-text, is what makes a log queryable at that granularity:

// logging.js — mounted before any route handler, alongside metrics.js
const pino = require("pino");
const { randomUUID } = require("crypto");

const logger = pino({ level: process.env.LOG_LEVEL || "info" });

function requestLogging(req, res, next) {
  req.id = req.headers["x-request-id"] || randomUUID();
  res.setHeader("x-request-id", req.id);
  const startedAt = Date.now();
  res.on("finish", () => {
    logger.info({
      request_id: req.id,
      route: req.route ? req.route.path : req.path,
      method: req.method,
      status: res.statusCode,
      duration_ms: Date.now() - startedAt,
    }, "request handled");
  });
  next();
}

module.exports = { logger, requestLogging };

Every field on that log line is deliberately something the metrics can't carry — request_id above all, since it's exactly the high-cardinality identifier the earlier cardinality warning says never belongs on a metric label. Because instances in an Auto Scaling Group come and go, and there's no central log store yet, pull logs by connecting to an instance directly (Session Manager, if the instance profile permits it, or SSH) and reading Docker's own log output for the container:

docker logs $(docker ps -q -f ancestor=registry.internal/checkout-svc) --since 10m | grep '"status":5'
docker logs $(docker ps -q -f ancestor=registry.internal/checkout-svc) --since 10m | jq 'select(.request_id=="'"$REQ_ID"'")'

That's deliberately as far as this page takes logging — reading one instance's own Docker logs is enough to prove structured logs work and to correlate one request end to end. Shipping every instance's logs to something centrally searchable, like the ELK Stack or CloudWatch Logs, is real production work this capstone doesn't require — and the moment request_id starts propagating across service boundaries as a trace ID instead of stopping at one service's log line, you've crossed into distributed tracing & telemetry, deliberately out of scope for a single-service capstone.

A dashboard with two panels, not twenty

☺ Like you're 10: The dashboard is a text file describing what to draw, so it travels with the Terraform in Git instead of living only inside one person's browser tab.

Following the "dashboards as code" discipline from the Grafana page, file-based provisioning is the natural fit here — no Kubernetes to run a ConfigMap sidecar, just a directory Grafana reads on startup, already bind-mounted by the compose file above:

// infra/envs/prod/observability/grafana/dashboards/checkout-svc.json
{
  "title": "checkout-svc — golden signals",
  "panels": [
    {
      "title": "Request rate by status",
      "type": "timeseries",
      "targets": [
        { "expr": "sum(rate(http_requests_total{job=\"checkout-svc\"}[5m])) by (status)" }
      ]
    },
    {
      "title": "p99 latency by route",
      "type": "timeseries",
      "targets": [
        { "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{job=\"checkout-svc\"}[5m])) by (le, route))" }
      ]
    }
  ]
}

# infra/envs/prod/observability/grafana/provisioning/dashboards/default.yaml
apiVersion: 1
providers:
  - name: checkout
    folder: Checkout
    type: file
    options: { path: /var/lib/grafana/dashboards }

Two panels, not a wall of them. Request rate by status shows traffic and errors on the same axis — a spike in the red band alongside the total tells you at a glance whether it's a real error surge or just more traffic overall; p99 latency by route is the tail number that an average would hide entirely. Saturation deliberately isn't a panel here — CPU and memory per instance already exist on whatever generic host dashboard you point at node_exporter's own output, and duplicating it per-service is exactly the kind of dashboard sprawl that makes nobody's dashboard the one people actually check. Redeploy the compose stack and the dashboard shows up under the Checkout folder within a restart — commit and push, don't click-and-forget in the browser.

Two alerts, not twenty

☺ Like you're 10: Only two warning lights are wired to page a person — everything else lives on the dashboard, where a human can look when they're curious, not when they're asleep.

These two are deliberately not the same kind of alert as Part 3's checkout-prod-canary-5xx CloudWatch alarm, and it's worth naming the difference precisely. Part 3's alarm is narrow and temporary on purpose — scoped to the canary target group's own dimension, watching only during a rollout's bake window, and gone (idle at zero) the rest of the time. The two alerts below are the opposite: broad and permanent, spanning whichever fleet or fleets are currently live via the fleet label above, watching around the clock rather than only during a release. The three alerting principles from monitoring & observability — actionable, symptom-based, fatigue-aware — turn directly into a rule file: page on the two symptoms real users actually feel, error rate and latency, and stop there. This is a plain Prometheus rule file, loaded via rule_files in prometheus.yml above, reusing the exact for-duration flap-suppression pattern from the Prometheus page:

# infra/envs/prod/observability/alerts/checkout-svc.yml
groups:
  - name: checkout-svc.alerts
    rules:
      - alert: CheckoutHighErrorRate
        expr: |
          sum(rate(http_requests_total{job="checkout-svc", status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{job="checkout-svc"}[5m])) > 0.05
        for: 10m
        labels: { severity: page, service: checkout-svc }
        annotations:
          summary: "checkout-svc error rate above 5% for 10m"
          runbook: "https://runbooks.acme.internal/checkout-svc-error-rate"

      - alert: CheckoutHighLatency
        expr: |
          histogram_quantile(0.99,
            sum(rate(http_request_duration_seconds_bucket{job="checkout-svc"}[5m])) by (le)
          ) > 1.5
        for: 10m
        labels: { severity: page, service: checkout-svc }
        annotations:
          summary: "checkout-svc p99 latency above 1.5s for 10m"
          runbook: "https://runbooks.acme.internal/checkout-svc-latency"

promtool check rules alerts/checkout-svc.yml   # validate before reloading a running server
curl -s -X POST http://localhost:9090/-/reload
# visit :9090/alerts — both rules should show as "inactive" with a healthy fleet

Watching an alert actually transition

☺ Like you're 10: Break it on purpose, on a day when breaking it doesn't matter, so the first time you watch a page fire isn't during a real incident.

Trust in a paging system is built by watching it work before it has to work for real — the same hands-on-proof discipline Part 2 used to destroy and rebuild dev. Take checkout-prod's capacity to zero on purpose and watch the state machine from the Prometheus page play out for real:

# force it — same ASG Part 2 provisioned
aws autoscaling set-desired-capacity --auto-scaling-group-name checkout-prod \
  --desired-capacity 0 --region us-east-1
sleep 60
aws autoscaling set-desired-capacity --auto-scaling-group-name checkout-prod \
  --desired-capacity 3 --region us-east-1

# watch the alert move: inactive -> pending -> firing -> inactive again
watch -n 5 'curl -s localhost:9090/api/v1/alerts | jq ".data.alerts[] | {alertname: .labels.alertname, state: .state}"'

Confirm it sits in pending for close to the full 10-minute for window before ever reaching firing — that gap is the flap-suppression working as designed, not a bug. Once it fires, it should show up in Alertmanager at :9093 too, grouped and ready to route.

Routing the page — the plumbing, not the response

☺ Like you're 10: This part wires the phone to actually ring; what the person does after picking up is a different lesson entirely.

The severity: page label on both alerts exists to be routed. Point it at a checkout-svc PagerDuty service, set up the same way the PagerDuty page walks through — one Service, one Escalation Policy, one Schedule:

# infra/envs/prod/observability/alertmanager.yml
route:
  receiver: default-null
  group_by: [alertname, service]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match: { severity: page }
      receiver: checkout-pagerduty

receivers:
  - name: default-null
  - name: checkout-pagerduty
    pagerduty_configs:
      - routing_key: "${PAGERDUTY_ROUTING_KEY}"

That's deliberately where this page stops. Detection and routing — Prometheus decides when, Alertmanager decides who — are what this part builds. What a human actually does in the minutes after their phone buzzes — acknowledging, triaging, running the runbook link in the alert's own annotation, declaring an incident — is a distinct discipline covered in full starting with incident management and put into practice next in Part 5.

What "done" looks like for Part 4

☺ Like you're 10: A service that shows its own vital signs, keeps a diary of every visitor, and rings exactly two bells — never more, never fewer.

At the end of this part, checkout-svc exposes the four golden signals through a real /metrics endpoint on every instance, found automatically through EC2 service discovery instead of a list someone has to maintain by hand; every request logs one structured JSON line carrying a request_id; every release through Part 2's terraform apply pushes a deploy marker through the Pushgateway exception to the pull model; a two-panel dashboard lives as code next to the Terraform that describes the fleet it watches; and exactly two Prometheus alerts — an error-rate page and a latency page — are wired through Alertmanager to a PagerDuty service. Nothing here is thrown away — each later part reads today's instrumentation directly:

Milestones

☺ Like you're 10: Tick each box only once you've actually watched it happen on your own AWS account, not because the step "sounds right."

Work these in order — each depends on the infrastructure state from the one before. Progress saves in this browser.

Part 4 gave checkout-svc eyes: real golden-signal metrics discovered automatically across an elastic fleet, structured logs correlated by request, and a deliberately short alert list wired all the way through to a paging service. Continue to Capstone Part 5 — Incident Response, where one of those two alerts fires for real. Or step back to the full lab track to see how this capstone fits the rest of the hands-on labs, revisit monitoring & observability and the Prometheus / Grafana tool pages for the concepts behind what you just built, and go tune a deliberately noisy alert set in Drill — Set Up Meaningful Alerts.