Datadog

Datadog is a commercial, full-stack observability SaaS: one Agent installed on a host or shipped as a container sidecar collects infrastructure metrics, distributed traces, and logs, and ships all three to one hosted backend where they're correlated under a shared set of tags — env, service, version — so a single dashboard, a single alert, and a single investigation can move between "the host is saturated," "this trace is slow," and "here's the exception in the log line that explains it" without switching tools. Founded in 2010 and public since 2019, it's the category-defining commercial answer to the same problem this course's monitoring & observability page frames tool-neutrally, and its most-discussed characteristic outside the product itself is its pricing: infrastructure hosts, APM hosts, custom metrics, and indexed log events are each billed separately and continuously, which makes Datadog's invoice a genuinely useful, and genuinely different, cost-visibility case to hold up against the CI/CD compute bill covered in FinOps for Delivery Pipelines. This page covers the Agent and Cluster Agent architecture, what real configuration and instrumentation look like, monitors and dashboards managed as code, the day-to-day CLI and API, the pricing model in enough detail to reason about it, and the gotchas that turn a fast rollout into a surprising bill.

What Datadog is and the problem it solves

☺ Like you're 10: Instead of running three separate tools for numbers, diaries, and request-maps and manually lining them up during an incident, Datadog collects all three and keeps them pre-lined-up by a shared label.

Before a platform like Datadog existed, "observability" in practice meant running several unrelated systems side by side — Nagios or Zabbix for metrics and thresholds, a separate ELK or Splunk deployment for logs, and, once distributed tracing became common, yet another tool like Zipkin for traces — each with its own agent, its own query language, its own retention policy, and no shared vocabulary between them. Correlating "the error rate spiked" (a metrics fact) with "here's the exception" (a log fact) with "here's which downstream call was slow" (a trace fact) meant a human manually cross-referencing timestamps across three UIs during an incident, which is exactly the kind of toil this course's monitoring & observability page argues against.

Datadog's core idea is unification through one mechanism: every signal — a metric point, a log line, a trace span, a process, a container, a host — carries the same set of key:value tags, and the same three tags in particular (env, service, version) are treated as first-class enough to have an official name, unified service tagging. Filter any view in Datadog to service:checkout env:prod and metrics, traces, logs, and even the underlying hosts and containers all narrow to the same slice at once, because they were tagged consistently from the moment the Agent collected them. That's the whole value proposition in one sentence: not that Datadog collects more data than a self-hosted stack can, but that it collects it pre-correlated, in one product, with one query surface, out of the box.

Where it fits in the delivery pipeline and the on-call loop

☺ Like you're 10: Datadog doesn't build or ship your code — it starts watching the moment your code is running, and it's usually the thing that decides whether an alert turns into a page.

Datadog sits downstream of everything covered in CI/CD pipelines and deployment strategies: it doesn't build, test, or deploy anything, but the moment a new version is running — on a VM, in a container, behind a load balancer — the Agent starts reporting on it, and a well-set-up canary or rolling deploy watches Datadog's own metrics (error rate, p99 latency) as the automated go/no-go signal for whether that rollout should continue. On the operations side, a Datadog monitor firing is very often the trigger that starts incident management: monitors can page directly through PagerDuty or Opsgenie, post to a Slack channel, or open a case, and the same tag scheme that ties a metric to a trace also ties an alert to the exact service and version that caused it. Distributed tracing specifically gets a much deeper treatment, vendor-neutral, in Distributed Tracing & Telemetry; SLO burn-rate alerting built on top of Datadog's own SLO objects is covered in SLOs, Error Budgets & Toil. Datadog is one of several products competing for this exact slot in the DevOps toolchain — Prometheus plus Grafana, the ELK stack, and New Relic all cover overlapping ground, and the comparison table near the end of this page lays out how to choose between them.

Architecture: the Agent, the Cluster Agent, and tags as the glue

☺ Like you're 10: One small program runs next to your code and phones home over an outbound connection only — nothing from outside ever has to reach in and touch it.

The Datadog Agent is a single install (a package on a host, a container image in Docker or Kubernetes) that bundles several cooperating processes under one supervisor: the core Agent runs infrastructure checks — CPU, memory, disk, and roughly 750-plus built-in integrations for things like PostgreSQL, Redis, and Kafka, each an integration-specific check that knows how to poll that system — and listens on UDP 8125 for DogStatsD, a StatsD-compatible protocol your application can emit custom metrics to directly. The trace Agent (APM) receives spans on port 8126 from a per-language tracing library (ddtrace for Python, Node, Java, Go, Ruby, .NET, PHP), batches and samples them, and forwards them on. The Log Agent tails files or a container's stdout/stderr and ships log lines. The Process Agent reports what's actually running on the host, which is what lets Datadog show you the process tree behind a saturated CPU metric. Every one of these processes only ever initiates outbound HTTPS connections to Datadog's intake — nothing has to reach back into your network, which is the same "agent dials out" posture Jenkins agents use for exactly the same reason.

On Kubernetes specifically, the Agent runs as a DaemonSet (one Agent per node, exactly like any other node-scoped daemon), but a second component — the Cluster Agent — runs as a single Deployment, once per cluster. Its job is to offload work that would otherwise be duplicated by every node's Agent: cluster-level checks like kube_apiserver_metrics and kube_scheduler only need to run once, not once per node; it aggregates Kubernetes events and Custom Resource state; and, most operationally useful, it exposes the External Metrics API, which lets a HorizontalPodAutoscaler scale a Deployment on a Datadog query — queue depth, a custom business metric — instead of only on raw CPU or memory, the same reconciler-consumes-a-metric pattern this course covers generally in infrastructure as code.

Instrumenting a real stack

☺ Like you're 10: Three small files — one for the Agent's own settings, one for how it runs next to your app, one for how your app talks to it — cover almost every real rollout.

A production rollout is usually three layers: the Agent's own configuration, how the Agent is deployed alongside the workload, and how the application emits traces and custom metrics into it.

The Agent's own config

# /etc/datadog-agent/datadog.yaml — the core Agent config on a host install
api_key: ${DD_API_KEY}
site: datadoghq.com          # or datadoghq.eu, us3.datadoghq.com, us5.datadoghq.com, ap1.datadoghq.com
                              # — pick your org's actual intake region; wrong site = "Agent is up" but nothing arrives
hostname: checkout-web-01
tags:
  - env:prod
  - service:checkout
  - team:payments

logs_enabled: true
apm_config:
  enabled: true
  receiver_port: 8126

process_config:
  process_collection:
    enabled: true            # real CPU/memory cost — see the gotchas section before enabling everywhere

Deploying it: Docker Compose and Kubernetes

# docker-compose.yml — Agent as a sidecar, autodiscovering the checkout container
services:
  checkout:
    image: ghcr.io/acme/checkout:1.4.3
    labels:
      com.datadoghq.ad.logs: '[{"source":"nodejs","service":"checkout"}]'
    environment:
      DD_AGENT_HOST: datadog-agent
      DD_ENV: prod
      DD_SERVICE: checkout
      DD_VERSION: "1.4.3"

  datadog-agent:
    image: gcr.io/datadoghq/agent:7
    environment:
      DD_API_KEY: ${DD_API_KEY}
      DD_SITE: datadoghq.com
      DD_APM_ENABLED: "true"
      DD_LOGS_ENABLED: "true"
      DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL: "true"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /proc/:/host/proc/:ro
      - /sys/fs/cgroup/:/host/sys/fs/cgroup:ro

# values.yaml for the official chart — helm repo add datadog https://helm.datadoghq.com
datadog:
  apiKeyExistingSecret: datadog-api-key   # never a plaintext key in a committed values file
  site: datadoghq.com
  logs:
    enabled: true
    containerCollectAll: true
  apm:
    portEnabled: true
  processAgent:
    enabled: true
  tags:
    - "team:payments"

clusterAgent:
  enabled: true
  metricsProvider:
    enabled: true             # exposes the External Metrics API for the HPA

The application side: APM and custom metrics

Most tracing libraries need zero code changes for the common frameworks — set the unified-tagging environment variables and wrap the process entry point, and the tracer auto-instruments the web framework, the database driver, and outbound HTTP calls:

# auto-instrumentation via environment variables — no source change required
$ DD_ENV=prod DD_SERVICE=checkout DD_VERSION=1.4.3 ddtrace-run python app.py

# a manual span for the one code path auto-instrumentation doesn't reach,
# and a custom business metric alongside it via DogStatsD
from ddtrace import tracer
from datadog import statsd

with tracer.trace("checkout.calculate_tax", service="checkout", resource="POST /cart/tax"):
    tax = calculate_tax(cart)

statsd.increment("checkout.orders.completed", tags=["payment_method:card"])  # fine
# statsd.increment("checkout.orders.completed", tags=[f"order_id:{order.id}"])  # DON'T — see cardinality below

Monitors, dashboards, and SLOs as code

☺ Like you're 10: Instead of clicking around a UI to set up an alert, you write it in a file, put it in Git, and let a plan-and-apply tool create it — the same discipline this course already teaches for infrastructure.

Datadog's own Terraform provider treats monitors, dashboards, and SLOs as ordinary managed resources, which is the same reviewable-diff argument Terraform makes for infrastructure generally, pointed at alerting instead of servers.

resource "datadog_monitor" "checkout_error_rate" {
  name    = "[checkout] error rate above SLO burn threshold"
  type    = "metric alert"
  message = <<-EOT
    {{#is_alert}}Error rate breached on checkout — page @pagerduty-checkout{{/is_alert}}
    {{#is_recovery}}Recovered.{{/is_recovery}}
  EOT
  query   = "sum(last_5m):sum:trace.http.request.errors{service:checkout,env:prod}.as_count() / sum:trace.http.request.hits{service:checkout,env:prod}.as_count() > 0.02"

  monitor_thresholds {
    critical = 0.02
    warning  = 0.01
  }

  notify_no_data    = true    # a monitor that goes silent when its host disappears is a blind spot, not a quiet win
  renotify_interval = 30
  tags              = ["service:checkout", "team:payments"]
}

resource "datadog_service_level_objective" "checkout_availability" {
  name = "Checkout availability"
  type = "metric"
  query {
    numerator   = "sum:trace.http.request.hits{service:checkout,env:prod}.as_count() - sum:trace.http.request.errors{service:checkout,env:prod}.as_count()"
    denominator = "sum:trace.http.request.hits{service:checkout,env:prod}.as_count()"
  }
  thresholds {
    timeframe = "30d"
    target    = 99.9
    warning   = 99.95
  }
}

See SLOs, Error Budgets & Toil for how that datadog_service_level_objective resource fits into a broader error-budget policy, and Secrets & Credential Management for keeping the API and application keys these examples reference out of committed files.

Day-to-day commands

☺ Like you're 10: A handful of commands tell you whether the little reporter is actually working, and a couple more let you talk to the newsroom directly without opening a browser.

# is the Agent alive, and is every check succeeding?
$ sudo datadog-agent status
$ sudo datadog-agent status collector          # just the check results, trimmed
$ sudo datadog-agent configcheck                # what config Autodiscovery actually resolved
$ sudo datadog-agent diagnose                   # connectivity + permissions self-test — run this FIRST
$ sudo datadog-agent check kubelet -t 5         # run one check by hand, print raw output, 5 iterations

# submit a custom metric straight to the API, bypassing DogStatsD entirely
$ curl -X POST "https://api.datadoghq.com/api/v2/series" \
    -H "DD-API-KEY: ${DD_API_KEY}" -H "Content-Type: application/json" \
    -d '{"series":[{"metric":"checkout.orders.custom","type":3,
         "points":[{"timestamp":'"$(date +%s)"',"value":1}],"tags":["env:prod"]}]}'

# dog — the CLI wrapper over the same REST API, good for scripting and CI
$ dog monitor show 123456
$ dog dashboard show abc-def-ghi
$ dog tag add host:web-01 env:prod service:checkout

The pricing model: what actually drives the bill

☺ Like you're 10: Datadog doesn't charge by how often you look at a dashboard — it charges by how many machines, containers, and unique labels you're generating data from, every single month, whether or not anyone ever looks.

Datadog's pricing is metered by footprint, not by activity, and every product surface bills on a different unit. Infrastructure Monitoring is priced per host per month (billed hourly and prorated, then summarized on the monthly invoice), with container-based tiers available once a host runs enough containers that per-host pricing stops making sense. APM is priced per APM-instrumented host, layered on top of the infra host price. Log Management bills two numbers separately: ingested volume (per GB, regardless of whether that log is ever searched) and indexed events (per million log events indexed per month, at a 15-day or 30-day retention tier) — which is why Datadog and most competing log platforms let you configure exclusion filters that ingest a log for short-term triage but never pay the indexing cost for lines nobody will query. Synthetic tests bill per run; Real User Monitoring bills per session. Treat every number in this paragraph as illustrative rather than current — Datadog revises tiers and included allowances often enough that you should read them straight off Datadog's own pricing page before building a budget on them, the same hedge this course's own certification pages give exam costs.

The sharpest edge is custom metrics. Datadog's billing unit for a custom metric isn't "one metric name" — it's one unique combination of metric name and tag values. A single statsd.increment("checkout.orders.completed", tags=[f"order_id:{order.id}"]) line looks like one metric in the code, but every distinct order_id it's ever called with creates a separate billed custom metric, because each tag combination is a separate time series under the hood. A counter tagged with an unbounded field — an order ID, a user ID, a request ID — can silently generate thousands of billed custom metrics a day from a single line of instrumentation, with no error, no warning, and no visible change in behavior until someone reads the invoice.

This is worth holding directly against FinOps for Delivery Pipelines's cost-visibility point, because it's the same underlying shape wearing an almost opposite cost driver. That page's CI runner fleet is billed by activity — rate per minute × minutes actually run — so its levers are about doing less redundant work: caching, selective testing, cancelling superseded runs. Datadog's bill is driven by footprint — how many hosts and containers exist, and how many unique tag combinations your instrumentation happens to generate — so "run it less often" isn't a lever at all; the levers that actually move a Datadog bill are reducing the number of priced units (right-sizing host and container counts, moving dense container hosts onto container-based pricing) and governing cardinality before it ships, not after. Attribution still matters in exactly the same way it does for a shared CI fleet — tag every host and service with a team or cost-center label and use Datadog's own usage-attribution views to turn one aggregate SaaS invoice into a per-team number, the same showback/chargeback choice that page walks through — but the mechanism for shrinking the total is genuinely different in kind, not just in tool, which is exactly what makes this a useful contrast case rather than a duplicate one.

Other gotchas and failure modes

☺ Like you're 10: A few habits that felt harmless on day one — an open UDP port, a missing setting, a copy-pasted DaemonSet — turn into real problems by month three.

DogStatsD drops packets silently. The default transport is UDP, and UDP has no delivery guarantee — under high throughput or a burst, packets can simply vanish with no error surfaced anywhere, so a metric that "should" be there just quietly under-reports. A Unix domain socket, where supported, or a larger client-side buffer reduces this; either way, don't assume a missing data point means the event didn't happen.

Process collection and log tailing aren't free. Enabling process_collection and full container log collection on every host adds real, measurable CPU and memory overhead to the Agent itself, which matters on a dense, cost-sensitive fleet. Enable what you'll actually use per host rather than turning every collector on everywhere by default, and remember that overhead is itself part of the cost equation the pricing section above only covers the invoice side of.

Query language lock-in. Datadog's metric query syntax isn't PromQL, its log query syntax isn't Lucene or LogQL, and its monitor and dashboard definitions are Datadog-specific JSON. That's not a defect — it's a real, honest trade for the unification this page opened with — but it means every monitor, dashboard, and SLO written against Datadog is a rewrite, not a straight export, if the platform ever changes. Weigh that switching cost the same way you'd weigh any other vendor lock-in decision, not as an afterthought discovered during a migration.

Alternatives and when to choose it

☺ Like you're 10: Every option here can watch your systems — the real choice is whether you'd rather pay a vendor to keep it simple or run the pieces yourself to keep the cost curve flatter.

The practical rule mirrors the one Jenkins reaches for self-hosted CI: choose Datadog on purpose, for a specific reason you can name — speed of rollout, unification across four telemetry types, no in-house platform-observability capacity — not because "everyone already knows it." A team with real platform engineering capacity and a very large, steady host count often finds the self-hosted OSS stack's cost curve, which scales with compute and storage the team already controls rather than a per-host SaaS meter, wins out at scale; a smaller team without that capacity usually comes out ahead paying for the unification. Neither choice is permanent — the tags, dashboards, and alerting logic on this page carry over conceptually to whichever platform a team lands on next, which is why monitoring & observability and distributed tracing & telemetry teach the concepts before any vendor's syntax. Practice wiring real observability hands-on in Capstone Part 4 — Observability, or go set up alerts that actually page someone in Drill — Set Up Meaningful Alerts.