Datadog

Datadog is a commercial, fully hosted observability and security platform: metrics, distributed tracing, log management, synthetic monitoring, real-user monitoring, and infrastructure/security monitoring, all sharing one tag model and one query surface. You install a small open-source agent on each host or as a Kubernetes DaemonSet; everything past that agent — ingestion, storage, indexing, query, dashboards, alerting — runs on Datadog's infrastructure, not yours. That single fact is this page's throughline: Datadog trades the engineering time of running your own metrics and logging stack for a monthly invoice shaped by how many hosts you run and how much data you send it, and almost every gotcha below is a consequence of that trade.

What Datadog is and the problem it solves

☺ Like you're 10: It's one company's app that watches your servers, your code's requests, and your log files all at once, so you're not stitching together five separate tools and hoping their timestamps line up.

Datadog was founded in 2010 by Olivier Pomel and Alexis Lê-Quôc, went public on NASDAQ (ticker DDOG) in 2019, and has grown from a metrics-and-dashboards product into a broad observability and security suite: Infrastructure Monitoring, APM & Distributed Tracing, Log Management, Synthetic Monitoring, Real User Monitoring (RUM), Network Performance Monitoring, Database Monitoring, Cloud Security Management, and an AI-assisted anomaly engine called Watchdog, plus the SLO tracking this page ends on. The pitch isn't any single one of those products — Prometheus does metrics well, Jaeger does tracing well, the Elastic Stack does logs well — it's that they all share one tag model. A request that shows up as a slow trace in APM, a spike in a host's CPU metric, and an error line in the logs can be correlated by service:checkout, env:prod, and a shared request ID, in one UI, without you building the plumbing that stitches three separate open-source tools' timestamps and label schemas together yourself.

The structural tradeoff that follows from that is the one covered by the SRE toolchain in one sentence: Datadog is "a commercial, fully hosted metrics/APM/logs platform that trades self-hosting effort for a managed pipeline and a large library of pre-built integrations." Everything below unpacks exactly what that trade costs and buys, concretely — not as a marketing claim but as commands, config, and a pricing model you can reason about before you commit a fleet to it.

Architecture: the Agent, the Cluster Agent, and the SaaS backend

☺ Like you're 10: A small helper program sits on every machine and quietly ships numbers home over the internet; the company's own computers do all the heavy remembering and drawing of graphs.

The Datadog Agent is open source (Apache-2.0), rewritten from Python to Go for its v6/v7 generation released in 2018, and it is the only piece of this architecture that runs on infrastructure you operate. It's a single process bundling several sub-components: the core collector, which runs integration checks — small Python or Go plugins, one per technology (Postgres, Redis, nginx, and roughly 700 others) — on a fixed interval; DogStatsD, a StatsD-compatible daemon listening on UDP 8125 that receives and locally pre-aggregates custom application metrics before forwarding them; the trace-agent, listening on TCP 8126, which receives spans from a language-specific tracing library (ddtrace) instrumented into your application and batches them upstream; a log-collection component that tails files or reads a container runtime's log driver; and a process and network agent for the corresponding monitoring products. In Kubernetes, the per-node Agent runs as a DaemonSet, paired with a separate Cluster Agent (one replica, or a small HA deployment) that runs cluster-wide checks once instead of once per node, caches API-server responses so a thousand-node cluster doesn't hammer the control plane, exposes an External Metrics API so the Horizontal Pod Autoscaler can scale on a Datadog metric, and runs an admission-controller webhook that can inject tracing libraries into pods automatically.

Everything the Agent collects leaves your infrastructure over HTTPS to Datadog's intake, authenticated by a per-org API key, addressed at whichever site your org was provisioned on (datadoghq.com, datadoghq.eu, us3.datadoghq.com, us5.datadoghq.com, and a couple of others). From that point on, storage, indexing, and query execution are entirely Datadog's problem: there's no metrics time-series database, log index, or trace store for your team to size, shard, upgrade, or page itself on — which is precisely the operational burden that shows up on the other side of the ledger as a bill, in the pricing section below.

The configuration you actually write

☺ Like you're 10: A handful of files: one that tells the Agent your account and what to turn on, one per thing it's watching, and — in Kubernetes — one Helm values file that does both at once for a whole cluster.

Four artifacts cover almost everything a team writes by hand, and the habits that separate a clean setup from a confusing one are the same habits that separate the gotchas below from things that never happen to you.

Core agent config and an integration check

# /etc/datadog-agent/datadog.yaml — the Agent's own settings
api_key: ${DD_API_KEY}            # injected from a secret store, never committed
site: datadoghq.com               # datadoghq.eu / us3.datadoghq.com / us5.datadoghq.com — pick ONE per org
env: prod
tags:
  - team:checkout
  - service:checkout

logs_enabled: true
apm_config:
  enabled: true
  receiver_port: 8126

process_config:
  process_collection:
    enabled: true

dogstatsd_non_local_traffic: true   # accept DogStatsD packets from other containers on the host
---
# /etc/datadog-agent/conf.d/postgres.d/conf.yaml — one integration check
instances:
  - host: db.internal
    port: 5432
    username: datadog_monitor
    password: ${POSTGRES_MONITOR_PASSWORD}
    tags:
      - service:checkout-db
init_config:

Kubernetes: the Helm chart

In Kubernetes almost nobody hand-writes a DaemonSet; the maintained Helm chart is the standard install path, and it configures the Agent, the Cluster Agent, and RBAC together from one values file.

# values.yaml
datadog:
  apiKey: ${DD_API_KEY}
  site: datadoghq.com
  logs:
    enabled: true
    containerCollectAll: true
  apm:
    portEnabled: true
  processAgent:
    enabled: true
  tags:
    - "env:prod"

clusterAgent:
  enabled: true
  metricsProvider:
    enabled: true          # exposes the External Metrics API so the HPA can scale on a Datadog metric

agents:
  containers:
    agent:
      resources:
        requests: { cpu: 200m, memory: 256Mi }
        limits:   { memory: 512Mi }  # the Agent's own footprint is real — size it, don't leave it uncapped

$ helm repo add datadog https://helm.datadoghq.com && helm repo update
$ helm upgrade --install datadog datadog/datadog -f values.yaml -n datadog --create-namespace

Unified service tagging and custom metrics

Unified service tagging is the single convention that makes the cross-product correlation in the previous section actually work: every product — APM, logs, infra metrics, RUM — recognizes the same three tags, env, service, and version, and a trace, a log line, and a host metric tagged identically get linked automatically in the UI. Set them via environment variables (picked up by the tracing library and the Agent alike) or, in Kubernetes, via pod labels that the Cluster Agent translates into the same environment variables for you.

# pod spec — either set env vars directly, or use these labels and let the
# Cluster Agent derive DD_ENV / DD_SERVICE / DD_VERSION automatically
metadata:
  labels:
    tags.datadoghq.com/env: prod
    tags.datadoghq.com/service: checkout
    tags.datadoghq.com/version: "1.4.3"
spec:
  containers:
    - name: checkout
      env:
        - { name: DD_LOGS_INJECTION, value: "true" }   # stamps trace IDs into log lines automatically

Application-level counters and gauges that don't come from an integration check go through DogStatsD, using a client library in your application's own language:

from datadog import statsd

statsd.increment("checkout.order.completed", tags=["payment_method:card"])
statsd.histogram("checkout.order.total_cents", order_total_cents, tags=[f"currency:{currency}"])
# do NOT tag a custom metric with order_id, user_id, or any other high-cardinality value —
# see "custom metrics and cardinality" in the gotchas section below before you do this

APM instrumentation is either explicit — wrapping the process start command with ddtrace-run (Python), an equivalent wrapper for Java/Node/Ruby/.NET/PHP/Go, or manual span creation in code — or, in Kubernetes with the admission controller enabled, entirely automatic: label a namespace or pod with admission.datadoghq.com/enabled: "true" and the tracing library is injected without touching the application image at all.

Monitors, dashboards, and SLOs as code

☺ Like you're 10: "Monitor" is Datadog's word for an alert rule, and just like everything else worth trusting in production, you write it in a file and let a tool apply it — not click it together by hand and hope nobody forgets why.

Datadog calls an alert rule a monitor, and it supports several types beyond a plain threshold: metric alert, anomaly detection (a machine-learned expected band), outlier detection (one host or pod behaving unlike its peers), forecast (projecting a metric forward and alerting before a hard limit is hit), composite (combining other monitors with boolean logic), log alert, and — the type this page cares about most — SLO alert, which fires on burn rate rather than on the raw metric. Clicking these together in the UI works for a handful of services; at fleet scale, teams manage monitors, dashboards, and SLOs the same way they manage infrastructure: as Terraform resources, reviewed in a pull request, applied by CI.

resource "datadog_monitor" "checkout_error_rate" {
  name    = "checkout: error rate above 2% (5m)"
  type    = "metric alert"
  message = <<-EOT
    {{#is_alert}}Error rate breached 2% for checkout.{{/is_alert}}
    @pagerduty-checkout
  EOT
  query = "sum(last_5m):sum:trace.http.request.errors{service:checkout,env:prod}.as_count() / sum:trace.http.request.hits{service:checkout,env:prod}.as_count() > 0.02"

  monitor_thresholds {
    critical = 0.02
    warning  = 0.01
  }
  notify_no_data    = true
  no_data_timeframe = 10
  tags = ["team:checkout", "service:checkout"]
}

resource "datadog_service_level_objective" "checkout_availability" {
  name        = "checkout: availability"
  type        = "metric"
  description = "99.9% of checkout requests succeed, 30-day rolling window"

  query {
    numerator   = "sum:trace.http.request.hits{service:checkout,env:prod}.as_count() - sum:trace.http.request.errors{service:checkout,env:prod}.as_count()"
    denominator = "sum:trace.http.request.hits{service:checkout,env:prod}.as_count()"
  }
  thresholds {
    timeframe = "30d"
    target    = 99.9
    warning   = 99.95
  }
  tags = ["team:checkout"]
}

Day-to-day commands and workflows

☺ Like you're 10: A handful of commands answer "is the Agent actually working," "what exactly did this one check collect," and "did my change to the alerts actually apply."

# Agent health, on the host itself
$ sudo datadog-agent status                        # full status: checks, forwarder queue, collector runs
$ sudo datadog-agent status collector               # just the check-collection summary
$ sudo datadog-agent check postgres --log-level debug   # run one check once, see exactly what it collected
$ sudo datadog-agent configcheck                    # the merged, resolved config the Agent actually loaded
$ sudo datadog-agent diagnose                        # connectivity + permissions self-test
$ sudo systemctl restart datadog-agent

# Kubernetes
$ kubectl exec -it <agent-pod> -n datadog -- agent status
$ kubectl exec -it <cluster-agent-pod> -n datadog -- datadog-cluster-agent status
$ helm upgrade datadog datadog/datadog -f values.yaml -n datadog   # roll out a config change

# monitors / dashboards / SLOs as code
$ terraform plan -target=datadog_monitor.checkout_error_rate
$ terraform apply
$ terraform import datadog_dashboard.checkout <dashboard-id>       # bring a click-built dashboard under Terraform

# tagging a CI pipeline or deploy for correlation with the monitors above
$ datadog-ci tag --level pipeline --tags team:checkout

Gotchas and failure modes

☺ Like you're 10: Almost every "why is this so expensive" or "why did nobody get paged" story on Datadog traces back to one of a handful of well-known traps.

Custom metrics and cardinality — the classic bill shock

A custom metric is billed per unique combination of metric name and tag values, not per metric name — Datadog calls this a custom metric time series. Emit checkout.order.total_cents tagged with currency and you have a handful of series. Emit the same metric tagged with order_id or user_id, and every single order becomes its own permanent billed time series, because each unique ID is a unique tag value. This is the single most common "why is our Datadog bill ten times what we expected" story in the industry, and it's entirely self-inflicted: nothing warns you at emission time, the cost only shows up on the next invoice or on the Usage page if someone thinks to check it.

Log ingestion and log indexing are billed — and configured — separately

Datadog splits logs into two independent stages with two independent prices: ingestion (every log line the Agent forwards, priced per GB) and indexing (the subset actually made searchable and retained, priced per volume of indexed events at a chosen retention period). The intended pattern is to ingest broadly and cheaply, then write exclusion filters that index only what's worth searching and retaining — debug-level noise from a chatty library, for instance, gets ingested (so it's available briefly and can feed metrics-from-logs) without being indexed at the higher price. Teams that don't know this distinction exists index everything by default and pay for retention they never intended to buy.

Host billing granularity and ephemeral infrastructure

Infrastructure and APM pricing is per host per month, but "host" doesn't always mean what you'd guess: containers on a host generally roll up under that host's license, but a serverless unit like a Fargate task is billed on its own basis rather than as part of a host count. Short-lived CI runners, autoscaled batch nodes, or a fleet of spot instances that scale up and down all day can quietly inflate the effective host count if they're not accounted for — worth checking against Datadog's current billing documentation for the exact rules, since the boundary between "counts as a host" and "doesn't" has shifted as the product line has grown.

Agent resource overhead, DogStatsD packet loss, and site mismatches

The Agent's default footprint is real, not negligible: with logs, APM, process, and network monitoring all enabled, a few hundred megabytes of RAM and a meaningful CPU slice per node is normal, and across a thousand-node fleet that's capacity you're spending to watch capacity. DogStatsD listens over UDP by design — fire-and-forget, no delivery guarantee — so a burst of real traffic that outpaces the Agent's buffer silently drops packets rather than erroring, which means a custom metric can quietly under-report exactly when you need it to be accurate. And because an org is provisioned on one site (datadoghq.com, datadoghq.eu, us3, us5, and others), an API key generated on the wrong site, or a service instrumented against the wrong site's intake, fails in a way that looks like "nothing is showing up" rather than a clear error — check site: in datadog.yaml first when telemetry from one service is mysteriously missing.

Monitor flapping from over-sensitive anomaly and outlier detection

Anomaly and outlier monitors are convenient because they need no hand-picked threshold, but at default sensitivity they're also the monitor type most likely to page on a benign, explainable shift — a deploy that legitimately changes a metric's baseline, a weekly traffic pattern the model hasn't seen enough history to learn yet. Left untuned across a large monitor fleet, this is exactly the mechanism behind alert fatigue: the fix isn't to abandon anomaly detection, it's to tune sensitivity deliberately and to require a sustained breach (not a single data point) before paging, the same discipline multi-window burn-rate alerting applies to SLO monitors specifically.

The real tradeoff: per-host/per-GB pricing vs. self-hosting Prometheus and Grafana

☺ Like you're 10: Renting the control room means someone else keeps it running, but you pay every month, forever, for every machine and every megabyte; building your own means no monthly bill for the software itself, but now the building, wiring, and repairs are your job.

Datadog's price shape, at a high level, is per-host per month for Infrastructure Monitoring and APM (with Free, Pro, and Enterprise tiers gating features), per-GB for log ingestion plus a separate per-volume charge for indexed/retained logs, and per-custom-metric-time-series beyond an included allotment — exactly the mechanism the gotchas above walk through. Treat the specific dollar figures as a moving target and verify them on Datadog's own pricing page before budgeting; what's stable is the shape of the bill, which scales with fleet size and data volume rather than being a flat platform fee.

The self-hosted alternative — Prometheus for metrics, Grafana for dashboards, typically paired with Loki for logs and a long-term-storage layer like Thanos, Mimir, or Cortex once a single Prometheus server's retention and HA limits are hit — has no per-host license fee at all. That doesn't make it free: the cost simply moves from a vendor invoice to compute, storage, and — the part teams chronically underestimate — the engineer-hours to run, upgrade, scale, and be on call for what is now a second production system, one whose own failure means you're blind during the incident you needed it for. High-cardinality metrics still cost you under Prometheus; the cost just shows up as memory pressure and query latency on infrastructure you operate, instead of as a line item someone can see on an invoice.

The honest way to make this decision is the one reliability economics covers in general: price the fully-loaded cost of the option nobody puts on an invoice — the engineer-hours a self-hosted stack consumes — against the vendor bill, at your actual fleet size and team size, not a hypothetical one. Below some crossover point of hosts and headcount, Datadog's invoice is cheaper than the salary-hours of running Prometheus/Grafana/Thanos well; above it, the math usually flips, which is exactly why the largest, most metrics-heavy organizations tend to self-host and small-to-mid-size teams tend to buy.

SLO tracking: Datadog's built-in SLOs vs. a dedicated tool like Nobl9

☺ Like you're 10: Datadog can grade your own homework using only the answers already sitting in its notebook; a dedicated SLO tool can grade homework from several different notebooks at once, even ones Datadog never sees.

Datadog's SLO object, introduced earlier on this page, comes in two shapes: metric-based (a numerator and denominator query over Datadog metrics, as in the Terraform example above) and monitor-based (the SLO is defined as the aggregate uptime of one or more existing Monitors — it's "good" whenever none of them are in an alert state). Both come with a status widget showing current attainment and error-budget remaining, and both can drive a slo alert monitor for burn-rate paging — see SLIs, SLOs & error budgets for the underlying math and SLO windows & composite SLOs for how rolling vs. calendar-aligned windows change what that attainment number even means.

The structural limit is exactly what you'd expect from a feature bundled into a single-vendor platform: a Datadog SLO can only be computed from data Datadog already has. If your actual reliability signal lives somewhere Datadog doesn't see — a payment provider's own status feed, a BigQuery table another team owns, a service still emitting to a Prometheus server nobody's migrated yet — the Datadog SLO object can't reach it until that data is piped into Datadog as a metric first.

Nobl9 is built the opposite way: it's explicitly data-source agnostic, ingesting from Prometheus, Datadog itself, CloudWatch, BigQuery, Splunk, Elasticsearch, and a growing list of others — Datadog is literally just one entry in Nobl9's list of supported Data Sources, not a competitor it replaces. It computes SLOs and composite SLOs centrally, defined in the vendor-neutral OpenSLO YAML spec via its sloctl CLI, and applies one uniform burn-rate alerting policy across every team's SLOs regardless of which backend a given service happens to emit its metrics to. The equivalent idea for a Prometheus-only shop that doesn't want a commercial SLO layer at all is Sloth, which generates the recording and alerting rules directly as Prometheus config from a short declarative spec — no separate platform, no separate bill, but also no cross-backend story if half your fleet is on Datadog.

The decision in practice: if Datadog is genuinely your organization's single source of telemetry truth, its built-in SLO objects are effectively free — already included in a platform you're paying for regardless — and are the pragmatic default with no reason to add a second vendor. Once an org is multi-backend — some teams on Prometheus, some on Datadog, a newly acquired team on a cloud-native monitoring service nobody's migrated off yet — or once you want SLOs defined as portable, version-controlled specs independent of whichever observability vendor happens to be under contract this year, a dedicated layer like Nobl9 earns the extra line item. Both approaches still need the alerting discipline in multi-window, multi-burn-rate alerting layered on top — neither tool makes that judgment call for you.

Where Datadog fits in the SREF blueprint

☺ Like you're 10: The exam won't quiz you on Datadog's pricing page — it wants you to recognize "this is a metrics-and-monitoring tool" from a description, whether or not the name in front of you is one you've used.

The DevOps Institute SRE Foundation (SREF) exam is closed-book and tests tool categories rather than vendor trivia, as SRE Tools & Automation covers in full — Datadog is one of several representative tools listed for the metrics-and-monitoring row alongside Prometheus, Grafana, and InfluxDB, and Monitoring & Service Level Indicators is the domain that actually gets tested. Know what a fully-managed, agent-based, multi-signal platform is for and how it differs structurally from a self-hosted, single-purpose tool — that's the transferable knowledge; the specific Terraform syntax and CLI flags on this page are for the job, not the exam.