New Relic

New Relic is a full-stack observability SaaS platform built outward from a single original idea: instrument the application first, not the server underneath it. Where a lot of monitoring history starts at infrastructure metrics and works inward toward the code, New Relic — launched in 2008 as "New Relic RPM," one of the first true application performance monitoring (APM) products delivered as a hosted service rather than installed software — starts at the transaction a user actually triggered and traces it wherever it goes: through your code, across the network, into a database, and back. Nearly twenty years later that transaction-first lens is still what separates it from a metrics-and-dashboards tool that later bolted tracing on. This page covers how that tracing actually works under the hood, the config and code you write to get it, how New Relic prices what you send it against Datadog's very different model, and the real question behind "New Relic or a self-hosted Prometheus and Grafana stack" — which, spoiler, is rarely about features.

What New Relic is and the problem it solves

☺ Like you're 10: It started as one clever idea — watch the actual request move through your code — and grew into one big shared notebook for every kind of signal your systems produce.

New Relic's founder, Lew Cirne, had already built and sold an earlier APM company (Wily Technology) before starting New Relic in 2008 around a bet that monitoring belonged in the cloud, sold as a service, not installed as on-premises software you patched yourself. That "APM as SaaS" bet is easy to take for granted now, but at the time it was the unusual part — the transaction tracing itself built on ideas that predated the company. What New Relic actually shipped first, under the "RPM" (Rails Performance Monitoring, later reframed as "Real Production Monitoring") name, was a Ruby agent that auto-instrumented a running application and showed a single chart nobody else was showing cleanly: which layer of this specific slow request — your code, the database, an external call — actually owned the time.

The product has since grown into what New Relic calls a single "observability platform" rather than a bundle of point tools: APM (application performance monitoring and distributed tracing), Infrastructure (hosts, containers, Kubernetes), Browser and Mobile (real user monitoring), Logs, Synthetics (scripted uptime and transaction checks from external locations), and Applied Intelligence (anomaly detection and alert correlation) — all writing into one underlying store rather than several siloed products glued together after acquisition. That "one store" claim is the architectural detail worth understanding before anything else on this page, because it's what the rest of New Relic's behavior — good and bad — actually follows from.

Where it fits in the delivery pipeline

☺ Like you're 10: It doesn't build or ship anything — it watches everything you already shipped and tells you, in one place, whether it's actually working.

New Relic sits downstream of CI/CD and deployment strategies entirely — it observes what's running in production (and staging, if you point agents at it), it doesn't build or deploy anything itself. Its most useful integration point with the pipeline is narrow but high-leverage: a Change Tracking event, recorded via one API call from your deploy job, draws a vertical marker on every APM chart at the exact moment a release went out, so an error-rate spike two minutes later is visibly correlated with the deploy that caused it instead of investigated as a mystery.

# record a deployment marker from CI — one line, huge payoff for the next incident review
$ curl https://api.newrelic.com/graphql \
    -H "API-Key: $NEW_RELIC_USER_KEY" -H 'Content-Type: application/json' \
    -d '{"query":"mutation { changeTrackingCreateDeployment(deployment: {entityGuid: \"'"$NR_ENTITY_GUID"'\", version: \"'"$GIT_SHA"'\", user: \"ci-bot\"}) { deploymentId } }"}'

Downstream of that, New Relic is where monitoring & observability and incident management actually happen day to day: its NRQL alert conditions are usually the thing that pages someone through PagerDuty or Opsgenie, and its dashboards are frequently where a deployment-frequency or change-failure-rate story gets told to feed the DORA metrics this course keeps circling back to. See Monitoring & Logging for how it sits alongside logging and metrics generally, and Distributed Tracing & Telemetry for the vendor-neutral theory behind what this page shows as one product's implementation.

Architecture: agents, NRDB, and how transaction tracing actually works

☺ Like you're 10: A small helper rides along inside your app, tags the request as it crosses into each new service, and every tag lands in one giant, fast notebook you can ask questions of later.

Every New Relic language agent — Java, .NET, Node.js, Python, Ruby, PHP, Go — does the same three jobs: auto-instrument the runtime (bytecode weaving for Java and .NET, monkey-patching or module hooks for Node, Python, and Ruby) so common frameworks and database drivers are traced with zero code changes; wrap the current unit of work as a transaction the moment an inbound request arrives; and, when that transaction calls another traced service, inject a trace-context header (New Relic's own format historically, now interoperable with the W3C Trace Context standard) so the receiving service's agent can stitch its own spans onto the same trace instead of starting a new, disconnected one. Since 2021 New Relic also accepts telemetry directly over OTLP, the OpenTelemetry wire protocol — you can skip the New Relic-branded agent entirely and point an OpenTelemetry SDK or Collector straight at New Relic's ingest endpoint, which matters a great deal for the vendor-lock-in question later on this page.

Everything that lands in NRDB — New Relic's proprietary, horizontally scaled, columnar telemetry store — is stored as an event: a timestamped bag of attributes, whether it started life as a metric data point, a log line, a span, or a custom business event. That uniform storage model is the payoff of the "one store" claim from the previous section: a single query language, NRQL (New Relic Query Language, deliberately SQL-shaped), can join a trace's duration against a deploy marker and a log line from the same five minutes, because to NRDB they're the same kind of thing wearing different attributes. Three vocabulary words are worth being precise about, because New Relic's UI uses them constantly: a transaction is one unit of inbound work in one service (a web request, a queue message handled); a span is one traced operation inside that work (a database call, an outbound HTTP call, a traced function); and a trace is the full multi-service chain of spans stitched together by a shared trace ID, which is what the transaction-tracing UI actually renders as a waterfall.

Distributed tracing has a well-known blind spot: sampling only 1% of traces at the point they're created ("head-based" sampling, what nearly every agent does by default to control cost) means the rare, expensive, error-carrying traces you most want to see are exactly the ones statistically likely to get thrown away. New Relic's answer is Infinite Tracing — an optional, Kubernetes-deployed trace observer that sits in front of ingest, holds 100% of spans just long enough to evaluate them, and then makes a "tail-based" keep/drop decision per trace based on rules like always keep errors and always keep the slowest 5%, discarding routine fast traces before storage rather than before they were even seen. It costs more ingest than plain sampling, which is exactly why it's opt-in rather than default.

The config and code you actually write

☺ Like you're 10: One line at the top of your app turns instrumentation on, and one small file describes the alerts you want — both of which belong in your repo, not clicked together in a browser.

Installing an agent is small on purpose. For Node.js, the entire commitment is requiring the agent as the literal first line your process executes, before anything else has a chance to load:

// newrelic.js — agent config, lives next to your entrypoint
'use strict'
exports.config = {
  app_name: ['checkout-api'],
  license_key: process.env.NEW_RELIC_LICENSE_KEY,
  distributed_tracing: { enabled: true },
  logging: { level: 'info' },
  attributes: {
    exclude: ['request.headers.cookie', 'request.headers.authorization']   // strip PII/secrets before they're ever sent
  }
}

// server.js — MUST be the very first require, before express or anything else
require('newrelic')
const express = require('express')
const app = express()

Java attaches as a JVM agent rather than a code change, with the equivalent settings in newrelic.yml, and custom spans use an annotation rather than a wrapper call:

$ java -javaagent:/opt/newrelic/newrelic.jar -jar checkout-service.jar

@Trace(dispatcher = true)
public void chargeCard(Order order) {
    NewRelic.addCustomParameter("order.id", order.getId());   // shows up as a queryable Span attribute
    paymentGateway.charge(order);
}

Bypassing the New Relic-branded agent entirely and shipping OpenTelemetry data straight to New Relic's OTLP endpoint is a common, deliberately vendor-neutral pattern — the instrumentation layer stays portable even if the backend later changes:

# otel-collector-config.yaml — no New Relic agent installed anywhere; OTel does the instrumenting
exporters:
  otlp/newrelic:
    endpoint: otlp.nr-data.net:4317
    headers:
      api-key: ${NEW_RELIC_LICENSE_KEY}
service:
  pipelines:
    traces:  { exporters: [otlp/newrelic] }
    metrics: { exporters: [otlp/newrelic] }

Alerts and dashboards belong in version control the same as everything else this course argues for — see infrastructure as code — and New Relic's official Terraform provider is the standard way to get them there:

terraform {
  required_providers { newrelic = { source = "newrelic/newrelic", version = "~> 3.0" } }
}

resource "newrelic_alert_policy" "checkout" {
  name                = "checkout-service"
  incident_preference = "PER_CONDITION_AND_TARGET"
}

resource "newrelic_nrql_alert_condition" "high_error_rate" {
  policy_id = newrelic_alert_policy.checkout.id
  name      = "Error rate above 5%"
  nrql {
    query = "SELECT percentage(count(*), WHERE error IS true) FROM Transaction WHERE appName = 'checkout-service'"
  }
  critical {
    operator            = "above"
    threshold           = 5
    threshold_duration  = 300
    threshold_occurrences = "at_least_once"
  }
}

Day-to-day commands

☺ Like you're 10: A handful of commands cover most days: ask a question in NRQL, check what's instrumented, and let Terraform ship the alert changes.

# the New Relic CLI — mostly a thin, scriptable front end over NerdGraph
$ newrelic profile configure --profile prod --apiKey NRAK-... --accountId 1234567
$ newrelic apm application list --format json | jq '.[] | {name, id}'
$ newrelic nrql query --accountId 1234567 \
    -q "SELECT apdex(duration, t: 0.5) FROM Transaction WHERE appName = 'checkout-service' SINCE 1 hour ago"

# NRQL directly — the query language behind every chart, alert, and CLI call above
# apdex + throughput per transaction name, last hour
SELECT apdex(duration, t: 0.5), rate(count(*), 1 minute) AS throughput
FROM Transaction WHERE appName = 'checkout-service' FACET name SINCE 1 hour ago

# average span duration by service, for traces that contained an error
SELECT average(duration) FROM Span
WHERE trace.id IN (SELECT trace.id FROM Transaction WHERE error IS true SINCE 1 hour ago)
FACET service.name

# ship alert/dashboard changes the same way you ship infrastructure changes
$ terraform plan -target=newrelic_nrql_alert_condition.high_error_rate
$ terraform apply

Consumption-based pricing: New Relic against Datadog

☺ Like you're 10: New Relic charges mostly for how much you send it and how many people log in; Datadog charges mostly for how many machines you're watching and how many separate features you turn on.

New Relic's pricing since its 2020 overhaul is built on two independent meters: data ingest — billed per gigabyte, per month, and counted the same way regardless of whether that gigabyte was metrics, logs, traces, or events, because it all lands in the same NRDB — and user seats, split between "Full" users (can build dashboards, write alert conditions, run ad hoc NRQL) and unlimited free "Basic" users (largely read-only). New Relic has historically included a meaningful free monthly ingest allowance plus one free full-platform seat; treat any specific gigabyte or dollar figure as something to confirm on New Relic's own pricing page, because consumption pricing on both sides of this comparison changes more often than course material can track reliably.

Datadog's model is structured almost the opposite way: per-host pricing for Infrastructure Monitoring and APM, each billed as its own add-on module per host; Log Management split into a per-GB ingest fee and a separate per-million-events fee just to index and retain what you ingested; and custom metrics billed per distinct metric-and-tag combination emitted — the single most notorious line item for teams that tag freely and find out later. The two models optimize for different shapes of workload:

New Relic against a self-hosted Prometheus and Grafana stack

☺ Like you're 10: Both can answer nearly the same questions — the real choice is whether your team has spare hands to run the answering machine yourselves.

It's tempting to compare New Relic and the open-source "LGTM" stack — Prometheus for metrics, Grafana for dashboards, Loki for logs, Tempo for traces — as a features checklist, and on that axis it's close to a wash: the OSS stack can do distributed tracing, log correlation, and long-term metric storage (via Thanos, Mimir, or Cortex) that looks a lot like what New Relic sells as one product. The decision that actually matters is a different question entirely: who is going to operate the observability platform itself, day two and every day after?

In practice, team size is the tell. A two- or three-person platform team supporting a hundred-plus services genuinely does not have the slack to also run Thanos or Mimir at the reliability the business needs — buying the SaaS wins almost by default, and the "extra" cost of consumption billing is smaller than the fully-loaded cost of the headcount running an equivalent OSS stack well. A thirty-person SRE organization with existing Kubernetes operational muscle, a strict data-residency requirement, and genuine cost sensitivity at very large scale can absorb that operational burden and come out ahead — but that's a staffing and compliance decision wearing a technology comparison's clothes. See FinOps for Delivery Pipelines for the cost side of that trade and On-Call Culture & Sustainable Operations for the staffing side of it.

Gotchas and failure modes

☺ Like you're 10: Most of the pain isn't New Relic breaking — it's a small setup choice quietly costing more, or a trace with a gap in it, months after nobody remembers making that choice.

Trace gaps from an unstitched hop. A trace is only as complete as the weakest link propagating its context. A service running without an agent, a message queue consumed asynchronously without header propagation, or a language New Relic doesn't instrument will silently break the chain — the waterfall view just stops, and it's easy to mistake "the trace ends here" for "the request ended here." Distributed Tracing & Telemetry covers context propagation in more general, vendor-neutral depth.

Full-user seat sprawl. Full users are the seat that costs money; Basic users are free but can't write a query, build a dashboard, or edit an alert condition. Provisioning everyone in engineering as a Full user by default — usually done once, during onboarding, and never revisited — is a quiet, recurring line item that a periodic seat audit catches immediately and nothing else will.

High-cardinality attributes bloat ingest and slow queries even without Datadog's separate per-metric fee. New Relic doesn't bill custom metrics per series the way Datadog does, but a custom attribute set to something unbounded — a request ID, a raw user ID — still inflates the GB you're ingesting and makes any FACET query touching that attribute noticeably slower. The fix is the same discipline either vendor rewards: tag with bounded, meaningful dimensions (service, environment, region), not anything that's effectively unique per event.