Loki

Grafana Loki is a log-aggregation system built around one deliberately narrow bet: don't index the content of the log line at all — index only the small set of labels that describe where the line came from, and leave the line itself sitting, compressed but otherwise untouched, in cheap object storage. Grafana Labs' own shorthand for this since Loki's 2018 announcement has been "like Prometheus, but for logs," and the comparison is exact — a Loki stream is architecturally the same idea as a Prometheus series, identified by a label set, and LogQL is PromQL's syntax bent to match lines instead of numbers. This page covers that indexing bet in full: why it makes Loki dramatically cheaper to run than Elasticsearch for the same log volume, what LogQL looks like once you already know PromQL, and the one place the analogy to Prometheus gets genuinely dangerous — Loki's cardinality limits break in almost the opposite direction from a full-text-indexed store's.

What Loki is, and the bet its architecture makes

☺ Like you're 10: It writes down where a box of letters is, not what's inside them — so filing is nearly free, but reading a specific box still means actually opening it.

Every log-aggregation system has to answer the same two questions eventually: where do I put the raw text, and how do I find a specific line again later without reading everything? Elasticsearch — and the broader ELK/EFK family covered on the Elastic Stack page — answers both at once with an inverted index: every log line is tokenized, every distinct token is indexed against every document containing it, and the index itself typically runs 25–100%+ the size of the raw data it describes. That's what buys sub-second, arbitrary full-text search across a whole fleet's logs regardless of which fields you tagged anything with — and it's also why Elasticsearch clusters are expensive to run at real log volume: the indexing work and the index's own storage footprint happen continuously, on every line ingested, whether or not anyone ever queries it.

Loki refuses that trade on purpose. It builds an index over exactly one thing — the label set attached to a stream, a small, low-cardinality set like {app="checkout", namespace="prod", pod="checkout-7d9f4-x2k1p"} — and stores the log lines themselves as gzip- or zstd-compressed chunks in object storage (S3, GCS, Azure Blob, or a filesystem for small setups), addressed by which stream and time range they belong to. Nothing about the line's actual text — the words, the JSON fields, the stack trace — ever gets tokenized or indexed. A query first uses the (tiny) index to find which streams and chunks are even in scope, then reads and greps the actual chunk bytes for anything past that point. Grafana Labs' own framing for this is exactly the phrase this page's title borrows: index the metadata, not the log line.

Architecture — the write path and the read path

☺ Like you're 10: An agent tags each line and ships it off; Loki batches matching lines into a box per stream, files the box's card in a tiny index, and only reads the actual box when you ask a specific enough question.

Loki is a set of components that can run as one binary (for a laptop or a small cluster) or scaled out independently at real volume. The shape of the pipeline is the same either way.

The four pieces worth knowing by name

A Distributor is the write-path front door: it validates incoming pushes, computes a hash of each stream's label set, and forwards it to the Ingester (or ingesters, at a configured replication factor) responsible for that hash range. An Ingester holds each stream's most recent chunk in memory — appending new lines to it, backed by a write-ahead log for crash recovery — until the chunk hits a size or age threshold and gets flushed as an immutable, compressed object to durable storage, at which point the ingester also writes an entry into the index recording which stream and time range that chunk covers. A Querier answers LogQL by combining two sources: whatever's still sitting unflushed in the ingesters (the most recent few minutes) and historical chunks read back from object storage, guided by the index. At real scale, a Query Frontend sits in front of the queriers and splits a large query by time range, runs the pieces in parallel, and caches results — the reason a Loki query over a week of data doesn't just serialize against one process.

Three deployment shapes, one binary

The same loki binary runs three ways, chosen by how much you scale: monolithic mode (a single process runs every component — target: all — fine for a laptop, a demo, or genuinely small production log volume), simple scalable deployment (SSD — the default recommendation for real production since Loki 2.4: exactly two deployment targets, read and write, each independently scaled, which covers the overwhelming majority of teams without the next tier's operational overhead), and full microservices mode (every component above — distributor, ingester, querier, query frontend, compactor, index gateway — deployed and scaled completely independently, reserved for the handful of organizations ingesting at genuinely enormous, many-terabytes-a-day volume). Start at SSD; don't reach for microservices mode until SSD has demonstrably run out of runway.

LogQL — PromQL's grammar, aimed at lines instead of numbers

☺ Like you're 10: First you point at the right box using labels — exactly like a PromQL selector — then you pipe what comes out through filters, the same way you'd pipe a shell command through grep.

If you've written a PromQL query in this course's SLIs, SLOs & error budgets page or on the Prometheus page, LogQL's shape is already familiar: a log stream selector in curly braces, using exactly the same =, !=, =~, !~ operators PromQL uses, is mandatory and must match on the indexed labels first. Everything after that selector is a pipeline of stages applied left to right, and it's the piping model — not the selector — that's LogQL's actual innovation over PromQL's syntax.

# A log query: find the raw lines
{app="checkout", env="prod"} |= "timeout" != "context canceled"

# Parse JSON, then filter on an extracted field — the parser runs BEFORE the label filter
{app="checkout"} | json | status_code >= 500 | line_format "{{.method}} {{.path}} → {{.status_code}}"

# A metric query — the LogQL equivalent of Prometheus's rate() on a counter, but the "counter"
# is just "how many matching lines showed up," computed at query time from raw chunk bytes
sum by (status_code) (
  rate({app="checkout", env="prod"} | json | status_code >= 500 [5m])
)

# unwrap — pull a numeric field out of every matching line and quantile it, same shape as
# histogram_quantile() in PromQL but sourced from a field embedded in the log line itself
quantile_over_time(0.99,
  {app="checkout"} | json | unwrap duration_ms [5m]
) by (route)

Config and agent setup you actually write

☺ Like you're 10: One file tells Loki where to keep the boxes and the index cards; a second file tells the collector which few labels are safe to write on the outside of the box.

A production loki-config.yaml is mostly storage and schema wiring — the interesting knobs are the schema's dated cutover row and the limits that keep a bad label choice from becoming an incident.

# loki-config.yaml
auth_enabled: false            # true + X-Scope-OrgID header for real multi-tenancy

common:
  path_prefix: /loki
  replication_factor: 3
  storage:
    s3:
      bucketnames: acme-loki-chunks
      region: us-east-1
  ring:
    kvstore: { store: memberlist }

schema_config:
  configs:
    - from: 2024-06-01              # schema changes apply from this date FORWARD only —
      store: tsdb                    # never edit an old row; append a new one with a new `from`
      object_store: s3
      schema: v13                    # check Grafana Labs' docs for the current recommended value
      index:
        prefix: loki_index_
        period: 24h

limits_config:
  retention_period: 720h            # 30d — enforced by the compactor, not by this number alone
  ingestion_rate_mb: 10
  ingestion_burst_size_mb: 20
  max_streams_per_user: 10000       # a hard ceiling — see Gotchas for what happens near it
  max_label_names_per_series: 15
  reject_old_samples: true
  reject_old_samples_max_age: 168h

compactor:
  working_directory: /loki/compactor
  retention_enabled: true            # forgetting this: chunks pile up in S3 forever, billed forever

On the write side, an agent tails logs and attaches labels before anything reaches Loki. Promtail was Loki's original purpose-built agent; Grafana Labs has since been consolidating its whole agent lineup — Promtail, the Grafana Agent, and an OpenTelemetry Collector distribution — into a single unified collector called Grafana Alloy, so check which one a given deployment is standardizing on before you copy a config verbatim. Either way, the discipline that matters is the same: keep the label set small and bounded.

# promtail-config.yaml — Kubernetes pod discovery, mirroring Prometheus's own kubernetes_sd_configs
server:
  http_listen_port: 9080
positions:
  filename: /tmp/positions.yaml
clients:
  - url: http://loki-gateway.monitoring.svc/loki/api/v1/push

scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    pipeline_stages:
      - docker: {}                          # unwrap the Docker/CRI log envelope first
      - json:
          expressions: { level: level }      # parse JSON but extract only ONE field as a label
      - labels: { level: }                   # level has ~5 values — safe to index
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        target_label: app
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      # deliberately NOT relabeling pod name, request path, user id, or trace id onto labels —
      # those stay in the line content, findable with |= / | json, not indexed as streams

Day-to-day commands

☺ Like you're 10: Mostly you ask questions through Grafana's Explore tab — but a small command-line tool exists for scripting the same questions.

# logcli — Loki's query CLI, the LogQL equivalent of promtool query
$ export LOKI_ADDR=http://localhost:3100

$ logcli labels app                                  # list every value seen for one label
$ logcli series '{app="checkout"}'                    # list every distinct stream matching a selector

$ logcli query '{app="checkout", env="prod"} |= "timeout"' --since=1h --limit=200
$ logcli query '{app="checkout"} | json | status_code >= 500' --since=30m -o raw

# a metric query, same as you'd paste into Grafana Explore
$ logcli query 'sum by (status_code) (rate({app="checkout"} | json [5m]))' --since=1h

# --- Helm, the standard install path ---
$ helm repo add grafana https://grafana.github.io/helm-charts
$ helm install loki grafana/loki -f values.yaml --namespace monitoring --create-namespace
# values.yaml sets deploymentMode: SimpleScalable, loki.storage.type: s3, retention, limits

# --- reaching a running instance directly ---
$ kubectl -n monitoring port-forward svc/loki-gateway 3100:80
$ curl -s localhost:3100/ready
$ curl -s -H 'X-Scope-OrgID: acme' localhost:3100/loki/api/v1/labels   # required when auth_enabled: true

In practice, most day-to-day querying happens through Grafana's Explore tab against a Loki data source rather than logcli — the same tab this course's Grafana page covers, including the trick where a trace ID spotted in a log line becomes a one-click jump into Tempo or Jaeger via a derivedFields configuration on the data source.

Gotchas and failure modes

☺ Like you're 10: The two ways Loki actually breaks are opposites of each other: writing too many different labels on the box, or asking it to check every box in the building at once.

Stream cardinality — the same failure mode as Prometheus, for a worse reason

This is the sharpest edge on this whole page, and it's exactly the concern the content brief for this page points at. A Loki stream is identified by its complete label set, precisely the way a Prometheus series is — so a label with effectively unbounded values (a raw request path, a user ID, a pod name that churns on every deploy, a trace ID) multiplies the number of streams the same way it multiplies Prometheus series, covered on the Prometheus page's own cardinality gotcha. But Loki's consequence compounds further: each new stream also gets its own chunk, and a chunk that never accumulates enough lines from its one narrow stream to hit its target size before it ages out gets flushed small and poorly compressed — "stream churn" or "the small chunks problem," visible as rising ingester memory, degraded compression ratios, and slower queries against a store now fragmented into far more objects than the log volume actually warrants. Past limits_config.max_streams_per_user, new streams are rejected outright with a 429 and a "per-stream rate limit exceeded" or "max streams per user exceeded" error, not silently absorbed.

A broad query is a distributed grep, not a lookup

This is where the "index the metadata, not the log line" bet sends the bill due. Because line content is never indexed, a query whose stream selector is broad — or a query spanning a large time range even against a narrow selector — can't skip past irrelevant data the way an inverted index would; it has to decompress and scan every chunk the selector and time range pull in. The query frontend parallelizes that scan across queriers and time-shards, and result and chunk caches absorb a lot of the repeated cost, but a genuinely unbounded query (a nearly-empty label selector over weeks of data) can still be slow, get load-shed with a 429 too many outstanding requests, or simply run past a configured query timeout. This is the precise mirror image of Elasticsearch's failure mode: Elasticsearch pays a steady, continuous tax on every line at ingest time so that any later search is fast regardless of scope; Loki pays almost nothing at ingest time and instead pays proportional to how broad and unselective the query turns out to be. Narrow with labels first, filter content second — always in that order.

The other operational sharp edges

• Loki is not a search engine. No relevance ranking, no fuzzy matching, no cross-field scoring — it answers "show me matching lines for a service/pod/trace I can already name," not "find anything in the fleet that might be about this." If genuinely unstructured, ad-hoc full-text discovery across arbitrary content is the actual requirement, that's Elasticsearch's real strength, at Elasticsearch's real cost — see the comparison table below.
• Chronological order per stream. Loki has historically required each individual stream's lines to arrive in non-decreasing timestamp order, rejecting late or out-of-order pushes with entry out of order; reject_old_samples_max_age and Loki's out-of-order ingestion window (relaxed starting around Loki 2.4) both bound how much slack you actually have — worth checking against your log shipper's batching and retry behavior before you're surprised by dropped lines during an agent restart.
• Schema changes are additive, never edited in place. Changing the index type (boltdb-shipper → tsdb) or the index period means appending a new dated row to schema_config.configs, not rewriting the existing one — old data stays queryable under its original schema entry. Editing history-affecting settings in place instead of appending a row is how teams orphan old index data by accident.
• Retention is the compactor's job, not a config number's. limits_config.retention_period does nothing on its own if compactor.retention_enabled is false or the compactor isn't running — chunks accumulate in object storage, and get billed for, indefinitely regardless of what the retention number says.
• Multi-tenancy is a header, not magic. With auth_enabled: true, every push and query needs an X-Scope-OrgID header identifying the tenant; forget it and requests fail immediately with a 401, which is usually the first thing to check when "Loki isn't receiving anything" turns out to mean "Loki is rejecting everything."

Where Loki sits, and the comparison that decides whether to reach for it

☺ Like you're 10: Loki is the cheap-by-default choice when you already know roughly which box you're looking in; the full-text catalog earns its cost back when you genuinely don't.

Loki is the logs corner of the same trio Grafana's own page walks through — metrics via Prometheus, logs via Loki, traces via Tempo or Jaeger — queried from one Explore pane, with a trace ID in a log line jumping straight to the matching span. It's the log-aggregation half of SRE Tools & Automation's tool landscape, sits next to the Elastic Stack as the two named log-aggregation options that domain calls out, and is exactly the kind of backend monitoring & observability assumes exists once a metric alone can't explain why something broke. If you want reps building this stack rather than just reading about it, Capstone Part 2 — build the monitoring & alerting is the exercise that puts a real Loki config and LogQL queries in front of you.

The practical rule most SRE teams converge on: if the query you'll actually run is almost always "show me the logs for this service, this pod, this trace, in this window" — which is the overwhelming majority of real on-call log usage — Loki's bet pays for itself immediately, because that's precisely the query shape its label-first index is built for. Reach for Elasticsearch specifically when the actual requirement is unstructured discovery across content you can't predict how to label in advance — security/audit log search across an entire estate is the classic case — and accept the cost that buys.

Going further

☺ Like you're 10: This page is enough to write a real config and a real LogQL query — the official docs are where the deployment-sizing and schema-migration detail lives.

The canonical source is Grafana Labs' documentation at grafana.com/docs/loki — the LogQL reference and the storage/schema pages are worth reading end to end once this page's shape is familiar — alongside the source at github.com/grafana/loki. Loki, Grafana, and Tempo moved from Apache 2.0 to the AGPLv3 license in 2021; verify current licensing terms on Grafana Labs' own site before making a packaging or redistribution decision that depends on it, the same way this course's other tool and certification pages recommend checking anything pricing- or licensing-adjacent against the vendor. Pair this page with Grafana for how Loki gets queried and alerted on day to day, the Elastic Stack for the full-text-indexed alternative worked through in equal depth, and monitoring & observability for where logs fit among the signals an SRE actually watches.