Loki

Loki is Grafana Labs’ log aggregation system, and its one big idea is a refusal: it does not index the contents of your log lines. It indexes only a small set of labels — namespace, app, pod, container — keeps the raw log text compressed in cheap object storage, and brute-force greps it at query time. That trade makes logs affordable enough to keep for every workload on a platform, and it solves the problem every platform team eventually hits: “the metric says errors went up at 14:32, and now I need the actual sentence the application wrote when it failed” — without buying an expensive search cluster or asking a single tenant to configure anything.

What Loki is and the problem it solves

☺ Like you’re 10: It keeps everybody’s log messages in a cheap warehouse, labelled on the outside, so you can go and read the exact sentences later.

Loki was started at Grafana Labs in 2018 with an explicit design brief: build the logging equivalent of Prometheus. Same label data model, same service discovery, same query feel, same operational simplicity — a system you can run yourself without a full-time team. It is open source (AGPLv3), it is not a CNCF project, and it has quietly become the default log backend of the cloud native stack, largely because it is the one people can actually afford to leave switched on.

The problem before Loki

The classic answer to “where do my logs go?” was the ELK/EFK stack: Elasticsearch with Fluentd or Logstash feeding it and Kibana on top. Elasticsearch is a genuinely excellent full-text search engine, and that is precisely the problem — it builds an inverted index over every token in every log line. On a busy cluster the index can approach or exceed the size of the logs themselves, it must live on fast SSDs, and it needs enough RAM to keep the hot parts resident. Teams responded the only way they could: by logging less, sampling, or dropping retention to three days. The observability data you most want during an incident is the data you deleted last Tuesday to save money.

The bet: index labels, not text

Loki’s bet is that you almost never search the whole world of logs. You search a slice — this namespace, this app, this hour — and then you filter. So Loki indexes only the label set, which is tiny, and stores the log lines themselves as compressed chunks in object storage (S3, GCS, Azure Blob, MinIO, or a filesystem for toy setups). A query resolves labels through the index to a handful of chunks, then the queriers download those chunks and run the filters in parallel, at gigabytes per second. You lose “find this UUID anywhere in the last year in under a second.” You gain logs that cost roughly what object storage costs, which is close to nothing.

What Loki deliberately is not

Loki is not a full-text search engine, not a SIEM, and not a metrics store. It will not do fuzzy matching, relevance ranking or aggregations over arbitrary fields the way Elasticsearch will. It is also not a tracing system: it can find you the log lines that mention a trace ID, but reconstructing the causal path across twelve services is Jaeger’s or Tempo’s job. And while LogQL can compute metrics from logs, doing so at scale is far more expensive than a counter that Prometheus already scrapes. The observability lesson lays out how the three pillars divide the work; Loki owns exactly one of them.

Where Loki fits in a platform

☺ Like you’re 10: Loki sits next to Prometheus in the platform’s watchtower. Prometheus keeps the numbers; Loki keeps the sentences.

In the reference architecture, Loki lives in the observability plane: a shared capability the platform team runs once, which every tenant consumes for free by the simple act of writing to stdout. That last part is the real platform win. A team onboards to logging by doing nothing — the collection agent already tails every container’s log file on every node. Compare that with metrics, where a tenant must instrument code and write a ServiceMonitor. Like everything else on the platform, Loki itself is installed and configured through GitOps, usually via its Helm chart.

Its neighbours, and who does what

Grafana is the front end: Loki has no UI of its own, and you query it through Grafana’s Explore view or a dashboard panel. Prometheus is the sibling that answers “how much, how fast, how bad” while Loki answers “what exactly happened.” OpenTelemetry can ship logs into Loki through the Collector’s OTLP/HTTP exporter, because Loki 3.x exposes a native OTLP ingestion endpoint — the older dedicated Loki exporter in the Collector was deprecated in favour of exactly that. Jaeger and Tempo hold traces, and Grafana stitches Loki to them with derived fields that turn a trace ID inside a log line into a clickable link. Loki even has a ruler component that evaluates alerting rules written in LogQL and fires them at the same Alertmanager Prometheus uses — so “no successful payment logged in ten minutes” can page someone.

CNPE domain relevance

Loki is not on the official CNPE tool list — the exam names Prometheus, Grafana, OpenTelemetry and Jaeger for observability. But the exam’s Observability domain is about concepts: the three pillars, structured logging, correlation, retention and cost. Loki is the tool the industry actually reaches for when the answer is “logs,” so it shows up constantly in real platform work and in scenario-style questions about aggregating container logs. Treat it as background you should understand well and be able to discuss, rather than YAML you must reproduce under time pressure. It also touches FinOps (log retention is a real line item), storage (object storage lifecycle), and incident response, where it is usually the second place anyone looks.

How it works — architecture and components

☺ Like you’re 10: One team of helpers catches log lines and packs them into boxes; another team fetches boxes back out and reads them fast.

Loki is a single Go binary that can wear different hats. You start it with a -target flag, and that flag decides which of its internal components it runs. Understanding the two paths — write and read — makes almost every failure mode obvious.

The components, one by one

Streams, chunks, and the index

Every log line arrives with a timestamp, the line itself, and a label set. The label set defines the stream. Loki appends lines to that stream’s current chunk — a compressed block, typically flushed at around 1.5 MB or after max_chunk_age (2h by default) — and writes an index entry recording “stream X, these labels, this time window, this chunk object.” Modern Loki uses a TSDB index (schema v13), the same technology as Prometheus’ head index, stored in the same object storage bucket as the chunks. This is the single store model: no Cassandra, no DynamoDB, no separate index database. One bucket holds everything.

Loki 3.0 added structured metadata: per-line key/value pairs that are stored with the chunk rather than in the index. This is the sanctioned home for high-cardinality attributes like a trace ID or a pod IP — you can filter on them, but they never create new streams. It is the escape valve that makes the “no high-cardinality labels” rule livable.

Deployment modes

The same binary scales three ways, and choosing correctly is most of the operational battle.

The resources you will actually write

☺ Like you’re 10: Three files: one that tells Loki where the warehouse is, one that tells the collector which boxes to label how, and a pile of little questions you type all day.

A Loki configuration — storage, schema, limits, retention

This is the shape of a simple-scalable Loki config, normally delivered through the Helm chart’s Loki config values (loki.config as a template, or loki.structuredConfig when you want to merge YAML in). The four blocks that matter are schema_config (which index format, from when), storage_config (which bucket), limits_config (the guard rails that stop one noisy tenant ruining the day), and compactor (which is where retention actually lives).

auth_enabled: true            # require the X-Scope-OrgID header = multi-tenancy on

common:
  path_prefix: /var/loki
  replication_factor: 3
  ring:
    kvstore: { store: memberlist }

schema_config:
  configs:
    - from: 2024-06-01        # schema periods are append-only: NEVER edit a past
      store: tsdb             # entry, always add a NEW one with a future date
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

storage_config:
  aws:
    bucketnames: acme-loki-chunks
    region: eu-west-1
    s3forcepathstyle: false   # credentials come from IRSA / workload identity
  tsdb_shipper:
    active_index_directory: /var/loki/tsdb-index
    cache_location: /var/loki/tsdb-cache

limits_config:
  ingestion_rate_mb: 8              # per tenant (default strategy is 'global',
                                    # i.e. shared across all distributors)
  ingestion_burst_size_mb: 16
  max_label_names_per_series: 15    # a hard stop on label sprawl
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  max_query_length: 721h            # refuse absurd time ranges
  max_query_parallelism: 32
  max_entries_limit_per_query: 5000
  retention_period: 720h            # 30 days — the DEFAULT for every stream
  volume_enabled: true              # powers Grafana's "log volume" explorer

compactor:
  working_directory: /var/loki/compactor
  retention_enabled: true           # WITHOUT this, retention_period does NOTHING
  delete_request_store: s3
  compaction_interval: 10m
  retention_delete_delay: 2h

runtime_config:
  file: /etc/loki/runtime/overrides.yaml
  period: 10s               # reloaded live, without restarting Loki

# ---------------------------------------------------------------------
# A SEPARATE FILE — /etc/loki/runtime/overrides.yaml. Per-tenant and
# per-stream overrides do NOT belong in the main config: keep audit logs
# long, throw debug noise away fast.
# ---------------------------------------------------------------------
overrides:
  acme-platform:
    retention_period: 720h
    retention_stream:
      - selector: '{namespace="security-audit"}'
        priority: 1
        period: 2160h                # 90 days
      - selector: '{namespace=~"dev-.*"}'
        priority: 2
        period: 72h                  # 3 days

Collecting Kubernetes pod logs with Grafana Alloy

Loki does not collect anything itself — an agent does. The agent runs as a DaemonSet, discovers pods through the Kubernetes API exactly as Prometheus does, tails the container log files under /var/log/pods, attaches labels derived from pod metadata, and pushes batches to the distributor. Promtail was the original agent and you will still meet it everywhere, but Grafana has frozen its features and declared end of life; Grafana Alloy is the successor. OpenTelemetry Collector, Fluent Bit and Fluentd can all write to Loki too.

// Grafana Alloy configuration. 1) discover pods, 2) rewrite metadata into
// a SMALL label set, 3) tail, 4) parse the container runtime envelope, 5) push.

discovery.kubernetes "pods" {
  role = "pod"
}

discovery.relabel "pods" {
  targets = discovery.kubernetes.pods.targets

  // Only tail pods on THIS node — the DaemonSet pattern.
  rule {
    source_labels = ["__meta_kubernetes_pod_node_name"]
    regex         = sys.env("HOSTNAME")
    action        = "keep"
  }
  rule {
    source_labels = ["__meta_kubernetes_namespace"]
    target_label  = "namespace"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
    target_label  = "app"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_container_name"]
    target_label  = "container"
  }
  // Build the on-disk glob to tail, i.e. /var/log/pods/*POD_UID/CONTAINER/*.log
  rule {
    source_labels = ["__meta_kubernetes_pod_uid", "__meta_kubernetes_pod_container_name"]
    separator     = "/"
    action        = "replace"
    replacement   = "/var/log/pods/*$1/*.log"
    target_label  = "__path__"
  }
  // NOTE: no pod_ip, no request_id, no version-with-git-sha. Those are
  // unbounded and would create a new stream per value. See the gotchas.
}

local.file_match "pods" {
  path_targets = discovery.relabel.pods.output
}

loki.source.file "pods" {
  targets    = local.file_match.pods.targets
  forward_to = [loki.process.default.receiver]
}

loki.process "default" {
  stage.cri {}                       // strip the containerd log envelope

  stage.json {                       // pull fields out of a JSON log line
    expressions = { level = "level", trace_id = "trace_id" }
  }
  stage.labels {
    values = { level = "" }          // BOUNDED: info/warn/error/debug only
  }
  stage.structured_metadata {
    values = { trace_id = "" }       // HIGH cardinality: metadata, not a label
  }
  stage.drop {
    expression = "GET /healthz"      // don't pay to store probe spam
  }
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url       = "http://loki-gateway.monitoring.svc/loki/api/v1/push"
    tenant_id = "acme-platform"
  }
}

LogQL — the query language you type all day

A LogQL query has two halves. First a stream selector in curly braces, which uses the index and is mandatory; then a pipeline of filters and parsers separated by |, which runs over the raw lines. Line filters are the cheap ones — run them before parsers so you parse fewer lines.

# --- Stream selectors (indexed; = != =~ !~ exactly as in Prometheus) ---
{namespace="checkout", container="api"}
{namespace=~"prod-.*", app!="healthcheck"}

# --- Line filters (a fast grep over the raw text) ---
# |=  contains        !=  does not contain
# |~  regex matches   !~  regex does not match
{namespace="checkout"} |= "error" != "timeout" |~ `(?i)connection (refused|reset)`

# --- Parsers: turn the line into labels you can filter and aggregate on ---
{namespace="checkout"} | json                      # all JSON fields become labels
{namespace="checkout"} | json level="lvl", user="usr.id"   # or pick and rename
{namespace="checkout"} | logfmt                    # key=value lines
{namespace="ingress"}  | pattern `<ip> - - <_> "<method> <uri> <_>" <status> <size>`
{namespace="legacy"}   | regexp `(?P<status>\d{3}) (?P<dur>\d+)ms`

# --- Label filters run AFTER a parser; they understand numbers and durations ---
{namespace="checkout"} | json | status >= 500 and duration > 250ms
{namespace="checkout"} | logfmt | level="error" | user_id != ""

# --- Formatting: rewrite the line, or synthesise a label ---
{namespace="checkout"} | json | line_format `{{.level}} {{.msg}} ({{.trace_id}})`
{namespace="checkout"} | json | label_format route=`{{ .path }}`

# --- METRIC QUERIES: turn logs into a graph. These return numbers, not lines. ---
# rate() = entries per second; count_over_time() = entries in the window
sum by (app) (rate({namespace="checkout"} |= "error" [5m]))
sum by (level) (count_over_time({namespace="checkout"} | json | __error__="" [1m]))
topk(5, sum by (app) (count_over_time({namespace="checkout"}[1h])))   # noisiest apps
bytes_over_time({namespace="checkout"}[1h])              # who is generating cost?

# unwrap turns a parsed numeric field into a real time series
quantile_over_time(0.99,
  {namespace="checkout"} | json | unwrap duration_ms [5m]) by (route)

# Error ratio, computed from logs alone
  sum(rate({namespace="checkout"} | json | status >= 500 [5m]))
/ sum(rate({namespace="checkout"} | json [5m]))

Two habits make LogQL fast. Narrow the stream selector first — every label you can add is index work instead of chunk-scanning work. Put line filters before parsers — |= "error" | json parses only the lines containing “error”, while | json |= "error" parses everything first. And remember | __error__="": when a parser fails on a line, Loki tags it rather than dropping the query, and that filter hides the failures.

Day-to-day commands

☺ Like you’re 10: Mostly you ask questions from a terminal, check which labels exist, and confirm the collector is actually shipping anything.

logcli — LogQL from the terminal

logcli is Loki’s official CLI. It is the fastest way to check whether data exists at all, and it is scriptable, which Grafana is not.

export LOKI_ADDR=http://localhost:3100
export LOKI_ORG_ID=acme-platform          # required when auth_enabled: true

logcli labels                              # which label NAMES exist at all?
logcli labels namespace                    # which VALUES does 'namespace' have?
logcli series '{namespace="checkout"}'     # which streams exist in that slice?

logcli query '{namespace="checkout"} |= "error"' --since=1h --limit=200
logcli query '{namespace="checkout"} | json | status >= 500' \
  --from="2026-07-19T09:00:00Z" --to="2026-07-19T10:00:00Z" --output=jsonl

logcli instant-query 'sum by (app) (rate({namespace="checkout"} |= "error" [5m]))'
logcli query '{namespace="checkout"}' --tail                   # live follow
logcli query '{namespace="checkout"} |= "error"' --stats        # what did it cost?

logcli stats '{namespace="checkout"}' --since=1h   # streams/chunks/bytes BEFORE you query
logcli query '{namespace="checkout"}' --since=15m --output=raw --no-labels

Checking Loki itself, from kubectl

kubectl -n monitoring get pods -l app.kubernetes.io/name=loki
kubectl -n monitoring port-forward svc/loki-gateway 3100:80

curl -s http://localhost:3100/ready                     # is it up?
curl -s http://localhost:3100/config | head -40         # the config it ACTUALLY loaded
curl -s http://localhost:3100/metrics | grep -E 'loki_(distributor|ingester)_'

# Ask the HTTP API directly — this is exactly what Grafana does.
curl -sG http://localhost:3100/loki/api/v1/query_range \
  -H 'X-Scope-OrgID: acme-platform' \
  --data-urlencode 'query={namespace="checkout"} |= "error"' \
  --data-urlencode 'limit=20' | jq '.data.result[].values[][1]'

curl -sG http://localhost:3100/loki/api/v1/labels -H 'X-Scope-OrgID: acme-platform'

# Is the AGENT the problem, or Loki? Check the collector first.
kubectl -n monitoring logs ds/alloy | grep -iE 'error|429|entry out of order'
kubectl -n monitoring exec ds/alloy -- ls /var/log/pods | head

# Ingester health and flush behaviour — the write path's own metrics.
curl -s http://localhost:3100/metrics | grep -E 'loki_ingester_(memory_streams|chunks_flushed_total)'

Gotchas and failure modes

☺ Like you’re 10: Here are the ways Loki gets slow, expensive, or quietly refuses your logs.

Label cardinality — the cardinal sin

This is the failure. Every unique combination of label values is a separate stream with its own in-memory buffer and its own chunks. Put a pod name, a pod IP, a request ID, a user ID, a full URL path or a git-SHA version label on your streams and the stream count goes from hundreds to millions. The symptoms arrive in order: ingester memory climbs, chunks flush tiny and half-empty (wrecking compression), the index bloats, queries slow to a crawl, and eventually ingesters are OOM-killed. Keep to roughly five to ten labels, all with bounded values — cluster, namespace, app, container, level. Everything else belongs in the log line, where a line filter or parser can reach it for free, or in structured metadata. Diagnose with logcli series, the loki_ingester_memory_streams metric, and the /loki/api/v1/index/volume endpoint.

Rejected writes: rate limits and out-of-order entries

Agents surface Loki’s complaints as HTTP 429 or 400 and then retry, so logs go missing quietly. Ingestion rate limit exceeded means limits_config.ingestion_rate_mb is too low for the tenant (or one app is spewing) — raise the limit or drop the noise at the agent. entry too far behind means a line’s timestamp is older than the accepted window; this happens when an app back-fills, when a node’s clock is wrong, or when you parse a timestamp out of the line with the wrong timezone. Loki accepts moderately out-of-order writes within a stream, but not arbitrarily old ones. And Maximum active stream limit exceeded is the cardinality problem announcing itself politely. Always read the agent’s logs before concluding Loki is broken — workload triage and the troubleshooting playbook use the same “check the client before the server” discipline.

Queries that time out or refuse to run

max entries limit per query exceeded and query time range exceeds limit are guard rails doing their job — narrow the selector or the window rather than raising the limit reflexively. too many outstanding requests means the query-frontend queue is full, usually because a handful of unbounded queries (a bare {namespace=~".+"} over seven days) are eating all the querier capacity. The structural fix is the query-frontend with splitting, sharding and result caching enabled, plus enough queriers; the cultural fix is teaching people to always name a namespace. Run logcli query --stats to see bytes processed — it is startling how much a lazy selector costs.

Unstructured logs, and retention that never happens

Loki is only as good as what your applications write. A free-text line like Something went wrong!! can be grepped and nothing more; a JSON or logfmt line with level, msg, trace_id and duration_ms can be parsed, filtered numerically, aggregated and linked to a trace. Pushing teams toward structured logging is a developer-experience job — ship it in the golden path libraries, do not send a memo. The other silent trap: limits_config.retention_period does nothing unless compactor.retention_enabled: true is also set. Plenty of teams discover this when the storage bill arrives with three years of logs on it. Belt and braces: set Loki retention and a bucket lifecycle policy, and check that the compactor pod is actually running.

Alternatives and when to choose it

☺ Like you’re 10: Other log warehouses exist. They mostly trade money for search power — pick where on that line you want to sit.

The honest way to choose a log backend is to decide how much you are willing to pay for full-text search you will use perhaps twice a month. Loki sits deliberately at the cheap end; Elasticsearch sits at the powerful end; most of the rest are variations on that axis.

A practical rule

If your platform already runs Prometheus and Grafana — which, if you have read the observability lesson, it should — Loki is the path of least resistance and least cost, because your teams already know the label model and the query feel. Choose Elasticsearch instead when a real, named requirement demands full-text search or long-lived audit search, and be honest about staffing it. Many mature platforms run both: Loki for all application logs, a smaller search cluster for security and compliance logs that governance requires be searchable for years. See The Tool Landscape for how Loki sits among the projects the exam does name.

Exam relevance and going further

☺ Like you’re 10: Loki isn’t on the exam’s tool list, and you can’t open its website on exam day either — so learn the ideas, not the YAML.

Loki is not named on the official CNPE tool list. Nothing will ask you to configure it. What is examinable is the reasoning around it: that logs are one of the three pillars, that container logs go to stdout and are collected by a node-level agent, that log storage is a cost decision with a retention policy attached, that structured logging is what makes logs useful, and that correlating logs to traces and metrics is the whole point of an observability strategy. If a scenario question describes aggregating container logs across a fleet, Loki-shaped thinking is the right answer even when the product is not named.

The documentation allowlist — read this twice

⚖ CNPA vs CNPE — That allowlist is a CNPE-only mechanic; CNPA is fully closed-book with zero external lookups of any kind, so there is no docs site to lean on there either. Even so, the log-aggregation reasoning above is worth knowing cold for CNPA's closed-book recall questions, not just for CNPE's hands-on tasks.

What to be able to do cold

Explain in one sentence why Loki indexes labels rather than log text, and what you trade away. Name the write path (distributor → ingester → object storage) and the read path (query-frontend → querier → chunks), and say what the compactor is for. Read a LogQL query aloud and describe it: which part uses the index, which part scans lines, and what |= versus |~ means. Say why pod or request_id must never be a stream label and where those values belong instead. Explain what a node-level DaemonSet agent does and why no application change is needed to get logs. And be ready to say how you would correlate a metric spike to a log line to a trace — that three-hop story is the answer the Observability domain is really fishing for. Cross-check the vocabulary against the glossary and rehearse the pillar boundaries in Observability.

Official resources for after the exam

Outside the exam, the canonical sources are grafana.com/docs/loki (the LogQL and Best practices pages repay slow reading), the deployment-mode guide at deployment modes, the agent documentation at grafana.com/docs/alloy, and the source at github.com/grafana/loki. Pair this page with Prometheus for the metrics half of the pair, Grafana for the query surface, Jaeger for the traces you will link to, and OpenTelemetry for the pipeline that increasingly carries all three.