Elastic Stack (ELK)

When an incident needs an answer that only exists as a sentence buried in some container's stdout three hosts ago, the Elastic Stack is still, by a wide margin, the tool most SRE teams reach for. Elasticsearch, Logstash, Beats, and Kibana — Elastic, Logstash, Kibana gave the stack its original "ELK" name, and Beats was folded in later as the lightweight shipper — together answer one question that metrics and traces structurally can't: what did the system actually say, in its own words, at that exact moment? This page covers what each of the four pieces actually does, how a raw log line becomes a searchable document, the index lifecycle management that keeps that searchability from bankrupting you, and the operational fact every team learns the hard way — that log volume, not query complexity, is what takes an ELK deployment down first.

What the Elastic Stack is and the problem it solves

☺ Like you're 10: It's four separate programs that each do one job — collect, tidy up, file away and index, and let you search — and together they turn a mountain of scattered text into something you can ask a question about.

Every service in a fleet writes logs, and by default those logs sit on whatever host or container produced them, in whatever format the developer who wrote that print statement happened to choose. That's fine until an incident spans a dozen services, at which point "grep every host by hand" stops being a strategy and becomes the incident itself. The Elastic Stack's job is to centralize that mess: ship every log line off its origin host, parse it into structured fields, index it for full-text search, and put a query interface in front of the result — so "show me every ERROR from the checkout service in the last twenty minutes that mentions timeout" is a five-second query instead of an SSH session into six containers that may already be gone.

Elasticsearch, Logstash, Beats, and Kibana — four names, one stack

Elasticsearch is the engine underneath everything: a distributed, JSON-document store built on top of Apache Lucene, which does the actual full-text indexing and search. Logstash is a general-purpose, plugin-based data pipeline — it reads from an input, transforms each event through a chain of filters, and writes it to an output, most commonly Elasticsearch. Beats are single-purpose, lightweight shippers written in Go — Filebeat tails log files, Metricbeat polls system and service metrics, Packetbeat inspects network traffic, Winlogbeat reads the Windows Event Log, and Auditbeat watches file integrity and the Linux audit framework — designed to sit on every host with a far smaller footprint than a JVM-based Logstash instance would allow. Kibana is the web UI: Discover for ad hoc searching, Dashboards for saved visualizations, Dev Tools for raw queries against Elasticsearch's API, and — via a bundled Alertmanager-adjacent engine — rule-based alerting of its own.

A brief history, and the license fork worth knowing about

Elasticsearch was first released in 2010 by Shay Banon, wrapping Lucene's mature but low-level search library in a distributed, JSON-native, horizontally scalable service. Logstash and Kibana were built separately and joined the same company shortly after; Beats arrived in 2015 as a lighter-weight answer to Logstash's JVM overhead, at which point the branding shifted from "ELK Stack" to "Elastic Stack" to reflect the fourth member. For most of that history the core software shipped under the permissive Apache 2.0 license. In January 2021, Elastic relicensed Elasticsearch and Kibana away from Apache 2.0 to a dual Elastic License / Server Side Public License (SSPL) model, citing cloud providers — chiefly AWS — offering the software as a managed service without licensing or contributing back. AWS responded within weeks by forking the last Apache-2.0 version into OpenSearch, which it has maintained since (governance later moved to the Linux Foundation). Elastic subsequently added AGPLv3 as a third licensing option for its own core products. Treat all of this as a moving target rather than settled fact — verify the current license terms for whatever version and distribution you're actually deploying on Elastic's own licensing page before you build a procurement or compliance decision on it.

Architecture: from a log line to a searchable document

☺ Like you're 10: A log line gets picked up at the source, walks through an optional tidying-up step, lands in the librarian's card catalog, and the reading room out front is where you ask about it.

Every deployment, however it's wired, is one variant of the same pipeline: collect at the source, optionally parse and enrich in flight, index into Elasticsearch, query and visualize through Kibana. The two decisions that actually distinguish real deployments from each other are where parsing happens and how long data stays searchable before it's aged out — the first is this section's subject, the second is Index Lifecycle Management, covered next.

Beats vs. Logstash vs. Elasticsearch ingest pipelines — where parsing happens

Raw text isn't queryable in any structured way; a log line has to be broken into fields — timestamp, service, level, message — before "show me only ERRORs" means anything to Elasticsearch. That parsing can happen in three different places, and the choice is a real architectural decision, not a style preference. Beats can do light, single-purpose parsing via built-in modules for known formats (nginx, Docker, syslog) but isn't a general transform engine. Logstash is the heaviest and most capable option — its grok filter matches log lines against named regex patterns (%{TIMESTAMP_ISO8601:timestamp} \[%{WORD:service}\] %{LOGLEVEL:level} %{JAVACLASS:class} - %{GREEDYDATA:message}), its dissect filter splits fixed-delimiter text without regex at all (faster, but only works when the format never varies), and it has plugins for geoip lookups, user-agent parsing, and arbitrary field mutation. Elasticsearch's own ingest pipelines run the same class of processors — grok, dissect, date, geoip, set, rename — but execute inside Elasticsearch itself, on the node handling the write, with no separate JVM process to run or scale. For a genuinely simple, well-structured log format, an ingest pipeline alone is often enough and removes a whole tier of infrastructure; Logstash earns its JVM overhead when you need multi-input fan-in, conditional routing to different outputs, or plugins ingest pipelines don't have.

Documents, indices, shards, and the inverted index underneath

Everything Elasticsearch stores is a JSON document, and every document lives in exactly one index — a logical namespace, roughly analogous to a database table. Physically, an index is split into one or more shards, and each shard is a completely self-contained Lucene index with its own inverted index: a map from every distinct term to the list of documents containing it, which is what makes "find every document containing the word timeout" a lookup instead of a scan. Replica shards are full copies of primary shards on different nodes, giving both fault tolerance and read throughput. Two field types matter more than any other mapping decision: text fields are analyzed — tokenized, lowercased, run through a language analyzer — and support full-text search but not exact-match filtering or sorting; keyword fields are stored verbatim and support exact-match, aggregation, and sorting but not full-text search. Mapping a log's service field as text instead of keyword is a common early mistake — it silently breaks the exact-match dashboards and aggregations built on top of it later.

Index Lifecycle Management: the mechanism that actually controls cost

☺ Like you're 10: A policy that automatically moves old logs from the librarian's fast front desk to a cheaper back room, then to a storage warehouse, then eventually shreds them — without anyone having to remember to do it by hand.

Full-text indexing every log line is expensive relative to just storing compressed raw text — the inverted index, the stored fields, and the doc-values structures Elasticsearch keeps for aggregations commonly add somewhere in the range of 1.1x to 3x overhead versus the raw log data, depending on settings. Keeping that overhead on your most expensive, fastest storage forever is how an ELK deployment's disk bill quietly becomes the biggest line item in the whole observability budget. Index Lifecycle Management (ILM) is Elastic's built-in answer: a declarative policy, attached to an index or a data stream, that automatically moves data through storage tiers as it ages, without anyone running a cron job or a manual migration.

The five phases: hot, warm, cold, frozen, delete

ILM defines up to five phases, each with its own actions and a min_age that gates when an index becomes eligible to enter it. Hot is where an index is actively written and queried, typically on the fastest local storage a cluster has; a rollover action here closes the current write index once it crosses a size, age, or document-count threshold and opens a new one, which is what keeps any single index from growing without bound. Warm holds indices that are read-mostly — typical actions here are a forcemerge (collapsing many small Lucene segments into fewer, larger ones for cheaper storage and faster search) and a shrink (reducing shard count, since a warm index no longer needs the write throughput its shard count was originally sized for). Cold holds rarely-queried data, often moved to cheaper storage tiers or nodes. Frozen is the cheapest tier that's still searchable at all — Elasticsearch's searchable snapshots keep the actual data in object storage like S3 and mount it on demand, trading query latency for a dramatically smaller local footprint. Delete removes the index outright once it's aged past whatever retention the organization actually needs.

Index templates, component templates, and data streams

ILM policies don't attach themselves — an index template matches a naming pattern (logs-checkout-*) and applies mappings, settings, and an ILM policy name to every new index that matches it automatically, so a rollover doesn't quietly leave the new index unmanaged. A data stream is the abstraction built specifically for this pattern: it presents as one logical, append-only name for writes and searches, while internally managing a sequence of auto-generated, hidden backing indices that rollover and progress through ILM behind the scenes — the modern replacement for the older, hand-rolled "index-per-day plus an alias" convention teams used to build themselves.

The config you actually write

☺ Like you're 10: A handful of files decide how a log line gets tidied up, what it's allowed to look like once it's filed, and how long it's allowed to stick around before it's thrown out.

A realistic pipeline for one service's logs touches four files: a Logstash filter (if Logstash is in the path at all), an Elasticsearch ingest pipeline, an ILM policy, and an index template that ties the ILM policy to a naming pattern.

# logstash.conf — parse checkout service application logs
input {
  beats { port => 5044 }
}
filter {
  # "2026-08-16T03:14:07.552Z [checkout] ERROR OrderService - Failed to charge card: timeout after 3000ms"
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} \[%{WORD:service}\] %{LOGLEVEL:level} %{JAVACLASS:class} - %{GREEDYDATA:log_message}" }
  }
  date {
    match => [ "timestamp", "ISO8601" ]
    target => "@timestamp"
  }
  mutate {
    remove_field => [ "timestamp", "message" ]
  }
}
output {
  elasticsearch {
    hosts => ["https://es-hot-01:9200"]
    data_stream => "true"
    data_stream_dataset => "checkout"
    data_stream_namespace => "prod"
  }
}

# PUT _ilm/policy/logs-checkout-policy
{
  "policy": {
    "phases": {
      "hot":    { "min_age": "0ms",
                  "actions": { "rollover": { "max_primary_shard_size": "50gb", "max_age": "1d" },
                               "set_priority": { "priority": 100 } } },
      "warm":   { "min_age": "3d",
                  "actions": { "forcemerge": { "max_num_segments": 1 },
                               "shrink": { "number_of_shards": 1 },
                               "set_priority": { "priority": 50 } } },
      "cold":   { "min_age": "14d",
                  "actions": { "set_priority": { "priority": 0 } } },
      "frozen": { "min_age": "30d",
                  "actions": { "searchable_snapshot": { "snapshot_repository": "logs-snapshots" } } },
      "delete": { "min_age": "90d",
                  "actions": { "delete": {} } }
    }
  }
}

# PUT _index_template/logs-checkout — ties the policy above to every matching index
{
  "index_patterns": ["logs-checkout-*"],
  "data_stream": {},
  "template": {
    "settings": { "number_of_shards": 1, "number_of_replicas": 1,
                  "index.lifecycle.name": "logs-checkout-policy" },
    "mappings": { "properties": {
      "@timestamp":  { "type": "date" },
      "service":     { "type": "keyword" },
      "level":       { "type": "keyword" },
      "log_message": { "type": "text" }
    } }
  }
}

Day-to-day commands and queries

☺ Like you're 10: A handful of questions cover almost every real workday: is the cluster healthy, how big are my indices, and can I ask it a specific question in its own language.

# cluster and index health — the first three commands during any "why is Kibana slow" report
$ curl -s localhost:9200/_cluster/health?pretty
$ curl -s "localhost:9200/_cat/indices?v&s=store.size:desc" | head -20
$ curl -s "localhost:9200/_cat/nodes?v&h=name,heap.percent,ram.percent,disk.used_percent"
$ curl -s "localhost:9200/_cat/shards/logs-checkout-*?v"

# see exactly what's stuck, and why, before you touch anything
$ curl -s localhost:9200/_cluster/allocation/explain?pretty

# clear a disk-watermark-triggered read-only block after freeing space — see Gotchas below
$ curl -X PUT "localhost:9200/logs-checkout-*/_settings" -H 'Content-Type: application/json' \
    -d '{ "index.blocks.read_only_allow_delete": null }'

# Query DSL — the checkout ERRORs from the last hour that mention "timeout"
POST logs-checkout-*/_search
{
  "query": {
    "bool": {
      "must":   [ { "match": { "log_message": "timeout" } } ],
      "filter": [
        { "term":  { "level": "ERROR" } },
        { "range": { "@timestamp": { "gte": "now-1h" } } }
      ]
    }
  }
}

# the same question, as KQL, typed straight into Kibana's Discover search bar
service:checkout and level:ERROR and log_message:"timeout"

Snapshots are how data leaves the cluster at all — for backup, and for the frozen tier's searchable snapshots above. Register a repository once, then snapshot on a schedule (Kibana's Snapshot Lifecycle Management, SLM, automates this the same way ILM automates tier movement):

# PUT _snapshot/logs-backups
{ "type": "s3", "settings": { "bucket": "acme-es-snapshots", "region": "us-east-1" } }

# PUT _snapshot/logs-backups/2026-08-16
# POST _snapshot/logs-backups/2026-08-16/_restore

Gotchas and failure modes — why log volume breaks it before query complexity does

☺ Like you're 10: A slow, complicated question rarely knocks the librarian over — but ten times the usual number of trucks showing up overnight absolutely will, and it happens quietly, not with a crash.

Nearly every serious ELK outage traces back to the same root cause: something upstream — a bug left in DEBUG logging, a traffic spike, a retry loop gone noisy — multiplies log volume far faster than anyone budgeted disk or heap for, and the stack degrades or stops accepting writes long before anyone has written a query complicated enough to be the actual problem.

Disk-based shard allocation watermarks: the silent ingestion stop

Elasticsearch protects itself from filling a node's disk with default watermark thresholds: at 85% used, the node stops receiving new shards; at 90%, shards actively relocate away from it; at 95% — the flood-stage watermark — every index with a shard on that node is forced read-only, and every write to it is rejected. This is, by a wide margin, the single most common "why did our logs just stop appearing" incident: nothing crashes, no alert necessarily fires on the application side, ingestion simply stops the instant a hot-tier node crosses 95% disk, exactly when a volume spike is already underway and logs matter most. Freeing disk space lets the watermark clear on its own in current versions, but under incident pressure most operators still explicitly unset index.blocks.read_only_allow_delete themselves — shown in the commands above — rather than wait and hope it clears in time.

Mapping explosion

Dynamic mapping — letting Elasticsearch infer a field's type from the first document it sees — is convenient until logs contain arbitrary, highly variable JSON (a raw error object, a user-supplied payload logged verbatim). Every previously-unseen field name creates a new mapping entry, and Elasticsearch enforces a default cap of 1000 fields per index for exactly this reason; past it, indexing starts failing outright. Cluster state, which includes every index's full mapping, is held in memory and broadcast to every node on every change — so a mapping that grows unbounded doesn't just risk hitting the field limit, it slows the whole cluster down before it gets there. The fix is a strict mapping wherever the field set is knowable in advance, and "dynamic": "strict" (rejecting unmapped fields outright) or "dynamic": false (accepting but not indexing them) wherever it isn't.

Shard sizing: oversharding and undersharding

Every shard is a separate Lucene index with its own file handles, memory overhead, and per-shard cost during a cluster state update or a rebalance — a widely cited rule of thumb keeps shard sizes in roughly the 10–50GB range for logs and caps shard count well below 20 per GB of heap on a node. Oversharding (too many small shards) wastes overhead on bookkeeping instead of data. Undersharding (too few, oversized shards) makes recovery and rebalancing slow, and concentrates query and indexing load onto fewer nodes than the cluster could otherwise spread it across. ILM's rollover action, sized correctly against real ingest volume, is the practical mechanism that keeps shard size in the healthy range automatically instead of as a one-time guess made on day one and never revisited.

JVM heap, GC pauses, and the 32GB ceiling

Elasticsearch runs on the JVM, and its guidance is specific: allocate roughly 50% of a node's available RAM to heap, and never set it above approximately 30–32GB regardless of how much RAM the node has. Past that boundary, the JVM loses the ability to use compressed ordinary object pointers, and the larger heap ends up storing fewer usable objects than a smaller one just under the line — a genuinely counterintuitive result that catches people sizing nodes for the first time. Under a sustained volume spike, heap pressure and garbage-collection pauses are the mechanism that actually takes a node down: a long enough GC pause can make a node miss cluster-state heartbeats and get ejected from the cluster, turning a log-volume problem into a cluster-stability incident — again, well before anyone's run a query complex enough to be blamed.

Logstash grok performance and backpressure

A grok filter compiles down to a regular expression, and a poorly written pattern — an unanchored .*, deeply nested optional groups — can hit catastrophic backtracking on a single unusual log line, stalling that pipeline worker thread. Because Logstash's internal queue is bounded, a stalled filter stage backs up into the input stage, which backs up into Beats' own outbound queue, which can eventually pause Filebeat's own file-tailing — turning one slow regex into a growing, unshipped backlog sitting on disk across the fleet. dissect avoids this entirely for genuinely fixed-delimiter formats, since it splits on literal positions rather than evaluating a regex engine at all; the standard practice is dissect first wherever the format is truly fixed, grok only for the genuinely variable remainder.

Alternatives and when to choose it

☺ Like you're 10: Other tools track logs too — some index everything the way Elasticsearch does and cost more for it, some only index a little and search the rest on the fly, which is cheaper but slower per query.

The real fork in the road is whether you index the full text of every log line at all. Elasticsearch does, which is what makes ad hoc full-text search fast — and what makes storage the thing you have to actively manage with ILM. Several real alternatives take a deliberately different trade.

Most mature SRE organizations don't pick exactly one forever — it's common to see Loki carrying the high-volume, low-cardinality bulk of routine service logs at low cost, with the Elastic Stack (or a smaller, deliberately scoped slice of it) reserved for the logs that genuinely need full-text search: security and audit trails, or a small number of services where "search everything" really is the requirement. Deciding which category a given service's logs fall into, before the ingest pipeline is built rather than after the storage bill arrives, is most of the architecture decision.

Where the Elastic Stack fits in the SREF blueprint

☺ Like you're 10: The exam wants you to recognize "this is the centralized-logging category" from a description, not memorize a specific ILM policy's JSON.

The DevOps Institute SRE Foundation (SREF) exam is closed-book and tests tool categories over vendor trivia, per SRE Tools & Automation — the Elastic Stack is the representative example for centralized log aggregation and search, alongside Loki as the lower-cost alternative model. Monitoring & Service Level Indicators is where the underlying practice actually gets tested: know that logs are one of the three observability pillars alongside metrics and traces, covered together in monitoring & observability, and that "index lifecycle management" as a concept — automatically aging data through cheaper tiers to control cost — recurs across nearly every stateful monitoring tool this course covers, not just this one. The specific grok syntax and ILM JSON on this page are for the job, not the exam.