ELK Stack (Elasticsearch, Logstash, Kibana)

"ELK" names three separate open-source projects that, put together, answer one question every team eventually asks the hard way: a service is misbehaving across forty hosts and a hundred containers, and nobody can find the one log line that explains why. Beats (and optionally Logstash) collect and ship log data off every machine, Elasticsearch indexes it so it's searchable by any word it contains within seconds of being written, and Kibana turns that index into dashboards, alerts, and an ad hoc search box. This page covers the architecture of that pipeline, the actual config files you write for each piece, the day-to-day commands and queries, the failure modes — disk watermarks, mapping explosions, oversized shards — that show up once real production volume hits it, and how running this stack yourself compares to paying a vendor to run an equivalent one for you.

What the Elastic Stack is and the problem it solves

☺ Like you're 10: Before this stack existed, finding one bad log line meant SSH-ing into servers one at a time and grepping by hand — this stack collects every log in one place first, so you search once instead of forty times.

Elasticsearch was created by Shay Banon in 2010, built on top of Apache Lucene (the same Java full-text search library that has powered search engines since the early 2000s), and released as open source under the Apache 2.0 license. Logstash, a data-pipeline tool for parsing and shipping logs, was created separately by Jordan Sissel in 2009. Kibana, a browser-based visualization layer for Elasticsearch, followed in 2013 from Rashid Khan. All three projects — plus the company behind Elasticsearch, which took the name Elastic — consolidated under one roof, and "ELK" became shorthand for running all three together: Logstash (or something like it) ships and parses logs, Elasticsearch indexes and stores them, Kibana visualizes and searches them. In 2015, Elastic added Beats — a family of small, single-purpose, Go-based shippers — to replace the older, heavier "Logstash Forwarder," and the product family was renamed the Elastic Stack to reflect a fourth component. "ELK" stuck as the name people actually say out loud, so this page uses both interchangeably, the same way the rest of this course does.

The problem this stack solves is centralized log aggregation, and it matters for a reason that's easy to underrate until you've hit it: logs that live only on the box that wrote them are logs you lose the moment that box is gone. A container's filesystem disappears when the container is rescheduled; an autoscaled instance is terminated and its disk goes with it; a fleet of forty hosts means forty separate places to search by hand, sequentially, while an incident is still active. Centralizing logs — shipping every line off the source host the moment it's written, into one indexed, queryable store — turns "SSH into each box and grep" into "type one query and get every match across the whole fleet in under a second," which is the entire reason this stack (or something functionally identical to it) sits underneath nearly every production system past a certain size.

Where it fits in an observability pipeline

☺ Like you're 10: Logs are one of the three things a running system tells you about itself — this stack's whole job is being the "logs" leg of that three-legged stool.

Monitoring & observability names the three telemetry pillars — metrics, logs, and traces — and ELK is the classic, still-dominant open-source answer to the logs pillar specifically. It doesn't replace Prometheus for metrics or an OpenTelemetry backend for distributed traces; most real stacks run all three side by side, and a well-instrumented incident response typically starts with a metric-based alert, narrows with a trace, and confirms the root cause with a log line pulled from exactly this kind of index. Where ELK earns a page of its own is depth: unlike a metric (a number with labels) or a trace (a timed span), a log line is unstructured or semi-structured free text, and making millions of those per day searchable by arbitrary keyword, in under a second, at scale, is a genuinely hard indexing problem — which is precisely what Elasticsearch, under the hood, is built to be good at.

In a containerized pipeline, the usual pattern is a Filebeat DaemonSet running one pod per Kubernetes node, reading every container's stdout/stderr straight off the node's /var/log/containers directory and auto-discovering which index to ship each line to based on pod annotations — so a team never edits application code to get logging working, and a log line from a crashed pod is still searchable after the pod itself is gone. Once logs land in Elasticsearch, Kibana's alerting rules can page a human through the same channels covered in incident management — Slack, email, or a direct PagerDuty/Opsgenie webhook — closing the loop from "a log line matched a bad pattern" to "someone got paged" without a human watching a terminal.

Architecture: Elasticsearch, Logstash, Beats, and Kibana

☺ Like you're 10: Each of the four pieces has exactly one job — carry, tidy, file, and display — and none of them can do the other three's job for it.

Elasticsearch: a distributed document store built on an inverted index

Elasticsearch stores JSON documents — one document per log line, in the logging use case — grouped into indices, the rough equivalent of a table. Each index is split into one or more shards, and each shard is itself a self-contained Lucene index on disk. Shards are what make Elasticsearch horizontally scalable: a cluster of many nodes distributes an index's shards across machines, so a query fans out in parallel instead of scanning one giant file sequentially. Every primary shard can have one or more replica shards — full copies living on different nodes, both for redundancy and to serve read traffic in parallel. Nodes take on roles: master-eligible nodes manage cluster state and shard allocation, data nodes hold and search shards, ingest nodes run lightweight pre-index transforms, and coordinating-only nodes just route requests — a small cluster often collapses all of these onto the same nodes, while a large one splits them out.

What makes full-text search fast is the inverted index: instead of storing "line 4,281 contains these words" the way a book stores pages, Lucene stores "the word 'timeout' appears in these documents," for every distinct word, so a search for a term is a direct lookup rather than a scan. New documents don't appear in search results the instant they're written — they land in an in-memory buffer first, and a background refresh (every 1 second by default) flushes that buffer into a new, immutable Lucene segment that becomes searchable. That's why Elasticsearch is described as near-real-time, not real-time: a one-second lag is normal and, for a logging use case, irrelevant.

Logstash: an input/filter/output pipeline

Logstash is a JVM-based data pipeline defined in three stages: input (where events come from — a Beats listener, a file, Kafka, HTTP), filter (transform each event), and output (where it goes — almost always Elasticsearch here, but it can just as easily be S3 or another Kafka topic). Its signature filter is grok: a library of named regular-expression patterns (%{IP:client}, %{TIMESTAMP_ISO8601:timestamp}) that turns an unstructured log line into named, typed fields, which is the actual work of making free text queryable rather than just stored. Other common filters include date (parse a field into the document's real @timestamp, rather than the time Logstash happened to receive it), mutate (rename, remove, or convert fields), and geoip (turn an IP address into a country and coordinates). A persistent queue — an on-disk buffer between input and filter — is what lets Logstash survive a restart or an Elasticsearch outage without silently dropping events already accepted; events that still fail to index after that (typically a mapping conflict) land in a dead letter queue instead of vanishing.

Beats: single-purpose, low-footprint shippers

Beats are small, statically compiled Go binaries, one per data source, designed to sit on every host or in every pod at near-zero resource cost — a deliberate contrast with Logstash's JVM footprint. Filebeat tails log files (or, in Kubernetes, container stdout); Metricbeat polls host and service metrics; Packetbeat captures network traffic; Winlogbeat reads the Windows Event Log; Auditbeat watches file integrity and audit subsystem events; Heartbeat does uptime/synthetic checks. Filebeat tracks its read position for every file it tails in a local registry, so a restart resumes exactly where it left off — deleting that registry file, deliberately or by accident, is how a host ends up re-shipping its entire log history as duplicate events. Beats can ship to Logstash for heavy parsing, or straight to Elasticsearch's own ingest pipelines (a lighter, index-time transform mechanism built into Elasticsearch itself) when the parsing need is simple enough not to justify a separate Logstash tier at all — the dashed bypass in the diagram above.

Kibana: search, dashboards, and alerting

Kibana is the browser UI on top of an Elasticsearch cluster, built around a handful of named workspaces worth knowing: Discover is ad hoc search over a Data View (a saved wildcard pattern matching one or more indices, formerly called an "index pattern"); Dashboards compose saved visualizations into one page; Lens is the drag-and-drop visualization builder most people actually use day to day; the Dev Tools console sends raw Elasticsearch REST API calls from the browser with autocomplete, which is how most engineers actually query and administer a cluster instead of hand-writing curl; and Alerting rules evaluate a query on a schedule and fire a connector — email, Slack, a generic webhook, or a PagerDuty integration — when a threshold is crossed. Spaces partition saved objects (dashboards, data views, alerts) into isolated tenants inside one Kibana instance, the usual way one cluster serves multiple teams without their dashboards colliding.

The config you actually write

☺ Like you're 10: Three files do almost all the work: tell Beats what to watch, tell Logstash how to tidy it up, and tell Elasticsearch how long to keep it before throwing it away.

A minimal but real setup: Filebeat tails an application's JSON log file and ships to Logstash; Logstash parses and enriches; Elasticsearch stores it behind an index template with a lifecycle policy that ages data out automatically.

# filebeat.yml
filebeat.inputs:
  - type: filestream
    id: checkout-app-logs
    paths:
      - /var/log/checkout/*.log
    parsers:
      - ndjson:
          target: ""            # app already logs structured JSON — parse it, don't re-grok it

processors:
  - add_host_metadata: {}       # hostname, OS, IP — free enrichment, no config needed
  - add_fields:
      target: ""
      fields:
        service: checkout
        env: production

output.logstash:
  hosts: ["logstash.internal:5044"]
  ssl.certificate_authorities: ["/etc/filebeat/ca.crt"]   # never ship logs over plaintext off-host

# /etc/logstash/conf.d/checkout.conf
input {
  beats { port => 5044 }
}

filter {
  # only needed for logs that AREN'T already structured JSON — this one is unstructured nginx access log
  grok {
    match => { "message" => "%{IPORHOST:client_ip} - - \[%{HTTPDATE:ts}\] \"%{WORD:method} %{URIPATHPARAM:path} HTTP/%{NUMBER:http_version}\" %{NUMBER:status} %{NUMBER:bytes}" }
  }
  date {
    match => ["ts", "dd/MMM/yyyy:HH:mm:ss Z"]
    target => "@timestamp"     # use the log's own time, not Logstash's receipt time
  }
  geoip {
    source => "client_ip"
    target => "geo"
  }
  mutate {
    remove_field => ["ts", "message"]   # drop the raw fields once they're parsed into real ones
  }
}

output {
  elasticsearch {
    hosts => ["https://es-01.internal:9200", "https://es-02.internal:9200"]
    index => "logs-checkout-%{+YYYY.MM.dd}"
    user => "${LOGSTASH_ES_USER}"
    password => "${LOGSTASH_ES_PASSWORD}"   # from environment, never hardcoded — see Secrets & Credential Management
  }
}

Notice the credentials in that output block come from an environment variable, never a literal string in the pipeline file — see Secrets & Credential Management for why that discipline matters even for a "just logging" service account. The index name above bakes in a date, the classic ELK pattern for time-series log data — but the more current, idiomatic mechanism is a data stream (introduced in Elasticsearch 7.9), which manages the underlying daily/rolled indices for you and pairs naturally with Index Lifecycle Management (ILM): a policy that moves an index through hot (actively written, fast storage), warm (read-only, still fast), cold, and eventually delete phases, rolling over to a new backing index automatically on a size, age, or document-count trigger.

PUT _ilm/policy/logs-checkout-policy
{
  "policy": {
    "phases": {
      "hot":   { "actions": { "rollover": { "max_primary_shard_size": "30gb", "max_age": "1d" } } },
      "warm":  { "min_age": "3d",  "actions": { "shrink": { "number_of_shards": 1 }, "forcemerge": { "max_num_segments": 1 } } },
      "cold":  { "min_age": "14d", "actions": { "set_priority": { "priority": 0 } } },
      "delete":{ "min_age": "90d", "actions": { "delete": {} } }
    }
  }
}

Day-to-day commands and queries

☺ Like you're 10: A handful of REST calls cover almost everything: is the cluster healthy, what's in it, and can I search it.

# cluster and index health — the first three commands for any "is ELK okay" question
$ curl -s "https://es-01.internal:9200/_cluster/health?pretty"
$ curl -s "https://es-01.internal:9200/_cat/indices?v&s=store.size:desc"   # every index, sorted by size on disk
$ curl -s "https://es-01.internal:9200/_cat/shards?v" | grep UNASSIGNED    # find unassigned shards fast
$ curl -s "https://es-01.internal:9200/_cat/nodes?v&h=name,disk.used_percent,heap.percent"

# search — the Query DSL, sent as a JSON POST body
$ curl -s -X GET "https://es-01.internal:9200/logs-checkout-*/_search?pretty" -H 'Content-Type: application/json' -d '{
    "query": { "bool": { "must": [
      { "match": { "status": 500 } },
      { "range": { "@timestamp": { "gte": "now-1h" } } }
    ]}},
    "size": 20,
    "sort": [{ "@timestamp": "desc" }]
  }'

# ILM and mapping introspection
$ curl -s "https://es-01.internal:9200/logs-checkout-*/_ilm/explain?pretty"
$ curl -s "https://es-01.internal:9200/logs-checkout-2026.08.16/_mapping?pretty"
$ curl -s "https://es-01.internal:9200/logs-checkout-*/_field_caps?fields=*&pretty" | head -50   # spot a field-count problem early

# operational commands
$ logstash --config.test_and_exit -f /etc/logstash/conf.d/checkout.conf   # validate config before restarting the service
$ filebeat test config
$ filebeat test output

In practice, most engineers run the same queries from Kibana's Dev Tools console instead of raw curl — same API, autocomplete included — and reach for curl mainly from scripts and CI health checks.

Gotchas and failure modes

☺ Like you're 10: A cluster that looks fine on day one can quietly set itself up to refuse writes on day ninety, purely from filling a disk or letting a schema grow without limits.

Disk watermarks turn a full disk into a read-only cluster, silently. Elasticsearch monitors disk usage per node against three thresholds: at 85% (the low watermark) it stops allocating new shards to that node; at 90% (high watermark) it tries to relocate shards away; at 95% (flood-stage) it forces every index with a shard on that node into read-only — index.blocks.read_only_allow_delete gets set automatically, and every subsequent write fails until an operator frees space and manually clears the block. This is one of the single most common ELK production incidents, and it looks nothing like a disk-full error — it looks like "logs stopped being indexed" hours after the disk actually filled.

Mapping explosion turns free-form data into a cluster-wide problem. Dynamic mapping is convenient until an application logs something like a customer ID or a session token as a JSON key rather than a value — each distinct key becomes a new field, forever, since fields are never removed from a mapping. Elasticsearch defaults to a hard limit of 1,000 fields per index (index.mapping.total_fields.limit); cross it and every new document with a new field is rejected, and even well under the limit, a mapping with thousands of rarely-used fields bloats cluster state and slows every node. The fix is either normalizing that data into a real field-value pair before it's indexed, or setting "dynamic": "strict" on the index template so an unrecognized field fails loudly at index time instead of silently growing the mapping forever.

JVM heap sizing has a hard ceiling most people don't expect. The rule of thumb is to set the Elasticsearch heap (Xms/Xmx in jvm.options) to 50% of the node's available RAM, leaving the rest for the OS page cache that Lucene relies on heavily — but never push the heap above roughly 30–32 GB regardless of how much RAM the box has. Past that boundary, the JVM loses compressed ordinary object pointers (compressed oops), and pointers silently double in size, so a 34 GB heap can hold less usable data than a 30 GB one. A node with 128 GB of RAM should still run something close to a 30 GB heap, not a 64 GB one.

Oversharding wastes resources you can't easily get back. Every shard is a Lucene index with its own file handles, memory overhead, and per-query cost — teams that create a fresh daily index per low-volume service accumulate thousands of tiny shards, and cluster overhead from managing them can exceed the cost of the data itself. The rule of thumb is to keep individual shards in roughly the 10–50 GB range and use ILM rollover (shown above) to hit that target automatically rather than a fixed daily index regardless of actual volume.

Self-hosted ELK vs. managed alternatives

☺ Like you're 10: You can run this whole workshop yourself, or pay someone else to run an equivalent one and just drop off your notebooks — the trade is control and cost versus time spent keeping the workshop running.

Running Elasticsearch well is a genuine operational discipline in its own right — cluster sizing, JVM tuning, shard management, and version upgrades are all real, ongoing work, not a one-time setup cost — which is exactly why a market of managed alternatives exists.

The practical rule most teams land on: reach for self-hosted ELK (or OpenSearch) when log volume is high enough that infrastructure cost clearly beats SaaS per-GB pricing and the team has the operational appetite for it; reach for a managed product — CloudWatch Logs on AWS, Datadog everywhere else — when the priority is standing up logging fast with nobody dedicated to running a search cluster; and reach for Loki specifically when a Prometheus/Grafana stack already exists and full-text search depth matters less than sharing one operational model across metrics and logs. None of these choices change the underlying concepts on this page — shards, indices, ingestion pipelines, retention — only who operates them. Practice the pipeline end to end in Capstone Part 4 — Observability, then drill turning a raw log stream into a real page in Drill — Set Up Meaningful Alerts.