HashiCorp Consul

Consul is the piece of the HashiCorp toolchain that answers a question the other three don't: once something is running, how does anything else find it, and how do the two of them talk without either one being handed a static address that goes stale the moment a container reschedules or an autoscaling group cycles? Terraform provisions the infrastructure, Vault secures the secrets living on it, Packer bakes the images it boots from — and Consul is the directory that stays correct in real time, the network layer that proves two services really are who they claim to be before letting them exchange a byte, and a small key/value store for the configuration that needs to change without a redeploy. Three jobs, one binary, one Raft-backed source of truth underneath all of them.

What Consul is, and where it sits in the HashiCorp stack

☺ Like you're 10: Consul keeps a constantly-updated list of what's running and where, checks that each thing is actually healthy, and lets services prove who they are to each other before they talk.

HashiCorp shipped Consul in 2014, and it's usually described as three products bundled into one agent: service discovery (a live, health-checked catalog of every registered service and where it's running), service mesh (Consul Connect — automatic mutual TLS and identity-based authorization between services via sidecar proxies), and dynamic configuration (a hierarchical key/value store). All three are backed by the same underlying data: a strongly-consistent catalog replicated across a small server cluster via the Raft consensus algorithm, exposed through a DNS interface, an HTTP API, a web UI, and — for the mesh — the sidecar proxies themselves.

The problem this replaces is one every team eventually hits at scale: a config file with DB_HOST=10.4.2.19 works fine until that instance is replaced by an autoscaling event, a failed health check triggers a reschedule, or a deploy moves the workload to a different node — at which point the hardcoded address is simply wrong, and nobody finds out until something times out in production. The related problem is trust: once you have dozens or hundreds of services on a shared network, "any pod can reach any other pod on any port" is the default in most environments, which means a single compromised service can probe or call anything else with no further authentication at all.

This page covers Consul alone. Where its territory overlaps a neighbor's — Vault can act as Connect's certificate authority instead of Consul's built-in one, and Consul itself can be Vault's storage backend, as covered in HashiCorp Vault — that boundary gets called out rather than re-explained.

Architecture: agents, gossip, and the Raft-backed catalog

☺ Like you're 10: A small number of "server" agents keep the official, agreed-upon record; every other machine runs a lightweight "client" agent that registers what's local and gossips with its neighbors to spot failures fast.

Every machine in a Consul deployment runs an agent, in one of two modes. Server agents — an odd number, typically 3 or 5 per datacenter, the same quorum math as Vault's Raft storage — hold the actual state: the catalog, the KV store, ACL tokens and policies, and Connect's certificate authority. They elect a leader via Raft, replicate every write to a majority before acknowledging it, and are the only agents anyone should trust for a strongly-consistent answer. Client agents run on every other node — one per host, not one per container — and do the workaday jobs: register local services, run health checks against them, and forward RPCs to the server cluster. A client agent holds no authoritative state of its own; killing one loses nothing except that node's local caching.

Underneath both agent types is a separate, lighter-weight system: gossip, built on HashiCorp's Serf library implementing the SWIM protocol (Scalable Weakly-consistent Infection-style Process group Membership). Gossip isn't how the catalog gets replicated — Raft does that, and only among servers — it's how agents discover each other and detect failure fast, by periodically pinging random peers and having peers relay suspicions about unresponsive nodes rather than everyone polling everyone. Every agent in one datacenter joins the same LAN gossip pool; server agents additionally join a WAN gossip pool with server agents in other datacenters, which is how traditional multi-datacenter federation works without every client agent needing WAN connectivity.

Autopilot, and the server-count discipline it still can't replace

Autopilot automates the housekeeping around that server cluster: it removes dead servers after a stable, configurable delay instead of leaving them cluttering Raft's peer set, waits for a newly joined server to catch up on replication before counting it toward quorum, and can be configured to keep servers spread across redundancy zones. It is genuinely useful, and it does not change the underlying math — quorum is still (N/2)+1 of whatever server count you actually run, and going from 3 servers to 2 live ones is already a failure state, not a warning.

Service discovery: registration, health checks, and the query interfaces

☺ Like you're 10: A service describes itself once in a small file, Consul keeps checking that it's actually alive, and anything else on the network can ask "where's checkout right now" over plain DNS or a simple HTTP call.

A service is registered either declaratively, by dropping a definition into an agent's config directory, or imperatively via the HTTP API — the same install-time-vs-runtime choice as most of this course's other tools. Registration alone only tells Consul a service exists; a check is what tells it whether to keep recommending that instance. Consul supports several check types: HTTP (poll a URL, 2xx/3xx is passing), TCP (can a connection open), gRPC, Docker (exec inside a container), and TTL (the inverse of the others — the application itself must actively check in before the TTL expires, or Consul marks it critical; useful for jobs that don't listen on a port at all). Script/exec checks still exist but are disabled by default on client agents (enable_local_script_checks must be explicitly set) precisely because "let anything registered on this node also run arbitrary shell commands on it" is a bigger attack surface than most teams intend to sign up for.

// services/checkout.json — dropped in the agent's -config-dir, or PUT to /v1/agent/service/register
{
  "service": {
    "name": "checkout",
    "id": "checkout-7f4c",
    "port": 8080,
    "tags": ["v2", "prod"],
    "meta": { "version": "2.4.0" },
    "checks": [
      { "http": "http://localhost:8080/healthz", "interval": "10s", "timeout": "2s" },
      { "ttl": "30s", "notes": "app heartbeats via PUT /v1/agent/check/pass" }
    ]
  }
}

Once registered, other services never need to know an IP address at all — they ask Consul. The DNS interface (default port 8600) is the lowest-friction path because almost every runtime already knows how to resolve a hostname: <service>.service.consul returns every currently-passing instance, tag-filtered lookups (<tag>.<service>.service.consul) narrow it further, and an explicit datacenter suffix reaches across a federated deployment. The HTTP API (default port 8500) gives you the full structured record — address, port, tags, per-check status — for anything that needs more than "just an IP."

$ dig @127.0.0.1 -p 8600 checkout.service.consul SRV        # SRV record includes the port
$ dig @127.0.0.1 -p 8600 v2.checkout.service.consul A        # only instances tagged "v2"
$ dig @127.0.0.1 -p 8600 checkout.service.dc2.consul A       # explicit cross-datacenter lookup

$ curl -s http://localhost:8500/v1/health/service/checkout?passing=true | jq '.[].Service.Address'
$ curl -s http://localhost:8500/v1/catalog/service/checkout | jq '.[] | {Node, Address, ServicePort}'

In practice, most production DNS setups don't point applications at Consul directly — they configure the resolver (e.g. a CoreDNS stub domain forwarding consul. queries to Consul's DNS, or dnsmasq on VM-based fleets) so ordinary hostname resolution just works, and the application code never has to know Consul exists.

Service mesh: Connect, sidecar proxies, and intentions

☺ Like you're 10: Every service gets a little bodyguard proxy next to it that checks a permission list and speaks encrypted to the other service's bodyguard — the application itself never has to touch TLS or an allow-list.

Connect is Consul's service mesh: every mesh-enabled service gets a sidecar proxy (Envoy in production; Consul ships a minimal built-in proxy too, meant for testing rather than real traffic) that terminates and originates all of that service's mesh traffic. Each sidecar holds a short-lived TLS certificate, issued and automatically rotated by Consul's built-in certificate authority (or by Vault, configurable as an alternate CA provider) — the identity in that certificate is the service's identity, not a host or an IP, which is what makes the model work even as instances get rescheduled onto different nodes constantly.

Authorization is a separate concept from encryption, and it's called an intention: an explicit allow or deny rule between a source service name and a destination service name. With ACLs enabled and a default-deny policy — the recommended production posture, covered below — two mesh-enabled services cannot exchange a single byte until an intention explicitly allows it, regardless of network reachability. That's the "zero trust" pitch made concrete: the network itself stops being the security boundary, service identity is.

$ consul intention create web checkout        # allow web -> checkout
$ consul intention check web checkout          # would this call be allowed right now?
$ consul intention delete web checkout         # remove it — falls back to the default policy

Above plain allow/deny, config entries give Connect real Layer 7 traffic management without touching application code — the same territory deployment strategies covers conceptually, expressed here as Consul objects instead of a load balancer's own configuration.

# service-defaults.hcl — declare the protocol so L7 features (routing, splitting) are even possible
Kind     = "service-defaults"
Name     = "checkout"
Protocol = "http"
---
# checkout-splitter.hcl — canary release: 90% to the stable subset, 10% to the new one
Kind = "service-splitter"
Name = "checkout"
Splits = [
  { Weight = 90, ServiceSubset = "v1" },
  { Weight = 10, ServiceSubset = "v2" }
]

$ consul config write service-defaults.hcl
$ consul config write checkout-splitter.hcl
$ consul config list -kind service-splitter

On Kubernetes, the same objects arrive as CRDs (ServiceDefaults, ServiceSplitter, ServiceIntentions) applied with plain kubectl, and sidecar injection happens automatically via a mutating admission webhook the consul-k8s Helm chart installs — a pod just needs one annotation, not a hand-written proxy config. This is also where Consul's cross-platform pitch is most concrete: the same mesh, the same intentions, and the same catalog can span Kubernetes pods, plain VMs, and workloads on Nomad at once, which is a genuinely different shape than a mesh that only understands one orchestrator's own service objects.

Dynamic configuration: the KV store and consul-template

☺ Like you're 10: Consul also keeps a small filing cabinet of settings that can change without redeploying anything, and a companion tool watches it and rewrites config files the instant something in it changes.

The KV store is a hierarchical key/value namespace living in the same Raft log as the catalog — strongly consistent, replicated, and queryable with the same consul kv CLI or the HTTP API. It's the right place for values an operator wants to flip without a deploy — a feature flag, a rate limit, which upstream a canary should point at — and explicitly the wrong place for two other things: it has a hard 512KB per-value size limit (the same class of Raft-log ceiling that trips up Helm's release Secrets), and it is not encrypted at rest by anything beyond the transport layer, so it is not a secrets store. That job belongs to Vault, and the two tools pair deliberately rather than compete: Consul KV for config that can be read in plaintext by anyone with catalog access, Vault for anything that genuinely needs a policy check and an audit trail before it's revealed.

$ consul kv put config/checkout/log_level debug
$ consul kv get config/checkout/log_level
$ consul kv get -recurse config/checkout/
$ consul kv delete config/checkout/log_level

Reading the KV store once is easy; the more common need is "keep a config file in sync with it forever." consul-template, a separate but tightly paired HashiCorp tool, watches KV keys and catalog entries via blocking queries — a long-polling HTTP call that holds open until the underlying index actually changes, instead of a client hammering the API on a fixed interval — re-renders a template the moment anything it references changes, and runs a reload command against the consumer, turning a load balancer's upstream list into a live view of the catalog instead of a file someone edits by hand.

# nginx.conf.tpl — consul-template syntax: {{ }} blocks against the live catalog
upstream checkout {
{{ range service "checkout" }}
  server {{ .Address }}:{{ .Port }};
{{ end }}
}

$ consul-template \
    -template "nginx.conf.tpl:/etc/nginx/nginx.conf:nginx -s reload"
# whenever `checkout` instances change, nginx.conf is rewritten and nginx is told to reload

The KV store also underpins Consul's session mechanism, which combines a KV check-and-set with a TTL to give you a distributed lock without standing up a separate coordination service — consul lock wraps this into a one-line "run this command only while holding the lock," the same leader-election primitive etcd and ZooKeeper are best known for, reachable here without adding a fourth system to the stack.

Day-to-day commands

☺ Like you're 10: A handful of commands cover most of it: start an agent, see who's in the club, ask what's registered, check on the server cluster, and manage keys.

# lifecycle
$ consul agent -dev                                     # single node, in-memory — local testing ONLY
$ consul agent -config-dir=/etc/consul.d                 # real server or client, from config files
$ consul reload                                           # re-read config/service files, no restart
$ consul leave                                             # graceful departure from the gossip pool

# membership and catalog
$ consul members                                          # LAN gossip pool, this datacenter
$ consul members -wan                                      # WAN pool, across federated datacenters
$ consul catalog services                                   # every registered service name
$ consul catalog nodes -service=checkout                     # which nodes are running it

# the server cluster itself
$ consul operator raft list-peers                           # leader, followers, voter status
$ consul operator raft remove-peer -address=10.0.1.11:8300   # after a server is permanently gone, not before

# gossip encryption
$ consul keygen                                              # generate a new 32-byte encryption key
$ consul keyring -install=NEW_KEY                             # roll it onto a LIVE cluster, in stages

# ACLs, KV, mesh
$ consul acl bootstrap                                       # ONCE — prints the initial management token
$ consul kv put config/checkout/log_level debug
$ consul intention create web checkout
$ consul connect proxy -sidecar-for checkout                  # run a sidecar manually, for testing

Gotchas and failure modes

☺ Like you're 10: Most Consul incidents are a cluster left open by default, an encryption key nobody planned to change, or someone assuming three servers means the same thing as one server running three times.

ACLs default to allow unless you explicitly say otherwise

A Consul cluster with ACLs enabled but no explicit default_policy — or with the legacy allow default some older configs still carry forward — leaves every unauthenticated request permitted by default. That means anyone who can reach the HTTP API can read the entire catalog, the entire KV store, and write intentions, with no token at all. Production guidance is default_policy = "deny" from the start, bootstrapped before any real workload is registered — the same lesson Vault's dev-mode gotcha teaches from the opposite direction: a security-relevant default that's convenient for a five-minute demo is exactly the wrong default to carry into production unexamined.

Gossip encryption is a bootstrap-time decision, not a quick retrofit

The symmetric key set via encrypt in agent config protects gossip traffic between agents, and every agent needs it before it can meaningfully join the pool — a mismatched or missing key doesn't produce a helpful error, it just produces an agent that silently can't see its peers. Turning encryption on for a cluster that's already running unencrypted isn't a one-line config change either; it's a staged rollout via consul keyring -install, -use, and eventually -remove of the old key, done gradually across the fleet so agents mid-rollout can still talk to both old and new peers. Plan for encryption from day one rather than retrofitting it under pressure later.

Losing Raft quorum stops writes, not the illusion that everything's fine

With 3 servers, losing 2 loses quorum — new registrations, health check transitions, KV writes, and new intentions all stop being committed, while DNS and HTTP reads can keep serving the last known state from whichever server (or a client's local cache) is still answering. That combination — reads that still work, writes that silently don't — is exactly the shape of failure most likely to go unnoticed until someone tries to register a new service during an incident and can't. Run 5 servers rather than 3 for anything where tolerating two simultaneous failures matters, and alert on Raft peer count directly rather than inferring cluster health from whether queries are still returning answers.

WAN federation across cloud or network boundaries is genuinely hard — which is why cluster peering exists

Classic multi-datacenter federation needs the WAN gossip pool's UDP and TCP ports open between every server, in every datacenter, across whatever firewalls, NAT, and cloud network boundaries sit in between — workable on a single provider's backbone, painful across multiple clouds or air-gapped networks. Cluster peering (Consul 1.14+) is HashiCorp's answer: two clusters establish a peering relationship over a single mTLS connection using a generated peering token, without joining a shared gossip pool at all, which is a far better fit for independently-operated Kubernetes clusters or multi-cloud topologies than trying to mesh every server together at the network layer. If you're designing for multi-region resilience more broadly, Resilient Cloud Solutions covers the same failover goal from the cloud-native side — Route 53 health-check routing rather than a gossip pool — worth reading as the comparison.

The proxy version matters as much as the Consul version

Envoy isn't bundled inside Consul — each Consul release supports a specific range of Envoy versions, and running a sidecar proxy version outside that supported matrix is a common source of mesh traffic silently misbehaving after what looked like a routine Consul upgrade. Check the supported-Envoy-versions table in HashiCorp's own release notes before bumping either piece independently, the same discipline Kubernetes upgrades demand of their own add-ons.

Consul vs. its alternatives

☺ Like you're 10: Other tools solve pieces of this same problem — the real question is whether your services all live in one place already, or are scattered across clouds, VMs, and clusters that need one shared answer.

The practical rule mirrors the one Vault reaches for on the secrets side: pick the cloud-native or orchestrator-native option when everything genuinely lives in one place already, and pick Consul when your estate is heterogeneous enough that a single-platform tool would need a second, third, or fourth tool bolted on to cover the rest of it. Plenty of real deployments run both — Kubernetes-native Services inside one cluster, Consul spanning that cluster, a fleet of VMs, and a second cloud region as the thing that actually ties them together.

Consul shows up as supporting infrastructure across the rest of this course wherever "many services, one network" becomes a real design problem: containers & orchestration names service discovery as a first-class requirement of any scheduler-driven fleet, distributed tracing & telemetry is where a mesh's sidecar access logs and trace headers usually end up, and Prometheus's own consul_sd_config can scrape targets straight out of the same catalog instead of a hand-maintained target list. If you're building toward AWS's DevOps Engineer Professional exam rather than a HashiCorp one, Consul itself won't appear by name — the DOP-C02 syllabus tests the AWS-native answer to this same problem instead, covered in Resilient Cloud Solutions and Configuration Management & IaC. HashiCorp has, at various points, offered a standalone Consul certification alongside Terraform Associate and Vault Associate — check HashiCorp's current certification catalog directly before planning around it, since which exams are active has shifted over time. Practice the mesh and discovery mechanics hands-on in Capstone Part 2 — Infrastructure as Code and Capstone Part 3 — Deployment Strategy. Official documentation lives at developer.hashicorp.com/consul, with the source at github.com/hashicorp/consul.