Envoy

Reliability patterns taught circuit breakers, retries with backoff and jitter, timeouts, and bulkheads as things a service builds into its own code — Resilience4j in Java, Polly in .NET, a hand-rolled retry loop everywhere else. Envoy is a high-performance L3/L4/L7 proxy, originally built at Lyft and open sourced in 2016, that implements nearly that entire toolkit as network-layer primitives configured declaratively instead of coded per service. It became the CNCF's second graduated project, after Kubernetes, in November 2018 — and it's also the reason "service mesh" became a mainstream idea at all: Istio, Consul Connect, and most of the CNCF mesh ecosystem are a control plane wrapped around Envoy's data plane, not a competing proxy implementation.

What Envoy is and the problem it solves

☺ Like you're 10: Instead of every app writing its own retry-and-circuit-breaker code, one proxy sitting next to every service does it identically for all of them, in every language at once.

Envoy was built at Lyft starting in 2015 by Matt Klein's infrastructure team, to solve a specific, unglamorous problem: Lyft ran services in half a dozen languages, and every one of them needed the same resilience behavior — retries, timeouts, circuit breaking, load balancing, TLS — reimplemented in whatever library that language happened to have. A Java service on Resilience4j and a Python service with a hand-rolled retry loop rarely behave identically under the same failure, and changing the retry policy meant coordinating a library bump and a redeploy across every repository that called the failing dependency. Lyft's answer was to move that logic out of application code entirely and into a proxy that sits next to every service, intercepts all of its inbound and outbound traffic, and applies the policy uniformly — no application code, no per-language library, no redeploy to change a timeout.

Envoy by itself is not a service mesh — it's the proxy a mesh is built from. A mesh needs a control plane deciding what Envoy should do (which routes exist, which clusters are healthy, which certificates to present) and pushing that decision to every Envoy instance live; Envoy is the uniform enforcement point that control plane targets. That split — data plane versus control plane — is the architectural idea the rest of this page builds on.

Architecture: the proxy and the xDS discovery APIs

☺ Like you're 10: A single fast proxy sits next to every service, and instead of hand-editing its config file, a separate control plane streams it live updates it applies without dropping a single connection.

Envoy itself is a single binary written in C++, using non-blocking event-driven I/O, deliberately optimized for low tail latency and predictable behavior under load rather than raw throughput. It ships in three deployment shapes that all run the identical binary: as a sidecar — one Envoy process per application pod, with inbound and outbound traffic transparently redirected through it via iptables rules or a CNI, so the application never has to know it's there; as an edge/gateway proxy — a fleet of Envoy instances fronting a cluster, terminating TLS and routing north-south traffic the way NGINX or HAProxy traditionally did; and standalone, as the engine inside API gateways like Envoy Gateway, Contour, Emissary-ingress, and Gloo Edge.

What makes Envoy usable as a live mesh data plane rather than a static reverse proxy is xDS — a family of gRPC discovery services Envoy polls or streams from a control plane: LDS (Listener Discovery Service), RDS (routes), CDS (clusters), EDS (endpoints — the actual healthy IPs behind a cluster), and SDS (Secret Discovery Service, for TLS certificates and keys, so certs rotate without a restart). In practice these are almost always multiplexed over one gRPC stream as ADS (Aggregated Discovery Service), which also fixes an ordering problem: pushing a new route before its cluster exists would otherwise create a brief 503 window. A control plane — Istio's istiod, Consul's built-in one, or a custom one built on the go-control-plane library — computes this config from its own source of truth and streams it down; Envoy applies additions and changes without dropping in-flight connections, which is the property that makes reconfiguring a fleet of proxies a live operation instead of a rolling restart.

The resilience toolkit, configured declaratively

☺ Like you're 10: The same four things reliability-patterns.html taught as app code — a way to notice a sick backend, a circuit breaker, retries, and timeouts — become a few dozen lines of YAML on a cluster or a route.

This is the direct payoff of putting a proxy in front of every service: the patterns from reliability patterns stop being a library each team adopts (or forgets to) and become fields on two Envoy resources — the Cluster (everything about a single upstream: how to find it, how to load-balance across it, when to eject a bad member of it) and the Route (everything about a single request path: which cluster it goes to, its timeout, its retry policy). Confirm exact field names and defaults against the current Envoy v3 API reference before shipping — Envoy's config surface has evolved across major API versions and individual fields do get deprecated.

Outlier detection — passive health checking

Outlier detection watches a cluster's real traffic — no separate health-check probe required — and ejects a member that looks unhealthy by its actual response pattern: a run of consecutive 5xx responses, a burst of consecutive gateway errors, or a success-rate outlier relative to its siblings using statistical outlier detection. An ejected host is pulled out of the load-balancing pool for a base ejection time that doubles on each subsequent ejection, then given another chance.

clusters:
- name: payments_service
  connect_timeout: 0.5s
  type: STRICT_DNS
  lb_policy: ROUND_ROBIN
  load_assignment:
    cluster_name: payments_service
    endpoints:
    - lb_endpoints:
      - endpoint: { address: { socket_address: { address: payments.internal, port_value: 8443 } } }
  outlier_detection:
    consecutive_5xx: 3            # eject after 3 consecutive 5xx from ONE host
    interval: 10s                 # how often the ejection sweep runs
    base_ejection_time: 30s       # doubles on repeated ejections of the same host
    max_ejection_percent: 50      # never eject more than half the cluster at once
    enforcing_consecutive_5xx: 100  # % of the time a qualifying host is actually ejected

max_ejection_percent is the single most important safety field in that block. If a shared dependency behind every member of a cluster fails at once — a database, a downstream all three replicas call — outlier detection watching only response codes would happily eject all of them, leaving nothing to route to. Capping the ejection percentage keeps at least some capacity in the pool even during a correlated failure; it doesn't fix the underlying dependency, but it stops the health check from making a bad outage total.

Circuit breaking — and why the name is misleading

  circuit_breakers:
    thresholds:
    - priority: DEFAULT
      max_connections: 1024
      max_pending_requests: 1024
      max_requests: 1024
      max_retries: 3
      retry_budget:
        budget_percent: { value: 20.0 }   # retries capped at 20% of active request volume...
        min_retry_concurrency: 3          # ...but never below this floor

Retries — with a budget, not just a count

Reliability patterns already covered why naive retries can turn a blip into an outage: if every one of ten thousand requests-per-second hitting a struggling cluster retries three times, the cluster's effective load doesn't stay flat, it roughly quadruples right when it can least absorb it. A fixed num_retries per request doesn't fix that; it just moves the multiplier into config. A retry budget (the retry_budget field shown above) fixes the actual problem by expressing the retry ceiling as a percentage of current active request volume rather than a flat per-request count — so total retry traffic scales with real traffic instead of amplifying without bound as volume grows.

routes:
- match: { prefix: "/" }
  route:
    cluster: payments_service
    timeout: 2s                  # total budget for the whole route, all attempts included
    retry_policy:
      retry_on: "5xx,reset,connect-failure,refused-stream"
      num_retries: 3              # a per-request ceiling — the budget above caps it fleet-wide
      per_try_timeout: 0.5s       # each individual attempt's own deadline
      retry_back_off:
        base_interval: 0.1s
        max_interval: 2s

Timeouts — three, not one

Envoy separates timeout concerns the way a careful reviewer would insist on by hand: connect_timeout on the cluster bounds how long establishing the TCP connection may take; route.timeout bounds the whole request, every retry attempt included; and per_try_timeout bounds a single attempt within that budget. Getting the relationship between the last two wrong is the most common timeout mistake — a per_try_timeout of 0.5s with three retries can consume up to 1.5s before the route-level 2s timeout ever fires, and if the caller's own client-side deadline is shorter than that, the caller gives up and disconnects while Envoy is still faithfully retrying into the void.

Why it became the default service-mesh data plane

☺ Like you're 10: Once every service already has this proxy standing next to it, that same connection can also carry a certificate, get encrypted, and report a trace span — a mesh is mostly just handing the proxy a few more jobs.

A service mesh needs exactly the shape Envoy already has: a proxy next to every service, capable of being reconfigured live, applying policy uniformly regardless of the application's language. Once that proxy exists, adding mutual TLS (via SDS-delivered certificates), consistent access logging, distributed tracing spans, and authorization checks is additive configuration on the same data plane rather than a second piece of infrastructure — see monitoring and observability for how that per-hop telemetry gets used. Istio, covered on its own page, is the clearest example: its sidecar container literally is Envoy, and istiod is the xDS control plane that translates Istio's own CRDs into the exact resources shown above — a DestinationRule's trafficPolicy.outlierDetection and connectionPool fields lower directly to Envoy's outlier_detection and circuit_breakers, and a VirtualService's retries block lowers to a route's retry_policy. Consul Connect and Gloo Mesh follow the same shape with their own control planes; Contour, Emissary-ingress, and the Envoy project's own Envoy Gateway use Envoy as an ingress data plane rather than a full mesh. Confirm current support status for any specific mesh product before committing to it — this is a fast-moving part of the ecosystem, and at least one major cloud provider's managed mesh offering has already been announced for deprecation since this course was written.

Day-to-day commands and operations

☺ Like you're 10: Almost nobody hand-edits Envoy's live config — instead you read it back through an admin endpoint, and change the source that generates it.

# standalone Envoy, started from a static bootstrap file
$ envoy -c bootstrap.yaml --service-cluster payments --service-node payments-1

# the admin interface — bound to 127.0.0.1:9901 by default, NEVER expose it publicly
$ curl -s localhost:9901/clusters | grep -E "health_flags|outlier"     # per-endpoint health state
$ curl -s localhost:9901/config_dump | jq '.configs[].dynamic_listeners'  # config actually applied
$ curl -s localhost:9901/stats/prometheus | grep retry                # retry/circuit-breaker counters
$ curl -s localhost:9901/ready                                        # readiness probe target
$ curl -X POST localhost:9901/logging?level=debug                     # bump log verbosity live

# Istio-managed Envoy sidecars: istioctl wraps the same admin data per pod
$ istioctl proxy-config cluster  checkout-7d6c9-abcde -n prod
$ istioctl proxy-config route    checkout-7d6c9-abcde -n prod
$ istioctl proxy-status                       # xDS sync state across the whole mesh: ACK / NACK / STALE

Standalone Envoy's classic reconfiguration mechanism — a hot restart, where a new worker process attaches to the old one's listening sockets over a shared-memory handoff so no connection drops during a binary upgrade — matters mostly for edge/gateway deployments managing their own lifecycle. Inside a mesh, config changes normally arrive over xDS with no restart of any kind; the hot-restart machinery is largely invisible unless you're operating standalone Envoy directly.

Gotchas and failure modes

☺ Like you're 10: Most Envoy surprises come from a safety knob nobody tuned, a config push that silently didn't apply, or forgetting that a proxy in front of every pod isn't free.

• The panic threshold, and why it exists. If outlier detection (or a failed active health check) marks too many hosts in a cluster unhealthy — by default, once fewer than 50% of the panic-relevant hosts are healthy — Envoy's load balancer stops honoring health status entirely and routes to all hosts, healthy or not. It looks like a bug the first time you see it; it's a deliberate safety valve against the worse outcome of black-holing an entire cluster because a shared dependency made every member look unhealthy at once. Design ejection thresholds knowing this valve exists, but don't rely on it as your actual resilience strategy.
• Retries without idempotency, again. Covered above and worth repeating as an operational failure mode: a route with retry_on: "5xx" in front of a non-idempotent write, with no idempotency key handled downstream, silently double-executes writes under exactly the failure conditions — a slow, overloaded backend — where it matters most.
• An xDS push that silently doesn't apply. If a control plane streams an invalid CDS or RDS update, Envoy NACKs it and keeps serving the last known-good config — the safe behavior, but also a quiet one. An engineer who shipped the change sees no error and assumes it's live; istioctl proxy-status showing STALE or NACKED for a pod, not the dashboard for the change itself, is where that gets caught.
• The sidecar tax is real. A per-pod Envoy sidecar costs real baseline memory and CPU, and adds roughly one to a few milliseconds of latency per hop in each direction — trivial for one hop, additive across a deep call chain. At high pod counts across a large fleet this becomes a genuine capacity planning line item, and it's the direct motivation behind "ambient" mesh designs (Istio's ambient mode, using a shared per-node ztunnel) that move some of this cost off the per-pod model.
• Timeout stacking. A route-level timeout shorter than per_try_timeout times the retry count, or longer than the calling client's own deadline, produces exactly the mismatch described above — Envoy still retrying a request the original caller already gave up on. Set the three timeout fields as one deliberate budget, not three independent guesses.
• Hand-writing raw xDS YAML doesn't scale. Every example on this page is what a control plane generates; almost no team maintains this by hand across a real fleet. Attempting to hand-roll and keep Cluster/Route resources in sync for more than a handful of services is exactly the kind of recurring manual work toil and automation exists to eliminate — it's the reason control planes like Istio exist in the first place, not an optional layer on top.

Alternatives and when to choose it

☺ Like you're 10: Other tools also route traffic and can retry or time out a call — they just trade away Envoy's live, uniform, cross-language configuration in different ways.

The practical rule: reach for application-level libraries when the system is small and single-language and an extra hop isn't worth it; reach for a lighter proxy like Linkerd2-proxy when Kubernetes-only mTLS and basic resilience is genuinely all that's needed; reach for Envoy — usually via a mesh control plane rather than hand-written config — once the fleet is polyglot, the resilience policy needs to be uniform and centrally changeable, or a mesh's other benefits (mTLS, tracing, authorization) are wanted on the same data plane. See the SRE toolchain for where Envoy sits alongside the rest of the observability and delivery stack, and Kubernetes reliability patterns for how the sidecar model interacts with pod scheduling and probes.

Going further

☺ Like you're 10: This page is the proxy; the control plane that usually drives it in production is a separate page.

The canonical source is the Envoy documentation and v3 API reference at envoyproxy.io/docs — confirm current field names, defaults, and API version against it, since Envoy's config surface iterates faster than a stable, rarely-changing OSS API — plus the source at github.com/envoyproxy/envoy and the CNCF project page at cncf.io/projects/envoy. Pair this page with reliability patterns for the application-level version of everything covered here, Istio for the control plane that drives most production Envoy fleets, chaos engineering for deliberately testing whether an outlier-detection or retry-budget configuration actually holds under a real fault, and the glossary for any term above that didn't stick on first read.