Istio

The ICA blueprint puts 60% of its weight on two things — 35% on Traffic Management, 25% on Securing Workloads — and this page turns both into running configuration. Istio is the CNCF-graduated, Envoy-based service mesh that takes encryption, identity, retries, timeouts and traffic shifting out of every application and makes them things the platform provides instead: a control plane called istiod that computes what every proxy should do, and a data plane — an Envoy sidecar riding in every pod, or a lighter ztunnel-plus-waypoint ambient mesh — that carries the bytes. Below: both data-plane shapes, the VirtualService/DestinationRule pair and the security trio you'll write for real, the istioctl verbs worth muscle memory, the failure modes that catch a happy-path-only read, and where Istio sits against Linkerd, Cilium's own mesh, and no mesh at all. This assumes the Kubernetes fluency the five-exam Kubestronaut foundation already gave you, from the sibling Kubernetes course; why any mesh looks the way it does has its own page.

Architecture: istiod, and two ways to carry traffic

☺ Like you're 10: One tower office computes every flight path and prints every badge; the planes themselves carry either their own dedicated co-pilot, or share a ground crew stationed at the runway instead.

Modern Istio collapsed what used to be three separate components — Pilot, Citadel, Galley — into one binary, istiod, running as a single Deployment in the istio-system namespace. It does four jobs and none of them involve a single byte of your traffic. It watches the Kubernetes API for Services, EndpointSlices, Pods and Istio's own custom resources. It translates that state into Envoy configuration — clusters, listeners, routes, endpoints, secrets — and pushes it to every proxy over the xDS protocol. It is the mesh's certificate authority, issuing every workload a short-lived X.509 certificate whose identity is a SPIFFE URI of the shape spiffe://cluster.local/ns/<namespace>/sa/<serviceaccount>, minted from something Kubernetes already vouches for: the pod's own ServiceAccount. And it runs the admission webhooks that validate your CRs and inject sidecars.

Sidecar mode: a co-pilot on every plane

In the classic model, an istio-proxy container — Envoy — is injected into every pod, and an init container programs iptables so inbound traffic redirects to port 15006 and outbound to 15001. The application never knows. Injection is opt-in per namespace, by label — either the simple istio-injection=enabled or, on a revisioned install, istio.io/rev=<revision>, which is what lets two control-plane versions run side by side while you migrate namespace by namespace. Injection only happens at pod creation, so existing pods are never retrofitted — they need a rollout restart before any of this applies to them.

Ambient mode: a shared ground crew at the runway

Ambient splits the mesh by layer instead of by pod. A per-node DaemonSet called ztunnel handles L4 only — mTLS, identity and basic telemetry for every ambient-enrolled pod on that node, tunnelling over HBONE (HTTP/2 CONNECT over mTLS). Only if a namespace or workload needs L7 features — HTTP routing rules, header matching, method/path-level AuthorizationPolicy — do you add a waypoint, an ordinary Envoy deployment that traffic for that scope is routed through. You enrol a namespace with the label istio.io/dataplane-mode=ambient, and — unlike sidecar mode — no restart is required, because there's no per-pod proxy to inject. The payoff is real: no per-pod memory tax, no pod restarts to upgrade the mesh, no sidecar lifecycle problems. You pay for L7 only where you actually deploy a waypoint.

One more thing every proxy does for free, sidecar or waypoint alike: because it already terminates and re-issues each hop, it emits uniform access logs and golden-signal metrics without asking the app for anything, and it propagates whatever trace headers the app already forwards — B3 or the W3C traceparent format — so a single request stitches into one distributed trace spanning every hop it crosses. That pipeline is what feeds Prometheus, Grafana, and the OpenTelemetry Collector the moment a namespace is enrolled, and it's exactly what Dot means below by "I didn't instrument a single span."

The VirtualService and DestinationRule pair you'll write constantly

☺ Like you're 10: One file decides where a request goes and how many copies get a shot; a second file names the delivery teams that first file is actually talking about.

A DestinationRule defines subsets — named groups of pods picked out by label — plus a trafficPolicy: load-balancer choice, connection pools, and outlierDetection, which is Istio's circuit breaker. A VirtualService then decides where a request actually goes: match on host, header, path or method, then route, split by weight, retry, time out, or inject a fault. Weights within one route must sum to 100, and match blocks are evaluated in order — first match wins.

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: payments
  namespace: prod
spec:
  host: payments.prod.svc.cluster.local
  subsets:
    - name: v1
      labels: { version: v1 }           # matches POD labels, not the Service's
    - name: v2
      labels: { version: v2 }
  trafficPolicy:
    loadBalancer: { simple: LEAST_REQUEST }
    connectionPool:                     # ---- circuit breaking ----
      tcp:  { maxConnections: 100 }
      http:
        http2MaxRequests: 200
        maxRequestsPerConnection: 10
    outlierDetection:                   # ---- eject unhealthy endpoints ----
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: payments
  namespace: prod
spec:
  hosts: ["payments"]                   # short name = in-mesh (east-west) traffic
  http:
    - match:                            # first match wins — order is meaning
        - headers:
            x-canary: { exact: "true" }
      route:
        - destination: { host: payments, subset: v2 }
    - fault:                            # ---- fault injection ----
        delay: { percentage: { value: 5 }, fixedDelay: 3s }
      timeout: 2s                       # ---- timeout ----
      retries:                          # ---- retries ----
        attempts: 3
        perTryTimeout: 500ms
        retryOn: 5xx,reset,connect-failure
      route:                            # ---- traffic shifting: must total 100 ----
        - destination: { host: payments, subset: v1 }
          weight: 90
        - destination: { host: payments, subset: v2 }
          weight: 10

The front door and reaching outward: Gateway, the Gateway API, and ServiceEntry

Istio's own Gateway binds a listener to an ingress or egress gateway deployment; a VirtualService then attaches routes to it with the gateways field — omit that field and the rule applies only to in-mesh traffic, the classic "why is my route being ignored?" The project's own stated direction is the vendor-neutral Gateway API instead (Gateway + HTTPRoute, gatewayClassName: istio), which new platforms should prefer. Reaching out of the mesh works the opposite way: a ServiceEntry registers an external host so policy can apply to it at all — pair it with outboundTrafficPolicy.mode: REGISTRY_ONLY, below, so anything not declared this way simply cannot leave.

apiVersion: networking.istio.io/v1
kind: ServiceEntry                      # teach the mesh about the outside world
metadata:
  name: stripe
  namespace: prod
spec:
  hosts: ["api.stripe.com"]
  ports: [{ number: 443, name: https, protocol: HTTPS }]
  resolution: DNS
  location: MESH_EXTERNAL

Every weight in that first code block is also the mechanism under a mesh-driven canary — Argo Rollouts watches metrics and rewrites exactly that weight field automatically, which is the separation of deploy from release this course keeps coming back to.

Locking it down: PeerAuthentication, RequestAuthentication, AuthorizationPolicy

☺ Like you're 10: One rule asks whether the plane itself is who it claims to be; a second asks whether the passenger's ticket is genuine; a third decides who's actually allowed to land.

Three resources, three different questions, and the exam rewards knowing exactly which one answers which. PeerAuthentication asks "must the caller present a mesh certificate?" — workload authentication, with mode: STRICT as the zero-trust setting; placed in istio-system with no selector it applies mesh-wide. RequestAuthentication asks "is this JWT valid, and who issued it?" — end-user authentication, which on its own rejects only invalid tokens, never absent ones. AuthorizationPolicy asks "is this caller allowed to do this?" — matching on SPIFFE principals (never an IP), requestPrincipals, HTTP method and path.

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: prod             # in istio-system with no selector = mesh-wide
spec:
  mtls:
    mode: STRICT               # STRICT | PERMISSIVE | DISABLE | UNSET
---
apiVersion: security.istio.io/v1
kind: RequestAuthentication
metadata:
  name: jwt-issuer
  namespace: prod
spec:
  selector:
    matchLabels: { app: payments }
  jwtRules:
    - issuer: "https://accounts.example.com"
      jwksUri: "https://accounts.example.com/.well-known/jwks.json"
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: deny-all
  namespace: prod
spec: {}                       # ALLOW with zero rules => nothing is permitted
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: checkout-may-call-payments
  namespace: prod
spec:
  selector:
    matchLabels: { app: payments }
  action: ALLOW
  rules:
    - from:
        # keys INSIDE one source are ANDed: identity AND a valid token, together
        - source:
            principals: ["cluster.local/ns/prod/sa/checkout"]
            requestPrincipals: ["https://accounts.example.com/*"]
      to:
        - operation:
            methods: ["GET", "POST"]
            paths: ["/api/cart*"]

Two evaluation rules matter more than any single field. DENY policies are evaluated before ALLOW policies, so a matching DENY wins outright over any ALLOW anywhere in the namespace. And the moment any ALLOW policy selects a workload, that workload becomes default-deny for everything else — which is exactly why the empty-spec policy above locks a namespace down with nothing else written. action: AUDIT logs what would have been denied without denying it, and is the correct way to roll a new policy out — PERMISSIVE before STRICT, AUDIT before DENY, every time.

AuthorizationPolicy governs which already-admitted requests may flow between live workloads; it's a different plane entirely from what Kyverno governs — which manifests get admitted to the cluster in the first place — and the two are complementary halves of the same zero-trust argument, not substitutes for each other. Kyverno, next in this section, picks up exactly that other half.

Installing, upgrading, and enrolling namespaces with istioctl

☺ Like you're 10: One command checks the airfield can even take Istio; a couple more install it, put a sticker on a namespace, and move a whole runway to a newer tower one step at a time.

istioctl install is the fast route and takes built-in profiles — default, demo, minimal, ambient — customized with --set or an IstioOperator-shaped file. Helm is the composable route, and what most GitOps setups use instead: three charts, in order — base (CRDs and cluster roles), istiod, then gateway per ingress or egress gateway — which means the whole datapath config becomes one reviewable values file, deliverable through Argo CD or Flux like anything else in this course's GitOps section. Ambient adds the cni and ztunnel charts on top.

The upgrade path names two strategies worth separating in one sentence each. In-place replaces the running control plane directly — simple, all-or-nothing. Canary installs a second istiod under a revision, moves namespaces onto it one at a time by changing the istio.io/rev label and restarting their pods, and rolls back by relabelling; a revision tag — a stable alias like prod-stable — moves a whole fleet at once by repointing the tag instead.

# Will this cluster even take Istio? Always run this first.
istioctl x precheck

# --- Path A: istioctl, with a profile ---
istioctl install --set profile=default -y
istioctl install --set profile=ambient -y          # ztunnel + CNI node agent

# --- Path B: Helm, three charts, in this order ---
helm repo add istio https://istio-release.storage.googleapis.com/charts
helm install istio-base istio/base -n istio-system --create-namespace
helm install istiod     istio/istiod -n istio-system --wait
helm install istio-ingressgateway istio/gateway -n istio-ingress --create-namespace

# --- Enrol a namespace — pick ONE, they are mutually exclusive ---
kubectl label namespace prod istio-injection=enabled        # sidecar, default rev
kubectl label namespace prod istio.io/rev=1-27-0             # sidecar, revisioned
kubectl label namespace prod istio.io/dataplane-mode=ambient # ambient — no restart needed
kubectl rollout restart deployment -n prod        # sidecar pods aren't retrofitted

# --- Canary upgrade: a second control plane, moved namespace by namespace ---
istioctl install --set revision=1-27-0 -y
istioctl tag set prod-stable --revision 1-27-0     # stable alias, moves a whole fleet
kubectl label ns prod istio.io/rev=1-27-0 --overwrite
kubectl rollout restart deployment -n prod
istioctl uninstall --revision 1-26-0 -y            # retire the old control plane

istioctl upgrade -y            # in-place: replaces everything, all at once
istioctl version                # client, istiod, and every data-plane proxy

Troubleshooting: the ladder from config to control plane to proxy

☺ Like you're 10: Instead of arguing about whether the tower or a pilot made the mistake, you just ask each one in turn — and one of them always answers with the exact reason.

The curriculum names the failure surface in a genuinely useful order — configuration, then the control plane, then the data plane — and working it in that order finds most failures fast. istioctl analyze is a real linter, and it catches missing subsets, conflicting policies and unreferenced gateways before they become a 503. For the control plane, ask whether istiod is healthy and whether its config actually reached the proxies. For the data plane, stop guessing and ask one proxy what it believes.

# 1. CONFIGURATION — lint before you blame anything else
istioctl analyze -n prod
istioctl analyze virtualservice.yaml destinationrule.yaml   # even before committing

# 2. CONTROL PLANE — is istiod healthy, and did the push land?
kubectl -n istio-system get pods
istioctl proxy-status              # alias: ps. CDS/LDS/EDS/RDS all SYNCED?
                                    # STALE = push stuck · NOT SENT = nothing to send

# 3. DATA PLANE — what does ONE proxy actually believe?
istioctl proxy-config route    payments-7d9f-abcde.prod --name 8080 -o json
istioctl proxy-config cluster  payments-7d9f-abcde.prod
istioctl proxy-config endpoint payments-7d9f-abcde.prod \
  --cluster "outbound|8080|v2|payments.prod.svc.cluster.local"
istioctl proxy-config secret   payments-7d9f-abcde.prod   # did the cert arrive?

# Plain-English summary: mTLS mode, policies and routes affecting one pod
istioctl x describe pod payments-7d9f-abcde.prod

# Turn up Envoy logging, then read what the proxy recorded
istioctl proxy-config log payments-7d9f-abcde.prod --level rbac:debug,router:debug
kubectl logs payments-7d9f-abcde -n prod -c istio-proxy

Learn the symptoms that map to a single cause, because recognition beats deduction under time pressure. RBAC: access denied is an AuthorizationPolicy, not a network problem. A 503 right after a routing change is almost always a subset the VirtualService names and no DestinationRule defines. Failures right after enabling STRICT mean something outside the mesh is calling in. And a pod with no sidecar at all is almost always a namespace label that got applied after the pod already existed.

Gotchas and failure modes

☺ Like you're 10: Most surprises come from forgetting who's still outside the mesh, a co-pilot that starts or stops at the wrong moment, or handing every messenger the whole flight schedule when they only needed three numbers.

STRICT mTLS cuts off everything unmeshed

The classic outage: flip mtls.mode: STRICT mesh-wide on day one and you sever every client the mesh doesn't know about — a monitoring scraper living outside the namespace, a legacy VM, a CronJob pod in a namespace nobody remembered to label, a cloud load balancer health check hitting a pod IP directly. Istio rewrites plain httpGet kubelet probes to route through the pilot-agent on port 15020 so they're exempt, but not every probe style qualifies that way, and headless Services and raw databases often need their ports explicitly named (http, grpc, tcp-mysql) or an appProtocol set before protocol detection stops guessing wrong. The safe sequence never changes: install PERMISSIVE, enrol and restart everything, confirm with istioctl x describe pod that traffic is genuinely mTLS, and only then flip to STRICT — one namespace at a time, never mesh-wide first.

Sidecar lifecycle races

Historically the ugliest bug class in the sidecar model. If the app container starts before istio-proxy is ready, its first outbound calls fail — a migration Job that runs at boot dies mysteriously. In the other direction, a Job or CronJob pod never reaches Completed, because the app exits but the sidecar keeps running forever. The old workarounds — holdApplicationUntilProxyStarts, or the app calling the proxy's /quitquitquit endpoint on exit — are largely superseded now by Kubernetes native sidecars: initContainers entries carrying restartPolicy: Always, on by default since Kubernetes 1.29 and GA in 1.33, which Istio uses to guarantee start-before, stop-after ordering. Ambient mode sidesteps the whole category — there is no sidecar to race against.

The cost of a proxy per pod, and config that grows with the mesh

Every sidecar costs memory, CPU, and a hop of latency in each direction — real money and real p99 at a few hundred pods. Worse, by default istiod tells every proxy about every service in the mesh, so each proxy's config grows with the size of the whole estate, and pushes get slow as the mesh does. The two levers are a Sidecar resource, restricting a namespace's proxies to the hosts they actually call, and exportTo/discoverySelectors to scope visibility further. If none of that appeals, that's precisely the argument for ambient mode, or for a lighter mesh entirely.

apiVersion: networking.istio.io/v1
kind: Sidecar
metadata:
  name: default
  namespace: prod          # applies to every proxy in this namespace
spec:
  egress:
    - hosts:
        - "./*"            # everything in my own namespace
        - "istio-system/*" # plus the control plane
        - "payments/*"     # plus the one namespace I genuinely call
  outboundTrafficPolicy:
    mode: REGISTRY_ONLY    # block egress to anything without a ServiceEntry

Istio vs the alternatives

☺ Like you're 10: A few different flight-crew arrangements exist. They trade off how much gear rides on every plane, how much the tower has to run, and how much you have to learn to operate one.

The honest version of that decision: like Cilium's eBPF commitment, adopting Istio is strategic, not a checkbox — it puts real weight on your platform team's plate in exchange for guarantees no app team could deliver by discipline alone, and it is genuinely hard to walk back once workloads depend on JWT-aware authorization and per-subset traffic policy. That's most of why the ICA exists as its own specialist credential, examining Istio by name the way the CCA examines Cilium by name, rather than a vendor-neutral spec the way CGOA does. Two other pages cover the same ground from different angles: Service Mesh Architecture compares Istio's own sidecar and ambient shapes against Linkerd's and Cilium's from first principles, and Platform Engineering's deeper operational reference goes further into multi-cluster identity and production-scale gotchas for anyone continuing on to the CNPE afterward.