Drill — Lock Down a Mesh Namespace

One namespace, one real mistake made on purpose: you'll enrol a tiny payments service into the mesh, confirm two different callers can reach it under PeerAuthentication mode PERMISSIVE, then flip the namespace to STRICT and watch one of those callers break — not because anything is wrong with your YAML, but because a legitimate external health-check was never going to run a sidecar. You diagnose the break instead of guessing at it, fix it with a scoped port-level override instead of backing the whole namespace off PERMISSIVE again, and then close the door that's still standing open: mTLS alone never restricted who could call payments, only whether the call was encrypted. The last third of the drill writes and proves an AuthorizationPolicy that finally answers that question. This is the hands-on counterpart to Istio and Service Mesh Architecture — if PeerAuthentication or AuthorizationPolicy don't already feel familiar, read those first, then come back and actually type it.

How this drill works

☺ Like you're 10: One namespace, flipped from friendly to strict, one caller that predictably breaks, one narrow fix, and then a second, completely separate gap you close on top of it.

This drill has two acts, and it's worth knowing that going in. Act one is an mTLS story: PeerAuthentication decides whether a workload will accept plaintext, encrypted, or only encrypted connections — and flipping a whole namespace to STRICT in one step is exactly the move that catches an unmeshed caller nobody remembered. Act two is a completely different question that mTLS never answers on its own: once every connection to payments is encrypted and authenticated, who is actually allowed to call it? The answer, until you write one, is "anyone in the mesh" — which is rarely what you meant. You'll prove that gap is real before you close it, the same habit this course keeps insisting on: don't declare a fix done until you've watched it actually stop the thing it's supposed to stop.

Set up the mesh and a namespace worth locking down

☺ Like you're 10: One small service with two doors — a main door for real traffic, and a little side window just for health checks — plus three different visitors to test it with.

Spin up a cluster and install Istio's demo profile — it includes an ingress gateway you won't need today, but it's the fastest reliable way to get a working control plane for a drill:

kind create cluster --name mesh-drill
istioctl x precheck
istioctl install --set profile=demo -y
kubectl create namespace prod
kubectl label namespace prod istio-injection=enabled
kubectl create namespace ops          # deliberately NOT labelled — stays outside the mesh

payments gets two ports on purpose: 8080 is the real API, 8081 is a lightweight health endpoint that an external monitor polls. Keeping them on the same workload is what makes the port-level fix later actually mean something. A tiny nginx config serves both:

# payments-nginx-conf.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: payments-nginx
  namespace: prod
data:
  nginx.conf: |
    events {}
    http {
      server {
        listen 8080;
        location / {
          default_type application/json;
          return 200 '{"service":"payments","status":"ok"}\n';
        }
      }
      server {
        listen 8081;
        location /healthz {
          return 200 'ok\n';
        }
      }
    }

# payments.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: payments
  namespace: prod
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: checkout
  namespace: prod
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments
  namespace: prod
spec:
  replicas: 1
  selector:
    matchLabels: { app: payments }
  template:
    metadata:
      labels: { app: payments }
    spec:
      serviceAccountName: payments
      containers:
        - name: nginx
          image: nginx:1.27-alpine
          ports: [{ containerPort: 8080 }, { containerPort: 8081 }]
          volumeMounts:
            - { name: conf, mountPath: /etc/nginx/nginx.conf, subPath: nginx.conf }
      volumes:
        - name: conf
          configMap: { name: payments-nginx }
---
apiVersion: v1
kind: Service
metadata:
  name: payments
  namespace: prod
spec:
  selector: { app: payments }
  ports:
    - { name: http, port: 8080, targetPort: 8080 }
    - { name: health, port: 8081, targetPort: 8081 }

Three callers, standing in for three real-world roles. checkout — in-mesh, its own ServiceAccount, the caller that's actually supposed to hit the API. ops-healthcheck — deliberately outside the mesh, standing in for an external monitor that polls /healthz and structurally can't run a sidecar. rogue — in-mesh, default ServiceAccount, standing in for "anything else that happens to be running in the cluster" — the caller nobody explicitly invited.

# callers.yaml
apiVersion: v1
kind: Pod
metadata:
  name: checkout-caller
  namespace: prod
  labels: { app: checkout-caller }
spec:
  serviceAccountName: checkout
  containers:
    - { name: curl, image: curlimages/curl:8.10.1, command: ["sleep", "infinity"] }
---
apiVersion: v1
kind: Pod
metadata:
  name: rogue
  namespace: prod
  labels: { app: rogue }
spec:
  containers:                        # default ServiceAccount — nobody invited this one
    - { name: curl, image: curlimages/curl:8.10.1, command: ["sleep", "infinity"] }
---
apiVersion: v1
kind: Pod
metadata:
  name: ops-healthcheck
  namespace: ops                     # unmeshed namespace — no sidecar, ever
  labels: { app: ops-healthcheck }
spec:
  containers:
    - { name: curl, image: curlimages/curl:8.10.1, command: ["sleep", "infinity"] }

kubectl apply -f payments-nginx-conf.yaml
kubectl apply -f payments.yaml
kubectl apply -f callers.yaml
kubectl -n prod rollout status deploy/payments
kubectl -n prod get pod payments-* -o jsonpath='{.spec.containers[*].name}'
# expect: nginx istio-proxy — the namespace label injected a sidecar automatically

Step 1 — confirm the baseline under PERMISSIVE

☺ Like you're 10: Badge reader installed, set to friendly mode — everyone still gets in, badge or no badge, so nothing looks any different yet.

Make the intent explicit rather than relying on Istio's own default. An explicit PERMISSIVE policy accepts both plaintext and mTLS on every port — which means, right now, all three callers should succeed identically, and you won't yet be able to tell from the outside which connections are actually encrypted:

# peer-auth.yaml
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: payments-mtls
  namespace: prod
spec:
  selector:
    matchLabels: { app: payments }
  mtls:
    mode: PERMISSIVE

kubectl apply -f peer-auth.yaml

kubectl -n prod exec checkout-caller -- curl -s -o /dev/null -w 'checkout  -> %{http_code}\n' http://payments.prod.svc.cluster.local:8080/
kubectl -n prod exec rogue          -- curl -s -o /dev/null -w 'rogue     -> %{http_code}\n' http://payments.prod.svc.cluster.local:8080/
kubectl -n ops  exec ops-healthcheck -- curl -s -o /dev/null -w 'ops       -> %{http_code}\n' http://payments.prod.svc.cluster.local:8081/healthz

All three should print 200. Hold that result in your head — it's the "before" picture for both acts of this drill: right now, anyone can reach payments, meshed or not, and encryption is optional rather than required.

Step 2 — flip to STRICT and watch what breaks

☺ Like you're 10: Flip the reader to strict mode. Everyone with a badge still walks straight in — the nurse without one is suddenly stuck outside, and she didn't do anything wrong.

Change one field and reapply:

  mtls:
    mode: STRICT   # was PERMISSIVE

kubectl apply -f peer-auth.yaml
kubectl -n prod exec checkout-caller -- curl -s -o /dev/null -w 'checkout  -> %{http_code}\n' http://payments.prod.svc.cluster.local:8080/
kubectl -n ops  exec ops-healthcheck -- curl -m 5 -s -o /dev/null -w 'ops       -> %{http_code}\n' http://payments.prod.svc.cluster.local:8081/healthz; echo "exit: $?"

checkout  -> 200
exit: 56

checkout never even noticed — its sidecar was already speaking mTLS to payments's sidecar the whole time; STRICT just makes that mandatory instead of optional. ops-healthcheck gets curl exit code 56, "recv failure: connection reset by peer": it never had a sidecar to upgrade the connection, so it's sending plain HTTP straight at a listener that now refuses anything that isn't mTLS. Nothing is broken in the sense of a bug — this is STRICT doing exactly its job. The question is what to do about a caller that was never going to be able to comply.

Step 3 — diagnose the break, don't guess at it

☺ Like you're 10: Before you touch anything, ask three questions in order: is the badge-reader's rulebook written correctly, did the rulebook actually reach the reader, and what did the reader do with the visitor who showed up?

Work the same three-layer ladder Istio and the ICA blueprint both teach — configuration, then control plane, then data plane — instead of jumping straight to a fix:

# 1. CONFIGURATION — is the PeerAuthentication itself valid and unambiguous?
istioctl analyze -n prod
# clean — analyze checks your YAML's shape, not runtime callers it never sees

# 2. CONTROL PLANE — did the STRICT policy actually reach payments's proxy?
istioctl proxy-status | grep payments
# payments-xxxxx.prod   SYNCED   SYNCED   SYNCED   SYNCED   istiod-...   1.2x.x

# 3. DATA PLANE — what does the proxy that rejected the call actually believe?
istioctl x describe pod -n prod $(kubectl -n prod get pod -l app=payments -o jsonpath='{.items[0].metadata.name}')

Pod: payments-7c9f8d-abcde
   Effective PeerAuthentication:
      Workload mTLS mode: STRICT
--------------------
Effective AuthorizationPolicies:
   (none, all traffic is allowed)

That output is the whole diagnosis in two lines. Nothing is misconfigured — istioctl analyze is clean because the policy itself is well-formed, and proxy-status confirms it genuinely reached the workload. The proxy is doing precisely what STRICT says: refuse anything that isn't mTLS. The problem was never the policy; it's that one real caller structurally can't meet it. Notice, too, the second block — (none, all traffic is allowed) under authorization. File that away; it's exactly the gap Step 6 exists to close.

Step 4 — fix it with a port-level override, not a namespace-wide retreat

☺ Like you're 10: Don't unlock the whole building again just because one visitor doesn't carry a badge — leave a small side window open for her specifically, and keep the main door locked.

The tempting fix is to drop the whole namespace back to PERMISSIVE — and that would work, in the sense that it makes the error go away while quietly un-doing the entire point of this drill. The correct fix is narrower: PeerAuthentication supports a portLevelMtls map that overrides the workload-wide mode for one named port only. Health checks on 8081 get to stay plaintext-tolerant; the real API on 8080 stays exactly as strict as it was:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: payments-mtls
  namespace: prod
spec:
  selector:
    matchLabels: { app: payments }
  mtls:
    mode: STRICT               # everything else on this workload — mTLS required
  portLevelMtls:
    8081:
      mode: PERMISSIVE          # the health port only — plaintext or mTLS, either is fine

kubectl apply -f peer-auth.yaml
kubectl -n ops  exec ops-healthcheck -- curl -m 5 -s -o /dev/null -w 'ops (8081)     -> %{http_code}\n' http://payments.prod.svc.cluster.local:8081/healthz
kubectl -n ops  exec ops-healthcheck -- curl -m 5 -s -o /dev/null -w 'ops on 8080!   -> %{http_code}\n' http://payments.prod.svc.cluster.local:8080/ ; echo "exit: $?"

The health check comes back 200. The second command — the unmeshed pod trying its luck against the real API port — still fails with the same connection reset as before. That second line is the proof this fix is actually scoped, not a disguised namespace-wide rollback: only the port you explicitly named changed behavior.

Step 5 — prove the mesh alone isn't an authorization boundary

☺ Like you're 10: Everyone with a badge can still open this door — the reader never actually checked whose badge it was, only that it had one.

Before writing anything new, confirm the gap istioctl x describe pod already told you about in Step 3 — (none, all traffic is allowed) — is real and not just a line of output you skimmed past:

kubectl -n prod exec rogue -- curl -s -o /dev/null -w 'rogue -> %{http_code}\n' http://payments.prod.svc.cluster.local:8080/

rogue -> 200

rogue was never given permission to call payments. Nobody wrote a ServiceEntry, a NetworkPolicy, or an AuthorizationPolicy naming it. It succeeds anyway, because STRICT mTLS only ever asked one question — is this connection encrypted and does the caller present a valid mesh certificate? — and rogue, being just another in-mesh workload with its own sidecar and its own SPIFFE identity, answers that question perfectly. Encryption and authentication are not authorization. That's not a bug in what you built in Step 4 — it's the whole reason AuthorizationPolicy exists as a separate resource in the first place.

Step 6 — write and prove a targeted AuthorizationPolicy

☺ Like you're 10: Now write the actual guest list — this exact badge and no other gets through the main door — and don't forget the side window still needs its own separate rule, or you'll accidentally lock the nurse back out.

Write one ALLOW policy with two rules: the real API, restricted to checkout's identity; the health port, left open to anyone, matching what Step 4 already decided about it. Match on principals — the workload's SPIFFE identity, derived from its ServiceAccount — never an IP address, which the mesh doesn't consider trustworthy for this purpose:

# authz.yaml
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: payments-allow-checkout
  namespace: prod
spec:
  selector:
    matchLabels: { app: payments }
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/prod/sa/checkout"]
      to:
        - operation:
            ports: ["8080"]
    - to:                              # no `from` here = open to any caller
        - operation:
            ports: ["8081"]            # keep the health port reachable — see below

kubectl apply -f authz.yaml

kubectl -n prod exec checkout-caller  -- curl -s -o /dev/null -w 'checkout on 8080 -> %{http_code}\n' http://payments.prod.svc.cluster.local:8080/
kubectl -n prod exec rogue            -- curl -s -o /dev/null -w 'rogue    on 8080 -> %{http_code}\n' http://payments.prod.svc.cluster.local:8080/
kubectl -n ops  exec ops-healthcheck  -- curl -m 5 -s -o /dev/null -w 'ops      on 8081 -> %{http_code}\n' http://payments.prod.svc.cluster.local:8081/healthz

checkout on 8080 -> 200
rogue    on 8080 -> 403
ops      on 8081 -> 200

rogue now gets 403 — a real RBAC: access denied refusal from the sidecar, not a network-level timeout. That's the entire drill's second act, proven in one line of curl output.

Clean up

☺ Like you're 10: Take the whole practice hallway down — nothing here needs to keep running once you've seen it work.

kind delete cluster --name mesh-drill

Both acts hold — an unmeshed caller can reach exactly the one port it needs, and exactly one identity can reach the real API? That's the whole drill. For the concepts behind every resource used here at full depth, see Istio and Service Mesh Architecture; for how AuthorizationPolicy sits beside the admission-time policy Kyverno enforces instead, see Policy-as-Code Philosophy. The full exam blueprint this drill's scope comes from is the ICA — the exam. Practice the mesh-and-policy skills again, at capstone scale, in Capstone Part 3 — Mesh & Policy — or step back to Build Your Cert Tracker — Start Here for the full five-part version. Want a different single skill next? Try Drill — Write an Enforcing Kyverno Policy or Drill — Diagnose a Stuck Argo CD Sync.