Capstone Part 3 — Mesh & Policy

This is the third of five parts building one continuous project: mission-api, the small tracker this capstone has been standing up since Part 1 gave it a GitOps pipeline and Part 2 gave it a canary. Two gaps have been sitting there quietly since Part 1, both easy to miss because nothing about them looked broken. mission-api has always called a second service, catalog-api, to fetch the canonical 16-exam ladder data — in plain HTTP, with no identity check on either end, because "it's inside the cluster" was treated as trust enough. And Part 2's canary weight was never actually a percentage of traffic; it was an approximation, the ratio of canary pods to stable pods, exactly as warned about on this course's own Argo Rollouts page. Today closes both gaps, and opens a third one on purpose: nothing has ever checked whether an image running in this namespace was actually built by this project's own pipeline. By the end of this page, Istio is carrying real weighted traffic and real mutual TLS, and Kyverno is standing at the door refusing anything unsigned.

What this part assumes, and what it adds

☺ Like you're 10: The two offices, the note-passing habit, and the loading dock with no inspector yet — nothing new to install until this page says so.

This part assumes Part 1's and Part 2's state is healthy: a Kubernetes cluster with an Argo CD install reconciling a manifests repository into the mission namespace, mission-api running as an Argo Rollouts Rollout behind mission-api-stable and mission-api-canary Services, and catalog-api running as a plain Deployment and Service — it never needed a canary of its own, because it only ever serves the same static 16-exam ladder data back out. You need kubectl, helm, and istioctl on your machine (the Istio page covers the install paths), the kyverno CLI from the Kyverno page, and cosign from Cosign & Sigstore. Nothing here replaces anything Part 1 or Part 2 built — every resource below is either new or a small, additive patch to an existing one.

Installing Istio and enrolling the mission namespace

☺ Like you're 10: Put the tube system's central hub in place first, then connect just the two offices that need it — everyone else on the station keeps passing notes by hand for now.

Install the control plane with the default profile — no ambient mode needed for two services this small, and sidecar mode is what the rest of this page assumes. Run the precheck first, the same habit Istio insists on before anything else:

istioctl x precheck
istioctl install --set profile=default -y

kubectl label namespace mission istio-injection=enabled
kubectl rollout restart deployment catalog-api -n mission
kubectl argo rollouts restart mission-api -n mission   # Rollouts' own restart — see below

Two things about that last line matter. Sidecar injection only ever happens at pod creation — a plain kubectl rollout restart works fine for catalog-api's ordinary Deployment, but mission-api is an Argo Rollouts Rollout, and restarting it through kubectl directly can race the controller's own view of the object. kubectl argo rollouts restart is the safe equivalent for a Rollout specifically — it bumps the same restart annotation Rollouts already understands, rather than fighting the controller for ownership of the pod template. Confirm the sidecar actually landed on both workloads before touching anything else:

kubectl get pods -n mission -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].name}{"\n"}{end}'
# mission-api-7d9f...    mission-api  istio-proxy
# catalog-api-6b4c...    catalog-api  istio-proxy

Two containers per pod, not one, is the only proof that matters here — a namespace label with no restart behind it leaves every existing pod exactly as unmeshed as it was five minutes ago.

From an approximation to an exact number: wiring the Rollout into a real VirtualService

☺ Like you're 10: Until today, "ten percent of visitors" really meant "one out of every ten rooms" — close, but not the same thing as actually counting visitors. Now something counts.

Part 2 already named mission-api-stable and mission-api-canary in the Rollout's strategy.canary block, which is exactly what a real traffic router needs to route between — nothing about those two Services changes today. What's been missing is the router itself, and the VirtualService it reads. Add it as a new file in the same manifests repository Argo CD already reconciles:

## mesh/mission-api-vsvc.yaml — new in Part 3
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: mission-api-vsvc
  namespace: mission
spec:
  hosts: ["mission-api"]
  http:
    - name: primary            # this name is what the Rollout's routes: [ primary ] points at
      route:
        - destination: { host: mission-api-stable }
          weight: 100
        - destination: { host: mission-api-canary }
          weight: 0

Then patch the one field Part 2's Rollout was always missing — trafficRouting, sitting right beside the canaryService/stableService pair that was already there:

## mission-api-rollout.yaml — the one block added to Part 2's existing strategy.canary
spec:
  strategy:
    canary:
      canaryService: mission-api-canary   # unchanged since Part 2
      stableService: mission-api-stable   # unchanged since Part 2
      trafficRouting:                     # NEW — this is the whole fix
        istio:
          virtualService:
            name: mission-api-vsvc
            routes: [ primary ]           # must match the http[].name above, exactly

Commit both files and let Argo CD sync them in — no kubectl apply by hand, the same discipline Part 1 established for everything in this namespace. Once it's synced, run one canary step and read the weight from the one place that can't be fooled by a pod count: the proxy itself.

kubectl argo rollouts set image mission-api mission-api=registry.kubestronaut.dev/mission-api:1.5.0 -n mission
kubectl argo rollouts get rollout mission-api -n mission --watch
# ...pauses at the first step, setWeight: 10

istioctl proxy-config route "$(kubectl get pod -n mission -l app=mission-api,version=canary -o name | head -1 | cut -d/ -f2)" \
  --name 80 -o json | jq '.[0].routeConfig.virtualHosts[0].routes[0].route.weightedClusters'
# canary cluster: weight 10, stable cluster: weight 90 — an exact Envoy weight, not a pod-count guess

That's the whole point of today's traffic-routing work, proven rather than asserted: before this page, a 10-replica rollout with one canary pod really was serving close to 10% by accident; a 3-replica rollout with one canary pod was actually running at 33% while the Rollout's own status still claimed 10. Every step from here forward is the number Envoy was told to enforce, not the number pod scheduling happened to produce.

Locking down mission-api ↔ catalog-api: mTLS, then who's allowed to call at all

☺ Like you're 10: First make sure both offices' notes are sealed so nobody outside can forge or read one; only after that, decide which office is even allowed to send notes to the other.

Roll out mTLS the safe way Istio's own gotchas insist on: PERMISSIVE first, so meshed and unmeshed traffic both keep working while you confirm the mesh is actually carrying encrypted traffic, then flip to STRICT once you've checked.

## mesh/mission-peer-auth.yaml
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: mission
spec:
  mtls:
    mode: PERMISSIVE   # step one — flip to STRICT once istioctl confirms real mTLS below

istioctl x describe pod "$(kubectl get pod -n mission -l app=catalog-api -o name | head -1 | cut -d/ -f2)" -n mission
# ...
# Effective PeerAuthentication:
#    Mode: PERMISSIVE
# ...calls FROM mission-api are already mTLS — confirmed before ever going STRICT

Once that's confirmed, change one field and commit it — Argo CD applies the flip on its next sync:

spec:
  mtls:
    mode: STRICT   # step two — every caller now needs a valid mesh certificate, full stop

STRICT mTLS answers "is this caller who it claims to be?" — it says nothing yet about "should this caller be allowed to do this." catalog-api currently accepts a call from anything inside the mesh with a valid certificate, which today means only mission-api — but nothing stops a third service added in a later part from calling it too, silently, the moment it's meshed. Close that with an explicit deny-everyone-else baseline plus one narrow allow, the same two-policy pattern Istio covers in full:

## mesh/catalog-api-authz.yaml
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: catalog-api-deny-all
  namespace: mission
spec:
  selector:
    matchLabels: { app: catalog-api }
  action: ALLOW
  rules: []          # ALLOW with zero rules => nothing is permitted — the baseline
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: catalog-api-allow-mission-api
  namespace: mission
spec:
  selector:
    matchLabels: { app: catalog-api }
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/mission/sa/mission-api"]   # identity, not IP — survives every pod restart
      to:
        - operation:
            methods: ["GET"]
            paths: ["/exams*"]

Prove the deny side first, before trusting the allow side at all — a policy that "does nothing" because the selector is wrong is the single most common mistake here:

kubectl run probe -n mission --rm -it --image=curlimages/curl --restart=Never \
  --overrides='{"spec":{"serviceAccountName":"default"}}' \
  -- curl -sS -o /dev/null -w '%{http_code}\n' http://catalog-api/exams
# 403 — RBAC: access denied. "default" is not sa/mission-api, and that's the point.

kubectl exec -n mission "$(kubectl get pod -n mission -l app=mission-api,version=stable -o name | head -1 | cut -d/ -f2)" \
  -c mission-api -- curl -sS -o /dev/null -w '%{http_code}\n' http://catalog-api/exams
# 200 — the one identity the policy names gets through

Kyverno: no signed image, no pod

☺ Like you're 10: The loading dock's first real inspector — every crate gets its seal checked before it's wheeled in, no exceptions for "it looked fine to me."

Everything above governs traffic between two services that are already running. It says nothing about how they got there in the first place — since Part 1, Argo CD has faithfully deployed whatever image tag a manifest named, signed or not, built by this project's own pipeline or not. Install Kyverno and close that gap the same way Kyverno covers in depth:

helm repo add kyverno https://kyverno.github.io/kyverno/ && helm repo update
helm install kyverno kyverno/kyverno -n kyverno --create-namespace \
  --set admissionController.replicas=2

## policy/require-signed-mission-images.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-signed-mission-images
spec:
  rules:
    - name: require-keyless-signature
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaces: [mission]
      verifyImages:
        - imageReferences: ["registry.kubestronaut.dev/*"]   # ours only — never verify public bases this way
          mutateDigest: true               # rewrite tag -> digest once verified, closes the re-tag race
          required: true
          attestors:
            - count: 1
              entries:
                - keyless:
                    subject: "https://github.com/kubestronaut/*/.github/workflows/build-and-sign.yml@refs/heads/main"
                    issuer: "https://token.actions.githubusercontent.com"
                    rekor: { url: https://rekor.sigstore.dev }

The subject pattern is deliberately scoped: it matches the build-and-sign.yml workflow, run from the main branch, in any repository under the kubestronaut GitHub organization — wide enough to cover both mission-api and catalog-api's own repositories, narrow enough that nobody else's signature, anywhere, satisfies it. Commit it through Git, the same as everything else on this page. Add one step to the build workflow Part 1 already wired up, so every image this project ships actually gets signed:

# .github/workflows/build-and-sign.yml — one step added, after the image push
      - name: Sign the image (keyless)
        run: |
          cosign sign --yes "registry.kubestronaut.dev/mission-api@${{ steps.push.outputs.digest }}"

Prove the gate refuses what it's supposed to before trusting it with anything real — push an unsigned image and watch the admission fail with a message that names exactly what's wrong:

kubectl run unsigned-test -n mission --image=registry.kubestronaut.dev/mission-api:untested --restart=Never
# Error from server: admission webhook "validate.kyverno.svc-fail" denied the request:
# policy require-signed-mission-images/require-keyless-signature fail:
#   failed to verify signature: no signatures found for registry.kubestronaut.dev/mission-api:untested
kubectl delete pod unsigned-test -n mission --ignore-not-found

# now sign it for real, the same command the CI step above runs
cosign sign --yes registry.kubestronaut.dev/mission-api:untested
kubectl run signed-test -n mission --image=registry.kubestronaut.dev/mission-api:untested --restart=Never
kubectl get pod signed-test -n mission -o jsonpath='{.spec.containers[0].image}{"\n"}'
# registry.kubestronaut.dev/mission-api@sha256:...  — the tag is gone; mutateDigest pinned it closed
kubectl delete pod signed-test -n mission

That digest in the last line is the proof, not the log message from the first attempt: Kyverno didn't just check a box next to the tag you asked for, it rewrote the reference to the exact bytes it verified, which is what stops someone re-pointing :untested at a different image after the check ran.

What "done" looks like for Part 3, and where Part 4 picks up

☺ Like you're 10: Sealed tubes between the two offices, a real inspector at the loading dock — proven with your own commands, not just applied and hoped for.

At the end of this part: Istio is installed and the mission namespace is enrolled for sidecar injection on both workloads; mission-api-vsvc exists and the Rollout's trafficRouting.istio block reads its weights from real Envoy configuration instead of a pod-count approximation; mission-api and catalog-api speak STRICT mutual TLS, and an AuthorizationPolicy means only mission-api's own ServiceAccount may ever call catalog-api; and no pod in this namespace runs an image that wasn't signed by this project's own pipeline, proven by watching Kyverno reject one and admit another. Nothing from Part 1 or Part 2 was thrown away:

Milestones

☺ Like you're 10: Tick each box only once you've watched it happen on your own screen — a command that "should" work isn't the same as one you've seen pass.

Work these in order — each depends on the mesh and policy state from the one before. Progress saves in this browser.

Part 3 gave this capstone a mesh and a policy engine that actually enforce something — a canary weight Envoy honors exactly, mutual TLS between the only two services that exist so far, an authorization policy naming exactly one allowed caller, and an image gate that refuses anything this project didn't sign itself. Continue to Capstone Part 4 — Observability, where the mesh's own signals join everything else this capstone emits. Or step back to Build Your Cert Tracker — Start Here for how all five parts fit together, revisit Service Mesh Architecture and Policy-as-Code Philosophy for the concepts behind what you just built, or get a faster, standalone rep of each half in Drill — Lock Down a Mesh Namespace and Drill — Write an Enforcing Kyverno Policy.