Hands-On Labs · The Capstone · Part 3 of 5

Capstone Part 3 — Networking & Ingress

This is the third of five parts building one continuous project: a real storefront running on Kubernetes, deployed the hard way so every mechanism is something you typed and watched work, not something a course video skipped past. Part 1 gave the cluster its shape — nodes, namespaces, and the storefront namespace this whole capstone lives in. Part 2 gave it something to run — the web and api Deployments, each with its own ConfigMap and Secret, each already passing its own liveness and readiness probes. Both have been reachable only from inside the cluster this whole time, one kubectl port-forward away from anyone who wanted to look. Today that changes, deliberately and in layers. This page gives the app stable Service identities, a public hostname with a browser-trusted TLS certificate that renews itself, and — the part most tutorials skip — a NetworkPolicy posture that starts from deny-everything and opens exactly four narrow, named holes. By the end, shop.example.com answers over HTTPS, and nothing in the cluster can reach api except web, on purpose.

☺ Explain it like I'm 10

Picture an apartment building that, until today, had no lock on the front door and no way for anyone outside to get in at all. First, every unit gets a proper buzzer with a name on it — press 4B and only 4B answers, no matter which actual tenant is home this week — that's a Service. Then the building gets one real, guarded front door with a working lock, a mailbox that only opens with a key nobody can forge, and a sign out front with the building's real, verified address — that's the Ingress and its certificate. Last, and most important: the building manager changes every door's lock to open for nobody by default, then hands out exactly the keys that make sense — the front door can reach the lobby, the lobby can reach 4B, and nothing else opens for anyone, no exceptions, until someone deliberately cuts a new key.

🐦🐢Your hosts for this part: Pip the Hummingbird & Timmy the Turtle — Pip is the messenger who gets a request to the right Pod no matter how often its address changes, and Timmy refuses to let any of it through until there's a named rule saying it's allowed.
⚠ Where you are arriving from, and where you're headed

Arriving: a storefront namespace with web and api Deployments running healthy Pods, reachable only via kubectl port-forward or a shell inside the cluster — no Service, no public entry point, no NetworkPolicy at all, which on a stock cluster means every Pod can already reach every other Pod. Leaving this page: web and api Services with stable ClusterIPs, a storefront Ingress terminating TLS for shop.example.com off a cert-manager-issued certificate that renews itself, and a default-deny-all NetworkPolicy with exactly four targeted allows layered on top of it. Part 4 picks up exactly here and gives the api Pod somewhere durable to keep its data; Part 5 comes back to this same namespace to lock down who's even allowed to change any of it.

What this part assumes, and what it produces

☺ Like you're 10: The building, the units, and the tenants already exist — today is only about the front door, the mailboxes, and the locks.

This part assumes Part 2's web and api Deployments already exist and are healthy in the storefront namespace — kubectl get pods -n storefront should show every Pod Running with READY 1/1 before you touch anything below. You'll also need ingress-nginx and cert-manager installed as cluster add-ons — both are single Helm installs, covered in full on their own tool pages, and neither is specific to this capstone. Nothing here touches the Deployments, ConfigMaps, or Secrets Part 2 built; every object on this page is new, additive, and namespaced to storefront or cluster-scoped in a way that doesn't collide with anything that already exists. For the concepts underneath everything below — the flat, no-NAT Pod network, why NetworkPolicy is deny-by-selection rather than deny-by-default globally, and the exact Service/Ingress manifests this page builds on — see Services & Networking, the CKA blueprint domain this whole page is really a hands-on rep of.

The world model this part adds

ThingNameIntroduced
The applicationstorefront namespace: web + api DeploymentsPart 1–2
web / api Servicesweb (ClusterIP :80→:8080), api (ClusterIP :8080→:8080)Part 3 — this page
TLS issuer + certificateletsencrypt-prod ClusterIssuer, shop-tls Certificate/SecretPart 3 — this page
Public entrypointstorefront Ingress, class nginx, host shop.example.comPart 3 — this page
Traffic-shape policiesdefault-deny-all, allow-ingress-to-web, allow-web-to-apiPart 3 — this page
Egress-shape policiesallow-dns-egress, allow-api-egress-paymentsPart 3 — this page

Services: giving web and api a name that outlives any one Pod

☺ Like you're 10: Pods get replaced constantly and their addresses change every time — a Service is the one buzzer number that never changes, no matter which actual tenant answers it today.

Neither Deployment has a stable network identity yet — a Pod's IP is assigned when it starts and gone the moment it's rescheduled, which happens routinely. A Service fixes that with one virtual IP and DNS name in front of a changing set of Pods, selected purely by label; membership is tracked automatically in EndpointSlice objects that kube-proxy on every node watches and turns into real packet-forwarding rules, exactly as the API-and-controllers model and Networking & the CNI cover in depth. Both Services here are plain ClusterIP — internal-only is correct today, since nothing outside the cluster should ever reach either one directly; that's the Ingress's job, next section.

apiVersion: v1
kind: Service
metadata:
  name: web
  namespace: storefront
spec:
  selector: { app: web }
  ports:
    - port: 80
      targetPort: 8080

---
apiVersion: v1
kind: Service
metadata:
  name: api
  namespace: storefront
spec:
  selector: { app: api }
  ports:
    - port: 8080
      targetPort: 8080

Apply both, then confirm each has real endpoints — not just a ClusterIP — before moving on: kubectl get endpointslices -n storefront -l kubernetes.io/service-name=web should list the actual Pod IPs behind it. A Service with a valid ClusterIP but zero endpoints is the single most common silent failure in this domain: the object exists, DNS resolves it, and every connection to it just times out, because its selector doesn't actually match any Pod's labels.

⚠ A Service selector typo doesn't error — it just goes quiet

kubectl apply never validates that a Service's selector matches anything real; app: web against Pods labeled app: webapp applies cleanly, resolves in DNS, and answers every connection with a silent timeout instead of a helpful error. Before debugging anything downstream of a Service, always check kubectl get endpointslices -n storefront -l kubernetes.io/service-name=<svc> first — an empty list means the selector, not the network, is the bug.

The public entrypoint: Ingress, and a certificate that renews itself

☺ Like you're 10: The Ingress is the front door with a real lock; cert-manager is the helper who quietly swaps in a fresh ID card for that door every month, weeks before the old one expires, without anyone asking.

With both Services in place, one ClusterIssuer gives the whole cluster a single, shared way to get browser-trusted certificates from Let's Encrypt — create it once, and every namespace can reference it by name, exactly as cert-manager covers in full:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: platform-team@acme.io
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
      - http01:
          ingress:
            ingressClassName: nginx

Next, an explicit Certificate — written out in Git next to the Ingress that consumes its Secret, rather than left to the cert-manager.io/cluster-issuer ingress-shim annotation to create implicitly. Both routes end at the same object; writing it explicitly is what cert-manager's own checkpoint calls out as the better default for anything running in production, because the relationship between hostname, issuer, and Secret stays visible and reviewable in a pull request instead of hiding inside an annotation:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: shop-tls
  namespace: storefront
spec:
  secretName: shop-tls
  duration: 2160h        # 90d — ignored by Let's Encrypt, requested anyway
  renewBefore: 720h       # renew 30d before expiry
  dnsNames:
    - shop.example.com
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer

Finally, the Ingress itself — one object naming both routes and the certificate that protects them:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: storefront
  namespace: storefront
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  ingressClassName: nginx
  tls:
    - hosts: [shop.example.com]
      secretName: shop-tls
  rules:
    - host: shop.example.com
      http:
        paths:
          - path: /api
            pathType: Prefix
            backend:
              service: { name: api, port: { number: 8080 } }
          - path: /
            pathType: Prefix
            backend:
              service: { name: web, port: { number: 80 } }

Apply all three in order — ClusterIssuer, then Certificate, then Ingress — and watch kubectl get certificate -n storefront shop-tls -w until READY flips to True. Behind that flip is the full ACME dance: a CertificateRequest, an Order, and an HTTP-01 Challenge that cert-manager solves through the Ingress it's already watching — the complete chain, and why evaluation can take a minute or two, is diagrammed on the cert-manager page itself; this page only needs the last mile.

shop.example.com ingress-nginx Service (LoadBalancer) ingress-nginx controller terminates TLS here Secret: shop-tls kept current by cert-manager path / path /api web Service :80 api Service :8080 web Pods api Pods
◆ Key idea

Longest-matching-path wins, not YAML order — /api and / above are written with the more specific rule first purely for human readability, not because the Ingress spec cares which line comes first. If / were listed above /api in the same object, a request for /api/orders would still route to the api Service, because /api is the longer matching prefix. See Services & Networking and ingress-nginx for the full matching rules, including what happens when two separate Ingress objects claim the same host.

Default-deny first, then four narrow allows

☺ Like you're 10: The building manager changes every lock to open for nobody, then hands out exactly the keys that make sense — front door to lobby, lobby to 4B, nothing else, no exceptions, until someone deliberately cuts a new key.

Right now, with zero NetworkPolicy objects in storefront, the flat Pod network means every Pod in the cluster can already reach both web and api directly — a Pod in an unrelated namespace, a compromised sidecar three teams over, anything. That's the default Kubernetes ships with, and it's the opposite of what a public-facing app with a real backend should allow. The fix isn't one policy — it's five, applied in a specific order of reasoning even though Kubernetes evaluates them all at once: first, deny everything for every Pod in the namespace; then, name exactly what's still allowed to happen, one narrow rule at a time.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: storefront
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]

An empty podSelector: {} matches every Pod in the namespace; naming both Ingress and Egress in policyTypes with no ingress or egress rules underneath means every Pod here now accepts nothing in and sends nothing out — including, easy to forget, DNS lookups. Apply this alone and both web and api go instantly, completely dark. That's expected, and it's exactly why the four policies below exist.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-ingress-to-web
  namespace: storefront
spec:
  podSelector: { matchLabels: { app: web } }
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: ingress-nginx }
          podSelector:
            matchLabels: { app.kubernetes.io/name: ingress-nginx }
      ports:
        - { port: 8080, protocol: TCP }

---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-web-to-api
  namespace: storefront
spec:
  podSelector: { matchLabels: { app: api } }
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector: { matchLabels: { app: web } }
      ports:
        - { port: 8080, protocol: TCP }

---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: storefront
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: kube-system }
          podSelector:
            matchLabels: { k8s-app: kube-dns }
      ports:
        - { port: 53, protocol: UDP }
        - { port: 53, protocol: TCP }

---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-api-egress-payments
  namespace: storefront
spec:
  podSelector: { matchLabels: { app: api } }
  policyTypes: [Egress]
  egress:
    - to:
        - ipBlock: { cidr: 203.0.113.0/24 }   # payments provider's published range
      ports:
        - { port: 443, protocol: TCP }

Read allow-ingress-to-web's single from entry carefully — namespaceSelector and podSelector sit inside the same list item, which means they combine with AND: only Pods matching both selectors at once. Split them into two separate entries in the from array instead and the meaning changes completely, to OR — any Pod in the ingress-nginx namespace regardless of its own labels, or any Pod anywhere carrying that controller's label. The single-entry, AND'd form here is deliberately the narrower one: only the actual ingress-nginx controller Pod, in its own namespace, gets in.

ingress-nginx namespace controller Pod web Pod allow-ingress-to-web :8080 api Pod allow-web-to-api :8080 CoreDNS kube-system namespace allow-dns-egress :53 Payments API ipBlock 203.0.113.0/24 allow-api-egress-payments :443 stray Pod default ns, or unlabeled in storefront blocked — no rule names this source default-deny-all denies Ingress + Egress for every Pod first — every solid arrow above exists only because a named policy allows it
⚠ ipBlock is an IP, not a promise about who owns it

allow-api-egress-payments allows traffic to 203.0.113.0/24 — not to "the payments provider," even though that's what it means today. NetworkPolicy has no concept of a hostname; if that provider rotates or expands its published IP range and nobody updates the CIDR here, the egress call starts failing with a NetworkPolicy silently in the way, and the failure looks exactly like a downstream outage rather than a stale allow-list. Treat any ipBlock rule against a third party's range as something to revisit on a schedule, the same way cert-manager's page treats a certificate's expiry — a thing that's correct today and silently wrong later unless someone owns re-checking it.

◆ Key idea

NetworkPolicies for the same Pod are always additive, never a stricter intersection — allow-web-to-api and any future policy that also selects app: api both apply at once, and the union of everything any of them allows is what gets through. That's why default-deny-all has to exist before any of this reasoning holds: without it, there's no baseline to add narrow exceptions on top of — every Pod starts open, and a NetworkPolicy can only ever take traffic away from an already-selected Pod, never restrict a Pod nothing has selected yet.

🐦 Pip's-eye view

"I don't carry a single packet myself — I just make sure the address you asked for lands on the Pod that's actually supposed to answer right now, even though that Pod didn't exist five minutes ago. Timmy's rules don't slow me down at all when they're right; they only ever stop the requests that were never supposed to reach that Pod in the first place. The two of us aren't in tension — a Service that finds the right Pod and a NetworkPolicy that guards it are solving completely different problems that happen to sit on the exact same wire."

Proving it: what should work, and what absolutely should not

☺ Like you're 10: Ring the buzzer from outside — it should work. Try walking straight into 4B from the hallway without buzzing anyone — it shouldn't, and now it can't.

Apply everything above in order — Services, then the ClusterIssuer/Certificate/Ingress, then all five NetworkPolicies — and prove both halves independently. First, the app is genuinely reachable:

curl -sv https://shop.example.com/ 2>&1 | grep -E "HTTP/|subject:|issuer:"
# HTTP/2 200
# subject: CN=shop.example.com
# issuer: C=US, O=Let's Encrypt, CN=R3
# -- a REAL issuer, not cert-manager's self-signed fallback cert

curl -s https://shop.example.com/api/healthz
# {"status":"ok"} -- routed through web's Ingress rule to the api Service, over the policy chain

Then prove the negative — the part a passing curl above can't show on its own. From a throwaway Pod with no matching label, in a completely different namespace:

kubectl run stray --rm -it --restart=Never --image=curlimages/curl -n default -- \
  curl -m 3 http://api.storefront.svc.cluster.local:8080/healthz
# curl: (28) Connection timed out after 3001 milliseconds
# -- default-deny-all blocks it; allow-web-to-api only names Pods labeled app: web, and this isn't one

kubectl run stray --rm -it --restart=Never --image=curlimages/curl -n storefront -- \
  curl -m 3 http://web.storefront.svc.cluster.local:80/
# curl: (28) Connection timed out after 3001 milliseconds
# -- same namespace isn't enough either: allow-ingress-to-web only names the ingress-nginx
# controller as a source, and this Pod carries neither its namespace nor its label

Both timeouts are the point, not a bug to chase. The first curl succeeding and both of these failing, together, is what "default-deny with targeted allows" actually means in practice — reachable exactly where it should be, and nowhere else, proven from the outside rather than assumed from reading YAML.

⚠ Confirm your CNI actually enforces NetworkPolicy before trusting any of this

The Kubernetes API accepts a NetworkPolicy object whether or not anything in the cluster enforces it. Calico, Cilium, and most managed cloud CNIs implement the NetworkPolicy API; plain Flannel by default does not — the object saves cleanly, kubectl get networkpolicy lists it, and it silently does nothing. If the "should be blocked" curl above succeeds instead of timing out, this — not a typo in the YAML — is the first thing to check. See Calico and Cilium for the two CNIs that do enforce it, and Networking & the CNI for why the CNI spec leaves enforcement optional in the first place.

Milestones

☺ Like you're 10: Tick each box only once you've actually watched it happen on your own screen, not because the step "sounds right."

Work these in order — each depends on the state left by the one before. Progress saves in this browser.

0 / 11 milestones complete
1Write and apply the web and api Services
Both manifests exactly as shown above, applied to the storefront namespace.
Done when: kubectl get endpointslices -n storefront shows real Pod IPs behind both.
2Create the letsencrypt-prod ClusterIssuer
Apply the ClusterIssuer above, with your own contact email.
Done when: kubectl describe clusterissuer letsencrypt-prod shows condition Ready: True.
3Write the Certificate and the storefront Ingress
Both manifests exactly as shown above, applied in that order.
Done when: kubectl get certificate -n storefront shop-tls shows READY True, SECRET shop-tls.
4Curl the live site over HTTPS
curl -sv https://shop.example.com/, reading the certificate's issuer: line.
Done when: the response is 200 and the issuer is Let's Encrypt, not cert-manager's self-signed fallback.
Concept: this page's proving section
5Apply default-deny-all, and watch the site go dark
Apply just this one NetworkPolicy, nothing else yet.
Done when: milestone 4's same curl now times out completely.
6Apply allow-ingress-to-web
Apply the policy exactly as shown, matching your real ingress-nginx namespace and controller labels.
Done when: the site answers again over HTTPS, while a direct pod-to-pod curl to web from an unrelated Pod still times out.
Concept: this page's AND-vs-OR selector note
7Apply allow-web-to-api
Apply the policy exactly as shown.
Done when: curl https://shop.example.com/api/healthz returns 200 through the full chain, not just directly against the api Pod.
Concept: additive policies, same Pod
8Apply allow-dns-egress
Apply the policy, matching your cluster's real CoreDNS namespace and label if they differ.
Done when: kubectl exec into any storefront Pod and nslookup kubernetes.default resolves.
Concept: CoreDNS
9Apply allow-api-egress-payments
Apply the policy, with the payments provider's real published CIDR in place of the example range.
Done when: a request to the payments provider from inside the api Pod succeeds, while a curl to any other external IP from that same Pod still times out.
Concept: this page's ipBlock caveat
10Prove the negative from a stray Pod, twice
Run both kubectl run stray ... commands from the proving section, unmodified.
Done when: both curls time out, and you can say out loud which specific policy's absence causes each one.
Concept: this page's proving section
11Say out loud what state you're leaving for Part 4
Confirm: Services stable, HTTPS live with a real cert, default-deny plus four named allows all proven independently.
Done when: you can describe this state without looking anything up — it's the exact starting point Part 4 assumes.
🎬 At the Pod Squad
🐦

Pip the Hummingbird: Buzzer's up, lock's on the door, real certificate — shop.example.com is live and it's actually us answering.

🐢

Timmy the Turtle: And nothing else can walk in the back. I checked — twice, from two different directions.

🦫

Benny the Beaver: QA just pinged me — a debug Pod they spun up in default can't reach api at all, not even for a health check.

👺

Gizmo the Gremlin: Easy fix — just delete default-deny-all. Back to everything-reaches-everything, ship it, nobody will notice. 😈

🐢

Timmy: Absolutely not. That debug Pod was never supposed to reach api — that's the policy working, not breaking. If QA needs it, they get their own narrow allow scoped to their own Pod's label, same as everyone else got.

🐿️

Nutty the Squirrel: I'll file that one — allow-qa-debug-to-api, port 8080 only, expires when the debug Pod does.

🦫

Benny: ...that's fair. A five-minute review beats an outage from a hole nobody remembers opening.

🐢 Timmy's checkpoint

1. Why does default-deny-all have to exist before any of the four "allow" policies actually mean anything? 2. In allow-ingress-to-web, why does putting namespaceSelector and podSelector in the same from entry matter, and what would change if they were two separate entries in the array instead? 3. Why does allow-dns-egress target the kube-system namespace with the k8s-app: kube-dns podSelector specifically, rather than a bare namespaceSelector: {} matching every namespace? 4. What's the real risk in using an ipBlock CIDR to allow api's egress to the payments provider, instead of something DNS-based? 5. Walk through, layer by layer, why the second stray-Pod curl in the proving section — from inside storefront itself — still times out. 6. If the "should be blocked" curl from a stray Pod unexpectedly succeeds, what should you check before assuming your YAML has a bug?

Check your answers
  1. Without it, every Pod in the namespace starts fully open by default — a NetworkPolicy can only take traffic away from a Pod some policy has already selected for deny; it can never restrict a Pod that no policy touches. default-deny-all is what makes every later "allow" mean something, by giving them a closed baseline to open narrow exceptions on top of.
  2. Selectors inside the same from list entry combine with AND — only a Pod matching both the namespace and the pod label gets through, which here means only the real ingress-nginx controller Pod. Splitting them into two separate entries changes the meaning to OR: any Pod in that namespace regardless of its labels, or any Pod anywhere carrying that label — a much wider, and much less intentional, hole.
  3. A bare namespaceSelector: {} would allow egress to every Pod in every namespace on port 53, not just to CoreDNS — a far wider hole than "let this Pod resolve DNS" actually requires. Naming the specific namespace and the CoreDNS Pod's own label keeps the allow scoped to exactly the one destination the rule is meant to permit.
  4. NetworkPolicy has no concept of a hostname — an ipBlock rule allows traffic to a fixed CIDR, not to "whoever currently owns that CIDR." If the payments provider rotates or expands its published IP range and the rule isn't updated, egress starts failing with the NetworkPolicy silently in the way, and the failure looks exactly like a downstream outage rather than a stale allow-list needing a refresh.
  5. Being in the same namespace isn't a NetworkPolicy exemption by itself. allow-ingress-to-web only names the ingress-nginx controller (by namespace and label) as an allowed source for web; a stray Pod inside storefront carrying neither that namespace nor that label matches no rule at all, so default-deny-all's baseline still applies to it and the connection times out.
  6. Confirm the CNI actually implements the NetworkPolicy API before trusting the object did anything — the Kubernetes API server accepts and stores a NetworkPolicy whether or not anything enforces it, and a CNI like plain Flannel that doesn't implement the API will let every policy on this page silently do nothing while kubectl get networkpolicy still lists them as if they were active.

Part 3 gave the storefront a real front door, a certificate that renews itself, and a network posture that denies everything until you've said otherwise. Continue to Capstone Part 4 — Storage & Stateful Apps, where api gets somewhere durable to keep its data behind everything built here. Or step back to Build a Cluster — Start Here for how all five parts fit together, revisit Services & Networking and Networking & the CNI for the concepts behind what you just built, cross-check the security framing against Security: Defense in Depth and, cross-course, DevSecOps's Kubernetes security deep dive, or drill this exact failure mode under timed conditions in Drill — Diagnose a Networking Failure.