Hands-On Labs · Guided Drills

Drill — Diagnose a Networking Failure

This drill hands you a working two-service app and then breaks its networking twice, in two genuinely different ways, and asks you to tell them apart using nothing but symptoms and your own kubectl commands. Act 1 looks deceptively small at first: the Service resolves fine in DNS, its EndpointSlice is healthy, and curl still never gets an answer back — it just hangs until it gives up. Act 2 looks like a much bigger fire: nothing resolves at all, as if DNS itself fell over. Neither failure is a broken app, a missing Pod, or a bad Service definition — both are a single NetworkPolicy, written for a defensible reason, missing exactly one line it needed. You'll stand up a real kind cluster with a CNI that actually enforces NetworkPolicy, reproduce both failures on purpose, diagnose each one the way you'd have to under real pressure, and fix it without deleting the security control that caused it. No capstone required — about 25–40 minutes, one focused skill.

☺ Explain it like I'm 10

Imagine two different reasons a phone call to your friend's house might fail. In the first, you dial the exact right number — you know it's right, you've called it a hundred times — and it just rings and rings with nobody ever picking up, forever, even though someone is definitely home. That's not a wrong-number problem; something between you and their phone is silently muting the ring before it ever reaches anyone. In the second, you can't even find the number to dial in the first place — you flip open the phone book and the page won't open at all. Those are two completely different failures that both end with "I couldn't reach my friend," and treating them as the same problem is exactly how you spend twenty minutes fixing the wrong thing. This drill hands you both broken phones, one right after the other, and makes you prove which one you're holding before you touch anything.

🐦🦊Your hosts for this drill: Pip the Hummingbird & Foxy — getting a request from a Service to the right Pod is Pip's entire job, so a request that leaves and never comes back is Pip's problem before it's anyone else's; Foxy refuses to accept "it's probably just flaky" as an explanation for a connection that quietly stopped answering.
⚠ Before you start

You need Docker running, plus kind and kubectl on your machine — nothing else. Every curl, nslookup, and nc below runs from inside a Pod via kubectl exec, so there's nothing extra to install locally. This drill stands up its own disposable cluster, named netdrill and unrelated to any cluster the capstone left running — the two won't collide, and kind delete cluster --name netdrill at the end leaves nothing behind. Kubernetes and CNI behavior does drift between minor versions; if a command below errors, check kubectl version and the Calico manifest version pinned below, and adapt.

How this drill works

☺ Like you're 10: Two broken things, one after another, on purpose — feel each symptom first, then go find its one real cause before you fix anything.

You'll build one small app — checkout calling payments inside a shop namespace — and prove it works before breaking anything, the same discipline any real incident deserves. Then a security-hardening NetworkPolicy lands on payments and Act 1's symptom appears: DNS is fine, the Service has real endpoints, and every request still times out. You'll diagnose it by ruling things out in order rather than guessing, fix it with a narrow allow rule instead of deleting the guardrail, and confirm it. Then a second hardening policy lands on checkout itself, and Act 2's symptom appears: nothing resolves, not even the Service you already fixed. Same method, different layer, same discipline. By the end you'll have a decision tree in your head for reading any "can't reach a Service" symptom and knowing which of four different causes it actually points to — only two of which this drill stages for you.

Stand up the scratch cluster with real NetworkPolicy enforcement

☺ Like you're 10: A network that can't say no can't teach you anything about the network saying no — so the very first thing this cluster needs is a CNI that actually enforces the rule you're about to write.

kind's default built-in CNI, kindnet, gives basic Pod-to-Pod connectivity but enforces zero NetworkPolicy objects — apply the policies below on top of it and nothing would happen at all, silently, which would make this entire drill dishonest. disableDefaultCNI: true skips kindnet on purpose so a real CNI goes in instead, the same move the capstone's own Part 1 makes for the identical reason. One node is enough for a 25–40 minute drill — no ingress controller, no multi-node scheduling story needed here.

# kind-netdrill.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: netdrill

networking:
  disableDefaultCNI: true          # skip kindnet — Calico goes in for real, next step
  podSubnet: "192.168.0.0/16"      # Calico's own default pool

nodes:
  - role: control-plane
$ kind create cluster --config kind-netdrill.yaml
Creating cluster "netdrill" ...
Set kubectl context to "kind-netdrill"

$ kubectl get nodes
NAME                     STATUS     ROLES           AGE
netdrill-control-plane   NotReady   control-plane   22s
# NotReady is correct — no CNI means no usable Pod network yet, nothing to chase here

$ kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml

$ kubectl -n kube-system get pods -l k8s-app=calico-node -w
NAME               READY   STATUS    RESTARTS   AGE
calico-node-9k2pl   1/1     Running   0          41s

$ kubectl get nodes
NAME                     STATUS   ROLES           AGE
netdrill-control-plane   Ready    control-plane   90s

Don't move on until that node reads Ready — the Calico tool guide covers what calico-node is actually doing underneath, but the fact that matters here is simpler: Ready is your proof the CNI is live and every policy from this point on will actually be enforced, not silently ignored.

Deploy checkout and payments, and prove the baseline works before breaking anything

☺ Like you're 10: You can't tell a broken phone call from a working one if you never heard what "working" sounds like.

payments is the backend — an nginx:1.27-alpine Deployment behind a ClusterIP Service, good enough to prove a real response arrived. checkout is the client — a single Pod running curlimages/curl, doing nothing but sleeping, that you'll kubectl exec into for every command in this drill. No NetworkPolicy exists yet, so on a stock cluster every Pod can already reach every other Pod — that default is exactly what the rest of this drill deliberately narrows.

# shop-app.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: shop
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments
  namespace: shop
spec:
  replicas: 2
  selector: { matchLabels: { app: payments } }
  template:
    metadata: { labels: { app: payments } }
    spec:
      containers:
        - name: payments
          image: nginx:1.27-alpine
          ports: [{ containerPort: 80 }]
---
apiVersion: v1
kind: Service
metadata:
  name: payments
  namespace: shop
spec:
  selector: { app: payments }
  ports: [{ port: 80, targetPort: 80 }]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
  namespace: shop
spec:
  replicas: 1
  selector: { matchLabels: { app: checkout } }
  template:
    metadata: { labels: { app: checkout } }
    spec:
      containers:
        - name: checkout
          image: curlimages/curl:8.10.1
          command: ["sleep", "infinity"]
$ kubectl apply -f shop-app.yaml
$ kubectl -n shop rollout status deploy/payments deploy/checkout

$ kubectl exec -n shop deploy/checkout -- nslookup payments.shop.svc.cluster.local
Server:    10.96.0.10
Address:   10.96.0.10:53
Name:      payments.shop.svc.cluster.local
Address:   10.96.211.44

$ kubectl exec -n shop deploy/checkout -- curl -s -m 5 -o /dev/null -w "http_code=%{http_code} time=%{time_total}s\n" http://payments.shop.svc.cluster.local/
http_code=200 time=0.014s

Keep that http_code=200 line in mind — it's the "before" every symptom below is measured against. Nothing gets broken until you've heard it work.

Act 1 — payments resolves fine, and never answers

☺ Like you're 10: The number was right, the phone rang, and somebody was home — but the ring got muted on its way through, and you can't tell that from the dial tone alone.

A security-hardening pass locks down payments. Apply it, and go straight back to checkout without reading ahead — feel the symptom before you know the cause:

# payments-default-deny.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: payments-default-deny
  namespace: shop
spec:
  podSelector: { matchLabels: { app: payments } }
  policyTypes: [Ingress]
  # no `ingress:` list at all = zero allowed rules = every Pod selected here
  # accepts nothing in, from anywhere, full stop
$ kubectl apply -f payments-default-deny.yaml

$ kubectl exec -n shop deploy/checkout -- nslookup payments.shop.svc.cluster.local
Server:    10.96.0.10
Address:   10.96.0.10:53
Name:      payments.shop.svc.cluster.local
Address:   10.96.211.44
# still resolves — this was never a naming problem

$ kubectl exec -n shop deploy/checkout -- curl -s -m 5 -o /dev/null -w "exit=%{exitcode} time=%{time_total}s\n" http://payments.shop.svc.cluster.local/
(hangs for the full five seconds, then:)
curl: (28) Operation timed out after 5000 milliseconds
exit=28 time=5.002s

Work it the way you'd have to for real — rule things out in order, don't jump to the policy just because it's the newest change:

# 1. Endpoints — is anything actually behind the Service at all?
$ kubectl get endpointslices -n shop -l kubernetes.io/service-name=payments -o wide
NAME             ADDRESSTYPE   PORTS   ENDPOINTS
payments-a1b2c   IPv4          80      192.168.107.4,192.168.107.9
# two live endpoints — this isn't the empty-EndpointSlice / selector-mismatch bug either

# 2. NetworkPolicy — what's actually selecting these Pods right now?
$ kubectl get networkpolicy -n shop
NAME                     POD-SELECTOR   AGE
payments-default-deny    app=payments   2m

$ kubectl describe networkpolicy payments-default-deny -n shop
Spec:
  PodSelector:     app=payments
  Allowing ingress traffic:
     (Selected pods are isolated for ingress connectivity)
  Allowing egress traffic:
     (Selected pods are isolated for egress connectivity)

DNS resolved. The endpoint was real. The packet had somewhere correct to go — and never got there. describe confirms it in one line: payments-default-deny selects these Pods for ingress and allows nothing at all in return. Calico's dataplane is dropping the SYN silently, node-side, before it ever reaches the container — no RST, no ICMP unreachable, nothing for curl to report except its own timeout.

Baseline — no NetworkPolicy in the way checkout sends SYN payments Pod SYN delivered, ACK sent back curl: 200 OK ~10ms — instant Act 1 — payments-default-deny, no allow rule (this drill) checkout sends SYN payments Pod SYN silently dropped by Calico curl: exit 28 hangs 5s, then times out A different bug (not staged here) — nothing listening on the port checkout sends SYN payments Pod packet delivered, nothing bound to :80 curl: exit 7 instant "connection refused" The wait is the tell — a silent drop hangs until your own timeout; a refusal is instant and loud.
◆ Key idea

curl's own exit code is a free diagnosis, if you bother to read it. Exit 28 ("Operation timed out") means the client waited its full timeout and heard nothing back at all — a packet delivered into a void, the exact signature of a NetworkPolicy silently dropping a SYN. Exit 7 ("Failed to connect... Connection refused") means a response came back fast, just a negative one — the kernel on the far end telling you plainly that nothing is listening on that port. Learn to read that number before you open a single log; it tells you which half of the decision tree below you're even in.

Fix it with a narrow allow rule — don't touch, don't delete, payments-default-deny:

# allow-checkout-to-payments.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-checkout-to-payments
  namespace: shop
spec:
  podSelector: { matchLabels: { app: payments } }
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector: { matchLabels: { app: checkout } }
      ports:
        - { port: 80, protocol: TCP }
$ kubectl apply -f allow-checkout-to-payments.yaml

$ kubectl exec -n shop deploy/checkout -- curl -s -m 5 -o /dev/null -w "http_code=%{http_code} time=%{time_total}s\n" http://payments.shop.svc.cluster.local/
http_code=200 time=0.011s
◆ Key idea

NetworkPolicy rules are additive across every policy selecting a given Pod — there's no "deny wins" precedence to reason about. A Pod's effective ingress is the union of every ingress rule from every policy that selects it, which is exactly why adding allow-checkout-to-payments fixed this without deleting payments-default-deny at all: the deny policy still contributes zero rules of its own, and the new policy's one rule is now in the union. DevSecOps's Kubernetes security deep dive states this rule formally — worth knowing cold, because "just delete the policy that's in the way" is the single most common wrong instinct under pressure, and it's also the one Gizmo reaches for below.

Act 2 — a second hardening pass, and now nothing resolves at all

☺ Like you're 10: This time the phone book itself won't open — and that's a much bigger, scarier-looking symptom for what turns out to be the exact same kind of mistake.

A week later, a different engineer locks down checkout's outbound traffic — a defensible move, since nothing about a checkout client should be reaching arbitrary destinations. Apply it, and try the fixed payments call again first:

# checkout-default-deny-egress.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: checkout-default-deny-egress
  namespace: shop
spec:
  podSelector: { matchLabels: { app: checkout } }
  policyTypes: [Egress]
  # no `egress:` list = zero allowed rules = checkout can now reach nothing
  # at all outbound, DNS included
$ kubectl apply -f checkout-default-deny-egress.yaml

$ kubectl exec -n shop deploy/checkout -- timeout 5 nslookup payments.shop.svc.cluster.local
;; connection timed out; no servers could be reached
command terminated with exit code 1

Everything looks broken now — not just payments, any name. Resist the urge to assume CoreDNS itself died; work outward the same disciplined way this course's own troubleshooting methodology argues for, ring by ring:

# 1. Is resolv.conf even still correct?
$ kubectl exec -n shop deploy/checkout -- cat /etc/resolv.conf
nameserver 10.96.0.10
search shop.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
# unchanged, and pointed at the right ClusterIP — this was never a resolv.conf problem

# 2. Can checkout reach CoreDNS at all, at the raw packet level?
$ CLUSTER_DNS_IP=$(kubectl get svc -n kube-system kube-dns -o jsonpath='{.spec.clusterIP}')
$ kubectl exec -n shop deploy/checkout -- nc -zvw2 "$CLUSTER_DNS_IP" 53
nc: 10.96.0.10 (10.96.0.10:53): Operation timed out
# a raw TCP probe straight at CoreDNS's own IP never gets there either — this
# isn't "the DNS server is unhealthy," it's "checkout can't send packets out at all"

# 3. What changed on checkout's own NetworkPolicy?
$ kubectl get networkpolicy -n shop
NAME                            POD-SELECTOR   AGE
payments-default-deny           app=payments   14m
allow-checkout-to-payments      app=payments    9m
checkout-default-deny-egress    app=checkout    38s

$ kubectl describe networkpolicy checkout-default-deny-egress -n shop
Spec:
  PodSelector:    app=checkout
  Allowing egress traffic:
     (Selected pods are isolated for egress connectivity)

resolv.conf was fine. The probe proved packets weren't leaving checkout at all, in any direction — not "DNS is down," but "everything outbound from this one Pod is now blocked." describe confirms it: checkout-default-deny-egress selects checkout for egress and allows nothing out, port 53 to CoreDNS included. This is the exact trap Services & Networking calls the single most common self-inflicted NetworkPolicy outage: an egress default-deny with no explicit rule carved out for DNS.

Fix it with one policy covering both destinations checkout legitimately needs — DNS, and payments itself, since the blanket deny above also re-broke the traffic Act 1 already fixed:

# allow-checkout-egress.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-checkout-egress
  namespace: shop
spec:
  podSelector: { matchLabels: { app: checkout } }
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: kube-system }
          podSelector:
            matchLabels: { k8s-app: kube-dns }
      ports:
        - { port: 53, protocol: UDP }
        - { port: 53, protocol: TCP }
    - to:
        - podSelector: { matchLabels: { app: payments } }
      ports:
        - { port: 80, protocol: TCP }
$ kubectl apply -f allow-checkout-egress.yaml

$ kubectl exec -n shop deploy/checkout -- nslookup payments.shop.svc.cluster.local
Server:    10.96.0.10
Address:   10.96.0.10:53
Name:      payments.shop.svc.cluster.local
Address:   10.96.211.44

$ kubectl exec -n shop deploy/checkout -- curl -s -m 5 -o /dev/null -w "http_code=%{http_code} time=%{time_total}s\n" http://payments.shop.svc.cluster.local/
http_code=200 time=0.013s

Both symptoms are gone, and every policy from both acts is still in place — three deny-shaped or scoping policies plus two narrow allows, exactly the layered posture the capstone's own Part 3 builds out at full scale for a real storefront.

The general method: let the symptom tell you which layer to check

☺ Like you're 10: You don't have to guess which of four different bugs you're looking at — three quick checks narrow it down every time, before you've fixed anything.

This drill only staged two of four common "can't reach a Service" root causes on purpose. The other two are just as real, and this platform covers each of them elsewhere — the tree below is the same reasoning you just did, generalized so it works on a bug you've never seen before:

Can't get a response from a Service — read the symptom before touching anything Does DNS resolve? nslookup / dig from inside a Pod no DNS ITSELF IS BROKEN check the egress NetworkPolicy for port 53 to CoreDNS — Act 2, this drill yes Does the Service have live Endpoints? kubectl get endpointslices -l kubernetes.io/service-name=<svc> no SELECTOR NEVER MATCHED Service selector doesn't match Pod labels — see Services & Networking yes Hangs, or refuses instantly? read curl's own exit code — 28, or 7 hangs / times out SILENT DROP ingress NetworkPolicy on the backend has no allow rule — Act 1, this drill refuses instantly NOTHING IS LISTENING targetPort points at a port no process has bound — see kube-proxy forwarding

Blue leaves are exactly what this drill just staged and fixed. The two gray leaves are real, common bugs this drill deliberately left alone: an empty EndpointSlice from a Service selector that doesn't match its Pods' labels is covered in Services & Networking's own exam-traps section, and the mechanics of how a Service's targetPort becomes a real forwarding rule — and what happens when nothing's bound to it — are covered in Networking & the CNI's section on kube-proxy.

🐦 Pip's-eye view

"I don't carry a single packet myself, and I never claimed to — I just make sure the name you asked for lands on the right Pod. Both times today, I did my job correctly: the name resolved, or it didn't, exactly the way the DNS record and the CoreDNS Service said it should. What actually stopped the traffic was never mine to control. If you come looking for me first every time a request goes quiet, you'll be right maybe a quarter of the time — the other three leaves on that tree belong to somebody else entirely."

✎ Try it — one more rep before you tear down

From a clean baseline (delete every NetworkPolicy in shop, confirm curl and nslookup both work again), apply payments-default-deny and checkout-default-deny-egress at the same time, in one kubectl apply. Before you run a single diagnostic command, predict out loud which symptom you'll see first and why. Then check yourself — the answer isn't obvious until you've actually reasoned through which one blocks the other's evidence from ever reaching you.

Clean up

☺ Like you're 10: This whole cluster only ever existed to break twice, on purpose — nothing about it should outlive the lesson.

$ kind delete cluster --name netdrill
Deleting cluster "netdrill" ...

$ docker ps -a --filter "name=netdrill"
# empty — nothing left behind
0 / 10 steps complete
1Stand up the netdrill kind cluster with Calico and confirm real enforcement
Done when: kubectl get nodes shows Ready and the calico-node Pod is Running.
2Deploy checkout and payments, and prove the baseline works
Done when: a curl from checkout to payments returns http_code=200 before any NetworkPolicy exists.
3Apply payments-default-deny and reproduce "resolves but never answers"
Done when: nslookup still succeeds but curl to payments times out with exit code 28.
4Diagnose Act 1 by ruling out DNS and endpoints before reading the policy
Done when: you can name payments-default-deny as the cause before opening the reveal.
5Fix Act 1 with a scoped ingress allow rule, without deleting the deny policy
Done when: curl from checkout to payments succeeds again with both policies still applied.
6Apply checkout-default-deny-egress and reproduce total DNS failure
Done when: nslookup itself times out and a direct nc to CoreDNS's port 53 also fails.
7Diagnose Act 2 by reading checkout's own NetworkPolicy
Done when: you can name the exact missing egress rule before opening the reveal.
8Fix Act 2 with a DNS-plus-payments egress allow rule
Done when: nslookup and curl from checkout both succeed again, all five policies still applied.
9Explain why Act 1 hung instead of refusing instantly
Done when: you can state the difference between curl exit 28 and exit 7 without checking the callout.
10Tear down the cluster
Done when: kind delete cluster --name netdrill leaves no netdrill containers in docker ps.
🎬 At the Pod Squad
🦊

Foxy: checkout can't reach payments. DNS is fine, the EndpointSlice is fine, and curl just... never comes back.

🐦

Pip the Hummingbird: That rules out a name problem and an empty-Service problem in one shot. Something's swallowing the packet after it's already been correctly delivered.

🦫

Benny the Beaver: That'd be me — I put payments-default-deny on during last week's hardening pass. Guess I only wrote half of it.

👺

Gizmo: Easy fix — just kubectl delete that policy, ship it, nobody has to know. 😈

🐢

Timmy the Turtle: No. Benny wrote that deny rule for a real reason, and the reason didn't stop being true. Add the allow next to it — don't tear down the guardrail to get past it.

🐦

Pip the Hummingbird: Added — checkout to payments, port 80, named and scoped. And while I was in there: checkout's own egress policy from yesterday is missing its DNS rule too. Fixing both before this pages someone at 2 a.m.

🐢 Timmy's checkpoint

1. payments resolved fine in DNS and had a healthy EndpointSlice, yet curl to it timed out instead of refusing instantly. What two causes does that specific symptom rule out, and what does it point to instead? 2. Why did Timmy refuse to just delete payments-default-deny, and what rule about multiple NetworkPolicy objects selecting the same Pod makes the actual fix work? 3. In Act 2, nslookup itself hung. Name the two things the diagnosis ruled out before reaching the real cause, and the one command that proved it directly. 4. curl exit code 28 and exit code 7 both mean "the request failed," but they point at two different bugs. What's the difference, and which bug produces which code? 5. This drill only reproduced two of the four leaves on the general decision tree. Name the other two failure causes it covers, and which existing page on this platform explains each one.

Check your answers
  1. It rules out a DNS/naming problem (the name resolved correctly) and a Service-selector problem (the endpoint was real and live). It points to something blocking the packet after it had a correct destination to travel to — in this case, an ingress NetworkPolicy on the backend with no matching allow rule, silently dropping the SYN.
  2. Because the deny policy was written for a real reason that didn't stop being true just because it was inconvenient right now. NetworkPolicy rules are additive across every policy selecting a Pod — a Pod's effective ingress is the union of every rule from every policy that selects it — so adding a second, narrower allow policy works without deleting or weakening the first one at all.
  3. The diagnosis ruled out resolv.conf being wrong (it was unchanged and correctly pointed at CoreDNS's ClusterIP) and CoreDNS itself being unhealthy (a raw nc probe at CoreDNS's own IP on port 53 also timed out, proving the block was on checkout's outbound side, not the DNS server). kubectl describe networkpolicy checkout-default-deny-egress -n shop proved the cause directly, showing zero allowed egress rules.
  4. Exit 28 ("Operation timed out") means the client waited its full timeout and got no response at all — a packet silently dropped, the signature of a NetworkPolicy blocking a SYN with no explicit refusal. Exit 7 ("Connection refused") means a fast, explicit negative response came back — the signature of a packet reaching its destination but finding nothing listening on that port. Act 1's bug produces exit 28; a targetPort/no-listener bug produces exit 7.
  5. An empty EndpointSlice from a Service selector that doesn't match its Pods' labels — covered in Services & Networking's exam-traps section — and a targetPort pointing at a port nothing is actually bound to, whose mechanics are covered in Networking & the CNI's kube-proxy section.

Both bugs in this drill are variations on one idea: a NetworkPolicy written for a real reason, missing one line it needed, and a symptom that looks nothing like "check the NetworkPolicy" until you've ruled out everything else first. For the concepts underneath both acts — the flat Pod network, how kube-proxy turns a Service into real rules, and CoreDNS's role in all of it — see Services & Networking and Networking & the CNI; for the general ring-by-ring method this drill borrowed for Act 2, see a troubleshooting methodology. The capstone's own Part 3 builds this same default-deny-plus-narrow-allows posture at full scale for a real storefront if you want the fuller build; DevSecOps's Kubernetes security deep dive covers the additive-policy rule and this exact DNS-egress trap from the security side. Ready for a different single skill? Try Drill — Debug a Stuck Pod or Drill — Harden an RBAC Configuration, or step back to Build a Cluster — Start Here for the full continuity version.