Triage — Networking, RBAC & Admission

Some failures are not inside a workload at all. The pods are Running, the logs are clean, the image is right — and yet the request never arrives, or the request is rejected before it can arrive. This page collects those failures into two families. The first is the network path: a Service that points at nothing, a name that will not resolve, a NetworkPolicy dropping packets in total silence, a proxy answering 404 or 503, a mesh refusing an unmeshed client. The second is the pair of guards that sit in front of the API server itself: RBAC, which decides whether a subject may perform a verb, and admission, which decides whether an object is even the right shape. Both produce dense one-line errors that, read carefully, contain the entire diagnosis. This is one branch of the triage hub — read it after the universal order, and drill it until each first command is automatic.

The universal triage order, in brief

☺ Like you’re 10: Before you guess what’s wrong, always look in the same places, in the same order. Guessing first is how you waste ten minutes.

Everything below assumes you have already run the standard opener from the triage hub. The single biggest time sink under exam pressure is forming a hypothesis before gathering evidence, so the discipline is: do not think, run the sequence. In short form it is get → describe → events → logs → controller status, and only then a hypothesis.

Networking failures

☺ Like you’re 10: When one app can’t talk to another, walk the path in order: does the address exist? does the name resolve? is a firewall in the way? is the door number right?

Network debugging is a path walk, and the path has four gates: endpoints (does the Service point at any pods?), DNS (does the name resolve?), policy (is a NetworkPolicy dropping it?), and ports (is the traffic arriving at the right door?). Walk them in that order and you will not get lost. The concepts behind each gate are in Platform Networking.

The classic: a Service with no endpoints

If you learn one networking check, learn this one. A Service is just a label selector plus a port mapping; if the selector matches nothing, the Service exists, resolves in DNS, and blackholes every connection. It is the most commonly planted networking fault anywhere, because it is invisible unless you look for it specifically.

kubectl get endpointslices -n app -l kubernetes.io/service-name=api
kubectl get endpoints -n app api          # older, shorter output: <none> means broken

# Compare the two sides directly — this is the whole diagnosis:
kubectl get svc api -n app -o jsonpath='{.spec.selector}{"\n"}'
#   {"app":"api"}
kubectl get pods -n app --show-labels | head
#   api-7c9...  Running  app=api-server,tier=backend      <-- mismatch!

# Prove it by selecting with the Service's own selector:
kubectl get pods -n app -l app=api          # "No resources found" == confirmed

# Fix one side or the other (fixing the Service is usually safer than relabelling pods):
kubectl patch svc api -n app -p '{"spec":{"selector":{"app":"api-server"}}}'

DNS resolution failures

Cluster DNS follows the pattern <service>.<namespace>.svc.cluster.local. Inside the same namespace the short name works; across namespaces it does not, which produces a great many “it worked in dev” failures. The default ndots:5 setting means short names get tried against each search-domain suffix first, so a genuinely external name like api.example.com (two dots, fewer than five) also goes through the search list before being tried as an absolute name.

# Get a shell with DNS tools; the pod's own image often has none.
kubectl run dnsutils --rm -it --restart=Never --image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.3 -- bash

nslookup api                      # same namespace only
nslookup api.payments             # cross-namespace
nslookup api.payments.svc.cluster.local
cat /etc/resolv.conf
#   nameserver 10.96.0.10
#   search app.svc.cluster.local svc.cluster.local cluster.local
#   options ndots:5

# Is CoreDNS itself healthy?
kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50
kubectl get svc -n kube-system kube-dns
kubectl get cm coredns -n kube-system -o yaml    # check the Corefile for a bad forward/rewrite

☺ Like you’re 10: Inside your own room you can shout “Sam!” and Sam hears you. From another room you have to shout “Sam in the kitchen!” — service.namespace. Forgetting the second half is one of the most common mistakes there is.

NetworkPolicy silently dropping traffic

NetworkPolicies fail silently — dropped packets look exactly like a hung service, with no log and no event anywhere. The rule to remember: policies are additive allow-lists, and the moment any policy selects a pod, that pod becomes default-deny for the direction(s) that policy names. A frequent planted bug is a default-deny egress policy that forgets to allow DNS, which breaks everything in a confusing, name-resolution-shaped way.

kubectl get networkpolicy -A
kubectl describe netpol default-deny -n app

# Which policies select this pod? (match podSelector against the pod's labels)
kubectl get pod -n app api-1 --show-labels
kubectl get netpol -n app -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podSelector}{"\t"}{.spec.policyTypes}{"\n"}{end}'

# The allow rule people forget: DNS egress. Without it, a default-deny-egress
# namespace cannot resolve ANY name, and every symptom looks like broken DNS.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: app
spec:
  podSelector: {}                 # every pod in the namespace
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }

Ingress and Gateway: 404 vs 503

The status code partitions the problem cleanly. 404 means the proxy could not match the request to a backend at all — the Host header does not match a rule, the path does not match, or the Ingress was never claimed by a controller because ingressClassName is missing or wrong. 503 means the routing matched but the backend has no healthy endpoints — so it is really a workload or endpoints problem wearing an HTTP costume. A third code is worth knowing: 502 means the proxy did reach an endpoint and the connection was refused or reset — endpoints exist, but nothing is listening on the port they name, which is the targetPort fault in the last section.

kubectl describe ingress web -n app
kubectl get ingress -n app -o custom-columns=NAME:.metadata.name,CLASS:.spec.ingressClassName,HOSTS:.spec.rules[*].host,ADDRESS:.status.loadBalancer.ingress[*].ip
kubectl get ingressclass                        # does the named class exist?
kubectl logs -n ingress-nginx -l app.kubernetes.io/component=controller --tail=50

# Reproduce with an explicit Host header — most 404s are a Host mismatch:
curl -sv -H 'Host: shop.example.com' http://<ingress-ip>/checkout

# Gateway API equivalents:
kubectl get gateway,httproute -A
kubectl describe httproute shop -n app          # status.parents[].conditions says Accepted/ResolvedRefs

Service mesh mTLS failures

A mesh with PeerAuthentication set to STRICT requires every client to present a workload certificate. Any client without a sidecar — a pod in a namespace missing the injection label, a bare Pod, a CronJob-spawned Job — is rejected at the proxy, producing errors like upstream connect error or disconnect/reset before headers. This is a favourite exam trap because everything looks healthy: pods Ready, endpoints present, DNS fine.

# Istio
kubectl get namespace -L istio-injection            # is the label there?
kubectl get pod -n app -o custom-columns=NAME:.metadata.name,CONTAINERS:.spec.containers[*].name
#   look for the istio-proxy sidecar; 1/1 instead of 2/2 means no sidecar
istioctl analyze -n app
istioctl proxy-status
istioctl proxy-config cluster deploy/api -n app
kubectl get peerauthentication -A
kubectl get authorizationpolicy -A

# Linkerd
linkerd check --proxy
linkerd viz stat deploy -n app                      # success rate + RPS per workload
kubectl get pod -n app -o jsonpath='{.items[*].metadata.annotations.linkerd\.io/inject}{"\n"}'

Remember that injection normally happens at pod creation, so after adding istio-injection=enabled to a namespace you must restart the workloads: kubectl rollout restart deployment -n app.

Port and targetPort mismatches

The last gate is the simplest and the easiest to overlook, because every other signal looks green. The endpoints exist, the labels match, DNS resolves, no policy is in the way — and traffic still fails, because the Service’s targetPort does not equal the port the container is actually listening on. The fix is to match targetPort to the container’s containerPort, either by number or by port name.

kubectl get svc api -n app -o yaml | grep -A 6 ports
kubectl get pod -n app api-7c9 -o jsonpath='{.spec.containers[*].ports}{"\n"}'

RBAC failures — who may do what

☺ Like you’re 10: Two different guards can stop you. One says “you’re not allowed” (RBAC), the other says “that shape isn’t allowed here” (admission). They sound similar and are fixed completely differently.

A forbidden message is one of the most information-dense strings in Kubernetes, and reading it precisely is a skill worth ten minutes of practice. It names the subject, the verb, the resource, and the namespace — four facts that between them tell you exactly which Role or binding to write. See Security & Policy Enforcement for the underlying model.

Reading a forbidden error

Error from server (Forbidden): pods is forbidden:
  User "system:serviceaccount:ci:builder" cannot create resource "pods"
  in API group "" in the namespace "prod"
       |                    |            |          |            |
    subject               verb       resource   API group    namespace

That single line tells you: the subject is the ServiceAccount builder in namespace ci; the missing verb is create; the resource is pods in the core (empty) API group; and the scope needed is namespace prod. Note the subject’s namespace (ci) and the target namespace (prod) differ — so the RoleBinding must live in prod and reference a subject in ci, which is precisely the mistake people make when they create the binding in the wrong namespace.

Proving it with kubectl auth can-i --as

Impersonation turns RBAC from guesswork into a two-second experiment. Run it before your fix to confirm the diagnosis and after your fix to confirm the grant landed — you never have to obtain the ServiceAccount’s token or log in as it.

# Impersonate to confirm, before and after your fix. This is the fastest RBAC loop there is.
kubectl auth can-i create pods -n prod --as=system:serviceaccount:ci:builder
kubectl auth can-i --list -n prod --as=system:serviceaccount:ci:builder
kubectl auth can-i '*' '*' --as=system:serviceaccount:ci:builder      # is it accidentally admin?

# Find what is currently bound to the subject:
kubectl get rolebindings,clusterrolebindings -A -o wide | grep 'ci:builder'
kubectl describe clusterrole edit | head -20

# Grant exactly what the error asked for, in the namespace the error named:
kubectl create role pod-creator -n prod --verb=create,get,list --resource=pods
kubectl create rolebinding builder-pods -n prod \
  --role=pod-creator --serviceaccount=ci:builder

kubectl auth can-i create pods -n prod --as=system:serviceaccount:ci:builder   # -> yes

Role vs ClusterRole, and the six binding mistakes

Nearly every RBAC fault that is not a plain missing grant is one of these six. Each one produces a binding that exists and looks correct, which is why they cost so much time.

Admission failures — what shapes are allowed

Admission is a different guard entirely: the request is authorised, but a policy engine rejects the object’s shape. The tell is that the error names a webhook and a policy rather than a subject and a verb.

Admission webhook denials

Kyverno and Gatekeeper both return the policy name and a human-written message, so the fix is usually to read the message and comply — add the missing label, drop the privileged flag, pin the image digest.

kubectl apply -f deploy.yaml
# Error from server: admission webhook "validate.kyverno.svc-fail" denied the request:
#
# resource Deployment/app/web was blocked due to the following policies
#
# require-run-as-nonroot:
#   autogen-check-containers: 'validation error: Containers must run as non-root.
#     rule autogen-check-containers failed at path
#     /spec/template/spec/containers/0/securityContext/runAsNonRoot/'

# Kyverno: what policies exist, and which are enforcing vs auditing?
kubectl get clusterpolicy,policy -A
kubectl get clusterpolicy require-run-as-nonroot -o yaml | grep -iE 'failureAction|validationFailureAction'
kubectl get policyreport -A                 # audit-mode results land here, not in your terminal
kubectl logs -n kyverno -l app.kubernetes.io/component=admission-controller --tail=50

# Gatekeeper: constraints carry their violations in status
kubectl get constrainttemplates
kubectl get constraints
kubectl describe k8srequiredlabels ns-must-have-owner | grep -A 20 'Total Violations'
kubectl logs -n gatekeeper-system -l control-plane=controller-manager --tail=50

Audit vs enforce

The audit-versus-enforce distinction matters for both engines. Kyverno expresses it per-rule as a failure action of Audit or Enforce (in older releases, spec.validationFailureAction at policy level — check the version you are given), and Gatekeeper uses spec.enforcementAction with deny, dryrun, or warn. In audit/dryrun mode nothing is blocked and no error reaches your terminal — violations appear only in a PolicyReport or in the constraint’s status.violations. So “my policy isn’t working” very often means “my policy is in audit mode and is working perfectly.”

When the webhook itself is down

This is the most alarming failure in the section, because it breaks things that have nothing to do with policy. A ValidatingWebhookConfiguration with failurePolicy: Fail means that if the webhook service cannot be reached, the API server rejects the request outright. If the policy controller is unhealthy and its webhook matches broadly, every create and update in scope starts failing — including, in the worst case, the pods that would restore the policy controller.

kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations
kubectl get validatingwebhookconfiguration kyverno-resource-validating-webhook-cfg \
  -o jsonpath='{range .webhooks[*]}{.name}{"\t"}{.failurePolicy}{"\t"}{.namespaceSelector}{"\n"}{end}'

# Error you will see:
#   Internal error occurred: failed calling webhook "validate.kyverno.svc-fail":
#   failed to call webhook: Post "https://kyverno-svc.kyverno.svc:443/validate?timeout=10s":
#   context deadline exceeded

kubectl get pods -n kyverno -o wide          # is the controller even running?
kubectl get svc,endpoints -n kyverno         # does the webhook service have endpoints?

# Emergency break-glass: remove the broken webhook config so the cluster is usable again.
# Back it up first, and restore it once the controller is healthy.
kubectl get validatingwebhookconfiguration <name> -o yaml > /tmp/webhook-backup.yaml
kubectl delete validatingwebhookconfiguration <name>

☺ Like you’re 10: If the guard who checks everyone’s badge faints, and the rule says “nobody enters unless the guard approves,” then nobody enters at all — including the doctor who could wake the guard up. That is failurePolicy: Fail.

Those are the failures that live between and in front of your workloads. For the pod-shaped ones underneath them, go to Triage — Workloads; for sync, rollout and delivery faults, Triage — Delivery; and for the full diagnostic sequence and the rest of the matrices, return to the triage hub. The underlying models are in Platform Networking and Security & Policy Enforcement, and every command on this page is indexed in the Command Reference.