Exam Prep · Triage · Delivery, Observability & Platform APIs

Triage — Delivery, Observability & Platform APIs

Pod, networking and storage faults are the ones everybody drills. The failures that actually eat your clock on a platform exam are one layer up: the delivery engine that reconciles your cluster, the telemetry pipeline that is supposed to prove it works, and the custom APIs your developers self-serve from. These tools all fail the same way — quietly, with the real error written somewhere you have not looked yet. This page is the map of where each one hides its complaint. It is a companion to the triage hub, which holds the universal diagnostic order and the pod, networking, storage, RBAC and admission matrices; here we take the three platform-layer sections in full: GitOps and delivery, observability, and platform APIs.

☺ Explain it like I’m 10

Imagine you posted a letter asking for a new bike, and no bike arrived. There are only two possible stories. Either nobody read your letter — it fell behind the desk, or you sent it to an address where nobody works. Or somebody read it and got stuck — the factory ran out of wheels and wrote you a note about it. Every tool on this page works like that factory: when it gets stuck, it writes a note. The note is called status conditions, and it is almost always sitting right there on the thing you asked for. Nearly all of this page is just knowing where the note is kept for each tool.

🦫🐘Your hosts for this topic: Benny the Beaver & Ellie the Elephant — Benny built the delivery rails and knows exactly which controller in the pipeline stopped turning, and Ellie never forgets a symptom she has seen before, so she can tell you which of the three label matches is the one that silently broke.

The triage order, in brief

☺ Like you’re 10: Before you guess what’s wrong, always look in the same places, in the same order. Guessing first is how you waste ten minutes.

Everything below assumes the same opening sequence, so here it is in compressed form. The full version, with the reasoning and the decision flow, lives on the triage hub.

Get → describe → events → logs → controller status

Steps 1–4 are the general Kubernetes ladder. Step 5 is the platform-engineering addition that matters enormously here, because so much of a platform is expressed as custom resources whose status.conditions block is the real error message. Only at step 6 do you get to have an opinion.

#	Step	Command	What you are looking for
1	Wide list	`kubectl get <kind> -o wide`	Phase, restarts, age, node, IP.
2	Describe	`kubectl describe <kind> <name>`	The Events block at the bottom. Then Conditions.
3	Namespace events	`kubectl get events --sort-by=.lastTimestamp`	Failures on objects you did not think to describe.
4	Logs	`kubectl logs <pod> --previous`	Why the process itself died.
5	Controller / CR status	`kubectl get <cr> -o yaml` → `status.conditions`	What the operator thinks. Argo CD, Flux, Crossplane and cert-manager all write the true error here.
6	Hypothesis	—	Only now — and prefer the cause that explains all the evidence.

◆ Key idea

For the tools on this page, step 5 is where you start, not where you finish. A crashing pod explains itself in Events; a stuck Argo CD Application, a non-reconciling Flux Kustomization and a never-Ready Crossplane claim all explain themselves in status.conditions. Every one of these controllers is contractually obliged to say why it is unhappy — the entire skill is knowing which object to ask. Drill the exact command shapes in the Command Reference.

GitOps & delivery failures

☺ Like you’re 10: The robot that keeps your cluster tidy will tell you exactly why it’s unhappy — you just have to ask it in its own language.

Argo CD reports two independent statuses and confusing them wastes time. Sync status answers “does the cluster match Git?” — Synced or OutOfSync. Health status answers “are the resources actually working?” — Healthy, Progressing, Degraded, Missing, or Suspended. They are orthogonal: you can be perfectly Synced and thoroughly Degraded, which means Git got its way and Git was wrong. The model behind all of this is in GitOps Workflows, and the rollout machinery in CI/CD & Progressive Delivery.

Argo CD: OutOfSync vs Degraded vs Missing

Status	Means	Look at
`OutOfSync`	Live state differs from the rendered manifests in Git	`argocd app diff` — is the diff yours, or drift?
`Degraded`	Applied fine, but a resource is unhealthy (crashing pods, failed Job)	The pods. This is a workload problem, not a delivery one.
`Missing`	A resource in Git does not exist in the cluster at all	Sync failed, was skipped, or a hook blocked it
`Progressing`	Rollout in flight, not yet settled	Wait, then re-check — do not fix what is still moving
`Unknown` / `ComparisonError`	Repo unreachable, bad path, or manifests fail to render	repo-server logs; credentials; the path in `spec.source`

When the status is Degraded, hand the problem straight over to the pod-level matrices on the triage hub — Argo CD has done its job correctly and is merely reporting that the manifests you asked for produce unhealthy resources.

Argo CD: sync failures

argocd app get checkout                        # sync status, health, and per-resource state
argocd app diff checkout                       # exactly what differs from Git
argocd app history checkout
argocd app sync checkout --prune --dry-run

# Same information without the CLI (useful if you are not logged in):
kubectl get application checkout -n argocd -o jsonpath='{.status.sync.status}{"\t"}{.status.health.status}{"\n"}'
kubectl get application checkout -n argocd -o yaml | sed -n '/conditions:/,/^  [a-z]/p'
kubectl logs -n argocd deploy/argocd-repo-server --tail=50          # render/auth errors
kubectl logs -n argocd statefulset/argocd-application-controller --tail=50   # sync errors

Three sync failures recur often enough to name:

A failed hook. PreSync hooks must succeed before anything is applied, so one failing migration Job leaves every resource Missing and the app never progresses. Find the hook Job and read its logs: kubectl get jobs -n app, then kubectl logs job/<name> -n app.
An immutable field. Changing spec.selector on a Deployment, the pod template of a Job, or a Service’s clusterIP produces field is immutable. Kubernetes will never accept the update, so no amount of re-syncing helps. Fix it with a replace: argocd app sync checkout --replace, or add Replace=true to syncOptions — or delete the resource and let Argo recreate it.
Self-heal fighting you. With selfHeal: true, your manual kubectl edit is reverted within seconds and it looks like your change “didn’t save.” Either change Git, or temporarily disable automation.

# Immutable-field error in the app conditions:
#   SyncError: Deployment.apps "web" is invalid: spec.selector: Invalid value: ...
#   field is immutable
argocd app sync web --replace

# Stop self-heal from reverting you while you investigate (remember to put it back):
kubectl patch application web -n argocd --type=merge \
  -p '{"spec":{"syncPolicy":{"automated":{"selfHeal":false}}}}'

# Pruning surprises: see what a prune WOULD delete before you enable it.
argocd app sync web --prune --dry-run
# Protect a resource that must never be pruned:
#   metadata.annotations: argocd.argoproj.io/sync-options: Prune=false

⚠ Prune deletes real things

Enabling prune: true on an app whose Git path is wrong — or which was pointed at the wrong branch — will cheerfully delete everything it believes is no longer declared. Always run argocd app diff or a --dry-run sync before enabling prune on anything you did not create thirty seconds ago. Related failure modes are catalogued in Anti-Patterns.

Argo Rollouts: a stuck progressive rollout

A Rollout that has stopped moving is in one of three situations: it is paused at a step and waiting for a human promotion; an AnalysisRun failed and it aborted; or the traffic shift never happened because the mesh or ingress integration is not wired up, so the canary weight is meaningless.

kubectl argo rollouts get rollout checkout -n app --watch
kubectl argo rollouts status checkout -n app
kubectl describe rollout checkout -n app | tail -25

# Did an analysis fail?
kubectl get analysisrun -n app
kubectl describe analysisrun checkout-abc123-2 -n app | grep -A 15 'Metric Results'
#   Phase: Failed   Message: metric "success-rate" assessed Failed due to failed (1) > failureLimit (0)

# Actions:
kubectl argo rollouts promote checkout -n app        # advance past a pause step
kubectl argo rollouts promote checkout -n app --full # skip all remaining steps
kubectl argo rollouts abort checkout -n app          # roll back to stable
kubectl argo rollouts retry rollout checkout -n app

# No traffic shift? The canary/stable Services and the traffic router must exist and be referenced.
kubectl get svc -n app checkout-canary checkout-stable
kubectl get rollout checkout -n app -o jsonpath='{.spec.strategy.canary.trafficRouting}{"\n"}'

Flagger, the other progressive-delivery engine on the tool list, reports the same class of problem through its Canary resource: kubectl get canary -A shows a status of Progressing, Succeeded or Failed, and kubectl describe canary <name> lists the metric checks that failed and the halt reason.

🦆 Dot’s-eye view

“The first time a canary stalled on me I assumed the analysis had failed, and I spent ten minutes staring at PromQL. It hadn’t failed — it was sitting on a manual pause step that nobody had told me about, waiting for a promote. Now the first thing I run is kubectl argo rollouts get rollout, because it draws the steps and puts an arrow on the one it is stuck at. Paused, failed, and never-shifted look identical from the deployment’s point of view, and completely different from the Rollout’s.”

Flux reconciliation failures

Flux splits the job across controllers, so first establish which stage failed: the source could not be fetched, the manifests could not be built, or the apply failed. The flux get commands print a Ready condition and a message for each stage, which is exactly the triage order in miniature.

flux get sources git -A                 # Ready=False here means fetch/auth/branch problem
flux get kustomizations -A              # Ready=False here means build or apply problem
flux get helmreleases -A
flux logs --level=error --all-namespaces --tail=50

# Force an immediate reconcile, pulling the newest commit first:
flux reconcile source git platform-config -n flux-system
flux reconcile kustomization apps -n flux-system --with-source

# Common messages and what they mean:
#   "failed to checkout and determine revision: ... authentication required"  -> secretRef / deploy key
#   "kustomize build failed: accumulating resources: ... no such file"        -> wrong path or missing file
#   "Helm upgrade failed: another operation (install/upgrade/rollback) is in progress"
#        -> a stuck release; flux suspend/resume, or roll back the Helm release
#   "dependency 'flux-system/infra' is not ready"                              -> dependsOn ordering

flux suspend kustomization apps -n flux-system
flux resume  kustomization apps -n flux-system      # suspend/resume clears many stuck states
kubectl get kustomization apps -n flux-system -o yaml | sed -n '/status:/,$p'

Tekton PipelineRun failures

Tekton failures cluster into two buckets: something about the workspace is missing or unbindable, or the ServiceAccount lacks the credentials to do its job — most often pushing the built image to a registry.

tkn pipelinerun list -n ci
tkn pipelinerun describe build-and-push-run-xyz -n ci
tkn pipelinerun logs build-and-push-run-xyz -n ci -f
tkn taskrun describe build-and-push-run-xyz-build -n ci

# Without the tkn CLI — a PipelineRun is just a CR with conditions, and the steps are pod containers:
kubectl get pipelinerun -n ci -o wide
kubectl get pipelinerun build-and-push-run-xyz -n ci -o jsonpath='{.status.conditions}{"\n"}'
kubectl logs -n ci <taskrun-pod> -c step-build

# Typical causes:
#   "Pipeline ci/build-and-push can't be Run; it contains Tasks that don't exist:
#    Couldn't retrieve Task 'kaniko': tasks.tekton.dev 'kaniko' not found"   -> task not installed in this ns
#   "doesn't bind Pipeline ci/build-and-push's Workspaces correctly: pipeline
#    expects workspace with name 'source' be provided by pipelinerun"        -> missing workspaces: binding
#   "denied: requested access to the resource is denied"                       -> SA lacks registry push creds
kubectl get sa -n ci pipeline -o yaml           # check secrets / imagePullSecrets
kubectl get pvc -n ci                           # workspace PVC bound?

The build-side practices that keep these pipelines boring — reproducible builds, signed artefacts, sane promotion — are covered in Release Engineering. To rehearse the delivery faults under time, work through Practice · GitOps.

Observability failures

☺ Like you’re 10: If the dashboard is empty, the problem is usually not the dashboard — it’s that nobody ever agreed to send it any numbers.

Observability breaks along a pipeline, and you debug it by walking the pipeline backwards from the empty panel: is the data in the dashboard? in the datasource? was it scraped? is it exposed? is the app even instrumented? Deeper treatment lives in Observability & Operations.

The observability failure matrix

Symptom	Likely causes	Where to look
Target missing from Prometheus entirely	ServiceMonitor selector does not match the Service · Prometheus’ own `serviceMonitorSelector` (often a `release` label) does not match the ServiceMonitor · namespace not selected	Prometheus UI → Status → Service Discovery
Target present but `DOWN`	Wrong port name · wrong metrics path · NetworkPolicy blocking the scrape · TLS mismatch	Prometheus UI → Targets (shows the error)
Target `UP` but no app metrics	App not instrumented · metrics on a different path or port	`curl` the pod’s metrics endpoint directly
Alert never fires	PromQL returns nothing · `for:` longer than the condition lasts · rule not loaded	Run the expression by hand in the UI
Alert fires, nobody is paged	Alertmanager route mismatch · active silence · inhibition rule	Alertmanager UI → Alerts / Silences
No traces in Jaeger	Wrong OTLP endpoint or port · exporter misconfigured · no context propagation · sampling at zero	Collector logs; the SDK’s endpoint env var

Prometheus: the ServiceMonitor label chain

This is the single most common observability fault on a platform exam, and it is a chain of three label matches, any one of which breaks silently. First, the ServiceMonitor’s spec.selector.matchLabels must match the Service’s labels (not the pods’). Second, the ServiceMonitor’s spec.endpoints[].port must be the name of a Service port, not a number. Third — the one people miss — the Prometheus custom resource has its own serviceMonitorSelector, and with the kube-prometheus-stack Helm chart that typically requires a release: <helm-release-name> label on every ServiceMonitor you create.

# Walk the chain, one link at a time:
kubectl get servicemonitor -n app api -o jsonpath='{.spec.selector.matchLabels}{"\n"}'
kubectl get svc -n app api --show-labels                   # link 1: do these match?
kubectl get svc -n app api -o jsonpath='{.spec.ports}{"\n"}'  # link 2: is the port NAMED?

kubectl get prometheus -n monitoring -o jsonpath='{.items[0].spec.serviceMonitorSelector}{"\n"}'
kubectl get prometheus -n monitoring -o jsonpath='{.items[0].spec.serviceMonitorNamespaceSelector}{"\n"}'
kubectl get servicemonitor -n app api --show-labels        # link 3: release label present?

# Then confirm from Prometheus itself:
kubectl port-forward -n monitoring svc/prometheus-operated 9090
#   http://localhost:9090/targets            -> is the target listed, and UP or DOWN?
#   http://localhost:9090/service-discovery  -> was it discovered and then dropped?
kubectl logs -n monitoring prometheus-kube-prometheus-stack-prometheus-0 -c prometheus --tail=40

# And confirm the app really exposes metrics where you think:
kubectl exec -n app deploy/api -- wget -qO- localhost:8080/metrics | head

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: api
  namespace: app
  labels:
    release: kube-prometheus-stack   # link 3 — Prometheus' serviceMonitorSelector
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: api    # link 1 — must match the SERVICE's labels
  namespaceSelector:
    matchNames: [app]
  endpoints:
    - port: http-metrics             # link 2 — the NAME of a Service port, never a number
      path: /metrics
      interval: 30s

☺ Like you’re 10: Three doors have to unlock in a row. The dashboard being empty doesn’t tell you which door is locked, so check all three in order instead of guessing.

Alerts that never fire, and alerts nobody sees

Split this in two immediately. If the alert is not visible as firing in Prometheus, the rule is at fault. If it is firing in Prometheus but no notification arrived, Alertmanager is at fault. Never debug both at once.

# Is the rule even loaded? (PrometheusRule needs matching labels too, same as ServiceMonitor)
kubectl get prometheusrule -A
kubectl get prometheus -n monitoring -o jsonpath='{.items[0].spec.ruleSelector}{"\n"}'
#   Prometheus UI -> Alerts: state should be Inactive / Pending / Firing

# Run the expression by hand — a typo'd label makes it return "no data" forever, silently.
#   sum(rate(http_requests_total{job="api",code=~"5.."}[5m]))
#     / sum(rate(http_requests_total{job="api"}[5m])) > 0.05
#   (a bare sum(rate(...)) is an absolute request rate, not a 5% error ratio —
#    dividing by the total is what makes the 0.05 threshold mean what you think)
# A `for: 10m` clause means a spike lasting 3m will NEVER fire. Check the duration.

# Firing in Prometheus but no page? It's routing.
kubectl port-forward -n monitoring svc/alertmanager-operated 9093
amtool alert query --alertmanager.url=http://localhost:9093
amtool silence query --alertmanager.url=http://localhost:9093     # an active silence?
kubectl logs -n monitoring alertmanager-kube-prometheus-stack-alertmanager-0 --tail=40
#   look for: matching labels in `route`, an `inhibit_rules` entry, or a receiver with no config

Traces that never arrive

The OpenTelemetry pipeline has three joints and each has a well-known failure. The endpoint and port are the first suspect: 4317 is OTLP/gRPC and 4318 is OTLP/HTTP, and sending gRPC payloads to the HTTP port (or vice versa) produces a connection that appears to work and delivers nothing.

kubectl logs -n observability deploy/otel-collector --tail=60
#   "failed to export traces: rpc error: code = Unavailable desc = connection refused"
#   service::pipelines::traces: references exporter 'otlp/jaeger' which is not configured
#        -> exporter declared but absent from the traces pipeline; the collector refuses to start

kubectl get cm otel-collector-conf -n observability -o yaml
#   verify: receivers.otlp.protocols.{grpc,http}, exporters.otlp.endpoint,
#           and that BOTH appear under service.pipelines.traces — a defined-but-unused
#           exporter is the classic silent failure.

kubectl exec -n app deploy/api -- printenv | grep OTEL
#   OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.observability:4317   (gRPC)
#   OTEL_EXPORTER_OTLP_PROTOCOL=grpc
#   OTEL_SERVICE_NAME=api

kubectl get svc -n observability otel-collector -o jsonpath='{.spec.ports}{"\n"}'
kubectl get pods -n observability -l app=jaeger

If spans arrive but are not joined into one trace, that is context propagation: each service must forward the traceparent header (W3C Trace Context). Broken propagation looks like many single-span traces rather than no traces at all — a useful distinction, because it tells you the collector is fine and the application code is not. Rehearse the whole pipeline against Practice · Observability.

Platform-API failures

☺ Like you’re 10: You filled in the order form and nothing arrived. Either nobody is reading the forms, or the factory read it and got stuck — and the factory writes its complaint on the form itself.

When self-service breaks, the developer’s experience is “I applied my claim and nothing happened.” There are only two shapes of cause: nobody is listening (no controller running, wrong CRD version or scope), or somebody is listening and is stuck (provider unhealthy, no credentials, composition mismatch). Distinguishing them takes one command. Background: Platform APIs & CRDs and Self-Service Platforms.

Is anyone listening?

kubectl get crd | grep -i database
kubectl api-resources | grep -i database        # shows APIVERSION, NAMESPACED, KIND
kubectl explain databaseinstance.spec           # if this errors, the CRD/version is wrong

# The two silent killers:
#  1. You applied v1alpha1 but only v1beta1 is served -> "no matches for kind"
#  2. The CRD is Cluster-scoped but you used -n app  -> object created somewhere you're not looking
kubectl get crd databaseinstances.platform.acme.io -o jsonpath='{.spec.scope}{"\t"}{range .spec.versions[*]}{.name}={.served},{end}{"\n"}'

# Is a controller actually reconciling it?
kubectl get pods -n crossplane-system
kubectl logs -n crossplane-system -l app=crossplane --tail=50

If the object exists, has no status block at all, and no events, that is the signature of nobody listening: the API server happily stored your YAML and no controller ever picked it up. A CR with a populated but unhappy status.conditions is the opposite case, and much easier.

A Crossplane claim that never becomes Ready

Crossplane composes: a claim (namespaced) creates a composite resource / XR (cluster-scoped), which creates one or more managed resources, which talk to a cloud provider. Debugging means following that chain downward and reading conditions at each level — the useful error is almost always at the bottom, on a managed resource, and it is almost always credentials or a field the provider rejected.

# Level 0 — the provider itself must be healthy before anything can work.
kubectl get providers
kubectl get providerconfig
kubectl get pods -n crossplane-system
#   Provider HEALTHY=False -> image pull, RBAC, or the provider pod is crashing

# Level 1 — the claim
kubectl describe databaseinstance orders-db -n app | tail -25
#   Conditions: Synced=False  Reason=ReconcileError
#   Message: cannot compose resources: no CompositeResourceDefinition ... / no matching Composition

# Level 2 — the composite (follow the resourceRef the claim points at)
kubectl get databaseinstance orders-db -n app -o jsonpath='{.spec.resourceRef}{"\n"}'
kubectl describe xdatabaseinstance orders-db-x7k2p | tail -25

# Level 3 — the managed resources: this is where the real error lives
kubectl get managed
kubectl describe rdsinstance orders-db-x7k2p-9fk3 | grep -A 12 Conditions
#   Synced=False  Reason=ReconcileError
#   Message: cannot initialize the AWS client: cannot get referenced Secret:
#            Secret "aws-creds" not found

kubectl get compositions
kubectl get compositeresourcedefinitions   # xrd
kubectl get xrd xdatabaseinstances.platform.acme.io -o jsonpath='{.status.conditions}{"\n"}'

Symptom	Likely cause	Check
Claim has no `status` at all	No controller watching this kind	`kubectl get pods -n crossplane-system`; is the provider installed?
`no matches for kind "X"`	CRD absent, or you used an unserved API version	`kubectl api-resources`, `kubectl get crd`
Claim Synced but never Ready	Managed resource still creating, or blocked at the cloud provider	`kubectl get managed` then describe the unready one
`cannot resolve composition`	Composition’s `compositeTypeRef` ≠ the XRD’s type, or the selector label missing	`kubectl get composition -o yaml`
Provider `HEALTHY=False`	Provider pod crashing, or ProviderConfig points at a missing Secret	Provider pod logs; `kubectl get secret` in the referenced namespace
Delete hangs forever	Finalizer waiting on external cleanup that keeps failing	`kubectl get <cr> -o yaml` → finalizers + events

Finalizers blocking deletion

Deleting a namespace that contains a stuck custom resource is how people accidentally wedge a cluster: the namespace enters Terminating and stays there, because a finalizer on one object inside it never completes. Diagnose before you brute-force — a finalizer usually exists to clean up something real, like a cloud database that will keep costing money if orphaned.

kubectl get ns app -o jsonpath='{.status.conditions}{"\n"}'
#   NamespaceContentRemaining / NamespaceFinalizersRemaining name the culprit

kubectl api-resources --verbs=list --namespaced -o name \
  | xargs -n1 kubectl get --show-kind --ignore-not-found -n app     # what is still in there?

kubectl get rdsinstance orders-db -o jsonpath='{.metadata.finalizers}{"\n"}'
kubectl describe rdsinstance orders-db | tail -20                   # why is cleanup failing?

# Last resort, after you understand the consequences (this can orphan cloud resources):
kubectl patch rdsinstance orders-db -p '{"metadata":{"finalizers":[]}}' --type=merge

⚠ Clearing a finalizer is not a fix

Stripping metadata.finalizers makes the object vanish from the cluster, which looks like success and can hide an expensive mistake: the cloud resource the finalizer was going to delete is now orphaned and still billing. Always read the failing cleanup first — it is usually a missing credential Secret or a revoked permission, and repairing that lets the finalizer complete honestly. Reach for the patch only when you have understood what will be left behind. Push the practice further in Practice · Platform APIs.

🐘 Ellie’s workshop · 20 min

Break the platform layer on purpose and time yourself finding each cause with nothing but the triage order. On a kind or minikube cluster with Argo CD and kube-prometheus-stack installed: (1) point an Argo CD Application at a path that does not exist in the repo and identify the status and the log that names the real error; (2) enable selfHeal: true, then kubectl edit the Deployment’s replica count and watch your change disappear — note how long it takes and what the app conditions say; (3) change a Deployment’s spec.selector in Git and find the field is immutable error, then fix it with --replace; (4) create a ServiceMonitor with a correct selector but no release label and confirm the target never appears in /service-discovery; (5) apply a custom resource whose CRD is not installed, and again with an unserved API version, and note that the two produce different errors. For each fault, write down the first command that revealed the cause. If it was not describe, get events, or a status.conditions read in at least four of the five, you are still guessing before gathering.

🎬 At the Platform Guild
🦆Dot: I merged the fix an hour ago and prod still has the old version. Argo says Synced. How can it be synced and wrong?
🦫Benny: Because Synced only means “the cluster matches Git.” Did you check the health status?
🦆Dot: …Degraded. So Git got exactly what it asked for and what it asked for is broken.
🐘Ellie: That is the whole lesson. Two statuses, two different problems. OutOfSync is a delivery bug; Degraded is a workload bug wearing a delivery costume.
👺Gizmo: Just kubectl edit the Deployment and set the image tag by hand. Thirty seconds, done, nobody has to review anything. 😈
🦫Benny: Self-heal will revert you in twelve seconds and you’ll spend the next ten minutes convinced the API server is haunted.
🐢Timmy: And the alert that should have caught this never fired, because its ServiceMonitor is missing the release label. Nothing is broken. Nothing is connected either.
🐘Ellie: Which is the theme of the whole page, really. These tools rarely scream. They just stop, politely, and write it down somewhere you haven’t opened.

Three tools, three hiding places: Argo CD and Flux put the truth in an Application or Kustomization condition, Prometheus puts it in Service Discovery, and Crossplane puts it on the managed resource at the bottom of the chain. Once you know that, the rest is just typing. Go back to the triage hub for the universal order and the workload-level matrices, take the sibling pages for workload failures and networking failures, keep the Command Reference open beside you, and then prove it under time in Practice Tasks and the Lab Track.

🐢 Timmy’s checkpoint

1. What does an Argo CD app that is Synced but Degraded tell you, and where do you go next? 2. Every resource in an app shows Missing and the sync never progresses — what is the first thing to suspect, and which object holds the evidence? 3. You get field is immutable on a Deployment. Why will re-syncing never help, and what are your two options? 4. A Flux GitRepository is Ready=False but the Kustomization looks fine — which stage failed, and which is the misleading one? 5. Your ServiceMonitor exists and the Service is correct, but the target never appears in Prometheus — name the third label match people forget. 6. An alert is visible as Firing in the Prometheus UI but nobody was paged. Which component do you debug, and which three things do you check in it? 7. Spans reach Jaeger but each request produces several one-span traces instead of one. What is broken, and what is not? 8. A Crossplane claim has no status block whatsoever. What does that specifically imply?

Check your answers

The cluster matches Git exactly, and the resulting resources are unhealthy — so the applied manifests are themselves wrong (crashing pods, failed Job). Fixing it means changing Git, not re-syncing, and the actual debugging is pod-level: describe the pods and read Events.
A failed PreSync hook. Hooks must succeed before anything is applied, so one failing migration Job leaves every resource Missing. The evidence is the hook Job: kubectl get jobs -n app, then kubectl logs job/<name> -n app.
Kubernetes will never accept the update — fields like a Deployment’s spec.selector, a Job’s pod template, or a Service’s clusterIP are immutable, so the same rejection repeats forever. Either sync with --replace (or add Replace=true to syncOptions), or delete the resource and let Argo recreate it.
The source stage failed — a fetch, auth, or branch problem, typically a missing secretRef/deploy key. A Kustomization can keep reporting its last successful build, so the source condition is the honest one; check flux get sources git -A before flux get kustomizations -A.
The Prometheus CR’s own serviceMonitorSelector — with kube-prometheus-stack this usually requires a release: <helm-release> label on the ServiceMonitor. Also confirm serviceMonitorNamespaceSelector covers the namespace, and that endpoints[].port is the Service port name, never a number.
Alertmanager — the rule is doing its job. Check the route for a label mismatch, look for an active silence, and look for an inhibit_rules entry suppressing it (a receiver with no real configuration is the fourth candidate). amtool alert query and amtool silence query answer this fastest.
Context propagation: the services are not forwarding the traceparent header (W3C Trace Context), so each hop starts a new trace. The collector, the endpoint, the port and the exporter configuration are all fine — spans are arriving, they are just not being joined. It is an application-code fix, not a pipeline one.
That no controller is watching that kind — the API server stored the object but nothing reconciled it. Check that the provider/operator pod is running and that the CRD and API version you used are actually installed and served, and check the CRD’s scope in case the object was created somewhere you are not looking.