Platform Observability, Security, and Conformance

One question in five comes from here, and the domain is two habits taught together. Observability is how a platform tells you the truth about itself — the four signals it emits, and which one answers which question. Security and conformance is how it keeps its promises when nobody is watching: encrypted traffic between services, policies that refuse bad manifests at the door, and a supply chain you can prove wasn’t tampered with. Because CNPA is multiple-choice, the goal is recall — naming which tool, which layer, which signal.

The official CNCF curriculum lists exactly five competencies in this 20% domain, each a section below: Observability Fundamentals: Traces, Metrics, Logs, and Events; Secure Service Communication; Policy Engines for Platform Governance; Kubernetes Security Essentials; Security in CI/CD Pipelines. At 20% it is the second-heaviest of the six domains, behind Platform Engineering Core Fundamentals at 36% — the published weights (36 + 20 + 16 + 12 + 8 + 8) total 100%. The domain title adds conformance: a platform should be provably standard and provably compliant, not merely “working on our cluster.” Questions almost always ask which signal fits a need, which layer a control lives at, or what order things happen in.

Observability fundamentals: traces, metrics, logs, and events

☺ Like you’re 10: Four instruments — numbers over time, a written diary, a parcel’s journey, and the caretaker’s notices.

The distinction the exam loves: monitoring answers questions you knew to ask in advance (“is CPU above 80%?”); observability lets you ask new questions of a running system without shipping new code. Cloud-native architecture forces the shift, because one user action now crosses a dozen ephemeral pods.

Metrics — cheap numbers over time

A metric is a number sampled at intervals and stored as a time series, identified by a name plus labels. Prometheus pulls (scrapes) an HTTP /metrics endpoint from each discovered target and queries with PromQL; Alertmanager routes, groups and silences the alerts. The types are counter (only goes up), gauge (up and down) and histogram/summary (distributions, for percentiles). Metrics aggregate beautifully — but are low-cardinality by design. Never put a user ID in a label.

Logs — the timestamped narrative

A log is a timestamped record of a discrete occurrence. Containers write to stdout/stderr, a node agent (Fluent Bit, Fluentd, Vector, Alloy) collects, and a store such as Loki or OpenSearch keeps it. Two exam-worthy facts: prefer structured logs (JSON, consistent keys) so they can be queried rather than grepped; and Loki indexes only labels, not the full log text, which is why it is cheaper than a full-text engine.

Traces — one request across many services

A distributed trace follows one request end-to-end as a tree of spans, each a unit of work with a duration, attributes and a parent. All spans share a trace ID propagated in a header — the W3C standard is traceparent. Tracing is the only signal answering “where did the time go, and which hop failed?” Jaeger and Tempo are the usual backends. Volume forces sampling: head sampling decides up front; tail sampling decides after seeing the whole trace, keeping every slow or errored one.

Events — Kubernetes narrating its own state changes

Events are the signal people forget, and the curriculum names them explicitly. A Kubernetes Event is a real API object a controller creates to explain why it acted: FailedScheduling, Pulled, BackOff, Unhealthy, Killing. They are namespaced, attached to an involved object, and short-lived — discarded after roughly an hour by default, so export them if you want them at review time.

Collecting the signals: OpenTelemetry and what to measure

☺ Like you’re 10: All the instruments now speak one language, so you can swap the thing that stores the readings without rewiring every workshop.

A platform team’s job is not to invent observability per team — it is to pave the road: instrument once, collect centrally, and let every tenant get dashboards and alerts for free. Mechanics go deeper in Observability & Platform Health.

The pipeline: instrument, collect, store, visualise

Every stack has the same four stages, and questions often ask you to place a tool in one. Instrumentation lives in or beside the app. Collection receives, batches and enriches. Storage keeps and indexes. Visualisation and alerting is where humans meet the data — Grafana as the single pane, Alertmanager as the pager.

OpenTelemetry — one standard, three signals

OpenTelemetry is the CNCF project that unifies instrumentation across traces, metrics and logs. It gives you a specification, per-language SDKs (plus zero-code auto-instrumentation for several runtimes), the OTLP wire protocol, and the Collector — a process configured as receivers, processors and exporters, runnable as a per-node DaemonSet, a gateway Deployment, or both. The payoff is vendor neutrality: change where data lands by editing Collector config, not application code.

# OpenTelemetry Collector: receive OTLP from apps, enrich, fan out to two backends
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }

processors:
  batch: {}                       # amortise network cost
  memory_limiter:                 # protect the collector from OOM
    check_interval: 1s
    limit_percentage: 80
  k8sattributes: {}               # enrich with pod / namespace / node metadata

exporters:
  otlp/traces:
    endpoint: jaeger-collector.observability.svc:4317
  prometheusremotewrite:
    endpoint: http://prometheus.observability.svc:9090/api/v1/write

service:
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters:  [otlp/traces]
    metrics:
      receivers:  [otlp]
      processors: [memory_limiter, batch]
      exporters:  [prometheusremotewrite]

What to measure: golden signals, RED and USE

Three mnemonics you may be asked to tell apart. Google’s four golden signals: latency, traffic, errors, saturation. RED — Rate, Errors, Duration — for request-driven services. USE — Utilisation, Saturation, Errors — for resources like CPU, memory and queues. On top: an SLI is the measurement, an SLO the target (99.9% of requests under 300ms over 30 days), and the error budget the allowed shortfall — the currency that lets a team say “we pause risky releases this week.”

Secure service communication

☺ Like you’re 10: Inside the building everyone still wears a badge, and every conversation happens in a sealed room — even between colleagues.

The old model was a hard perimeter with a soft, trusting interior. Zero trust replaces it with “never trust, always verify”: every workload proves who it is on every connection. Two mechanisms carry that, and the exam expects you to separate them.

TLS, mTLS and workload identity

Ordinary TLS encrypts the channel and lets the client verify the server. Mutual TLS (mTLS) adds the other direction — the server also verifies the client — giving three properties at once: encryption in transit, authentication of both parties, and a cryptographic identity to authorise against. That identity increasingly follows SPIFFE, written as spiffe://cluster.local/ns/payments/sa/checkout: derived from namespace and ServiceAccount rather than an IP, so it survives rescheduling.

A service mesh supplies mTLS without touching application code. It injects a proxy — a sidecar, or a per-node ztunnel in Istio’s ambient mode — that terminates TLS transparently, rotates short-lived certificates, and can enforce L7 authorization: “only the frontend ServiceAccount may call POST /checkout.” Linkerd does the same with a smaller footprint; Cilium offers encryption and identity-aware policy at the eBPF layer. Ingress certificates are separate — that is cert-manager’s job.

NetworkPolicy — the L3/L4 complement

By default every Pod can reach every other Pod. A NetworkPolicy changes that, and four facts are worth memorising: it is namespaced; it is allow-only (no deny rules — you restrict by omission); a Pod selected by any policy for a direction becomes default-deny for that direction; and it is enforced by the CNI plugin, so on a CNI that doesn’t implement it your policy is silently inert. It knows nothing of HTTP paths or methods.

# 1) Default-deny all ingress in the namespace: an empty podSelector selects every Pod
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: payments
spec:
  podSelector: {}                 # every Pod in this namespace...
  policyTypes: [Ingress]          # ...becomes default-deny for ingress
---
# 2) Then punch one precise hole: frontend -> checkout, port 8080 only
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-checkout
  namespace: payments
spec:
  podSelector:
    matchLabels: { app: checkout }
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: storefront }
          podSelector:
            matchLabels: { app: frontend }
      ports:
        - protocol: TCP
          port: 8080

Policy engines for platform governance

☺ Like you’re 10: A polite inspector stands at the door; anything breaking the house rules is turned away before it gets inside.

Governance at scale cannot be a wiki page saying “please set resource limits.” It has to be policy as code — rules in Git, reviewed like code, enforced by the cluster itself. The enforcement point is admission control.

The admission path — where policy plugs in

Every write to the API server follows a fixed pipeline, and its order is a reliable exam question: authentication (who are you?) → authorization (RBAC: may you?) → mutating admission (webhooks that may change the object — sidecar injection, default labels) → schema validation → validating admission (accept or reject only) → etcd. Mutation always runs first, so validation sees the final object. Generally available since Kubernetes 1.30, there is also a webhook-free option built into the API server: ValidatingAdmissionPolicy, expressed in CEL and evaluated in-process.

Kyverno and OPA/Gatekeeper

Two CNCF engines dominate; recognise both on sight. Kyverno is Kubernetes-native: policies are YAML custom resources (ClusterPolicy/Policy) with no new language, and it does more than say no — its rule types are validate, mutate, generate (a default NetworkPolicy in every new namespace), verifyImages and cleanUp. OPA/Gatekeeper wraps Open Policy Agent: logic in Rego inside a ConstraintTemplate, instantiated by Constraint resources — steeper, but usable well beyond Kubernetes.

# Kyverno: reject any Pod that would run as root. Audit first, enforce later.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-run-as-non-root
spec:
  validationFailureAction: Audit     # switch to Enforce once the report is clean
                                     # (recent Kyverno moves this per-rule:
                                     #  rules[].validate.failureAction)
  background: true                   # also report on pre-existing resources
  rules:
    - name: check-runasnonroot
      match:
        any:
          - resources:
              kinds: [Pod]
      exclude:
        any:
          - resources:
              namespaces: [kube-system, kyverno]
      validate:
        message: "Pods must set spec.securityContext.runAsNonRoot=true."
        pattern:
          spec:
            securityContext:
              runAsNonRoot: true       # required at Pod level — no anchor
            containers:
              - =(securityContext):    # =( ) is a conditional anchor: only checked
                  =(runAsNonRoot): "true"   # if the container overrides the default

Audit before enforce

Every serious engine has two modes under different names — Kyverno’s Audit vs Enforce, Gatekeeper’s dryrun/warn/deny, PSA’s audit/warn/enforce. The rollout pattern never changes: ship in audit mode, read the reports, fix or exempt offenders, then flip to enforce. Kyverno emits PolicyReport resources and Gatekeeper audits existing objects periodically — the evidence trail auditors want, and the conformance half of this domain. More in Governance & Compliance.

Kubernetes security essentials

☺ Like you’re 10: Give everyone the smallest key that opens the door they need — and no key at all if they don’t need a door.

This is defence in depth, framed by the 4Cs of cloud native security: Cloud → Cluster → Container → Code. Each layer sits inside the previous one, so weakness in an outer layer can’t be fixed by hardening an inner one. Depth lives in Security, Policy & Guardrails.

RBAC, ServiceAccounts and least privilege

A Role (namespaced) or ClusterRole (cluster-wide) lists permitted verbs on resources; a RoleBinding or ClusterRoleBinding attaches it to a User, Group or ServiceAccount. Two behaviours catch people out: RBAC is purely additive — there are no deny rules; and a RoleBinding may reference a ClusterRole, granting it only within that namespace. Every Pod gets a ServiceAccount — default if you don’t choose — and its token is mounted unless you set automountServiceAccountToken: false.

Pod Security Admission and securityContext

PodSecurityPolicy was removed in Kubernetes 1.25 and replaced by Pod Security Admission, a built-in controller driven entirely by namespace labels. Three levels from the Pod Security Standards — privileged, baseline, restricted — in three modes: enforce, audit, warn. Beneath it, each workload sets its own securityContext:

apiVersion: v1
kind: Namespace
metadata:
  name: payments
  labels:                                          # PSA is label-driven
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/warn: restricted
---
apiVersion: v1
kind: Pod
metadata: { name: checkout, namespace: payments }
spec:
  serviceAccountName: checkout
  automountServiceAccountToken: false              # no API token unless needed
  securityContext:
    runAsNonRoot: true
    runAsUser: 10001
    seccompProfile: { type: RuntimeDefault }
  containers:
    - name: app
      image: registry.example.com/checkout@sha256:9f2c...   # digest, not :latest
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        privileged: false
        capabilities: { drop: ["ALL"] }
      resources:
        requests: { cpu: 100m, memory: 128Mi }
        limits:   { memory: 256Mi }

Secrets — what Kubernetes does and does not give you

A Kubernetes Secret is base64-encoded, not encrypted: anyone who can read the object, or etcd on disk, has the plaintext. Name three mitigations — encryption at rest via the API server’s EncryptionConfiguration (ideally KMS-backed), tight RBAC on get/list/watch, and a real secret manager synced in by the External Secrets Operator (or encrypted in-repo with Sealed Secrets or SOPS). The GitOps rule: the reference lives in Git; the plaintext never does. See Secrets Management.

Security in CI/CD pipelines

☺ Like you’re 10: Check the crate before it leaves the factory, seal it with a sticker only you can print, and refuse any crate whose sticker doesn’t match.

Attackers stopped attacking production directly; they attack the pipeline that builds it. Two halves: scanning what you build, and proving what you ship. The delivery machinery itself is the next CNPA domain, hands-on in CI/CD & Progressive Delivery.

Shift left: scan early, fail fast

“Shift left” moves checks toward the developer, where fixes are cheapest. A good pipeline runs scanners for several problem classes: vulnerability scanning of dependencies and base images (Trivy, Grype), secret scanning, SAST on source, IaC and manifest scanning, and often the same policy engine you run in-cluster executed against manifests in CI, so rejection appears in a pull request rather than at deploy time. Re-scan images on a schedule: a CVE published tomorrow applies to what you built yesterday.

The supply chain: SBOM, signing, provenance

Three artefacts, three questions. An SBOM (SPDX or CycloneDX) answers “what is inside this image?” — indispensable when the next Log4Shell lands and you must know which of 400 services are affected. A signature answers “who built this, and was it altered?” — Sigstore/Cosign, whose keyless mode uses an OIDC identity, a short-lived Fulcio certificate and the Rekor transparency log, so there is no long-lived key to leak. Provenance answers “how was it built, from which commit?” — an in-toto attestation, graded by SLSA levels.

# Build → scan → SBOM → sign → attest, then verify before anything may run
trivy image --exit-code 1 --severity HIGH,CRITICAL registry.example.com/checkout:1.4.3
syft registry.example.com/checkout:1.4.3 -o spdx-json > sbom.spdx.json

# Keyless signing: OIDC identity -> short-lived cert (Fulcio) -> transparency log (Rekor)
cosign sign   registry.example.com/checkout@sha256:9f2c...
cosign attest --predicate sbom.spdx.json --type spdxjson \
              registry.example.com/checkout@sha256:9f2c...

# The gate: verify it came from OUR pipeline identity, not merely that *a* signature exists
cosign verify \
  --certificate-identity-regexp 'https://github.com/acme/.*' \
  --certificate-oidc-issuer     'https://token.actions.githubusercontent.com' \
  registry.example.com/checkout@sha256:9f2c...

Least-privilege pipelines and the GitOps advantage

The pipeline is a production system, so treat it like one: prefer short-lived federated credentials (OIDC, exchanged per run) over long-lived keys in CI variables, pin third-party actions by digest, and isolate runners so one tenant’s job can’t read another’s cache. The strongest control is structural: with GitOps, CI pushes an image and a commit while an in-cluster controller pulls — so CI never holds cluster credentials at all.

Conformance: proving the platform behaves

☺ Like you’re 10: It isn’t enough to be safe and standard — you have to be able to show it, with receipts.

“Conformance” carries two meanings here, and both appear in questions: conforming to the Kubernetes standard, and conforming to your own policies and regulations with evidence.

Kubernetes conformance and portability

The CNCF runs the Certified Kubernetes Conformance Program: a distribution earns the mark by passing a defined suite of end-to-end tests (commonly run with Sonobuoy) and may then use the Kubernetes name. The point is portability — a conformant cluster exposes the same core APIs everywhere, so workloads and platform tooling move between clouds without rewriting. That is what makes a multi-cloud reference architecture credible.

Audit logs and compliance evidence

The kube-apiserver audit log records who did what, when, and whether it was allowed. An audit policy chooses detail per rule — None, Metadata, Request, RequestResponse — because logging full bodies everywhere is expensive and a leak risk. Ship the logs off-cluster, where an attacker who compromises the cluster cannot edit them. Alongside, PolicyReport resources give continuous, machine-readable evidence: an auditor gets a live answer to “are all production namespaces running restricted Pod Security?”

Runtime security — the last line

Admission control governs what is allowed to start; it cannot see what a container does five hours later. Falco, the CNCF runtime security project, watches kernel syscalls via eBPF and fires on suspicious behaviour — a shell spawned inside a container, an unexpected outbound connection, a write to a sensitive path. Pair it with read-only container filesystems and you have prevention and detection, which is the story you tell in an incident review.

Twenty percent of the exam lives in one habit: a platform that can explain itself and defend itself without a human in the loop. Next, follow the paved road into Continuous Delivery & Platform Engineering, or return to the CNPA hub. For the hands-on version, the CNPE track goes deeper in Observability and Security & Policy, with the cram sheet in Know Cold.