Governance, Compliance & Tenancy at Scale

Every growing platform reaches the same crossroads: dozens of teams now share the same clusters, the auditors are asking for evidence, and the security team wants a “no” on risky changes — while the whole point of the platform was to let developers move without waiting for a “yes.” Governance is how you square that circle. Done badly, it becomes a ticket queue and a review board that people route around. Done well, it disappears into the golden path — the compliant thing is simply the easy thing, enforced by machines, inherited by every workload, and provable to an auditor without a human ever screenshotting a config. This deep dive covers the whole arc: tenancy models from soft to hard, policy-as-code at scale, compliance frameworks turned into controls-as-code, the audit and evidence trail, access governance, hardened baselines, and how to keep self-service both autonomous and safe.

Governance vs gatekeeping — guardrails, not gates

☺ Like you’re 10: A gate makes you stop and ask a grown-up every time. A guardrail just quietly keeps you on the road while you drive as fast as you like. Good governance is the second kind.

The single most important idea on this page is a distinction, not a tool. A gate is a synchronous human checkpoint: before your change lands, a person — a change-advisory board, an architecture review, a security sign-off — must say “yes.” A guardrail is an asynchronous, automated constraint: the system itself allows anything inside a safe envelope and rejects (or fixes) only what falls outside it, with no human in the hot path. Both aim at the same outcome — risk contained — but they have opposite effects on flow. Gates serialise every team behind a queue; guardrails let every team move in parallel and only stop the genuinely unsafe change.

Make the compliant path the easy path

The failure mode of governance is that “secure,” “compliant,” and “fast” get framed as a trade-off, so teams quietly choose fast and route around the rules. The way out is to collapse the trade-off: make the paved, supported golden path also the most compliant one, so doing the right thing requires less effort than doing the wrong thing. When the platform’s scaffolded service template ships with network policies, a hardened Pod spec, resource limits, and an audit-logged pipeline already wired in, a developer gets compliance for free by using the template — and has to actively work to escape it. That is the whole thesis of platform-as-product applied to governance: the control is a feature of the product, not a tax on it.

Automate policy so it never becomes a ticket queue

A rule that lives in a wiki page (“all images must come from our registry”) is a suggestion; a rule enforced at admission time is a control. The moment a policy depends on a human remembering to check it, three things happen: it is applied inconsistently, it becomes a bottleneck, and it produces no evidence. Encoding the same rule as policy-as-code fixes all three at once — it runs on every change, identically, and it emits a record. The goal is to move as many controls as possible from detective and manual (someone notices later) to preventive and automated (the platform refuses at the door), and to reserve human review only for the rare, genuinely novel decision that no policy yet covers.

The anti-pattern: governance as a review board

When governance is implemented as committees and manual approvals, it doesn’t make the platform safer — it makes it slower, and slowness breeds shadow IT. Teams spin up their own clusters, their own pipelines, their own cloud accounts precisely to escape the queue, and now you have less visibility and control than before. This is one of the classic anti-patterns: mistaking friction for safety. The cure is not to remove governance but to change its shape — from a gate a human operates to a guardrail the platform enforces — so that the safe path is also the fast path and there is nothing to route around.

Multi-tenancy models in depth

☺ Like you’re 10: Lots of teams share one big apartment building. You can give each team a room, a floor, a whole fake building inside the building, or a real separate building — the more you separate them, the safer but the more expensive it gets.

Multi-tenancy is the art of letting many teams share infrastructure without letting them see, starve, or step on each other. Kubernetes gives you a spectrum of isolation strengths, and the architect’s job is to pick the weakest isolation that meets the actual trust and compliance requirement — because stronger isolation always costs more money, more operational surface, and more developer friction. The key mental model is soft multi-tenancy (tenants are trusted teammates; the walls stop accidents and noisy neighbours) versus hard multi-tenancy (tenants may be hostile or untrusted; the walls must stop deliberate attacks and satisfy a compliance boundary).

Namespace-as-tenant: quotas, RBAC, and NetworkPolicy

The workhorse of soft multi-tenancy is the humble namespace, hardened with three companions. A ResourceQuota caps how much CPU, memory, and object count a tenant can consume, so one team can’t starve the cluster; a LimitRange supplies sane per-Pod defaults so workloads without explicit requests don’t escape the quota accounting. Role-based access control (RBAC) scoped with a RoleBinding limits who can act inside the namespace. And a default-deny NetworkPolicy stops cross-tenant traffic so team A can’t reach team B’s pods (see networking for the packet-level detail). Miss any one of the three and the isolation leaks: quotas without network policy means teams can talk to each other; RBAC without quotas means an authorised team can still take down the cluster.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: tenant-quota
  namespace: team-payments
spec:
  hard:
    requests.cpu: "20"          # cap total CPU requested by the tenant
    requests.memory: 40Gi
    limits.cpu: "40"
    pods: "150"
    count/services.loadbalancers: "2"   # stop surprise cloud LB bills
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: team-payments
spec:
  podSelector: {}               # every Pod in the namespace
  policyTypes: [Ingress]        # deny all inbound not explicitly allowed

Hierarchical Namespaces and Capsule

Flat namespaces don’t model an organisation: you want “the payments group has a policy that all its namespaces inherit,” not a hand-maintained copy in each. Two projects solve this. The Hierarchical Namespace Controller (HNC), a Kubernetes SIG project, lets a parent namespace propagate RBAC, network policies, quotas, and other objects down to child namespaces, so a team lead self-services sub-namespaces that automatically inherit the guardrails. Capsule takes a fleet approach: a Tenant custom resource groups many namespaces under one owner and enforces tenant-wide quotas, allowed registries, ingress hostnames, and node selectors — all while the tenant still talks to the single shared API server. Both let one platform team govern hundreds of namespaces declaratively instead of by copy-paste, which is the difference between tenancy that scales and tenancy that becomes a full-time job.

Virtual clusters (vCluster) and cluster-per-tenant

When soft isolation isn’t enough, you climb two more rungs. A virtual cluster (vCluster) runs a tenant their own syntactic Kubernetes control plane — its own API server and its own etcd — as a workload inside a host namespace, while the real pods are synced down to the host’s nodes. The tenant gets cluster-admin in their virtual cluster, can install their own CRDs and operators, and can’t see other tenants’ API objects — a big jump in isolation without the cost of a whole real cluster, since the nodes are still shared. At the far end, cluster-per-tenant gives each tenant a fully separate control plane and nodes: the strongest isolation, a clean compliance boundary, and total blast-radius containment — at the highest cost in money, fleet-management overhead, and the multi-cluster operational burden of keeping dozens of clusters patched and consistent.

Policy-as-code at scale

☺ Like you’re 10: Instead of a person checking every toy before it enters the playroom, a smart door reads the label and either lets the toy in, fixes it, or turns it away — the same way, every single time.

Policy-as-code turns rules into version-controlled, testable artifacts enforced by a Kubernetes admission controller — a webhook the API server calls on every create/update before the object is persisted. Two CNCF projects dominate, and a senior platform engineer should be fluent in both. OPA/Gatekeeper uses the general-purpose Rego language: extremely powerful, reusable across Kubernetes, Terraform, and APIs, but a language your team must learn. Kyverno is Kubernetes-native: policies are themselves YAML resources, so there’s no new language, and it does something Gatekeeper historically couldn’t do easily — mutate and generate resources, not just validate them.

Validate, mutate, generate — the three verbs

The power of policy-as-code isn’t just saying “no.” There are three distinct verbs, and mutation is the quiet hero. Validate rejects a non-compliant resource (“this Pod runs as root — denied”). Mutate silently fixes a resource as it’s admitted (“no runAsNonRoot set? I’ll add it”), so users get a compliant object without having to know the rule existed. Generate creates companion resources automatically (“a new namespace appeared? Stamp a default-deny NetworkPolicy and a ResourceQuota into it”). Mutating defaults are the most user-friendly control you can build, because they make misconfiguration structurally impossible rather than merely forbidden — the compliant path becomes the only path, with zero developer effort.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: baseline-hardening
spec:
  rules:
    # 1) MUTATE: inject a hardened default so users can't ship a root container
    - name: default-run-as-non-root
      match:
        any:
        - resources: { kinds: [Pod] }
      mutate:
        patchStrategicMerge:
          spec:
            securityContext:
              runAsNonRoot: true
    # 2) VALIDATE: refuse images from outside the trusted registry
    - name: only-internal-registry
      match:
        any:
        - resources: { kinds: [Pod] }
      validate:
        message: "Images must come from registry.acme.internal"
        pattern:
          spec:
            containers:
            - image: "registry.acme.internal/*"
    # 3) GENERATE: every new namespace gets a default-deny NetworkPolicy
    - name: add-default-netpol
      match:
        any:
        - resources: { kinds: [Namespace] }
      generate:
        kind: NetworkPolicy
        apiVersion: networking.k8s.io/v1
        name: default-deny
        namespace: "{{request.object.metadata.name}}"
        data:
          spec:
            podSelector: {}
            policyTypes: [Ingress, Egress]

Audit vs enforce — roll policy out without breaking prod

You never turn a new policy straight to “deny” on a live cluster — you’d block half the fleet on day one. Both engines support a two-phase rollout. In audit mode (Gatekeeper’s enforcementAction: dryrun, Kyverno’s validationFailureAction: Audit) the policy runs and records every violation as a report but admits the resource anyway, so you can measure the blast radius and let teams fix existing workloads. Only once the violation count reaches zero do you flip to enforce, where new violations are actually rejected. This audit-then-enforce ramp is how you introduce guardrails to a running platform without an outage — and the audit reports themselves become the evidence that the control exists and is being followed.

Policy libraries and exceptions at scale

At scale you don’t write policies from scratch — you adopt a library (the Gatekeeper policy library, Kyverno’s curated policies, or a Pod Security Standards bundle) and curate it. Just as important is a first-class exception mechanism: real systems always have a legitimate outlier (a legacy workload that genuinely needs a host mount), and if the only way to ship it is to weaken the global policy, your governance collapses. Kyverno’s PolicyException and Gatekeeper’s excluded-namespaces / config let you grant a narrow, named, time-boxed, audited carve-out for exactly one workload — the exception is itself declared in Git, reviewed, and visible, rather than a silent hole. Governance that can’t say a documented “yes, but only here” eventually gets bypassed entirely.

Compliance frameworks & what the platform provides

☺ Like you’re 10: A compliance framework is a big checklist of safety rules someone important made up. The platform’s trick is to build the checklist into the building, so every room passes inspection automatically.

Compliance frameworks are structured sets of controls — required safeguards — that an organisation must implement and prove. A platform engineer doesn’t need to become an auditor, but must understand what each framework governs so the platform can satisfy it centrally. The magic move is this: instead of each of fifty teams implementing “encrypt data in transit” and “restrict privileged containers” themselves, the platform implements each control once, and every workload running on it inherits the control automatically. One hardened baseline can satisfy a control across the entire fleet.

The alphabet soup — what each framework actually is

The names blur together until you anchor each to what it protects. Knowing the shape of each tells you which platform controls matter.

Controls-as-code — bake it in once, inherit it everywhere

The senior insight is that most technical compliance controls map cleanly onto platform mechanisms you already have. “Restrict privileged containers” is a Pod Security Admission label. “Encrypt data in transit” is mesh mTLS. “Enforce least-privilege access” is RBAC. “Log all administrative actions” is the Kubernetes audit log. “Encrypt secrets at rest” is etcd encryption and a KMS (see secrets & workload identity). When you express each control as code — a Kyverno policy, a NetworkPolicy, an audit policy — you get controls-as-code: the control is version-controlled, continuously enforced, and self-documenting. The framework requirement stops being a binder someone updates once a year and becomes a policy that fails loudly the moment reality drifts from it.

Audit & evidence — proving it to the auditor

☺ Like you’re 10: It’s not enough to be safe — you have to be able to show you were safe. So the platform keeps a tidy logbook of who did what, and takes regular safety photos on its own.

Compliance is a two-part claim: the control exists, and you can prove it existed continuously. The second half — evidence — is where platforms either shine or scramble. The goal is continuous compliance: instead of a frantic evidence hunt the week before an audit, the platform emits proof as a natural by-product of operating, so an auditor can be handed a live dashboard rather than a folder of stale screenshots.

Kubernetes audit logs — the who-did-what

The Kubernetes API server can emit a structured audit log of every request: who (the authenticated user or service account), what (verb and resource), when, and the response. An audit policy tunes the verbosity per resource so you capture the security-relevant events (secret access, RBAC changes, exec-into-pod) at Metadata or Request level without drowning in noise. Shipped to a tamper-evident store, this log is the backbone of nearly every framework’s “log administrative actions” control — and the first thing an incident responder or auditor asks for.

# apiserver audit policy: capture the security-relevant, skip the noise
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  # Log every access to Secrets at Metadata level (who read what, not the value)
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets"]
  # Log all changes to RBAC — the classic privilege-escalation surface
  - level: RequestResponse
    resources:
      - group: "rbac.authorization.k8s.io"
        resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
  # Don't log high-volume, low-risk read traffic
  - level: None
    verbs: ["get", "list", "watch"]
    resources:
      - group: ""
        resources: ["events", "endpoints"]

PolicyReports and SBOM inventories

Two more evidence sources come from the layers you’ve already built. Policy engines emit PolicyReports (a shared CNCF custom-resource format) — a live, queryable record of which resources pass or fail which policy, right there in the cluster. And a software bill of materials (SBOM) per image gives you an inventory of every component you run, so when the next critical CVE lands you can answer “are we affected, and where?” in minutes instead of days. Together these turn “are we compliant right now?” from a research project into a query.

Continuous compliance scanning — Kubescape and kube-bench

Finally, scanners check the cluster against published benchmarks on a schedule. kube-bench runs the CIS Kubernetes Benchmark — the canonical hardening checklist for the control plane, etcd, kubelet, and node config — and reports pass/fail per line item. Kubescape scans workloads and cluster config against multiple frameworks at once (NSA/CISA hardening, MITRE ATT&CK, CIS) and produces a risk score and remediation guidance, and it can run continuously in-cluster or as a CI gate. Wire these into a scheduled job feeding a compliance dashboard and you have exactly what an auditor wants: dated, framework-mapped, automatically generated evidence that the controls hold, day after day.

Access governance — who can do what, and proving it

☺ Like you’re 10: Everyone gets a name badge that opens only the doors they truly need — and once in a while a grown-up checks that nobody’s badge opens doors they stopped needing months ago.

Access governance is the discipline of granting the least privilege necessary, tying it to real identities, and continuously proving that the grants still make sense. It’s the control auditors probe hardest, because over-broad access is the root of most breaches and the easiest thing to let rot.

RBAC at scale and ClusterRole aggregation

Kubernetes RBAC is the primitive: Role/ClusterRole define permissions, RoleBinding/ClusterRoleBinding attach them to subjects. At scale, hand-maintaining sprawling roles becomes its own risk. ClusterRole aggregation is the mechanism that keeps it sane: you define small, labelled capability roles, and an aggregated role automatically unions everything carrying a matching label. Grant a team the aggregated role, and adding a new capability later is a matter of labelling one small role — the grant updates itself, with no risk of a giant edited-by-hand role quietly accumulating * permissions.

# An aggregated role that unions any ClusterRole labelled as tenant-developer.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: tenant-developer
aggregationRule:
  clusterRoleSelectors:
    - matchLabels: { rbac.acme.io/aggregate-to-developer: "true" }
rules: []          # intentionally empty — filled by aggregation
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: developer-deployments
  labels: { rbac.acme.io/aggregate-to-developer: "true" }
rules:
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]

SSO/OIDC and groups, not users

Never bind roles to individual humans — bind them to groups, and let your identity provider own group membership through OIDC/SSO. The cluster trusts an OIDC token; the token carries the user’s groups; RBAC binds to the group. Now the entire joiner-mover-leaver lifecycle lives in one place: someone joins the payments team in the IdP and inherits its access automatically; they leave and lose it automatically. Binding to individual users instead means a departing engineer’s access lingers in a dozen RoleBindings nobody remembers to clean up — the classic audit finding. Groups-not-users is the difference between access that self-maintains and access that silently rots.

Just-in-time and break-glass access

Standing privileged access is a liability even for the people who need it occasionally. Just-in-time (JIT) access grants elevated rights only when needed, for a bounded window, with an approval and a logged reason, then revokes automatically — so nobody carries cluster-admin around all day. Break-glass is the emergency variant: a rarely-used, heavily-audited path to elevated access for a real incident, loud by design (it pages the security team and writes a prominent record) so that using it is always noticed and reviewed after the fact. Both shrink the window in which a compromised credential is dangerous, and both generate exactly the evidence an auditor wants to see.

Periodic access review

Grants that made sense a year ago accumulate into a mess of over-privilege — the phenomenon auditors call “access creep.” Periodic access review (recertification) is the scheduled control that fights it: on a cadence, owners re-confirm that each grant is still needed, and anything unconfirmed is revoked. Because your access is defined declaratively — RBAC in Git, group membership in the IdP — the review can be largely automated: diff the current grants against what’s justified, surface the deltas, and route only the exceptions to a human. This closes the loop that makes least-privilege actually stay least over time.

Secure & compliant baselines

☺ Like you’re 10: Instead of hoping each team remembers every safety rule, you hand them a room that already has the padded walls, the locked cupboards, and the smoke alarm built in.

A baseline is the hardened default state every workload and namespace starts from — the concrete embodiment of “make the compliant path the easy path.” Rather than documenting what secure looks like and hoping teams comply, you ship secure as the starting point, so a team has to actively opt out of safety rather than opt in.

Pod Security Standards and Pod Security Admission

The Pod Security Standards define three tiers — privileged (unrestricted), baseline (blocks the known-dangerous, like host namespaces and privileged containers), and restricted (the hardened best-practice profile: non-root, no privilege escalation, seccomp, dropped capabilities). Pod Security Admission (PSA) is the built-in enforcer, configured with a single label on each namespace. It’s deliberately coarse — for anything richer than the three tiers you reach for Kyverno or Gatekeeper — but it’s the zero-dependency floor every cluster should set. Applying enforce=restricted to tenant namespaces means a Pod that tries to run as root is simply refused at admission, no external controller required.

# Pod Security Admission via namespace labels — no extra controller needed.
apiVersion: v1
kind: Namespace
metadata:
  name: team-payments
  labels:
    # Reject anything that violates the hardened "restricted" profile...
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    # ...and surface violations against a stricter bar in audit + warnings
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

CIS Benchmarks and kube-bench

Where Pod Security governs workloads, the CIS Kubernetes Benchmark governs the cluster itself — the API server flags, etcd permissions, kubelet configuration, and node hardening that a Pod-level control can’t reach. It’s the community consensus on “what does a hardened cluster look like,” and kube-bench is the tool that checks a running cluster against it line by line. Managed control planes (EKS, GKE, AKS) handle much of the control-plane section for you, but the node and kubelet checks are still yours — and running kube-bench on a schedule turns the benchmark from a document you read once into a control you continuously verify.

Golden namespace and config templates

The baseline becomes real through templates. A golden namespace template bundles the whole starter kit — the PSA labels, a default-deny NetworkPolicy, a ResourceQuota and LimitRange, the tenant RBAC bindings, and the standard policy exceptions — into one declarative unit that the platform stamps out whenever a team is onboarded. Combined with the generate policies from earlier, this means “create a new tenant” provisions a fully hardened, fully governed environment in seconds, identically every time. The golden path and the compliance boundary become the same artifact, which is exactly the outcome this whole page has been building toward.

Governing self-service safely

☺ Like you’re 10: Kids can pick and build their own rides — but only from safe pieces, only so many, and never off the edge. Freedom inside a fence.

The final synthesis: self-service and governance are not opposites — governance is what makes self-service safe enough to offer. Without guardrails, self-service means handing developers a loaded cluster and hoping; with them, it means handing developers real autonomy inside a fence they can’t accidentally cross. The platform’s job is to make the fence invisible when you’re inside it and impassable when you approach the edge.

Safe defaults on every golden path

Every self-service action — “new service,” “new database,” “new environment” — must carry its guardrails with it, not bolt them on later. The scaffolded output is already hardened: the generated manifests set non-root and resource limits, the created namespace already has its network policy and quota, the pipeline already logs to the audit trail. Because the safe configuration is the default the golden path produces, autonomy never has to mean insecurity — a developer moving fast on the paved road is compliant without a single deliberate act, which is the entire point.

Quotas and policy on every tenant, automatically

The mechanisms are the ones assembled across this page, composed into a single onboarding flow. When a tenant is created, the platform generates their namespace with PSA labels, stamps in a ResourceQuota and default-deny NetworkPolicy, binds the aggregated tenant RBAC role, and registers them with the policy library and compliance dashboard — all declaratively, all in Git, all reviewed through GitOps. The tenant then self-serves freely inside that envelope: they can deploy anything the policies allow, scale within their quota, and reach only what their network policy permits. Autonomy and control stop being a trade-off because they’re produced by the same act.

Governance, done this way, is the quiet layer that lets everything else in the course scale: the self-service portal can be generous because the guardrails are firm, the GitOps pipeline can move fast because the policies catch mistakes, and the security controls hold across a thousand tenants because they’re inherited, not re-implemented. It’s the difference between a platform that grows and a platform that becomes a bottleneck.