Flux — Multi-Tenancy & Security

A single Flux installation can serve twenty teams from twenty repositories — but only if you turn on the mechanisms that stop team A’s manifests from creating a ClusterRoleBinding, reading team B’s Secrets, or quietly pointing at a source nobody signed. This page is about those mechanisms: the tenant as a namespace with its own source and reconciler, ServiceAccount impersonation as the one control that actually bounds a tenant, the lockdown flags that close the cross-namespace side door, the supply-chain verification that decides whether a source is trustworthy at all, and the notification plumbing that makes sure a broken tenant is noticed by a human rather than discovered in a quarterly audit.

What it is and the problem it solves

☺ Like you’re 10: One helper, many classrooms — so the helper has to borrow each classroom’s key instead of using the master key.

This page assumes you already know what Flux is. If the words GitRepository, Kustomization and source-controller are new, read the parent overview at Flux first; it covers the GitOps Toolkit, the reconciliation loop, and the CLI. What follows is the part that overview deliberately leaves alone: how one Flux installation safely serves many independent teams, and how it decides that the config it is about to apply came from somewhere legitimate.

The default install is one enormous blast radius

Here is the uncomfortable fact that every multi-tenancy conversation starts from. A default flux bootstrap creates a ClusterRoleBinding that binds the built-in cluster-admin ClusterRole to the kustomize-controller and helm-controller ServiceAccounts in the install namespace. That is not a mistake — those two controllers have to be able to apply arbitrary Kubernetes objects, including CRDs, namespaces and RBAC, because that is literally the job. But it means that, by default, anything that reaches a Kustomization is applied with cluster-admin.

Follow that through. If a team can commit to a repository that a Kustomization points at, they can commit a ClusterRoleBinding that makes their own ServiceAccount cluster-admin, and Flux will apply it without a murmur. No alert fires. Nothing turns red. The Kustomization reports Ready: True, because from the controller’s point of view everything worked perfectly. That is the single most important thing to understand on this page, and it is the reason the rest of it exists.

What a “tenant” actually is in Flux

Flux has no Tenant CRD and no tenancy layer of its own. This is a deliberate design choice and it is the sharpest contrast with Argo CD, which invented the AppProject as a first-class tenancy object with its own RBAC grammar. Flux instead composes tenancy out of primitives Kubernetes already has, which means there is nothing new to learn and nothing new to trust — but also nothing that will stop you if you simply don’t configure it.

Concretely, a Flux tenant is four ordinary objects plus one field:

Soft tenancy, not hard tenancy — say so out loud

Flux’s own documentation is refreshingly honest about this and you should be too. What this page describes is soft multi-tenancy: many tenants on one cluster, separated by namespaces, RBAC and impersonation. It is a real boundary against accidents and ordinary misuse, and it is what almost every platform team actually runs — but it is not a boundary against an attacker who already has code execution inside a tenant pod, because that attacker now shares a kernel, a node, a CNI and a control plane with everyone else. Hard multi-tenancy means separate clusters, and Flux’s answer there is not a feature but a shape: a Flux per cluster, the fleet managed from a repository, which is where Cluster API and the multi-cluster deep dive take over. Stating that distinction cleanly is worth more in an exam scenario than memorising any flag on this page.

Where it fits in a platform

☺ Like you’re 10: Flux guards the door marked “what can this team ask Kubernetes to do?” — three other guards watch three other doors.

Multi-tenant Flux sits in the delivery layer of the platform architecture, but the boundary it enforces is an authorization boundary, not an isolation boundary. That distinction decides which tool you reach for when someone asks “can tenants hurt each other?”, and it is exactly the kind of question the governance conversation turns on.

Four doors, four guards

Read this table as a checklist for any shared cluster. Flux owns exactly one row. A platform that has done the Flux row and skipped the others has a tenancy story that looks good in a diagram and fails in practice — the classic shape catalogued in Anti-Patterns.

The last column is the honest bit. Impersonation cannot stop a tenant whose allowed Deployment is running a compromised image — that is why the second half of this page is about verifying where config and images came from, and why Sigstore & cosign and Trivy are neighbours rather than optional extras.

Its neighbours on the paved road

Upstream, CI builds and signs artifacts and Flux verifies those signatures before using them. Beside Flux, External Secrets is the alternative to in-repo SOPS when you want an external store as the source of truth — Secrets Management makes that call, and this page covers only the Flux-side mechanics. Downstream, Flagger rolls a tenant’s workload out progressively; because it is a cluster-wide controller with its own permissions, a tenant’s Canary is one more object your admission policy should have an opinion about. For onboarding, Backstage turns “create a tenant” into a form that raises a pull request against the platform repository — self-service in its most literal form.

The siblings carry the rest of the Flux surface: Flux — Helm & OCI Delivery and Flux — Image Automation. Both touch tenancy — a HelmRelease takes the same spec.serviceAccountName, and an ImageUpdateAutomation holds write credentials to a tenant’s repository — so this page states the tenancy rule and those pages state the mechanism.

CNPE domain relevance

This material straddles two domains of the exam blueprint. The reconciliation half belongs to GitOps & Continuous Delivery; the impersonation, RBAC and verification half belongs to Security & Compliance, and overlaps the Security & Policy lesson. That double weighting is why tenancy questions are good value to study: the same fact — “a Kustomization without spec.serviceAccountName applies as the controller” — can be asked as a delivery question or as a security question, and it is the answer either way.

How it works — impersonation, scoping and verification

☺ Like you’re 10: Before Flux applies your file, it puts on your name badge, checks the wax seal, and refuses to look in other people’s cupboards.

Three independent mechanisms combine to make a tenant boundary. They fail independently too, which is why the troubleshooting section later is organised by symptom: each mechanism has its own characteristic error string.

Impersonation — the one mechanism that matters

When a Kustomization sets spec.serviceAccountName: flux, kustomize-controller does not apply the built manifests with its own credentials. It builds a Kubernetes client configured to impersonate system:serviceaccount:<kustomization-namespace>:flux, and every create, patch, update and delete in that reconciliation is authorised as if the tenant’s ServiceAccount had issued it. The API server does the authorization; Flux is not making a policy decision at all, it is simply declining to use its own power.

Two consequences follow, and both are exam-shaped. First, the named ServiceAccount must exist in the same namespace as the Kustomization — you cannot impersonate an identity from another namespace, which is why tenant Flux objects live in the tenant’s namespace rather than in flux-system. Second, for impersonation to be permitted at all, the controller’s own identity needs the impersonate verb on the serviceaccounts resource. With the default cluster-admin binding that is satisfied automatically; if you strip that binding as a hardening step, you must grant impersonate explicitly or every tenant reconciliation fails at once.

HelmRelease takes the same field and behaves the same way: helm-controller impersonates the named ServiceAccount for the install, upgrade, rollback and uninstall actions, so the chart’s rendered objects are subject to the tenant’s RBAC exactly as a Kustomization’s would be.

Which identity actually applies — the precedence chain

Three settings decide the identity, and they are evaluated in a fixed order. Knowing the order is what lets you answer “why did this apply succeed when I expected it to be forbidden?” in one step instead of five.

Row 2 is the single highest-value hardening change on this page, and it is one line in a patch. Setting --default-service-account=flux on kustomize-controller and helm-controller converts the failure mode from silent over-privilege into a loud, obvious error: any Kustomization in a namespace without a flux ServiceAccount stops reconciling and says so. You want your safety mechanisms to fail noisily, and this one does.

Cross-namespace references and why a hardened install forbids them

By default, several Flux fields accept a namespace alongside a name. A Kustomization in namespace apps can set spec.sourceRef.namespace: flux-system and reconcile the platform team’s source. An Alert in one namespace can subscribe to events from another. An ImagePolicy can reference an ImageRepository next door.

That flexibility is convenient on a single-team cluster and corrosive on a shared one, for a reason worth stating precisely: Kubernetes RBAC has no way to express “only namespace X may reference this object.” RBAC controls who may read or write a GitRepository; it says nothing about who may point at one. So a tenant who can create a Kustomization in their own namespace can consume any source anywhere in the cluster — including private repositories whose credentials they were never given, because the credential is used by source-controller, not by them.

The fix is a controller flag, --no-cross-namespace-refs=true, which makes every reference resolve strictly within the referring object’s own namespace. Set it on the controllers that resolve references: kustomize-controller, helm-controller, notification-controller and image-reflector-controller. Once it is on, the fields still exist in the schema but populating them makes the object fail with a clear message rather than silently working.

Its companion flag is --no-remote-bases=true on kustomize-controller, which forbids a kustomization.yaml from listing a remote URL in its resources. A remote base is config fetched outside the source-controller artifact chain — so it is never signature-verified, never checksummed into the artifact, and never visible in flux diff. On a tenant cluster that is a hole straight through everything else on this page, and closing it costs nothing except telling tenants to vendor what they need into their own repository.

Verification — deciding a source is trustworthy at all

Impersonation answers “what may this config do?” Verification answers the prior question: “is this config what its author actually wrote?” Flux offers three distinct gates, all of them enforced by source-controller or kustomize-controller before anything is applied, and it is worth being precise about which artefact each one protects.

Flux’s own RBAC posture

Each controller runs with its own ServiceAccount in the install namespace, and their permissions are wildly asymmetric. source-controller, notification-controller and the two image controllers are narrow: they read and write their own CRDs and status, read Secrets and ConfigMaps for credentials, emit events and hold leader-election leases. Nothing about them needs cluster-admin. kustomize-controller and helm-controller are the broad ones, for the structural reason that a tool which applies arbitrary manifests needs arbitrary permission.

So the hardening ladder has three rungs, and most teams should climb at least two:

1. Impersonate per tenant. Set spec.serviceAccountName on every tenant Kustomization and HelmRelease. Cheap, immediate, no controller changes.
2. Make the default safe. Add --default-service-account, --no-cross-namespace-refs and --no-remote-bases so a forgotten field fails loudly instead of quietly escalating.
3. Remove the master key. Replace the cluster-admin ClusterRoleBinding with a purpose-built ClusterRole for kustomize- and helm-controller that grants the impersonate verb on serviceaccounts plus whatever the platform’s own root Kustomization genuinely needs. This is real work and it will break things the first time; do it on a non-production cluster and expect an afternoon.

The resources you will actually write

☺ Like you’re 10: A tenant is a folder of five small files, and none of them are magic.

Everything below is ordinary YAML you commit to the platform repository. There is no Tenant kind and no installer to run — the whole tenancy model is composition, which is exactly why it survives Flux upgrades so well.

The tenant itself — namespace, identity, permissions

Three objects, committed by the platform team, one file per tenant. Note what is not here: no ClusterRoleBinding, and no ClusterRole of your own invention. The RoleBinding below references a built-in ClusterRole but binds it inside a single namespace, so the grant is namespace-scoped no matter how powerful that ClusterRole is elsewhere.

apiVersion: v1
kind: Namespace
metadata:
  name: apps
  labels:
    toolkit.fluxcd.io/tenant: checkout-team
    # Pod Security Admission — the second door. Impersonation does not do this.
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flux                        # the identity Kustomizations in this namespace impersonate
  namespace: apps
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding                    # RoleBinding, NOT ClusterRoleBinding — this is the whole boundary
metadata:
  name: flux-reconciler
  namespace: apps
subjects:
  - kind: ServiceAccount
    name: flux
    namespace: apps
roleRef:
  kind: ClusterRole
  name: admin                        # built-in: full control of namespaced objects in THIS namespace
  apiGroup: rbac.authorization.k8s.io

Two deliberate choices in that file are worth defending. Binding the built-in admin rather than cluster-admin matters because a namespace-scoped binding of cluster-admin also carries the escalate and bind verbs, letting the tenant construct new RoleBindings inside their namespace referencing any ClusterRole — still namespace-bounded, but a wider surface than most teams intend. The built-in admin role deliberately withholds role escalation and quota writes, which is usually what you actually want. And the Pod Security Admission labels sit on the Namespace rather than being enforced by Flux, because Flux is not an admission controller: a tenant with namespace-admin can otherwise schedule a privileged, host-mounting pod that walks straight out of the boundary you just drew.

The flux CLI can scaffold the trio, which is a good way to learn the shape even if you end up hand-writing the file:

# Generate the Namespace + ServiceAccount + RoleBinding and print it; do not apply it.
# Read what it produces rather than trusting anyone's description of it — the generated
# names and the default ClusterRole have varied between releases.
flux create tenant checkout-team \
  --with-namespace=apps \
  --export > ./tenants/apps/rbac.yaml

# Then commit it. The tenant is created by the platform's own Kustomization on the
# next reconcile, like everything else. Onboarding is a pull request, not a ticket.
git add ./tenants/apps/rbac.yaml
git commit -m 'feat(tenants): onboard checkout-team'

The tenant’s source and reconciler — owned by the platform

These two objects live in the tenant’s namespace but are committed to the platform’s repository. That is the ownership split made concrete: the tenant changes what is inside ./deploy in their own repo; only a platform pull request can change the terms below.

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: checkout-team
  namespace: apps                    # in the TENANT namespace, not flux-system
spec:
  interval: 1m
  url: https://github.com/acme/checkout-team.git
  ref:
    branch: main
  secretRef:
    name: checkout-team-auth         # must also live in the apps namespace
  verify:                            # see "Verifying Git commit signatures" below
    mode: HEAD
    secretRef:
      name: checkout-team-signing-keys
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: checkout-team
  namespace: apps
spec:
  interval: 10m
  retryInterval: 2m
  sourceRef:
    kind: GitRepository
    name: checkout-team              # NO namespace field — same namespace only
  path: ./deploy
  prune: true
  wait: true
  timeout: 5m
  serviceAccountName: flux           # ← the impersonation. Without this line: cluster-admin.
  targetNamespace: apps              # force every namespaced object into this namespace
  decryption:
    provider: sops
    secretRef:
      name: sops-age-apps            # a key that decrypts ONLY this tenant's secrets

targetNamespace and serviceAccountName are belt and braces, and it is worth knowing why you want both. targetNamespace runs the Kustomize namespace transformer over the build, so every namespaced object lands in apps regardless of what the tenant wrote in metadata.namespace. It does nothing at all to cluster-scoped kinds — a ClusterRole has no namespace to rewrite. That is exactly the gap serviceAccountName closes, because the API server refuses the create outright. Set only targetNamespace and you have tidied the harmless objects while leaving the dangerous ones untouched.

Note also that secretRef, decryption.secretRef and postBuild.substituteFrom all resolve within the object’s own namespace. That is quietly important: a tenant cannot use postBuild substitution to lift values out of another tenant’s Secrets, and each tenant can hold a different SOPS key without any of them being able to reach the others.

A HelmRelease under the same impersonation

Everything above applies unchanged to Helm. helm-controller impersonates the same way, so a chart that installs a ClusterRole, a CRD or a webhook configuration — and a great many public charts do — will fail against a namespace-scoped tenant identity. That is correct behaviour, and it is the moment most teams discover which of their charts are cluster-scoped in disguise.

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: checkout-cache
  namespace: apps
spec:
  interval: 10m
  releaseName: checkout-cache
  serviceAccountName: flux           # same field, same meaning, same protection
  chart:
    spec:
      chart: redis
      version: "20.x"
      sourceRef:
        kind: HelmRepository
        name: bitnami                # must exist in the apps namespace under lockdown
        # namespace: flux-system     # ← rejected when --no-cross-namespace-refs is set
  install:
    createNamespace: false           # a Namespace is cluster-scoped: the tenant SA cannot make one
    remediation:
      retries: 3
  upgrade:
    remediation:
      retries: 3
      remediateLastFailure: true
  valuesFrom:
    - kind: Secret
      name: checkout-cache-values    # decrypted from SOPS by the Kustomization that applied it
      valuesKey: values.yaml

install.createNamespace: true is the field that catches people. Creating a Namespace is a cluster-scoped operation, so under impersonation it is forbidden and the release never installs — with an error that reads like a Helm problem and is actually an RBAC one. The tenant’s namespace was created by the platform’s root Kustomization long before; leave the field off. Chart sourcing, OCI charts and spec.chartRef are the sibling page’s territory — see Flux — Helm & OCI Delivery for the full treatment, including how chart verification hangs off an OCIRepository.

The bridge from SOPS to Helm values deserves spelling out, because it trips everyone once. SOPS decryption is a kustomize-controller capability: it is Kustomization.spec.decryption that turns an encrypted file into a real Secret in the cluster. The idiomatic pattern is therefore two-step — a Kustomization decrypts the file into a Secret, and the HelmRelease reads that Secret through valuesFrom. Order it with dependsOn, or the HelmRelease spends its first few intervals retrying against a Secret that does not exist yet.

Locking down the controllers

The flags live on the controller Deployments, so the GitOps-native way to set them is a Kustomize patch in the very directory flux bootstrap writes its own manifests into. Commit this and the next reconcile applies it — Flux hardens Flux, and the hardening is in the audit trail like everything else.

# clusters/prod/flux-system/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - gotk-components.yaml
  - gotk-sync.yaml
patches:
  - target:
      kind: Deployment
      name: "(kustomize-controller|helm-controller)"
    patch: |
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --no-cross-namespace-refs=true
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --default-service-account=flux
  - target:
      kind: Deployment
      name: kustomize-controller
    patch: |
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --no-remote-bases=true
  - target:
      kind: Deployment
      name: "(notification-controller|image-reflector-controller)"
    patch: |
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --no-cross-namespace-refs=true

Recent flux bootstrap releases also expose these as CLI flags, so the bootstrap command can write the patched manifests for you. Either route ends in the same committed YAML; the patch is shown here because it is the version a reviewer can read. Always confirm what actually landed with kubectl -n flux-system get deploy kustomize-controller -o jsonpath='{.spec.template.spec.containers[0].args}' — a patch whose target.name regex matched nothing fails silently, which is the worst possible outcome for a security control.

Verifying Git commit signatures

The verify block on the tenant source above makes source-controller check the OpenPGP signature on the commit before it will publish an artifact. The public keys live in an ordinary Secret in the same namespace, one armoured key per data entry; the entry names are arbitrary, the values are what matter.

# Collect the public keys of everyone allowed to author commits on this tenant's repo.
gpg --export --armor alice@acme.example > alice.asc
gpg --export --armor bob@acme.example   > bob.asc

kubectl -n apps create secret generic checkout-team-signing-keys \
  --from-file=alice.asc --from-file=bob.asc

# GitRepository.spec.verify, in full:
  verify:
    # What is verified: the commit at the tip of the checked-out ref, the tag, or both.
    # The accepted spellings changed when the source API went GA — older beta versions
    # used a lowercase "head". ALWAYS confirm on the cluster with:
    #   kubectl explain gitrepository.spec.verify.mode
    mode: HEAD
    secretRef:
      name: checkout-team-signing-keys   # Secret in the SAME namespace

Two honest caveats. A signature proves who signed, not that what they signed is correct — it is an authorship control, not a review control, and it is only as strong as the branch protection in front of it. And Flux’s spec.verify is built around OpenPGP; if your organisation has standardised on SSH-signed or certificate-based commit signing, check what your release supports before promising anyone this works.

Verifying OCI artifacts with cosign

The more modern shape is to stop pulling config from Git at reconcile time and instead publish an immutable, signed OCI artifact from CI — the same registry, the same signing story and the same digest pinning you already use for container images. Flux ships the CLI verbs for it, and Sigstore & cosign covers the signing side properly.

# In CI, after the tests pass: package the manifests as an OCI artifact, then sign it
# keylessly using the pipeline's own OIDC identity — no private key anywhere.
flux push artifact oci://ghcr.io/acme/checkout-team-config:$(git rev-parse --short HEAD) \
  --path=./deploy \
  --source="$(git config --get remote.origin.url)" \
  --revision="main@sha1:$(git rev-parse HEAD)"

cosign sign --yes ghcr.io/acme/checkout-team-config:$(git rev-parse --short HEAD)

apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: checkout-team
  namespace: apps
spec:
  interval: 5m
  url: oci://ghcr.io/acme/checkout-team-config
  ref:
    tag: latest                      # or semver: / digest: — a digest is the strongest pin
  secretRef:
    name: ghcr-pull
  verify:
    provider: cosign
    # KEYLESS: accept only artifacts signed by a specific workflow, attested by a specific
    # OIDC issuer. Both fields are regular expressions and both are anchored on purpose.
    matchOIDCIdentity:
      - issuer: '^https://token\.actions\.githubusercontent\.com$'
        subject: '^https://github\.com/acme/checkout-team/\.github/workflows/release\.yaml@refs/heads/main$'

The keyed variant swaps matchOIDCIdentity for a secretRef naming a Secret that holds a cosign.pub public key. Keyless is generally the better default, for the reason the cosign page argues at length: there is no private key to leak, rotate or lose, and the thing you pin is a workflow identity rather than a file somebody could copy onto a laptop. Later Flux releases added verification providers beyond cosign — kubectl explain ocirepository.spec.verify.provider will tell you which ones your install actually offers.

Decrypting secrets with SOPS

SOPS encrypts the values in a YAML file and leaves the structure readable, so a pull-request diff still shows which fields changed without showing what they changed to. Secrets Management compares it against Sealed Secrets and External Secrets; what matters here is the Flux-side mechanics, which are fussier than they look because the Secret key names carry meaning.

# age: generate a keypair. The PUBLIC key (age1...) goes in .sops.yaml and is committed.
# The PRIVATE key goes into the cluster and nowhere else.
age-keygen -o age.agekey

# The data key MUST end in .agekey — that suffix is how kustomize-controller recognises
# an age identity. A data key named "age.key" is silently ignored and decryption fails.
kubectl -n apps create secret generic sops-age-apps \
  --from-file=age.agekey=age.agekey

# For OpenPGP the suffix is .asc instead. For cloud KMS the well-known data keys are
# sops.aws-kms, sops.azure-kv, sops.gcp-kms and sops.vault-token — or omit the Secret
# entirely and let the controller authenticate with workload identity.

# .sops.yaml in the repo root — creation rules decide which key encrypts which path.
# This file is where per-environment and per-tenant key separation actually happens.
creation_rules:
  # Encrypt ONLY the values, never the whole document, or kustomize cannot read
  # kind/metadata and the build fails long before decryption is even attempted.
  - path_regex: tenants/apps/.*\.enc\.yaml$
    encrypted_regex: '^(data|stringData)$'
    age: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p

  - path_regex: clusters/prod/.*\.enc\.yaml$
    encrypted_regex: '^(data|stringData)$'
    kms: 'arn:aws:kms:eu-west-1:111122223333:key/2f3a0000-prod-flux'

# Encrypt in place before committing. The file is now safe to push.
sops --encrypt --in-place ./tenants/apps/db-credentials.enc.yaml

# Sanity-check what a reviewer will see: keys visible, values ciphertext, sops: block appended.
git diff --cached ./tenants/apps/db-credentials.enc.yaml

The Kustomization that applies the tenant’s path already carries the decryption block shown earlier. Because that block resolves a Secret in the Kustomization’s own namespace, giving each tenant its own key is simply a matter of creating a different Secret per namespace — and it means an encrypted file lifted from one tenant’s repository cannot be opened by another tenant’s cluster identity. Newer releases also let the controller authenticate to a cloud KMS with workload identity instead of a stored credential; kubectl explain kustomization.spec.decryption is the authority on what your install supports.

age or KMS — the key-management decision

This is the choice teams agonise over, and it reduces to two questions: who do you need to be able to revoke, and does someone need to be able to prove when a secret was decrypted?

SOPS can encrypt to several recipients at once, and the mature answer usually uses that: encrypt production files to the prod KMS key and to an offline age key kept somewhere physical, so a cloud-account catastrophe does not also mean losing every secret you own. If you take one rule from this section, take that one — the KMS-only setup is strictly more secure right up until the day it locks you out of your own cluster.

Alerting so a tenant failure is never silent

Here is the operational reality that makes notification part of tenancy rather than an afterthought. On a single-team cluster a broken reconciliation gets noticed, because the person who pushed the commit is watching for it. On a twenty-tenant cluster, a tenant whose Kustomization has been Ready: False for eleven days is invisible: the platform team’s dashboard is green because their objects are fine, and the tenant assumed it deployed. Silence is the failure mode, and it is the one that actually bites.

notification-controller answers it with three small objects, and under cross-namespace lockdown each tenant needs their own set in their own namespace. That is a feature rather than a chore — it routes each team’s failures to that team’s channel, without the platform team becoming a switchboard.

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: checkout-slack
  namespace: apps                     # same namespace as the Alert that references it
spec:
  type: slack                         # also msteams, discord, googlechat, telegram,
                                      # generic, generic-hmac, alertmanager, grafana,
                                      # opsgenie, pagerduty, sentry — plus the Git commit
                                      # status providers github / gitlab / bitbucketserver
  channel: checkout-alerts
  secretRef:
    name: checkout-slack-webhook      # data key: address
---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: checkout-failures
  namespace: apps
spec:
  providerRef:
    name: checkout-slack
  eventSeverity: error                # "info" also forwards every successful reconcile
  eventSources:
    - kind: Kustomization
      name: '*'                       # no namespace field — this namespace only
    - kind: HelmRelease
      name: '*'
    - kind: GitRepository
      name: '*'                       # catches signature-verification failures too
  exclusionList:
    - 'waiting for rollout to finish'   # regex; silence known-transient chatter
  eventMetadata:                        # newer API versions replaced the older single
    cluster: prod-eu-west-1             # "summary" string with this metadata map —
    tenant: checkout-team               # confirm with kubectl explain alert.spec

Pick eventSeverity per tenant rather than as a blanket policy: error is an on-call channel, while info also forwards every successful reconcile — a useful deploy log in a quiet tenant, unbearable in a busy one. The exclusionList regexes are the middle ground.

The third object closes the loop in the other direction. A Receiver is an inbound webhook that lets the tenant’s Git host tell Flux “new commit” instead of waiting out the interval — for the tenant that means a deploy landing in seconds instead of minutes, and for the platform it means you can raise intervals and stop twenty tenants polling the Git provider every sixty seconds.

apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
  name: checkout-webhook
  namespace: apps
spec:
  type: github                        # also gitlab, bitbucket, harbor, dockerhub,
                                      # quay, gcr, acr, nexus, generic, generic-hmac
  events: [ "ping", "push" ]
  secretRef:
    name: checkout-webhook-token      # data key: token — shared with the Git host
  resources:
    - kind: GitRepository
      name: checkout-team             # no namespace field — this namespace only

The URL you hand the Git host is not something you construct by hand: read it from kubectl -n apps get receiver checkout-webhook -o jsonpath='{.status.webhookPath}' and append it to the public address of the webhook-receiver Service. That path is derived from the token, so rotating the token changes the URL — update the Git host in the same change, or pushes quietly stop triggering and the tenant drops back to interval speed without anyone noticing. A small and extremely common own-goal.

Day-to-day commands

☺ Like you’re 10: Two habits: ask “what is red?” and ask “what could this tenant actually do?”

The flux CLI is a friendly wrapper over the CRDs and it will carry most of the day. But the commands that matter most for tenancy are kubectl ones, because the questions you are asking are RBAC questions and Flux does not answer those — the API server does. The parent Flux page has the general reconcile/suspend/trace vocabulary; what follows is the tenancy-specific layer on top.

Onboard and inspect a tenant

# Scaffold the tenant RBAC and read it before committing.
flux create tenant checkout-team --with-namespace=apps --export

# What Flux objects exist in this tenant's namespace, and are they healthy?
flux -n apps get all

# Fleet-wide sweep — the first command of any morning on a shared cluster.
flux get all -A --status-selector ready=false

# What did this Kustomization actually put in the cluster?
flux -n apps tree kustomization checkout-team

# Reverse lookup: which tenant, repo, path and commit produced this live object?
flux trace deployment/checkout -n apps

Prove the boundary holds — the commands that matter most

These are the highest-value commands on the page, and they are pure kubectl. --as makes the API server evaluate authorization as the impersonated identity and tell you the answer without you having to break anything. Run them the day you create a tenant, and again whenever anybody changes a RoleBinding.

# The single most useful check: enumerate everything the tenant identity can do.
kubectl auth can-i --list \
  --as=system:serviceaccount:apps:flux -n apps

# The three that must come back "no" on a correctly bounded tenant:
kubectl auth can-i create clusterrolebindings --as=system:serviceaccount:apps:flux
kubectl auth can-i create namespaces          --as=system:serviceaccount:apps:flux
kubectl auth can-i get secrets -n other-tenant --as=system:serviceaccount:apps:flux

# And the one that must come back "yes", or nothing will ever deploy:
kubectl auth can-i create deployments -n apps --as=system:serviceaccount:apps:flux

# Does the controller still hold the master key? (Expect a match on a default install.)
kubectl get clusterrolebinding -o wide | grep -E 'kustomize-controller|helm-controller'

# Are the lockdown flags actually on the Deployments?
kubectl -n flux-system get deploy kustomize-controller \
  -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n'

# Which Kustomizations are missing the impersonation field entirely?
kubectl get kustomizations -A \
  -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,SA:.spec.serviceAccountName'

That last one is worth putting in a scheduled job. A blank SA column on any row outside flux-system is either a deliberate exception you documented, or a tenant boundary that quietly does not exist. On a cluster with --default-service-account set the blank is harmless; without it, the blank is the finding.

Debug an impersonation or verification failure

# Why is this tenant not Ready? Conditions first, logs second.
kubectl -n apps get kustomization checkout-team -o yaml | yq '.status.conditions'
flux -n apps events --for Kustomization/checkout-team
flux logs --namespace=apps --kind=Kustomization --name=checkout-team --level=error

# Signature and decryption problems surface on the SOURCE, not on the Kustomization.
kubectl -n apps get gitrepository checkout-team -o yaml | yq '.status.conditions'

# Dry-run the tenant's change against the live cluster before it merges.
flux -n apps diff kustomization checkout-team --path ./deploy

# Verify an OCI config artifact by hand, using the same identity pin the cluster uses.
cosign verify ghcr.io/acme/checkout-team-config:abc1234 \
  --certificate-oidc-issuer='https://token.actions.githubusercontent.com' \
  --certificate-identity-regexp='^https://github\.com/acme/checkout-team/.*$'

Watch for absence, not just for errors

The controllers export Prometheus metrics, and two families of series turn the dead-man’s-switch idea from the previous section into an actual alert. Confirm the exact names on your install — curl a controller’s /metrics endpoint rather than trusting a page — then wire the rules into Prometheus and put the fleet view on a Grafana dashboard beside your delivery panels.

# Any Flux object that has been un-Ready for 15 minutes, labelled by tenant namespace.
sum by (namespace, kind, name) (
  gotk_reconcile_condition{type="Ready", status="False"}
) > 0

# The one people forget: suspended and nobody resumed it.
sum by (namespace, kind, name) (gotk_suspend_status) > 0

Gotchas and failure modes

☺ Like you’re 10: The scary bugs here are the ones where nothing turns red.

Tenancy failures split cleanly into two families, and they need opposite reactions. Loud failures — a forbidden error, a verification failure — are the system working: read the message, it is usually literally telling you the answer. Silent failures — a missing field, a patch that matched nothing, an alert nobody configured — are the dangerous ones, because the cluster looks healthy the entire time. Every item below is labelled with which kind it is.

Symptoms and their causes

This table is the one to drill. Under exam pressure — and at 3am — the useful reflex is error string to cause, not concept to definition.

Two signature traps worth memorising

The merge-commit trap: when you merge a pull request through a Git host’s web UI, the merge commit is created and signed by the host with its own key, not by anyone on your team. A GitRepository verifying the tip of main against a Secret of developer keys will therefore reject every web merge. Three ways out — add the host’s public signing key to the Secret, switch the repository to rebase or fast-forward merges so the tip is human-signed, or verify a signed tag instead of the branch tip. Choose deliberately: the first means trusting the Git host as a signer.

The mirrored-registry trap: a cosign signature lives in the registry as a separate object alongside the artifact, not inside it. Promote an artifact between registries with a plain copy — dev to prod, public to an internal mirror, an air-gap transfer — and the signature stays behind. Verification then fails on a perfectly legitimate artifact, and the natural-but-wrong reaction is to switch verification off. Use cosign copy, which brings the signature and any attestations along.

Ways a tenant escapes the namespace anyway

Impersonation bounds what a tenant may ask the API server for. It does nothing about these four, and a tenancy design that has not considered them is incomplete:

• A privileged pod. Namespace-admin includes creating Pods. Without Pod Security Admission or an equivalent, a tenant can run a privileged container with hostPath: / and read every secret on the node. This is the big one, and it is Kyverno’s job, not Flux’s.
• A kubeConfig reference. Kustomization and HelmRelease both accept spec.kubeConfig.secretRef, naming a kubeconfig for a different cluster. The control is not a flag — it is that creating a Kustomization is itself a privilege, which is another argument for keeping tenant Flux objects out of tenant-writable namespaces.
• A remote base. Without --no-remote-bases, a tenant’s kustomization.yaml can pull manifests from an arbitrary URL that no spec.verify ever saw.
• A cluster-wide operator they can talk to. If a controller watches a namespaced CR and acts with cluster-scoped power on its behalf — and many do — creating that CR is an escalation path even under perfect Flux RBAC. Audit which CRDs tenants may create, not just which core objects.

Alternatives and when to choose it

☺ Like you’re 10: Flux borrows Kubernetes’ own locks; Argo CD brought its own. Neither can turn one house into two.

The real choice is not “Flux or Argo CD” — it is “how much isolation does this actually need?”, and the honest answer sometimes rules out both.

Four tenancy models, compared

Flux against Argo CD, on tenancy specifically

The general comparison lives on the Flux overview and on Argo CD. On tenancy alone the difference is philosophical and it cuts both ways. Argo CD invented a tenancy object: AppProject restricts which repositories, destinations and cluster-scoped kinds an Application may use, and Argo’s own RBAC layer maps SSO groups to roles over projects. That is far more discoverable, and it lets a tenant see their own deployments in a browser without holding a cluster credential. The cost is that it is Argo CD’s policy layer rather than Kubernetes’ — one more authorization system to configure, audit and get right — and by default the apply itself still runs as Argo CD’s own privileged ServiceAccount. Flux takes the opposite bet: no new policy language, no new audit surface, and the apply authorised by the same API server as everything else — but nothing discoverable, nothing enforced unless you configure it, and the field you forgot is invisible. Argo CD’s risk is a misconfigured project; Flux’s risk is an unconfigured field. Know which mistake your team is likelier to make.

When one cluster is simply not the answer

Say this plainly, because it is the most valuable judgement on the page: if your tenants are mutually distrusting — different customers, different regulatory regimes, or code you did not write and cannot review — then no arrangement of namespaces and RBAC is sufficient, because they still share a kernel, a scheduler, a CNI and a control plane. That is a Cluster API and fleet problem, laid out in Multi-Cluster, and Flux is very good at it: a Flux per cluster, all reconciling from one fleet repository, is a standard and durable shape. Reserve soft tenancy for what it is actually good at — many teams inside one organisation who broadly trust each other, where the boundary exists to prevent accidents rather than attacks.

Exam relevance and going further

☺ Like you’re 10: One field, one flag, one signature check — and knowing which one fixes which problem.

Tenancy questions are unusually good value because they sit across two domains of the CNPE blueprint at once: GitOps & Continuous Delivery for the reconciliation half, Security & Compliance for the RBAC and supply-chain half. And they are practical by nature — expect to be asked to make a boundary hold, or to explain why one does not, rather than to define multi-tenancy.

⚖ CNPA vs CNPE — CNPA is fully closed-book: no allowlist, because there is nothing to allow. That makes the conceptual half of this page the part that transfers — why a delivery engine that applies as itself is a security problem, and what a namespace boundary does and does not contain. CNPE is where the specific fields and flags earn their keep, because CNPE asks you to configure things rather than describe them.

What to be able to do cold

Without notes, you should be able to: create a tenant namespace, ServiceAccount and namespace-scoped RoleBinding from a blank file; write a GitRepository and Kustomization in that namespace with serviceAccountName, targetNamespace and prune; explain in one sentence what happens if serviceAccountName is omitted; name the two hardening flags and what each one closes; say where SOPS decryption is configured and which controller performs it; name the three notification objects and which direction each one points; and read a forbidden error and say immediately whether it means the boundary is working or broken. Drill the reconciliation half in Practice: GitOps and the RBAC half in Practice: Security, keep the Command Reference beside you, and use Know Cold for the shapes you must be able to type from memory. When something is stuck, Triage: Delivery has the decision tree.

Where to go next on this site

Start from the parent Flux overview if the GitOps Toolkit vocabulary felt shaky, then take the siblings — Flux — Helm & OCI Delivery and Flux — Image Automation, the latter of which has its own tenancy question since an ImageUpdateAutomation holds write access to a tenant’s repository. For the theory underneath, the GitOps lesson covers the OpenGitOps principles and why pull-based delivery changes the credential story. Then Security & Policy and Kyverno for the admission half of the boundary, Sigstore & cosign for signing in depth, Secrets Management for the SOPS-versus-ESO decision, and Flagger for what happens to a tenant’s workload after Flux delivers it. Weighing engines rather than configuring one? Argo CD and the Tool Landscape.

Official sources — for study time, not exam time

Read these before the exam, never during it: the multi-tenancy guidance and reference implementation at fluxcd.io/flux/installation/configuration/multitenancy and github.com/fluxcd/flux2-multi-tenancy, the security documentation at fluxcd.io/flux/security, the component API references at fluxcd.io/flux/components for the exact verify and decryption schemas, SOPS itself at github.com/getsops/sops, and Kubernetes’ own RBAC and impersonation reference at kubernetes.io/docs/reference/access-authn-authz/rbac — which, unlike all the others, is on the exam allowlist and is worth knowing your way around.