A Fintech's Multi-Tenant Platform

This is a composite case study, not a real, identifiable company. "Ledgerly" is a stand-in — a card-issuing and banking-as-a-service fintech built from patterns that repeat across many real regulated platforms, in and out of finance. No single company is Ledgerly, but the shape of what happened to it is real: a platform team that let a shared cluster grow from four tenant squads to forty without ever revisiting the isolation model, until a routine penetration test found a workload in one squad's namespace could read Secrets that belonged to another. This page walks through what Ledgerly changed — namespace isolation sized to actual blast radius, RBAC that stops assuming trust, NetworkPolicy that defaults to deny, and an audit trail built to answer an assessor's question without anyone touching a keyboard that week — and, honestly, what about that fix does and doesn't transfer to a platform that isn't Ledgerly.

The starting shape: one fleet, forty squads, and a growing card-data blast radius

☺ Like you're 10: More families moved into the building every month, but nobody ever went back and added the locks the first four families never needed.

Ledgerly began the way a lot of fast-growing fintechs do: four founding engineering squads sharing one production Kubernetes cluster, each with its own namespace, and a platform team small enough that "just ask in Slack" was a genuinely reasonable way to get a permission granted. Eighteen months and one Series C later, that same cluster carried workloads for over forty squads — card issuing, the ledger core, KYC and onboarding, fraud and risk scoring, partner integrations, and a long tail of internal tooling — with two of those namespaces now squarely in PCI-DSS scope for cardholder data, and the whole estate in scope for an annual SOC 2 Type II audit. The isolation model had not grown with it. Namespaces existed, but the RBAC bindings inside them had accreted ad hoc: a team lead asking for "edit access so I can stop waiting on the platform team" usually got a RoleBinding to cluster-admin, because writing a scoped Role took longer than the incident already in progress. There was no NetworkPolicy object anywhere in the cluster — traffic between any two Pods, in any two namespaces, simply worked, which nobody had actively decided so much as never gotten around to preventing.

The gap surfaced the way these gaps usually do: not from a design review, but from an external penetration test ahead of the SOC 2 renewal. The finding was blunt — a compromised service account in the internal-tools namespace, simulating a low-severity supply-chain compromise in a third-party dashboard, could enumerate and read Secret objects in the card-issuing namespace, because the cluster-admin binding handed out eight months earlier for an unrelated incident had never been revoked. Nothing in the cluster's own dashboards had flagged it — every Deployment showed healthy, every Pod was Running, and the finding was invisible to any check that only asks "is the cluster currently green." That's exactly the throughline this course's own anti-patterns page names directly: a cluster can look perfectly healthy while carrying a boundary that was never real.

Namespace isolation, sized to blast radius, not org chart

☺ Like you're 10: Not every apartment needs a bank vault door — but the one with everyone's account numbers in it definitely does.

The rebuild's first decision was refusing to pick one isolation model for all forty squads. Following the same logic Multi-Cluster & Fleet Management lays out for when a second cluster is and isn't the right call, Ledgerly's platform team mapped each squad's data classification — not its place on the org chart — onto the isolation model that actually matched its blast radius:

The discipline worth stealing isn't the specific tiers — it's the refusal to default every squad to the top row "because compliance." Ledgerly's platform lead considered a full cluster-per-tenant model for the PCI-scoped squads, the strongest rung on the multi-cluster ladder, and rejected it: the fleet-management overhead of running and patching a second control plane for two squads wasn't buying a boundary that a dedicated node pool, taints, and enforced NetworkPolicy didn't already provide for their actual threat model. That's a decision to revisit if the regulatory scope ever tightens — it is not a decision to make once and never look at again.

# Every tenant namespace gets a quota and default limits the moment
# it's created, so one noisy squad can't starve its neighbors on a
# cluster forty teams now share.
apiVersion: v1
kind: ResourceQuota
metadata:
  name: tenant-quota
  namespace: onboarding-svc
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 40Gi
    limits.cpu: "40"
    limits.memory: 80Gi
    pods: "150"
    persistentvolumeclaims: "10"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: tenant-default-limits
  namespace: onboarding-svc
spec:
  limits:
    - type: Container
      default: { cpu: "500m", memory: "512Mi" }
      defaultRequest: { cpu: "100m", memory: "128Mi" }
      max: { cpu: "4", memory: "8Gi" }
---
# The PCI-scoped tier pins its Pods to a dedicated, tainted node pool —
# no unrelated squad's workload can be scheduled onto these nodes at all.
apiVersion: v1
kind: Namespace
metadata:
  name: card-issuing
  labels:
    tenant.ledgerly.io/name: card-issuing
    tenant.ledgerly.io/tier: pci-scope
    pod-security.kubernetes.io/enforce: restricted

The full mechanics of what that pod-security.kubernetes.io/enforce label actually does — and why a namespace with no label at all defaults to the loosest setting, not the strictest — belong to RBAC & Admission Control's coverage of Pod Security Admission; this page's job is where the label sits in the bigger isolation decision, not the admission mechanics themselves.

RBAC: tenant-scoped by default, cluster-admin by exception only

☺ Like you're 10: Instead of handing every family a master key "just in case," each family gets a key that only opens their own door — and the one master key that still exists gets logged every single time it's used.

The finding that started this whole rebuild was a stale cluster-admin binding, so RBAC was where the fix had to be most disciplined. Ledgerly's new baseline, templated through the same GitOps reconciler that manages the rest of the fleet: every squad gets a Role scoped to exactly its own namespace, bound to an SSO group rather than individual users, generated from a template the moment the namespace is created — no human hand-writes a fresh RBAC object for the fortieth squad any more than for the fourth. cluster-admin is no longer granted to a human identity at all, full stop; the only path to cluster-wide write access is a break-glass ClusterRoleBinding, explicitly time-boxed and tied to an incident number, that a small controller deletes automatically once its expiry annotation passes — so "we forgot to revoke it eight months later" stops being a mode of failure the model even allows.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: onboarding-svc
  name: onboarding-svc-engineer
rules:
  - apiGroups: ["", "apps", "batch"]
    resources: ["pods", "deployments", "services", "configmaps", "jobs", "pods/log"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]   # never create/delete by hand — GitOps owns that path
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: onboarding-svc
  name: onboarding-svc-engineers
subjects:
  - kind: Group
    name: "oidc:eng-onboarding-svc"     # SSO group, not individual users
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: onboarding-svc-engineer
  apiGroup: rbac.authorization.k8s.io
---
# Break-glass: cluster-wide, but time-boxed, incident-tagged, and
# auto-revoked — never a standing grant a person has to remember to undo.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: break-glass-oncall-2026-03-14
  annotations:
    ledgerly.io/expires-at: "2026-03-14T18:00:00Z"
    ledgerly.io/incident: "INC-4821"
subjects:
  - kind: Group
    name: "oidc:break-glass-oncall"
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: edit
  apiGroup: rbac.authorization.k8s.io

Verifying a binding before it ships, rather than after a pen test finds it, is exactly the impersonation check RBAC & Admission Control covers in full: kubectl auth can-i get secrets -n card-issuing --as=oidc:eng-onboarding-svc answers "can this group actually reach what I think it can't" in one command, and Ledgerly's platform team now runs exactly that check as a CI gate on every RBAC pull request, not as a one-off audit exercise months later.

NetworkPolicy: default-deny first, explicit allow for the few real crossings

☺ Like you're 10: Every apartment gets soundproof walls on move-in day. If two families genuinely need a shared doorway between their units, they ask for exactly one door, in exactly one spot — not a wall with no wall at all.

RBAC governs who can ask the API server for what; it has no opinion at all about which Pods can talk to which Pods over the network, which is a completely separate question Networking & the CNI covers in full. Ledgerly's cluster ran on Calico, which does enforce NetworkPolicy — worth checking explicitly, since a policy applied on top of a CNI that doesn't enforce it (plain Flannel, notably) is accepted by the API server and then silently does nothing, with no error telling anyone it was ignored. Every tenant namespace now gets a default-deny policy the moment it's created, both directions, and the only traffic allowed back in is a small number of explicitly named, explicitly scoped crossings between squads that genuinely need to talk to each other:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: card-issuing
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
# The one real crossing this tier needs: card-issuing calling the
# ledger's write API, on one port, from one labeled workload — nothing else.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-card-issuing-to-ledger-write-api
  namespace: ledger-core
spec:
  podSelector:
    matchLabels: { app: ledger-write-api }
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels: { tenant.ledgerly.io/name: card-issuing }
          podSelector:
            matchLabels: { app: card-issuer }
      ports:
        - protocol: TCP
          port: 8443

Building the first default-deny policy for a live namespace, rather than a fresh one, is exactly where the workshop below earns its twenty minutes — a policy that looks complete on paper always misses at least one crossing a workload actually depends on, and finding that gap in a test environment beats finding it in card-issuing at 2 a.m.

Audit logging: evidence as a query, not a fire drill

☺ Like you're 10: Instead of scrambling to remember who visited last month right before the inspector shows up, the building just keeps a logbook every single day — so there's nothing left to reconstruct.

A control that isn't provable is, to an assessor, indistinguishable from a control that doesn't exist. Ledgerly's audit design follows the same shape the CKS blueprint covers at the mechanics level — a kube-apiserver Policy resource wired in via --audit-policy-file, with rules evaluated first-match-wins — tuned specifically to the two PCI-scoped namespaces so the log captures what actually matters without drowning the platform team in routine GET traffic:

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages: ["RequestReceived"]
rules:
  # Full request + response for anything touching a Secret in
  # PCI-scoped namespaces — this is the row the pen-test finding lives in.
  - level: RequestResponse
    namespaces: ["card-issuing", "ledger-core"]
    resources:
      - group: ""
        resources: ["secrets"]

  # Every RBAC change, cluster-wide — segregation-of-duties evidence,
  # not scoped to PCI namespaces because a binding anywhere can reach them.
  - level: RequestResponse
    resources:
      - group: "rbac.authorization.k8s.io"
        resources: ["rolebindings", "clusterrolebindings", "roles", "clusterroles"]

  # Metadata only for routine object changes in the PCI tier — who and
  # when, without logging every field of every Pod spec.
  - level: Metadata
    namespaces: ["card-issuing", "ledger-core"]
    resources:
      - group: ""
        resources: ["pods", "configmaps"]

  # Drop the noisiest, lowest-value traffic before it ever hits the log.
  - level: None
    users: ["system:kube-scheduler", "system:kube-controller-manager"]
    verbs: ["get", "watch", "list"]

  # Catch-all for everything else, cheap and low-detail.
  - level: Metadata

Rule order matters here exactly as much as it does in an admission chain — the specific Secrets and RBAC rules have to come before the catch-all, or the catch-all's broader level would apply to them first and the fine-grained rows above would never fire. Logs ship off-cluster within seconds to a write-once store, deliberately, so an attacker who compromises the cluster itself can't also edit their own trail after the fact — the same evidence-design principle Platform Engineering's governance & compliance coverage puts as "evidence has to be a by-product of normal operation, never a task someone remembers to do before an audit." The day Ledgerly's SOC 2 assessor asked "show me every access to a Secret in the card-issuing namespace for the last quarter," the answer was a query against that store, not a week reconstructing it from memory and Slack threads.

What changed, honestly

☺ Like you're 10: The locks got real, the walls got soundproof, and the logbook stopped needing anyone to fill it in by hand.

Because Ledgerly is a composite, there is no single published metric to quote here — treat what follows as the shape of outcome that recurs across platforms doing this kind of rebuild, not a specific verified number. Four shifts are the ones worth internalizing:

• The finding that started this became structurally harder to repeat. A stale cluster-wide binding can't quietly survive eight months any more, because cluster-wide bindings no longer exist outside a time-boxed, auto-revoked exception path.
• A compromised low-trust workload stopped being a path into a high-trust one. The same internal-tools compromise the pen test simulated now hits a default-deny wall at the network layer before it ever reaches RBAC's decision at all — two independent controls, not one.
• Audit prep stopped being a fire drill. The SOC 2 assessor got a queryable, write-once log instead of a week of manual reconstruction, the same shift Northbridge's case file describes in a different regulated setting.
• Onboarding a new squad got faster, not slower. A namespace now arrives with its Role, its default-deny policy, and its quota already templated in — a new squad's first PR ships without anyone hand-writing RBAC for them, which was never true under the old ad hoc model.

None of that came from adding friction Ledgerly's engineers had to route around. It came from making the safe default cheaper to use than the workaround — the same reframing this platform's own best-practices & operating model page argues for generally, applied here specifically to tenancy.

What to steal, and what not to copy blindly

☺ Like you're 10: Even if your platform isn't a bank, the same tricks make any shared cluster safer — but a few things about Ledgerly's story won't be true for everyone.

Worth taking: size isolation to data classification, not org-chart position or fear — most of Ledgerly's forty squads never needed more than a namespace, RBAC, and a NetworkPolicy, and paying for a dedicated node pool or a second cluster everywhere would have bought nothing. Default every new tenant to the narrow, safe configuration and make the exception the thing that's loud, logged, and time-boxed — not the other way round. And design evidence as a query from day one: if proving a control held requires a human to do work the week before an audit, the control isn't actually automated yet, whatever the dashboard says.

Worth reading with real caution: no metric in the "what changed" section above is a verified, published number — this is a composite built to show the shape of the fix, not a citation. This also depends on genuine platform-team authority to templatize RBAC and revoke standing cluster-admin grants; if individual squads can opt out of the default-deny rollout "just for now," the model degrades back toward Ledgerly's original state one exception at a time. A dedicated node pool and a write-once audit pipeline both carry a real, ongoing operational cost — reaching for them because "we're regulated" without a specific compliance boundary that demands it is its own kind of waste. And a pen test finding the gap, rather than a design review catching it first, is itself worth noticing: Ledgerly got lucky that the finding came from a scheduled test and not a real incident. For the deeper mechanics behind every control in this case file — the exact admission chain, the full Pod Security Standards, and the CNI-enforcement caveat on NetworkPolicy — see RBAC & Admission Control, Security: Defense in Depth, and DevSecOps' Kubernetes Security Deep Dive. For the org-design side of running policy as code at this scale, Platform Engineering's regulated-enterprise case file and its underlying security & policy enforcement coverage go considerably further than this page does — worth reading next if RBAC and NetworkPolicy alone won't cover your compliance surface. And if the next step for you is proving this depth of Kubernetes security on paper, the CKS blueprint is the certification this case file's controls map onto most directly, with the wider CNCF ladder covered in the Golden Astronaut course.