Capstone Part 5 — Capacity Plan & Load Test

There are two honest ways to answer "how much traffic can checkout-api handle": do the math from data you already have, or push real traffic at it until it breaks. Good capacity planning refuses to pick one — it does both, and treats any gap between them as the finding. This part forecasts checkout-api's November peak from nine months of real traffic history, then points a k6 sweep at the same service, finally running somewhere closer to how the fact table in Part 1 always described it. Along the way it also closes the first of the two action items Part 4's postmortem left open — the k6 concurrency test that was due, by name, before this page — and it finds a second ceiling waiting quietly behind the first one, in the exact ten-connection pool that caused Part 4's incident in the first place. Reconciling all of that, not the forecast alone and not the load test alone, is what this page is actually about.

Where Part 5 picks up

☺ Like you're 10: A quick recap table so you're not hunting back through four earlier pages for the numbers this one needs.

Everything below assumes the exact artifacts the earlier parts produced, by name, not a rough paraphrase of them:

Notice the gap in that table: a 30-day average of ~1.9 req/s tells you almost nothing about the one number capacity planning actually cares about — the busiest five minutes of the busiest day. Part 4 already showed what a concurrency-shaped limit does when nobody's measured it on purpose: a config flip that was invisible at idle turned into a 920×-burn-rate incident the instant real concurrent traffic hit it. This part measures checkout-api's limits deliberately, in a controlled test, instead of finding the next one live in front of a customer. Keep your Part 2–4 docker compose stack around for reference — you're about to replatform it, not throw it away.

Standing up checkout-api on a real cluster

☺ Like you're 10: Move the exact same two containers you've been running on your laptop into a small practice Kubernetes cluster — same code, same database, a more realistic room for them to live in.

Every part so far ran checkout as a single docker compose container, while Part 1's own fact table described a real checkout-api running "behind a load balancer across two availability zones" — a gap between the lab and the story it's been telling. This part closes it, partway: not by adding redundancy yet (that's still Part 6's question), but by moving onto the kind of platform a real deployment would actually run on, since Litmus — the chaos-engineering tool Part 6 needs — only targets Kubernetes objects, not docker compose containers. Install kind and kubectl if you don't have them (Docker, which you already have from Part 2, is kind's only dependency), and stand up a small single-node cluster:

kind create cluster --name sre-dev
kubectl config current-context
# kind-sre-dev

kubectl create namespace checkout-api
kubectl config set-context --current --namespace=checkout-api

Build the exact same image Part 2 and Part 4 have been running — nothing in checkout-svc/ changes yet — and load it into the cluster (kind clusters can't pull from your local Docker daemon directly):

docker build -t checkout-svc:part5 ./checkout-svc
kind load docker-image checkout-svc:part5 --name sre-dev

Write k8s/order-db.yaml and k8s/checkout-api.yaml, translating Part 4's docker-compose.yml service-for-service — same image, same env vars, same DATABASE_URL hostname, because Kubernetes Service DNS resolves a bare order-db the same way docker compose's own DNS already did:

# k8s/order-db.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-db
spec:
  replicas: 1
  selector: { matchLabels: { app: order-db } }
  template:
    metadata: { labels: { app: order-db } }
    spec:
      containers:
        - name: order-db
          image: postgres:16
          env:
            - { name: POSTGRES_PASSWORD, value: "checkout" }
            - { name: POSTGRES_DB, value: "orders" }
          ports: [{ containerPort: 5432 }]
---
apiVersion: v1
kind: Service
metadata: { name: order-db }
spec:
  selector: { app: order-db }
  ports: [{ port: 5432, targetPort: 5432 }]

# k8s/checkout-api.yaml — replicas: 1 on purpose; see the callout below
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
spec:
  replicas: 1
  selector: { matchLabels: { app: checkout } }
  template:
    metadata: { labels: { app: checkout } }
    spec:
      containers:
        - name: checkout
          image: checkout-svc:part5
          imagePullPolicy: Never   # loaded via kind load, not pulled
          env:
            - { name: CHAOS_ERROR_RATE, value: "0" }
            - { name: DB_POOL_MODE, value: "pooled" }
            - { name: DATABASE_URL, value: "postgresql://postgres:checkout@order-db:5432/orders" }
          ports: [{ containerPort: 8080 }]
          resources:
            requests: { cpu: 250m, memory: 128Mi }
            limits:   { cpu: "1",  memory: 256Mi }
---
apiVersion: v1
kind: Service
metadata: { name: checkout-api }
spec:
  selector: { app: checkout }
  ports: [{ port: 80, targetPort: 8080 }]

Apply both manifests, port-forward the Service to the exact same local URL every earlier part already used, and confirm the migration didn't change anything about how the service behaves:

kubectl apply -f k8s/order-db.yaml -f k8s/checkout-api.yaml
kubectl rollout status deploy/checkout-api
kubectl get pods

kubectl port-forward svc/checkout-api 8080:80 &
curl -s -X POST http://localhost:8080/checkout
# {"status": "ok"}  <- same endpoint, same port, same response shape as every earlier part

Re-point Prometheus, Alertmanager, and Grafana at the new cluster too, since Part 6 needs all three already running there — reuse Part 2 and Part 3's exact rule files and Alertmanager config as-is, mounted as ConfigMaps, rather than rewriting a single threshold:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
kubectl create configmap checkout-burn-rate-rules --from-file=prometheus/checkout-burn-rate.rules.yml
kubectl create configmap checkout-alertmanager-config --from-file=alertmanager/alertmanager.yml

helm install monitoring prometheus-community/kube-prometheus-stack \
  --set additionalPrometheusRulesMap.checkout-burn-rate.configMap=checkout-burn-rate-rules \
  --set alertmanager.alertmanagerSpec.configSecret=checkout-alertmanager-config

kubectl get pods -l release=monitoring

Confirm the exact same four alert names from multi-window, multi-burn-rate alerting are loaded, unchanged, before moving on — a threshold that silently didn't survive the port is worse than an obvious one that never deployed at all:

kubectl exec -it deploy/monitoring-kube-prometheus-prometheus -- \
  wget -qO- http://localhost:9090/api/v1/rules | grep -o '"name":"Checkout[A-Za-z]*"'
# CheckoutErrorBudgetBurnFast, Moderate, Slow, Slowest — all four, same names as Part 2

The traffic dataset

☺ Like you're 10: Nine-ish months of "how busy did checkout-api get on its worst five minutes each month" — the file you're handed to forecast from.

Save this as reliability-capstone/capacity/traffic-history.csv. It's the peak five-minute request rate checkout-api actually served each month, pulled from the same Prometheus your dashboard already queries, plus the p99.5 latency observed during that peak window:

month,peak_req_s,p99_5_ms,notes
2025-09,9.1,58,
2025-10,9.4,59,
2025-11,28,268,Cyber Week 2025 (Nov 24-30) - closer to the line than anyone realized at the time
2025-12,10.2,60,
2026-01,10.6,61,
2026-02,11.1,62,
2026-03,11.5,63,
2026-04,12.0,64,
2026-05,12.5,65,
2026-06,31,382,Flash-sale email blast Jun 19 (~11 min) - first time p99.5 actually crossed 300ms
2026-07,13.5,67,
2026-08,14.0,68,through Aug 16 (partial month - today's data pull)

Two rows don't belong in a trend line, and spotting them is part of the exercise: November and June are both single, short-lived events — a planned promotion and an unplanned email blast — not organic growth. Fit a trend to the other ten months only. Mixing an event spike into a growth-rate calculation is a common, quiet way to badly overstate how fast "normal" traffic is actually growing.

Fitting the trend and forecasting November's peak

☺ Like you're 10: Figure out how fast an ordinary month is growing, then use that same rate to guess what an ordinary November would look like — before layering the promotion on top.

Excluding the two event months, growth from September (9.1) to August (14.0) — eleven months later — is compounding, not flat. Solve for the monthly rate the same way you'd solve for compound interest:

growth factor over 11 months = 14.0 / 9.1 = 1.538
monthly rate = 1.538^(1/11) - 1 ≈ 0.040   → ~4.0% / month, organic

Project that same 4.0%/month forward three months from today (August) to an ordinary November — what the month would look like with no promotion running at all:

ordinary Nov 2026 baseline = 14.0 × 1.040^3 = 14.0 × 1.1249 ≈ 15.7 req/s

Now the event adjustment, sourced from the team running it rather than derived from the trend line — exactly the distinction capacity planning & performance draws between organic growth and known future events. Last year's Cyber Week (28 req/s) landed against an ordinary-November baseline of about 9.8 req/s (interpolating one month past October's 9.4), so the promotion realized roughly a 2.9× multiplier over ordinary traffic. Marketing's plan for this year adds two new partner storefronts to the same campaign and states their own expectation directly: roughly 3.3× this time, not a number this page derives, a number marketing owns.

forecast Nov 2026 peak = 15.7 req/s (ordinary baseline) × 3.3 (Cyber Week multiplier)
                        ≈ 52 req/s

Fifty-two requests per second is the forecast. It is not, on its own, the number you provision against — see the next section for why, and read Drill — Forecast the Bottleneck first if you want more reps on this exact trend-plus-multiplier method in isolation before combining it with a live load test.

Turning the forecast into a required-capacity number

☺ Like you're 10: Add a safety margin on top of the guess, because a guess that's exactly right on paper still fails the moment reality is one percent worse than expected.

A forecast peak is not a provisioning target — it's the traffic you expect on an ordinary version of the busiest day, and it accounts for neither normal demand volatility (the forecast being a little low) nor a capacity-side loss (a replica dying mid-surge, precisely the question Part 6 will finally ask on purpose). Capacity planning & performance gives the standard fix: hold 30–50% headroom above the forecast. A mid-range 40% is what this page uses:

required capacity = forecast peak × (1 + headroom)
                   = 52 req/s × 1.40
                   = 72.8 req/s   →  round UP, never down: 73 req/s

checkout-api needs to sustain at least 73 req/s without breaching its 300ms p99.5 SLO by the time Cyber Week 2026 opens. That's the target the rest of this page tests against — not 52, not "whatever it can currently do."

Finding the real ceiling: the k6 breaking-point sweep

☺ Like you're 10: Stop guessing and start throwing real, increasing traffic at the exact Deployment you just stood up, until something actually gives.

The forecast is arithmetic against historical data; it says nothing about what checkout-api's actual implementation does under load. Point k6 at the same http://localhost:8080/checkout the port-forward above already exposes, and sweep the rate upward instead of holding it steady — Locust would work here too if you'd rather script the sweep in Python instead of JavaScript, but every command below assumes k6, matching the tool this capstone's already used since Part 4. Save this as reliability-capstone/capacity/checkout-breaking-point.js:

// checkout-breaking-point.js — run once per RATE, against the kind-sre-dev cluster
import http from 'k6/http';
import { check } from 'k6';

const RATE = Number(__ENV.RATE || 15);

export const options = {
  scenarios: {
    fixed_rate: {
      executor: 'constant-arrival-rate',   // open model - see the warning below
      rate: RATE,
      timeUnit: '1s',
      duration: '3m',
      preAllocatedVUs: Math.ceil(RATE * 2),
      maxVUs: RATE * 6,
    },
  },
  thresholds: {
    http_req_duration: ['p(99.5)<300'],   // Part 1's exact latency SLO, not p95
    http_req_failed:   ['rate<0.001'],    // Part 1's exact availability budget: 99.9%
  },
};

export default function () {
  const res = http.post('http://localhost:8080/checkout');
  check(res, { 'non-5xx': (r) => r.status < 500 });
}

Leave CHAOS_ERROR_RATE at 0 for this whole part. You're not injecting synthetic errors — you're finding out what happens when normal, error-free traffic simply arrives faster than checkout-api can drain it. Concurrency alone will do that on its own; no chaos flag needed.

Sweep across the rates that bracket where a two-worker gunicorn process should theoretically start to hurt — the math for that number is worked out in the next section, but run the sweep first, blind, the way Sol would insist on:

mkdir -p reliability-capstone/capacity/results
cd reliability-capstone/capacity

for RATE in 15 20 25 28 30 33 40; do
  k6 run -e RATE=$RATE --summary-export=results/rate-$RATE.json checkout-breaking-point.js
done

# pull the two numbers that matter out of each summary
for f in results/rate-*.json; do
  echo "$f  p99.5=$(jq '.metrics.http_req_duration.values["p(99.5)"]' "$f")ms  failed=$(jq '.metrics.http_req_failed.values.rate' "$f")"
done

Check dropped_iterations in each summary before trusting any row below it. If it's nonzero, preAllocatedVUs or maxVUs ran out before the configured rate was actually sustained, and that row tested a lower rate than its label claims — the same silent-under-delivery gotcha k6 warns about.

What the sweep found

☺ Like you're 10: The exact rate where "fine" turns into "broken," measured, not guessed — and it's a lot lower than the forecast needs.

Against the 73 req/s the forecast requires, checkout-api holds its SLO cleanly only to ~25 req/s and falls over completely by ~33 req/s — a validated safe ceiling covering barely a third of what Cyber Week needs.

Closing the gap: the worker fix, the retest, and the wall behind it

☺ Like you're 10: Give checkout-api more workers, run the exact same trucks-across-the-bridge test again — and then check whether the rope holding the boards together was ever tested at all.

The fix follows directly from the diagnosis: two workers isn't a tuning choice anyone made deliberately, it's gunicorn's own conservative default sitting untouched since Part 2. The standard sizing formula gunicorn's own docs recommend is workers = (2 × cpu_count) + 1; on a typical 8-core dev machine that's 17. Edit the last line of checkout-svc/Dockerfile, rebuild, reload the image into the cluster, and roll the Deployment:

CMD ["gunicorn", "-b", "0.0.0.0:8080", "-w", "17", "app:app"]

docker build -t checkout-svc:part5-fix ./checkout-svc
kind load docker-image checkout-svc:part5-fix --name sre-dev
kubectl set image deploy/checkout-api checkout=checkout-svc:part5-fix
kubectl rollout status deploy/checkout-api

for RATE in 25 33 52 73 100 150; do
  k6 run -e RATE=$RATE --summary-export=results/fixed-$RATE.json checkout-breaking-point.js
done

By the L ÷ W math, 17 workers at the same ~60ms average should hold roughly 17 ÷ 0.06 ≈ 283 req/s before hard failure. But checkout-api doesn't only hold 17 gunicorn worker slots — it also holds Part 4's exact SimpleConnectionPool(1, 10, DATABASE_URL), still capped at 10 connections, and every request now holds that connection for the same ~60ms it holds a worker for (Part 4's code acquires it "before the simulated work, releases after"). Run the same math against the pool instead of the workers:

pool ceiling  = L ÷ W  =  10 connections ÷ 0.06s  ≈  166.7 req/s
pool "knee"   ≈ 75% of that                       ≈  125 req/s

Two fixes, not one, and both go in before this part is done. First, size the pool with the same discipline the workers just got — not "10, because that's what Part 4 shipped," but a number with real headroom under the required capacity:

_pool = SimpleConnectionPool(1, 30, DATABASE_URL)
# 30 connections: 30 ÷ 0.06 ≈ 500 req/s theoretical, ~375 req/s at the 75% knee -
# comfortable margin above the 73 req/s Cyber Week requires, and still well under
# Postgres 16's own out-of-the-box max_connections of 100, leaving room for
# migrations, psql, and a monitoring exporter alongside checkout-api itself.

Second — and this is the piece that actually closes Part 4's action item, not just today's specific number — build the CI gate that keeps testing this on every future change, not just once, today, by hand. Reuse Part 4's own chaos/checkout-load.js exactly as forward-referenced there ("the same generator, run much bigger, in Part 5"), scaled well past pool capacity with a closed-model VU count instead of an open-model rate, since a CI gate needs to be fast and deterministic, not a full capacity sweep:

// chaos/checkout-load.js — Part 4's generator, scaled for Part 5's CI gate
import http from 'k6/http';
import { check } from 'k6';

export const options = {
  vus: 60,          // well past the 30-connection pool - forces real contention
  duration: '2m',
  thresholds: {
    http_req_failed: ['rate<0.001'],   // any pool-exhaustion error trips this
  },
};

export default function () {
  const res = http.post(
    'http://checkout-api/checkout',
    JSON.stringify({ cart_id: 'cart_9182' }),
    { headers: { 'Content-Type': 'application/json' } }
  );
  check(res, { 'non-5xx': (r) => r.status < 500 });
}

# .github/workflows/checkout-concurrency-gate.yml
name: checkout connection-pool concurrency gate
on: [pull_request]
jobs:
  concurrency-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: docker compose up -d --build      # cheap, fast - Part 2's compose stack, not the full cluster
      - run: docker run --rm --network=host -v $PWD/chaos:/scripts grafana/k6 run /scripts/checkout-load.js

Notice the deliberate split: the CI gate runs against Part 2's lightweight docker compose stack, on every pull request, in about two minutes — cheap enough to run constantly. The full capacity sweep on this page runs against the higher-fidelity kind-sre-dev cluster, occasionally, before events like Cyber Week. Shift-left (the CI gate, catching a regression before it merges) and a periodic capacity validation (this page, catching whatever CI can't) are complementary, not redundant — the same relationship Part 6 draws again between its own scheduled chaos experiment and this gate. With both landed, re-verify the pool bump against the sweep one more time before calling this section closed:

docker build -t checkout-svc:part5-final ./checkout-svc
kind load docker-image checkout-svc:part5-final --name sre-dev
kubectl set image deploy/checkout-api checkout=checkout-svc:part5-final
kubectl rollout status deploy/checkout-api

k6 run -e RATE=150 checkout-breaking-point.js
# p99.5 well under 300ms, 0.00% failed, pool utilization now ~30% at this rate - real margin, not luck

Record all of it — the dataset, the trend fit, the forecast, the pre-fix sweep, both root causes, both fixes, the retest, and the CI gate — in reliability-capstone/capacity/capacity-plan.md. That document, and the CI gate itself, are what Part 6 inherits, the same way this part inherited Part 4's incident and action items.

What "done" looks like for Part 5

☺ Like you're 10: A number you can defend on paper, a number you proved on a real cluster, a second number nobody had checked before, and a test that keeps checking it from now on.

At the end of this part checkout-api runs on a real kind-sre-dev cluster with Prometheus, Alertmanager, and Grafana already there for Part 6 to find; you have a forecast (52 req/s raw, 73 req/s required with headroom); a measured pre-fix ceiling (25 req/s safe, 33 req/s hard failure) with a named root cause confirmed by Little's Law (two blocking gunicorn workers); a fix, a retest that clears the actual requirement; a second, quieter ceiling found by running the exact same math against the database pool Part 4's incident already made infamous; a pool resized with real headroom; and a CI gate — built from Part 4's own chaos/checkout-load.js — that closes action item #1 for good instead of just for today. Nothing here gets thrown away: Part 6 — Chaos Engineer It starts from exactly this state, on this cluster, and asks the harder question this page deliberately left open — does the capacity you just proved still hold once a replica disappears mid-surge, the way a real availability zone eventually will.

Part 5 gave you a real cluster, a forecast, two measured ceilings instead of one, both fixed, and a CI gate that keeps testing the one Part 4 already broke once — the exact artifacts Part 6 — Chaos Engineer It picks up next, to find out whether any of this survives a fault, not just a straight line of traffic. Or step back to Run a Reliable Service — start here for how all six parts fit together, drill the trend-and-multiplier method alone in Drill — Forecast the Bottleneck, or read the math behind both knees this whole page rests on in queueing theory for SRE.