Capacity planning & performance

Reliability isn't just about handling failures gracefully — it's about having enough machine, database, and network capacity to serve the traffic you actually get, before you get it. This page covers how to forecast demand, find your system's real breaking point before real users do, plan a buffer that survives spikes and partial outages, choose between scaling out and scaling up, and use performance budgets to stop regressions from shipping in the first place. By the end you should be able to reason about a capacity plan the way you'd reason about an error budget: as a number chosen deliberately, not discovered during an incident.

Forecasting demand

Capacity planning starts with a forecast, not a guess. The baseline input is historical growth: pull request-rate, storage, or active-user trends over the past several months and fit a trend line — linear for a service growing at a steady absolute rate, exponential for one compounding month over month. Most production services blend both regimes at different points in their lifecycle, so it's worth plotting the data before assuming which one applies. A forecast built purely on the trend line is necessary but not sufficient, because it only captures organic growth and misses everything the business already knows is coming.

That's why forecasts also fold in known future events: a marketing campaign with a announced launch date, a partner integration going live, a pricing change expected to shift usage patterns, or a seasonal peak like a retail Black Friday or a tax-season deadline. These events don't show up in historical data at all, so they have to be added as explicit adjustments — often expressed as a multiplier on top of the organic trend ("marketing expects a 4x spike in signups during launch week") sourced directly from the teams running those events. Good capacity planning treats the forecast as a living document, revisited on a cadence (commonly monthly or quarterly) rather than produced once and filed away, because both the trend and the calendar of known events keep moving.

Load testing: finding the breaking point on purpose

A forecast tells you how much traffic to expect; it doesn't tell you what your system does when it gets there. Load testing closes that gap by deliberately driving synthetic traffic — via tools like k6, Gatling, Locust, or JMeter — against a system until it fails, so you learn the failure mode and the exact threshold under controlled conditions instead of during a real incident. There are a few distinct testing shapes worth naming: a load test holds traffic at an expected level to confirm the system meets its latency and error-rate targets there; a stress test pushes traffic up past expected levels until something breaks, to find the actual ceiling; and a soak test holds elevated traffic for an extended period (hours, not minutes) to catch slow failure modes like memory leaks, connection pool exhaustion, or disk fill that a short burst test would never surface.

The output that matters most from a stress test isn't a single number — it's the shape of the failure. Does p99 latency degrade gracefully as load climbs, or does the system fall off a cliff at a specific request rate? Does it fail closed (rejecting excess requests cleanly, ideally with backpressure or load shedding) or fail open (queuing everything until memory is exhausted and the process crashes)? A system that degrades gracefully and sheds load predictably is far safer to run close to its limit than one that behaves fine right up until it doesn't. Load testing results feed directly back into the headroom decision below, and they're also the input that turns a capacity plan into a genuinely testable claim rather than a spreadsheet extrapolation — the same discipline chaos engineering applies to failure modes instead of load.

Headroom: the buffer between baseline and the ceiling

Once you know the ceiling from load testing and the expected baseline from forecasting, capacity planning is the gap you deliberately leave between them. That gap is headroom, and it exists to absorb two different kinds of surprise: a demand spike the forecast didn't fully predict (a piece of content going viral, a competitor's outage sending their traffic your way), and a capacity-side event where you have less of your own fleet than planned — a failed availability zone, a bad deploy that has to be rolled back while nodes are draining, or routine maintenance taking a fraction of the fleet offline. Headroom sized only for demand spikes and not for partial capacity loss is a common planning gap: a system running at 80% of its measured ceiling looks safe until you remember that losing one of three availability zones instantly pushes the remaining two zones to well over 100% of their fair share unless the headroom was sized with that loss in mind.

There's no single correct headroom percentage — it depends on how volatile demand is, how quickly you can add capacity, and how bad an overload actually is for that service. A common starting heuristic is enough headroom to absorb the loss of one redundant zone or region (frequently landing around 30-50% above steady-state baseline for a system built on N+1 or N+2 redundancy) plus additional margin for demand volatility on top of that. The schematic below shows the three bands a capacity plan is reasoning about at any moment: the baseline load actually being served, the headroom zone reserved as buffer, and the hard ceiling established by load testing.

Horizontal vs. vertical scaling

Once you know how much extra capacity you need, there are two structurally different ways to add it. Vertical scaling makes existing instances bigger — more vCPUs, more RAM, a faster disk on the same node. Horizontal scaling adds more instances of the same size and spreads load across them, typically behind a load balancer. Each has real trade-offs, and mature systems usually use both at different layers rather than picking one dogmatically:

• Vertical scaling is operationally simple — no distributed-systems concerns, no need to shard state or handle cross-node coordination — but it has a hard ceiling (the largest instance type your cloud provider offers), usually requires a restart or brief downtime to resize, and does nothing for availability: a bigger single instance is still a single point of failure. It's the natural first move for stateful systems that are hard to shard, like a single primary database, at least until that ceiling is reached.
• Horizontal scaling has effectively no ceiling — add another instance — and directly improves availability, since losing one of many instances degrades capacity rather than taking the service down. The cost is added complexity: the workload has to be stateless or shardable, you need a load balancer and service discovery, and coordination problems (cache consistency, distributed locking, connection pool sizing per node) appear that a single bigger box never has to deal with.

In practice, stateless application tiers (API servers, web front ends) are the easy horizontal-scaling case and are usually paired with autoscaling that reacts to a live signal like CPU utilization or request queue depth. Stateful tiers — primary databases especially — are harder to scale horizontally (sharding, read replicas, and multi-primary setups all add real operational cost) and are frequently scaled vertically first, with horizontal approaches introduced only once vertical headroom is exhausted. See reliability patterns for how load balancing and sharding fit into the broader architecture picture.

Performance budgets: catching regressions before they ship

Everything above manages capacity at the infrastructure level. Performance budgets manage it at the feature level, and they catch a different failure mode entirely: not "we didn't provision enough," but "a single change quietly made everything more expensive." A performance budget is an agreed ceiling — p99 latency for a request path, CPU-milliseconds per request, memory footprint, bytes transferred, database queries per page load — that a feature is not allowed to exceed. The budget is checked in CI or in a pre-release gate, the same way a failing test blocks a merge, so a regression is caught the day it's introduced rather than surfacing weeks later as an unexplained capacity shortfall or a slow creeping SLO burn.

performance_budget:
  route: /api/v1/search
  p99_latency_ms: 250        # fail build if exceeded
  db_queries_per_request: 3  # catches N+1 regressions
  response_size_kb: 80
  cpu_ms_per_request: 40

The value of a performance budget is that it converts a fuzzy, hard-to-attribute problem — "the service got slower sometime last quarter" — into a build-time signal attributable to one commit. It also gives feature teams a concrete number to design against up front, instead of discovering a cost problem only after the feature is in production and traffic has scaled it up. Budgets should be set from real load-tested baselines and revisited alongside the capacity forecast, not picked arbitrarily, or they either block legitimate work or let real regressions through.

The central tension: over-provisioning vs. under-provisioning

Every technique on this page is in service of the same trade-off, and it's worth naming directly because it's the thing capacity planning actually optimizes. Over-provisioning — running far more capacity than you need — wastes money: idle compute, idle storage, and idle licenses are a direct, recurring line item, and at scale that waste compounds every billing cycle. Under-provisioning risks an outage: insufficient headroom turns a routine traffic spike or a single zone failure into a customer-facing incident, with all the costs that follow — burned error budget, an on-call page (see incident management & on-call), and a postmortem.

Neither extreme is free, which is why capacity planning is a continuous balancing act rather than a one-time decision. Forecasting demand, load testing to find the real ceiling, sizing headroom deliberately, choosing the right scaling shape, and enforcing performance budgets are all mechanisms for pushing that trade-off in a defensible direction: enough buffer to survive the failures and spikes you can reasonably anticipate, without paying indefinitely for capacity that sits idle. The right amount of headroom is a decision made in advance, backed by data, and revisited on a schedule — not a number discovered for the first time during an incident.