Drill — Forecast the Bottleneck

rec-api is the service behind every personalized row on Wavelength's home screen — "Because you watched…", "Top picks for you." It has been handed six months of its own monitoring history: peak weekly traffic, app-tier CPU, database connections in use, and primary-disk write IOPS, one row per month. Today is 2026-08-16. Next quarter — October through December — brings rec-api's usual organic growth plus a known, recurring event: a holiday content launch that has driven a predictable traffic surge every November for the last two years running. Your job: fit the growth trend from six months of history, forecast next quarter's peak demand including that known surge, and work out which one of three resources — the autoscaled app tier's CPU, the Postgres connection pool, or the primary's provisioned disk IOPS — actually runs out first. Work the arithmetic yourself before opening the answer key; the method is the entire point, and the answer is not the resource most dashboards would have had you watching.

The scenario: rec-api heading into next quarter

☺ Like you're 10: One service, three different "how much is left" gauges — and only one of them is the number everyone's actually staring at.

rec-api runs as a stateless fleet behind an autoscaling group (ASG): a target-tracking policy holds CPU at 70%, with a floor of 6 instances and a hard ceiling of 100 instances, max. Each instance is sized to handle about 130 requests/second at that 70% CPU target before the policy adds another one. Every instance talks straight to a single Postgres primary — no pooling layer in front of it yet — and each instance's database client opens a fixed local pool of 12 connections at process startup, a number set in internal/db/pool.go when the service first shipped and never revisited since it moved to autoscaling:

// internal/db/pool.go — set two years ago, never revisited
const dbPoolSizePerInstance = 12

func NewPool(dsn string) (*sql.DB, error) {
    db, err := sql.Open("postgres", dsn)
    if err != nil {
        return nil, err
    }
    db.SetMaxOpenConns(dbPoolSizePerInstance)
    db.SetMaxIdleConns(dbPoolSizePerInstance)
    return db, nil
}

Postgres itself caps out at max_connections = 550, with 50 reserved for replication and admin per platform convention — 500 usable connections for the app fleet. Every rec-api request that misses the in-process cache also writes one impression event to a feature-store table on that same primary, measured at 2.1 write IOPS per request plus a roughly constant 150 IOPS of background WAL and checkpoint traffic; the primary's EBS gp3 volume is provisioned for 12,000 IOPS. Three numbers, three hard ceilings: 100 instances, 500 connections, 12,000 IOPS. None of them autoscale together, and nobody has checked whether they still leave enough room since the service was this small.

Six months of history, read correctly

☺ Like you're 10: Same day, same hour, every month — so the numbers below are actually comparable to each other, not just six random Tuesdays.

rec-api's traffic follows a stable weekly pattern that hasn't changed in over a year even as absolute volume has grown: Friday–Saturday evening peaks run consistently around 30% above the weekly average. Because that shape is stable, every row below is measured at the same point in that weekly cycle — the Friday 8pm peak — so month-to-month differences reflect real growth, not weekly wobble contaminating the trend.

Look at the CPU column before anything else: 69%, 69%, 69%, 70%, 68%, 70% — flat, unremarkable, exactly what a well-tuned autoscaler is supposed to produce, even while request volume grew 45% over the same six months. That flatness isn't good news by itself; it's the autoscaler doing its one job (adding instances to hold CPU near its target), and it's also exactly why a team watching only a CPU or instance-count dashboard would see nothing alarming here at all. The connections and IOPS columns, which nothing autoscales, tell a different story: both are climbing in lockstep with traffic, with no policy watching them at all.

Step 1 — fit the organic growth rate

☺ Like you're 10: Turn "it grew from 3,200 to 4,650 over five months" into one clean monthly growth percentage you can project forward.

Compute the compound monthly growth rate implied by the six months of peak req/s, from February's 3,200 to July's 4,650 — five monthly steps:

Growth factor = 4,650 ÷ 3,200            = 1.453125
Monthly rate  = 1.453125 ^ (1/5) − 1
              = e^(ln(1.453125) / 5) − 1
              = e^(0.373716 / 5) − 1
              = e^0.074743 − 1
              = 1.07760 − 1
              = 0.0776   →   7.76% per month

7.76% a month is the organic growth rate every forecast below is built on. Applying it forward from July's 4,650 req/s:

Aug 2026 = 4,650 × 1.0776 = 5,011
Sep 2026 = 5,011 × 1.0776 = 5,400
Oct 2026 = 5,400 × 1.0776 = 5,819
Nov 2026 = 5,819 × 1.0776 = 6,270   (organic, before any launch adjustment)
Dec 2026 = 6,270 × 1.0776 = 6,757

That's the organic baseline for next quarter. It isn't the whole forecast yet — the known November launch event still has to be layered on top, in Step 3.

Step 2 — turn traffic into resource demand

☺ Like you're 10: Requests per second alone don't tell you anything about chairs and cars — you need the conversion rate from "kids at the party" to "chairs needed" for each supply separately.

Each of the three ceilings from the scenario converts to req/s through a different ratio, derived from the fixed constants already given:

Instances per req/s   = 1 ÷ 130 instance         (130 req/s per instance)
Connections per req/s = 12 ÷ 130 = 0.09231 conn   (12-conn pool per instance)
IOPS per req/s         = 2.1 IOPS, + a ~150 IOPS fixed baseline

Sanity-check the connections ratio against the historical table instead of trusting the algebra blind: 300 ÷ 3,200 = 0.09375 in February, 432 ÷ 4,650 = 0.09290 in July — both close to the derived 0.09231, with the small remaining wobble explained by instance count being a whole number (an autoscaler can't add 0.6 of an instance, so the real ratio steps rather than glides). Close enough to trust for a forecast.

Step 3 — forecast next quarter, launch event included

☺ Like you're 10: Add the cousin's soccer team on top of the trend — the guest list this time isn't just "a bit more than last time."

Wavelength's Holiday Watch Party content-and-device launch goes live November 12 — it has driven a similar spike each of the last two Novembers, a known seasonal event rather than a guess, adding roughly 40% over that week's organic baseline, sustained for about ten days before traffic reverts to the underlying trend. Layer that onto Step 1's organic forecast:

8,778 req/s — the November launch week — is the quarter's single highest forecast point. But notice October, the very first month of next quarter, already sits at 5,819 req/s from organic growth alone, three weeks before the launch has even started.

Step 4 — find out which ceiling breaks first

☺ Like you're 10: Line the party's guest count up against cups, chairs, and driveway space, and see which one runs dry first — it's rarely the one you'd guess.

Compare Step 3's forecast against Step 2's three maximum-sustainable-req/s figures — 13,000 / 5,417 / 5,643:

Both the pool and the disk cross into October's forecast, so calendar-month resolution alone doesn't separate them — but the underlying req/s thresholds do: 5,417 is a lower bar to clear than 5,643. As traffic ramps smoothly through that range, the connection pool is mathematically guaranteed to hit its ceiling about 226 req/s — roughly two and a half weeks at this growth rate — before disk IOPS does, regardless of which calendar month either lands in. The app tier's CPU never enters the conversation at all: even at the quarter's single worst moment, it's using barely two-thirds of its maximum instance count.

Even after the launch window closes, the quarter doesn't get safe again: December's organic 6,757 req/s still puts the pool at 124.8% and disk IOPS at 119.5% of ceiling. This isn't a ten-day spike problem — it's a structural shortfall for the entire quarter unless one of the two actual bottlenecks is addressed before October begins.

The fix: raise the ceilings the autoscaler doesn't manage

☺ Like you're 10: Buy more chairs and widen the driveway before the party — but check the chairs will actually fit in the garage first.

The correct fix for rec-api is not "raise SetMaxOpenConns per instance" — that number is already fixed regardless of instance count, and raising it just makes the shortfall arrive sooner. The real fix is a pooling layer that decouples fleet size from real Postgres backend connections. Introduce PgBouncer in transaction-pooling mode between the app fleet and Postgres:

# pgbouncer.ini — app instances connect here, not to Postgres directly
[databases]
recdb = host=pg-primary.internal port=5432 dbname=recdb

[pgbouncer]
pool_mode = transaction
max_client_conn = 20000       # cheap: every app instance's local pool lands here
default_pool_size = 450       # the expensive resource: real backend connections to Postgres
reserve_pool_size = 50

Nothing in internal/db/pool.go has to change — only the DSN it dials. Each instance still opens its local pool of 12, but now against PgBouncer instead of Postgres, and PgBouncer multiplexes many app-side connections onto a much smaller, roughly flat set of real Postgres backends — the fleet can keep growing without the backend connection count growing in lockstep with it. Raise the disk ceiling separately, on gp3's own dial:

aws ec2 modify-volume --volume-id vol-0abcdef1234567890 --iops 16000

Check the current maximum provisioned IOPS for this instance family and volume size on AWS's own EBS documentation before committing to a number — gp3 ceilings have moved over time and vary by configuration, the same hedge this course gives about any vendor-specific limit or exam detail that can drift after this page was written.

One more gap this forecast exposes: nothing pages on pool or disk utilization today — only CPU has an alert wired to it, and CPU was never going to be the thing that broke. See alert design & alert fatigue for adding a burn-style alert on pool and IOPS utilization percentage, not just the metric the autoscaler already watches. And before trusting any of these new ceilings in production, validate the forecast against a real k6 load test that drives synthetic traffic to the forecast peak under controlled conditions — the same discipline the course's capacity plan & load test capstone walks end to end, and the only way to confirm the model in Step 2 holds up against a real cluster instead of just this drill's arithmetic.

Try it yourself: raise the pool, then what?

☺ Like you're 10: Buying more chairs doesn't help if the driveway still can't fit everyone's car.