Foundations · Toil & automation

Toil & automation

Not all operational work is equal, and the SRE discipline gives the bad kind a name: toil. This page gives you the book's precise five-part definition, the 50% cap Google enforces as policy rather than good intentions, and a concrete ladder for walking any toil source from fully manual to a system that needs no human at all. By the end you should be able to look at a recurring task and say exactly why it's toil, and what the next rung on the ladder looks like.

The precise definition of toil

"Toil" is a specific technical term in the Site Reliability Engineering book (Google, 2016), not a synonym for "annoying work." Work only counts as toil when it has all of the following properties:

• Manual. A human is directly executing the task by hand, not merely supervising something that runs itself.
• Repetitive. You've done this exact task before and will do it again — it's not a one-off.
• Automatable. A machine could do it as well as or better than a human, with no judgment call that genuinely requires a person.
• Tactical. It's interrupt-driven and reactive — a disk filled up, a certificate expired — not something you chose to do as part of a plan.
• Devoid of enduring value. The service is in the same state after the task as it would have been if the task had never accumulated — you restored the status quo, you didn't improve it.
• O(n) with service growth. The work scales linearly with traffic, users, or machine count. Twice the fleet means twice the manual provisioning steps, unlike engineering work, which tends to scale sub-linearly once it's done.

That last property is the one people miss most often. Writing a design doc, debugging a novel outage, or building a dashboard is hard, sometimes unpleasant work — but none of it is toil, because it produces something that keeps paying off: knowledge, a fix, a tool. Toil is work you'll be doing again next week in the same shape, forever, and doing more of as the system you support grows.

Why Google caps toil at 50%

Google's policy is that no SRE should spend more than 50% of their time on toil, averaged over a quarter. This is a hard operational rule, not a nice-to-have: teams track the ratio (commonly self-reported via time-tracking during on-call and ops shifts, or extracted from ticket queues) and it feeds directly into headcount and prioritization conversations. If a team's toil ratio is consistently above 50%, that's treated as a staffing and engineering-investment problem to fix, not a fact of life to tolerate — the fix is assigning engineering time to automate the highest-volume toil sources, not hiring more people to carry more boxes.

The cap exists because toil left unchecked grows faster than the team that handles it. Since toil scales with service size and engineering headcount usually doesn't scale at the same rate, an SRE team that lets toil creep past 50% is on a trajectory toward 100% — a team that does nothing but firefighting, has no time left for the reliability engineering that would have prevented the fires, and burns out. The remaining 50%+ of time is explicitly reserved for engineering project work: exactly the kind of work that has enduring value and doesn't scale linearly with the service, which is the antidote to toil in the first place. This is the same trade-off error budgets make explicit for reliability spend — a number chosen in advance, tracked, and defended, not an aspiration.

The automation ladder

Toil doesn't get eliminated in one leap — it climbs a ladder, and it's normal, even correct, for different toil sources to sit at different rungs at the same time. Each rung removes a specific kind of human dependency:

1. Fully manual. An engineer does every step by hand, from memory or improvisation, each time. Slowest, most error-prone, and the only rung with zero repeatability guarantee.
2. Documented runbook. The steps are written down — exact commands, exact order — so any on-call engineer can execute them correctly under pressure, without inventing the procedure from scratch. Still fully manual execution, but consistent and transferable.
3. Partially automated (human-triggered script). The steps are encoded in a script or tool that a human runs and supervises — for example, running a provisioning script and watching it complete. Execution is now consistent by construction; a human still decides when it runs and confirms it worked.
4. Fully automated. The system detects the trigger condition itself and runs the fix without a human initiating it, though a human may still get notified and could intervene. This is where most mature toil-reduction work lands.
5. Self-service / autonomic. The system handles the entire loop — detection, decision, and remediation — with no human in the path at all, even as a notified bystander. The task has been fully absorbed into the platform.

Climbing a rung is a real engineering investment with a real payoff, and the ladder gives you a way to reason about where that investment is worth making first: a toil source that's high-frequency and still on rung 1 or 2 is usually the best ROI, since every step up removes a proportional slice of recurring manual load. Not every toil source needs to reach rung 5 — the right stopping point depends on frequency, blast radius, and how much judgment the task genuinely requires.

Three toil sources, worked through the ladder

Concrete examples make the ladder easier to apply. Each of these appears repeatedly in real on-call rotations — see incident management & on-call for how they surface as pages.

• Restarting a crashed service. Manual: an engineer gets paged, SSHes in, confirms the process is down, and restarts it by hand. Runbook: the exact restart commands and health checks are documented. Partially automated: an engineer runs a restart script and confirms the health check passes. Fully automated: a process supervisor (systemd, a Kubernetes liveness probe, a supervisord config) restarts the process automatically and pages only if it fails to recover within a bounded number of attempts. This is one of the most commonly fully automated toil sources on any mature platform.
• Provisioning a new customer's resources. Manual: someone hand-creates a database, IAM role, and storage bucket per new signup, copying values between consoles. Runbook: a checklist of exact resource names and settings. Partially automated: an engineer runs a Terraform apply or a provisioning script against a template, then verifies the output. Self-service: the customer's signup flow itself triggers infrastructure-as-code that provisions everything, with no engineer touching it at all — the common end state for SaaS onboarding at scale.
• Rotating a certificate. Manual: an engineer requests a new cert from the CA, uploads it to the load balancer, and updates references before the old one expires — a classic source of self-inflicted outages when someone forgets. Runbook: documented steps and an expiry calendar. Partially automated: a script generates a CSR and applies the new cert on command. Fully automated / autonomic: ACME-based automation (Let's Encrypt via certbot, or a service mesh's built-in cert rotation like Istio's) renews and redeploys certificates on a schedule with no human step at all — arguably the cleanest example of toil reaching rung 5, since certificate expiry is exactly the kind of purely calendar-driven, zero-judgment task automation handles perfectly.

Building the case for automation investment

Automating toil competes for the same engineering time as feature work, so it needs the same kind of justification. The strongest case combines three things: the toil's current frequency and per-occurrence cost (measured in engineer-hours, ideally from actual ticket or on-call logs), the O(n) growth property — projecting what the toil bill looks like at 2x or 5x current scale — and the risk profile of manual execution, since hand-run procedures under incident pressure are exactly where mistakes compound into bigger outages. A toil source costing two hours a week today but scaling linearly with a fleet that's doubling every two quarters is a much stronger automation case than one costing ten hours a week but capped, because it will never grow no matter how the service scales.

It's also worth being honest about false economies. Automation that's brittle, undocumented, or that nobody maintains can quietly become a new toil source of its own — someone now has to babysit the automation. The goal of climbing the ladder is durable removal of manual, repetitive, tactical work, not the appearance of automation. A script that a human must still watch, restart when it fails silently, or patch every time the environment shifts hasn't actually left rung 3.

Check your answers

1. Manual, repetitive, automatable, tactical (interrupt-driven, not strategic), devoid of enduring value, and scales linearly (O(n)) with service growth.
2. Toil ratio is measured (via time tracking or ticket data) and tracked against the 50% ceiling per SRE per quarter; teams consistently over the cap trigger a staffing/prioritization response — automation work gets assigned and toil sources get fixed — rather than the overage simply being accepted.
3. Rung 4 still has a human in the loop as a notified bystander who could (or must) intervene; rung 5 removes the human from the loop entirely — detection, decision, and remediation all happen without anyone being paged or asked to confirm.
4. Certificate expiry is purely calendar-driven and requires no human judgment call — there's no ambiguous case to reason about, which is exactly the profile of task that automation (like ACME/certbot-based renewal) can handle unattended with no loss of correctness.