Incident response & forensics
A security incident runs through the same six-phase lifecycle as any other incident, but every phase carries extra obligations that a pure availability outage doesn't: you can't just restore service and move on, because the how and the who of a breach matters as much as the fix. This page walks through the PICERL lifecycle, then covers the three things that make security incident response genuinely different from operational incident management — evidence preservation, legal notification duties, and containment decisions that trade speed for intelligence.
If your store's window gets smashed by a rock, you sweep up the glass and fix the window — done. If your store gets broken into, you don't sweep up first: you call the police, leave the footprints and the pried lock exactly as they are, write down who touched what and when, and you might even let the alarm company quietly keep watching the back door for a night instead of welding it shut immediately, because catching where the burglar goes next is worth more than kicking them out five minutes sooner. A security incident is the break-in, not the broken window — the evidence and the decision to watch-versus-block are the whole point, not an afterthought.
The PICERL lifecycle
PICERL — Prepare, Identify, Contain, Eradicate, Recover, Lessons learned — is the SANS Institute's six-phase incident-handling model, and it maps directly onto the four-phase lifecycle in NIST SP 800-61 Rev. 2 (Preparation; Detection & Analysis; Containment, Eradication & Recovery; Post-Incident Activity). The two are the same process at different granularity, and both are cyclical, not linear: the output of Lessons learned feeds back into Prepare for the next incident.
- Prepare — build the capability before you need it: runbooks, an on-call rotation, logging and detection coverage, forensic tooling, and a tested communication plan.
- Identify — confirm an event is actually a security incident, scope what's affected, and establish a rough timeline from the first available signal.
- Contain — stop the incident from getting worse, choosing a strategy (isolate now vs. observe) appropriate to what's actually at stake.
- Eradicate — remove the root cause: the malware, the compromised credential, the vulnerable code path the attacker used to get in.
- Recover — restore affected systems to normal operation, with verification that the threat is actually gone before traffic returns.
- Lessons learned — a blameless post-incident review that turns what happened into updated runbooks, detections, and controls.
Not just an outage: how a security incident differs from an operational one
General operational incident management — the pager going off because a service is down or latency has spiked — shares the same shape as security incident response but optimizes for a single goal: restore availability as fast as possible, then figure out the root cause afterward. A security incident adds three obligations on top of that, and getting any one of them wrong can outlast the technical fix by months.
First, a security incident requires preserving evidence, not just restoring service — see the next section. Second, it may trigger legal and regulatory notification obligations that a pure availability incident never does; see compliance and governance for the frameworks that govern this. Third, the containment decision itself is a different trade-off: an availability incident has one clear goal (bring the service back), while a security incident has to weigh stopping the damage against understanding it, which is covered in its own section below. None of this replaces the operational muscle memory of on-call response — it layers on top of it.
Preserving evidence and chain of custody
The instinct during an outage is to fix things — reboot the box, roll back the deploy, clear the queue. During a security incident, that same instinct destroys evidence: a reboot flushes volatile memory that might hold the only copy of an attacker's decryption key or in-memory malware, and rotating a log file before it's exported deletes the record of what actually happened. RFC 3227's order-of-volatility principle exists for this reason — collect the most volatile evidence first (CPU registers and cache, routing tables and ARP cache, running processes, RAM) before touching anything less volatile (disk, backups, archival logs), because volatile state disappears the moment power or process state changes.
Chain of custody is the paperwork that makes evidence usable later, in a courtroom, an insurance claim, or a regulator's inquiry: who collected an artifact, when, from where, with what tool, and every hand it passed through afterward, each transfer logged and each artifact hashed (SHA-256 is standard) at collection time so any later tampering is detectable. Forward logs to write-once storage — S3 Object Lock or an equivalent WORM target — before your normal retention policy rotates them out, and image disks with a tool that doesn't modify the source (a hardware write blocker plus dd or FTK Imager) rather than copying files off a live, still-mounted volume.
Killing a compromised container or host the moment you spot it is the single most common evidence-destroying mistake. It feels like containment, but it also wipes the process list, open network connections, and any memory-resident payload before anyone captures them — cordon and isolate at the network layer first, snapshot disk and memory, and only then terminate the workload.
Containment is a different trade-off during a security incident
For an availability incident, containment is unambiguous: isolate the failing component and restore service, because there's no downside to acting immediately. For a security incident, immediate isolation has a real cost — it tips off the attacker, cuts off the telemetry you're using to understand what they've touched, and can permanently forfeit the chance to trace lateral movement, identify a full blast radius, or attribute the intrusion. Watching an active intruder for a bounded window, inside a segment you control, is sometimes the higher-value move: security teams do this deliberately to map which systems were reached and what data left the network before shutting the door.
The trade-off tips the other way fast once the incident involves active, irreversible loss: ransomware mid-encryption or confirmed data exfiltration in progress calls for immediate isolation, full stop, because every additional minute of observation is measured in encrypted or stolen data. Reconnaissance-stage access with a contained blast radius is where deliberate, monitored containment can be justified. Either way, this isn't a call an individual on-call engineer should make alone under pressure — it belongs in the runbook, decided in advance, with legal and incident-command sign-off baked into the escalation path rather than improvised at 2 a.m.
Runbooks make the first response fast, not improvised
The value of a runbook is that it converts a high-stakes, low-frequency decision into a checklist someone can execute correctly under pressure, without re-deriving the right sequence of steps from scratch. Two scenarios come up often enough that every team should have a runbook for them before the first real incident, not during it.
- Leaked credential — revoke or rotate the credential immediately (see secrets management for how short-lived credentials shrink this window by default), invalidate any active sessions it authorized, pull the access logs for the exposure window to scope what it actually touched, and rotate anything else the same credential had access to before declaring the incident closed.
- Compromised container or host — isolate at the network layer (security group or Kubernetes NetworkPolicy) rather than killing the workload outright, snapshot disk and memory for forensics, cordon and drain the node instead of deleting the pod, and trace the entry point — see container and supply chain security for how a compromised base image or dependency typically gets in — before rebuilding from a known-good image and rotating any credentials the host held.
A runbook doesn't replace judgment during the incident, but it does remove the two most expensive failure modes: forgetting a step that only matters once every few years, and inventing the containment strategy live instead of applying one the team already agreed on.
1. What are the six phases of PICERL, and how do they map onto the four phases of NIST SP 800-61? 2. Name two obligations a security incident carries that a pure availability incident does not. 3. Why might a team deliberately delay isolating a compromised host instead of cutting it off immediately? 4. Why does the leaked-credential runbook need to exist before the incident happens, rather than being written during it?
Check your answers
- Prepare, Identify, Contain, Eradicate, Recover, and Lessons learned — they map onto NIST's Preparation; Detection & Analysis; Containment, Eradication & Recovery; and Post-Incident Activity, which group the same six activities into four broader stages.
- Preserving evidence and chain of custody, and legal or regulatory notification obligations (such as breach-notification deadlines) — a routine availability outage requires neither.
- Because immediate isolation cuts off telemetry and tips off the attacker, forfeiting the chance to trace lateral movement or scope the full blast radius before shutting the door — a trade-off that's only justified when there's no active, irreversible loss like ransomware encryption or ongoing exfiltration.
- Because deciding the exact sequence of steps — revoke, invalidate sessions, check the access logs, rotate downstream credentials — takes time and judgment that isn't available under the pressure of an active leak; writing it in advance turns a high-stakes improvisation into a checklist anyone on call can execute correctly.