Incident Management Maturity
From improvisation to a rehearsed loop: clear severities and the Incident Commander role, practiced communication and game days, then blameless postmortems whose actions actually close.
Every step is executable work: what, why, prerequisites, implementation, validation you can check, and the failure modes to avoid. Dependencies make the order explicit — a step marked Requires waits on those steps completing first.
Phase 1
Roles and severities exist
Before anything can be practiced, the words must mean the same thing to everyone — severity thresholds, an IC role, a sustainable rotation.
Define a severity scale with activation criteria
P0SFull step — what, why, how, failure modes
WHAT. Three or four severity levels, each defined by user impact (not by which system broke), with objective activation criteria and an example incident attached. WHY. Sev labels drive everything downstream — who wakes up, who is told, how fast — and undefined levels get gamed into everything-is-Sev1 until the scale means nothing. PREREQUISITES. None — a working session and the last incidents' history. IMPLEMENTATION. Draft levels from customer impact (all customers down / subset degraded / internal only / cosmetic), attach a real past incident as each level's example, write the objective trigger ('>5 percent error rate for 5 minutes'), and publish where incidents get declared. TOOLS. A doc; your incident channel. COMMON FAILURE MODES. Severity keyed to the broken component instead of impact — 'the cache is down' is Sev3 if users never notice; impact is the measure. EXPECTED IMPACT. The first minute of an incident stops being a debate about labels. EVIDENCE. The published scale and the 3 declaration messages.Validation — before you call it done
The scale exists with objective criteria and a worked example per level, and the next 3 declared incidents each cite their matching criterion in the declaration message.
Create the Incident Commander role card
P0SRequires: Define a severity scale with activation criteriaFull step — what, why, how, failure modes
WHAT. A one-page card: what the IC does (coordinate, communicate, decide), what they never do (fix the thing themselves), and the phrases to open an incident channel with. WHY. The IC pattern works because one mind holds the state; without the card, the loudest engineer becomes de facto IC while also being the only one debugging — the classic double-duty failure. PREREQUISITES. The severity scale, so the IC's first act (declare a level) has meaning. IMPLEMENTATION. Write the card — responsibilities, anti-responsibilities, the opening template (what we know, what we're doing, who's on what, next update time), and escalation criteria. Review it with the whole on-call population, not just seniors. TOOLS. A doc; the incident channel template. COMMON FAILURE MODES. A card that names the IC role but grants no decision authority — the IC becomes a scribe; say explicitly that during the incident the IC's coordination calls go. EXPECTED IMPACT. Someone always owns the incident's shape, so responders can own its parts. EVIDENCE. The card and the incident timeline showing separate IC and resolver roles.Validation — before you call it done
The card is published, every on-call engineer has acknowledged reading it, and the next Sev1/Sev2 incident log shows a named IC distinct from the primary fixer.
Stand up a sustainable on-call rotation
P0MRequires: Create the Incident Commander role cardFull step — what, why, how, failure modes
WHAT. A staffed, escalated rotation with clear handovers, follow-the-sun or compensating rest, and a paging policy that matches severity to response time. WHY. Unsustainable rotations quietly fail: responders mute the pager, decline swaps, and quit — and the failure surfaces during the worst possible week; sustainability is an operational requirement, not a perk. PREREQUISITES. Severity scale and IC card — the rotation pages people INTO a defined structure. IMPLEMENTATION. Define the rotation (primary + secondary, minimum 4-5 people for 24/7), write the handover template (open incidents, risky changes, watch items), set the paging policy per severity, and give escalations a named next-hop when primary is unresponsive for 5 minutes. TOOLS. PagerDuty/Opsgenie or equivalent; the shared calendar. COMMON FAILURE MODES. Two-person hero rotations — it works until one person takes a vacation, which is exactly when it stops working. EXPECTED IMPACT. Someone rested and prepared is accountable at 3 a.m., and everyone knows who. EVIDENCE. The schedule export and a sample of handover notes.Validation — before you call it done
The rotation shows 100 percent coverage for the next 8 weeks with no single responder holding more than 1 primary week in 4, and handover notes exist for every shift boundary in the last month.
Phase 2
Practice and communication
The skills that only exist after rehearsal: status updates during, and game days before.
Master internal updates and stand up a status page
P1SRequires: Create the Incident Commander role cardFull step — what, why, how, failure modes
WHAT. A fixed update cadence during incidents (every 30 minutes for Sev1) to a known channel, plus an external status page for customer-visible outages with its own update rules. WHY. Silence is the #1 complaint in every incident retro — internal stakeholders start investigating on their own and customers open tickets; scheduled updates collapse both loops. PREREQUISITES. IC role in place — updates are the IC's clock, not whoever's free. IMPLEMENTATION. Add the cadence to the IC card, template the update (impact, progress, next update time — always the next time), stand up the status page with pre-drafted component states, and rehearse the customer-facing wording rules (plain language, no blame, no ETA you cannot support). TOOLS. The incident channel; a status page (Statuspage, Cachet, or a static page you can update fast). COMMON FAILURE MODES. Waiting for good news to post — the update that says no change yet, next update at :30 IS the deliverable. EXPECTED IMPACT. Stakeholders and customers stop ping-pong during incidents, and the org's trust survives the outage. EVIDENCE. The channel transcript and the status page history.Validation — before you call it done
The next Sev1/Sev2 timeline shows updates at or within 5 minutes of every promised interval, and the status page carried at least one customer-visible incident to resolution with 3 or more updates.
Run quarterly game days
P1MRequires: Stand up a sustainable on-call rotation, Master internal updates and stand up a status pageFull step — what, why, how, failure modes
WHAT. Scheduled rehearsals injecting a realistic failure into a safe environment, walked through with full process — declare, IC, comms, resolve, debrief. WHY. The first time you use the severity scale, the IC card, and the status page cannot be a real incident; game days make the process fail in rehearsal, where failure is free (NIST's exercise guidance and SRE practice agree). PREREQUISITES. Roles and rotation defined, so the rehearsal exercises real structure. IMPLEMENTATION. Quarterly, in staging or a sacrificial environment: inject a failure (kill the primary DB, revoke a key, blackhole a dependency), let the on-call run the real process end to end, keep a referee log of friction points, and close the debrief actions before the next game day. TOOLS. A staging environment; fault injection (chaos tooling or plain kill commands); the referee's log. COMMON FAILURE MODES. Announcing the failure in advance — the drill must surprise the process, not confirm it. EXPECTED IMPACT. The process gets exercised 4 times a year at zero customer cost, so real incidents hit muscle memory. EVIDENCE. The four debriefs with closed action lists.Validation — before you call it done
Four game days per year are on the calendar and held, each producing a debrief whose actions are 100 percent closed before the next scheduled one.
Phase 3
The learning loop
Blameless postmortems with teeth: a standard, tracked actions, and a repeat-incident metric that tells the truth.
Adopt a blameless postmortem standard
P0MRequires: Define a severity scale with activation criteriaFull step — what, why, how, failure modes
WHAT. Every Sev1/Sev2 gets a postmortem within 5 business days, from a template that asks for timeline, contributing factors, and what would have caught it — never who messed up. WHY. Blameful reviews teach people to hide information, and hidden information guarantees the incident repeats; blameless is not softness, it is the only format that keeps the causal data flowing (the SRE book's postmortem chapter is the canonical argument). PREREQUISITES. IC role and severities — the postmortem consumes the timeline the IC kept. IMPLEMENTATION. Publish the template (summary, impact, timeline, root and contributing factors, what went well, action items with owners and dates, lessons), schedule the review meeting inside 5 days, and make the blameless norm explicit in the doc plus enforced by the facilitator in the room. TOOLS. The template doc; the review meeting. COMMON FAILURE MODES. Root cause: human error — that is where the analysis should start, not stop; ask why the error was possible and what system made it consequential. EXPECTED IMPACT. Incidents become the org's best source of systems intelligence instead of its best source of fear. EVIDENCE. The postmortem index with dates and the sampled documents.Validation — before you call it done
Every Sev1/Sev2 in the last two quarters has a completed postmortem within 5 business days, and a sampling of them shows contributing-factor language (no individual blame) throughout.
Track postmortem actions to closure
P0SRequires: Adopt a blameless postmortem standardFull step — what, why, how, failure modes
WHAT. Every postmortem action item becomes a tracked ticket with an owner and due date, reviewed weekly until closed — closure rate reported next to open count. WHY. The learning loop fails at the boring end: postmortems full of actions nobody owns produce beautiful documents and identical incidents; tracking is the difference between ritual and prevention. PREREQUISITES. Postmortem standard producing actions. IMPLEMENTATION. At the review's end, convert each action to a ticket (owner, date, severity-linked priority), add the open-actions report to the weekly ops review, and age anything past due with escalation to the owning team's manager. TOOLS. The ticket tracker; the weekly ops review slot. COMMON FAILURE MODES. Actions like improve monitoring — unactionable and eternal; require each action to name a change to a system, doc, or process. EXPECTED IMPACT. The org's stated lessons visibly turn into changed systems. EVIDENCE. The tracker's aging report.Validation — before you call it done
The current open-action list shows 90 percent of items closed within their due dates over the trailing quarter, and zero items older than 60 days without a documented extension.
Track the repeat-incident rate
P1SRequires: Track postmortem actions to closureFull step — what, why, how, failure modes
WHAT. A monthly metric: the share of incidents whose postmortem cites a cause already seen in a previous postmortem — plus a review of every repeat. WHY. The repeat rate is the learning loop's report card; a flat repeat rate means the postmortems are being written for compliance, and only the number makes that undeniable. PREREQUISITES. Postmortems with actions tracked — repeats are defined against them. IMPLEMENTATION. Tag postmortems with cause categories, compute the monthly repeat share (incidents matching a prior postmortem's cause within 12 months), review every repeat in the ops review with the question why did the earlier action not prevent this, and adjust the action (not the blame). TOOLS. The postmortem index; the ops review. COMMON FAILURE MODES. Cause categories too fine to ever match — the metric reads zero repeats forever; start coarse (deployment, capacity, dependency, config, security) and refine only when a category dominates. EXPECTED IMPACT. The program gets the one number that cannot be gamed: the same fire, again, or not. EVIDENCE. The monthly metric history and the repeat-review notes.Validation — before you call it done
6 monthly reports exist with the repeat rate computed, and each repeat incident in the window has a written why-the-action-missed note with a follow-up action.
Sources
- Tier 1 — PrimaryGoogle SRE — Managing Incidents ↗— Google(verified 2026-09-19)
- Tier 1 — PrimaryNIST SP 800-61r2 — Computer Security Incident Handling Guide ↗— NIST(verified 2026-09-20)
- Tier 1 — PrimaryGoogle SRE — Postmortem Culture ↗— Google(verified 2026-09-19)
Start from where YOU are
This roadmap is the general path. An assessment tells you which steps you can skip and which are critical — then generates a version of this plan from your actual findings, tracked to done in your workspace.