Incident Command and Postmortems
Blameless Postmortems
In one sentence
The purpose of a postmortem is to keep the same incident from happening again, and the action items keep that promise. So a good postmortem names the system, not a person, as the cause, backs every claim with lines from the actual records, and attaches to every action a verification command that "fails now and passes once fixed".
Why this was needed
Chapter 15 of Google's SRE book (Postmortem Culture) stresses blameless postmortems. The reason is simple. If you name a person as the cause, nobody will tell the facts at the next incident, and without facts you cannot find the cause. The sentence "the person who deployed made a mistake" fixes nothing: the system that made that mistake possible (deploys without config review, a disabled health check, an alert that fires late) remains as it was.
There is a quieter failure too: an action item is closed as "done", and the same incident happens again. Open it up, and the verification only checked "whether the key exists", covered failure with || true, or is checking an old copy the service does not read. If the verification does not catch the defect, that action was never done.
How it works
Separate the causes. The trigger is the change that started the incident, usually a single deploy. The root cause is the defect that change created, without which there would have been no incident. For example, deploy v3.2.0 is the trigger, and that deploy shrinking the connection pool from 64 to 5 is the root cause. Rolling back the deploy stops the incident, but the root cause, "the pool size can change without review", remains. Actions are tied to the root cause and the contributing factors. A contributing factor is a defect that existed all along, one that made the incident noticed late (a high alert threshold), kept it from being prevented (a disabled health check), or made it grow (retries without backoff). Mix the three and the actions go to the wrong place.
Attach evidence as lines. Not "the first error was around 06:24", but that line from the log, attached as is. Anyone must be able to find the same line again and check it, and plausibly made-up lines get caught in verification.
An action has three things. An owner (a team, not a person), what it changes, and a verification command. The verification must have three properties.
| Property | Meaning | Common fake |
|---|---|---|
| Fails now | Catches that the defect remains | Checking only that the key exists, covering with ` |
| Passes once fixed | Accepts other correct fixes too | grep -x that pins down a single value |
| Sensitive to its own factor | Looks at its own defect, not someone else's | Checking a similar other key or an old copy |
If you write the verification to take the target tree as an argument ($ROOT), you can run it the same way on the production config, on the copy from the time of the incident, and on the fixed copy.
One action item looks like this (the key and threshold in the example only show the format).
action: 세션 저장소 연결 수를 팀 기준 이상으로
owner: platform-team
factor: example-too-small
verify: awk -F= '$1=="max_sessions" {v=$2} END {exit !(v != "" && v+0 >= 100)}' "$ROOT/etc/session.conf"
awk -F= splits key=value in two, and when the same key appears several times it overwrites v so the last value remains, which is the same way the service reads the config. Since it compares against the threshold in END and answers with the exit code, as long as you do not attach it after a pipe or wrap it in || echo, that result is the verification result as is. Running this one line with ROOT=<incident-copy> (the copy from the time of the incident) and with ROOT=<fixed-copy> (the fixed copy) tells you within seconds whether the verification is honest.
What it looks like in the field
At a quarterly review meeting, the action list is all green, yet the same incident happens again. Open it up, and the verifications of the three actions are each lying in a different way. The order of fixing is always the same: first fix the verification so it fails now, exposing the fact that the defect remains, and then actually fix the system so the verification passes honestly. If you do not fix the verification first, you cannot even know whether you fixed the system.
Blameless writing also takes practice. Changing "A put in the wrong config" to "a config change could reach production without review or a health check" moves what the sentence points to as the thing to fix from a person to the pipeline. People's actions can remain in the timeline, but as a record that tries to understand "why that person judged that way with the information they had at that moment", not as a cause.
What you will do in the next lab
From the logs, deploy records, and alerts of a payment outage, you find the triggering deploy and the three pillars of the timeline as actual lines, separate the root cause from the contributing factors, and write a verification command for each action. The grader actually runs those commands against the copy from the time of the incident, two fixed copies, and a copy with only one factor left. You write a postmortem without people's names, actually fix the production config, and correct last quarter's actions that were "verified" yet saw the same outage.