Blameless postmortems — actions that fail now and pass once fixed

Goal

From the records of a payment outage, separate the triggering deploy, the root cause, and the contributing factors together with actual lines, and write for each action a verification command that "fails now and passes once fixed". Write a postmortem without people's names, actually fix the production config, and correct last quarter's actions whose verifications were lying.

Why it matters

What a postmortem promises is "it will not happen again", and what keeps that promise is not the writing but the actions. Even if an action is closed as "done", if its verification does not catch the defect, it was never done. This lab's grader actually runs your verification commands against the copy from the time of the incident, two fixed copies, and a copy with only one factor left: every way verifications lie in the field (checking only the key, masking the exit code, looking at an old copy or another key, pinning down one value) gets caught here. This lab is planned at 60 minutes, the same as the default session length, so extend your time in advance when you start.

Steps

Unpack the materials with python3 /opt/fixtures/ic/ickit.py open postmortem and read /root/ic/postmortem/README.txt. Write the commit sha (first 7 characters) of the deploy that went in just before the first ERROR in app.log, and the config file that deploy changed, in /root/ic/postmortem/answers/trigger.txt as two lines, deploy=<sha> and changed=<etc/...file>.
Write three lines in /root/ic/postmortem/answers/evidence.txt: trigger: <trigger-line> (the commit line or subject line of deploys.log pointing to the triggering deploy), first_error: <first-error-line> (the first ERROR line of app.log), and recovery: <resolved-line> (the line of alerts.csv where this incident's alert resolved). Each line must be exactly as it appears in the materials.
Line up the team thresholds in the README, the config in /srv/shop/etc/, and the deploy records, and write one root cause (what the triggering deploy changed to outside the threshold) and two contributing factors (what was outside the threshold all along) in /root/ic/postmortem/answers/factors.txt. Directly below each cause line (root_cause: <code> or contributing: <code>), attach evidence: <line> (one line of the materials). Codes: alert-threshold-high (the alert threshold is high), cache-ttl-zero (cache TTL 0), deploy-no-healthgate (deploy health check disabled), pool-too-small (the connection pool is small), retry-no-backoff (retries without backoff), timeout-too-long (the gateway timeout is long).
Write one action item for each root cause and contributing factor in /root/ic/postmortem/answers/actions.txt. Each action is four lines: action: <what>, owner: <team> (a team from teams.txt), factor: <factor-code>, and verify: <shell-command>. verify must take the config tree as the environment variable $ROOT (for example, "$ROOT/etc/pool.conf") and exit with a nonzero code if that factor is outside the team threshold, and with 0 if it is within. The grader actually runs it against the copy from the time of the incident (must fail), two fixed copies (must pass even with different values), and a copy with only that factor left (must fail).
Write the postmortem in /root/ic/postmortem/answers/postmortem.md. It must have four sections, ## Summary, ## Impact, ## Root cause, and ## Actions (the Korean section names ## 요약, ## 영향, ## 원인, and ## 조치 are also accepted). In the impact section write duration: <N>m (minutes from the first ERROR to the alert resolving, truncated) and failed_requests: <M> (the number of ERROR lines in app.log); in the cause section, the root cause and contributing factor codes; and in the actions section, a verify: line for each action. Do not write people's names (channel handles) on any line; the only exception is lines quoting the materials, if they start with evidence:.
Fix the production config /srv/shop/etc/ to bring the three defects within the team thresholds (do not break config that was already within them). Then confirm that the verify commands from step 04 all pass with ROOT=/srv/shop.
Running incident ic-unverifiable raises the incident. Actions A1, A2, and A3 in /srv/billing/postmortems/2026-q3.md are all done, and /srv/billing/bin/verify-actions gives PASS for all three, yet the config in /srv/billing/etc/ is outside the team thresholds (/srv/billing/README.txt) for all three. (1) Fix each action's verify so it fails while the defect remains and passes once fixed (keep the factor and the action order), and (2) actually fix the config so verify-actions gives PASS honestly. The grader runs your fixed verify commands against the copy from the time of the incident (including the old copy etc/old/), two fixed copies, and a copy with only that factor left. In /root/incident/ic-unverifiable/report.txt, write three lines, A1: <code>, A2: <code>, and A3: <code>, and evidence: <original-verify-line> (one original verify line or one line of verify-actions output). Each action's code is one of the following: always-passes (checks only that the key exists, not the value), exit-masked (covers a failed exit code afterwards), flaky-timing (the result changes with the time), wrong-file (looks at a file the service does not read), wrong-key (looks at another key with a similar name).

Notes

Materials tool: python3 /opt/fixtures/ic/ickit.py open <name> unpacks the materials into /root/ic/<name>/. The materials are generated from a random seed that differs per person, and the grader recomputes the answers from the same seed, so someone else's numbers will not match. Unpacking again gives the same materials; add --fresh to start over with new materials.
Write your answers under /root/ic/<name>/answers/. They disappear when the session ends, so keep a copy elsewhere if you need one.
Incident tools: incident list shows the incidents you can raise and a one-line symptom for each, and incident show <name> shows the ticket again. Raising the same incident again puts it into its original broken state with new random materials (answers you wrote earlier will no longer match).
The report is /root/incident/<name>/report.txt. Write one line <target>: <cause-code> per target, and at the end add evidence: <line> (one line that showed the cause) as is. The evidence must be one line that exists as is in the materials or the observation records (20 characters or more). The grader looks first at the actual state after your fix and uses the report to confirm your judgment.
verify runs as bash -c '<command>', with the config tree path in the environment variable ROOT and that tree as the current directory. To test it yourself, run it like ROOT=/srv/shop bash -c '<command>'; echo $? and look at the exit code.
Config files are one key=value per line, and when the same key appears several times the lower line wins. The form awk -F= '$1=="key" {v=$2} END {exit !(<condition>)}' <file> matches that rule.
Two common mistakes: appending || echo … to the end of a verification, erasing the failure, and hard-coding the production path (/srv/shop) into the verification, ignoring $ROOT.

Unpack the materials: find the triggering deploy

Unpack the materials with python3 /opt/fixtures/ic/ickit.py open postmortem and read /root/ic/postmortem/README.txt. Write the commit sha (first 7 characters) of the deploy that went in just before the first ERROR in app.log, and the config file that deploy changed, in /root/ic/postmortem/answers/trigger.txt as two lines, deploy=<sha> and changed=<etc/...file>.

app.log is UTC, and git's Date is in the committer's time zone. It is not "the most recent deploy" but "the deploy just before the first error". The change: line in the deploy commit body tells you the changed file.

The three pillars of the timeline as actual lines

Write three lines in /root/ic/postmortem/answers/evidence.txt: trigger: <trigger-line> (the commit line or subject line of deploys.log pointing to the triggering deploy), first_error: <first-error-line> (the first ERROR line of app.log), and recovery: <resolved-line> (the line of alerts.csv where this incident's alert resolved). Each line must be exactly as it appears in the materials.

Paste the line you found with grep as is (a grep -n line number prefix is fine). Similar other lines or lines edited by hand fail.

Root cause and contributing factors

Line up the team thresholds in the README, the config in /srv/shop/etc/, and the deploy records, and write one root cause (what the triggering deploy changed to outside the threshold) and two contributing factors (what was outside the threshold all along) in /root/ic/postmortem/answers/factors.txt. Directly below each cause line (root_cause: <code> or contributing: <code>), attach evidence: <line> (one line of the materials). Codes: alert-threshold-high (the alert threshold is high), cache-ttl-zero (cache TTL 0), deploy-no-healthgate (deploy health check disabled), pool-too-small (the connection pool is small), retry-no-backoff (retries without backoff), timeout-too-long (the gateway timeout is long).

Config within the threshold is not a defect (it is a decoy). Choose as evidence a line of the logs, deploy records, or alerts that shows the factor: a single line of a config file is too short to serve as evidence.

Actions you can verify

Write one action item for each root cause and contributing factor in /root/ic/postmortem/answers/actions.txt. Each action is four lines: action: <what>, owner: <team> (a team from teams.txt), factor: <factor-code>, and verify: <shell-command>. verify must take the config tree as the environment variable $ROOT (for example, "$ROOT/etc/pool.conf") and exit with a nonzero code if that factor is outside the team threshold, and with 0 if it is within. The grader actually runs it against the copy from the time of the incident (must fail), two fixed copies (must pass even with different values), and a copy with only that factor left (must fail).

Config is key=value, and when the same key appears again the lower line wins. Checking only that the key exists, pinning down one value literally, or covering the exit code with || fails. A convenient form is holding the last value with awk -F= and comparing it against the threshold in END, then exiting. Run it yourself with ROOT=/srv/shop bash -c '<command>'; echo $?.

Write a blameless postmortem

Write the postmortem in /root/ic/postmortem/answers/postmortem.md. It must have four sections, ## Summary, ## Impact, ## Root cause, and ## Actions (the Korean section names ## 요약, ## 영향, ## 원인, and ## 조치 are also accepted). In the impact section write duration: <N>m (minutes from the first ERROR to the alert resolving, truncated) and failed_requests: <M> (the number of ERROR lines in app.log); in the cause section, the root cause and contributing factor codes; and in the actions section, a verify: line for each action. Do not write people's names (channel handles) on any line; the only exception is lines quoting the materials, if they start with evidence:.

Write the cause as "what" (config, procedure, missing checks), not "who". Recount the numbers from the materials. Write owners as team names.

Actually carry out the actions

Fix the production config /srv/shop/etc/ to bring the three defects within the team thresholds (do not break config that was already within them). Then confirm that the verify commands from step 04 all pass with ROOT=/srv/shop.

When the same key appears several times, the service uses the lower line: whether you edit the line or append to the end, it is fine as long as the last value is within the threshold. After fixing, run verify yourself and look at the exit code.

Incident: "verified", yet the same outage happened again

Running incident ic-unverifiable raises the incident. Actions A1, A2, and A3 in /srv/billing/postmortems/2026-q3.md are all done, and /srv/billing/bin/verify-actions gives PASS for all three, yet the config in /srv/billing/etc/ is outside the team thresholds (/srv/billing/README.txt) for all three. (1) Fix each action's verify so it fails while the defect remains and passes once fixed (keep the factor and the action order), and (2) actually fix the config so verify-actions gives PASS honestly. The grader runs your fixed verify commands against the copy from the time of the incident (including the old copy etc/old/), two fixed copies, and a copy with only that factor left. In /root/incident/ic-unverifiable/report.txt, write three lines, A1: <code>, A2: <code>, and A3: <code>, and evidence: <original-verify-line> (one original verify line or one line of verify-actions output). Each action's code is one of the following: always-passes (checks only that the key exists, not the value), exit-masked (covers a failed exit code afterwards), flaky-timing (the result changes with the time), wrong-file (looks at a file the service does not read), wrong-key (looks at another key with a similar name).

Direction: there are several ways a verification lies: what it reads (file, key), what it compares against (does it even look at the value), and whether the exit code survives to the end. For each action, see which of the three is off. Tools: run the original verify yourself with ROOT=/srv/billing bash -c '<command>'; echo $?, the thresholds in the README, grep -n. Concretely: first fix verify and see with your own eyes that it fails on the current config, then fix the config so it passes. If you reverse the order, you cannot know whether the verification catches the defect.