The payment incident channel — who was in command, and what was missing

Goal

Find from data, in the incident channel record (chat, pages, orders), the flow of roles, the time to declaration, and decisions without owners, and write a status update in the set format with numbers you computed. At the end, fix in the configuration an incident in which pages did not reach a person.

Why it matters

Incident command is not a resolution to "do well" but a habit you can verify from the record. How many minutes it took to declare, whether the IC put hands on, and how many decisions had no owner all remain in the channel record as they are, and only teams that count them get better next time. The numbers in a status update are values customers and executives take at face value, so they must not be estimates. And all of this begins only when the page reaches a person; the paging configuration is almost the only command mechanism you can fix before an incident happens.

Steps

Unpack the payment incident materials with python3 /opt/fixtures/ic/ickit.py open roles and read /root/ic/roles/README.txt. Count how many people wrote in the incident channel (chat.jsonl) (excluding the bot icbot, and counting the same person once), and write that single number in /root/ic/roles/answers/participants.txt.
From this incident's first alert (the time an alert whose name starts with pay- was first triggered in pages.csv), write how many seconds it took until (1) the first response (the first acked of a pay- alert) and (2) the channel's /declare, in /root/ic/roles/answers/declare.txt as two lines, ack_delay_sec=<seconds> and declare_delay_sec=<seconds>.
Write who held IC, Comms, and Ops when the incident ended (/resolve) in /root/ic/roles/answers/roles.txt as three lines, ic: <name>, comms: <name>, and ops: <name> (names without @).
Find the decisions on which the IC typed /doing personally while holding command at that moment, and write their decision numbers (in the form D1) in /root/ic/roles/answers/ic-hands.txt, one per line (if there is no such decision, write none). What the former IC did after handing over IC does not count here.
Among the decisions recorded with /decide, find (1) those with no owner set by /own and (2) those where another person typed /doing on the same decision before the previous person typed /done (worked on simultaneously), and write them in /root/ic/roles/answers/work.txt as two lines, unowned: <decisions> and concurrent: <decisions> (leave it empty after the colon if there are none).
Using the time Comms typed /status as the reference, write the first status update in /root/ic/roles/answers/status.txt as five lines. The first line is [SEV<n>] <title> — status update (the severity and the title from /declare), followed by time: <YYYY-MM-DD HH:MM> KST (the update time, to the minute), impact: failed_orders=<N> affected_customers=<M> since=<HH:MM> KST (the number of failed orders, their customers, and the first alert time), command: ic=@<name> comms=@<name> (IC and Comms at the update time), and next_update: <YYYY-MM-DD HH:MM> KST (within the interval for the severity in the README). Count the impact numbers from orders.csv between the first alert time and the update time. The older format with the keys written in Korean is also accepted.
Running incident ic-slow-page raises the incident. Last night, alerts for three teams (payments, search, identity) each fired once, but for all three, the page reached nobody for the first 20 minutes (page log /var/log/pager/pages.log, configuration /srv/pager/, simulator python3 /opt/fixtures/ic/pagersim.py <alert> <time>). Find the cause for each team and fix it so that whenever it fires, it reaches that team's on-call person at minute 0. Keep the first step of each policy as calling that team's on-call (oncall) at minute 0, do not load more than 60% of a week's on-call onto one person, and do not delete people from people.csv. Assign on-call from among the people listed in people.csv under that team before the incident (do not create new people or bring back people who have left). Contacts must take a form the paging service can use: sms:<number>, voice:<number>, push:<id>, or email:<address>. When fixing the route rules, add to them rather than changing them (alerts with the original names must still go to that team). In /root/incident/ic-slow-page/report.txt, write three lines, payments: <code>, search: <code>, and identity: <code>, plus evidence: <pages.log-line> (one line of the page log). Each team's code is one of the following: ack-timeout (the page arrived but acknowledgment was late), contact-missing (the on-call person's contact method is empty), override-departed (an on-call override points to someone who has left), route-mismatch (the alert name does not match the routing rules and falls to the default policy), schedule-gap (the on-call schedule does not cover that time).

Notes

Materials tool: python3 /opt/fixtures/ic/ickit.py open <name> unpacks the materials into /root/ic/<name>/. The materials are generated from a random seed that differs per person, and the grader recomputes the answers from the same seed, so someone else's numbers will not match. Unpacking again gives the same materials; add --fresh to start over with new materials.
Write your answers under /root/ic/<name>/answers/. They disappear when the session ends, so keep a copy elsewhere if you need one.
Incident tools: incident list shows the incidents you can raise and a one-line symptom for each, and incident show <name> shows the ticket again. Raising the same incident again puts it into its original broken state with new random materials (answers you wrote earlier will no longer match).
The report is /root/incident/<name>/report.txt. Write one line <target>: <cause-code> per target, and at the end add evidence: <line> (one line that showed the cause) as is. The evidence must be one line that exists as is in the materials or the observation records (20 characters or more). The grader looks first at the actual state after your fix and uses the report to confirm your judgment.
Time zones in the materials: pages.csv is UTC, chat.jsonl is KST (+09:00), and orders.csv is KST but carries no marker.
Filtering to read only people's messages, as in jq -r 'select(.user != "icbot") | .text' chat.jsonl, makes the flow easy to see.
Two common mistakes: counting the bot's echoes (icbot) as people's commands, and reading commands written in backticks inside sentences as real commands.

Unpack the materials: who was in the channel

Unpack the payment incident materials with python3 /opt/fixtures/ic/ickit.py open roles and read /root/ic/roles/README.txt. Count how many people wrote in the incident channel (chat.jsonl) (excluding the bot icbot, and counting the same person once), and write that single number in /root/ic/roles/answers/participants.txt.

Each line is one JSON object. Extract only the user field with jq, remove duplicates, and count. The bot is not a person.

From the first alert to response and declaration

From this incident's first alert (the time an alert whose name starts with pay- was first triggered in pages.csv), write how many seconds it took until (1) the first response (the first acked of a pay- alert) and (2) the channel's /declare, in /root/ic/roles/answers/declare.txt as two lines, ack_delay_sec=<seconds> and declare_delay_sec=<seconds>.

The two files have different time zones (pages.csv is UTC, the chat is KST). Convert to the same instant units, then subtract. pages.csv also has alerts unrelated to this incident earlier on. date -d reads both forms.

The roles when the incident ended

Write who held IC, Comms, and Ops when the incident ended (/resolve) in /root/ic/roles/answers/roles.txt as three lines, ic: <name>, comms: <name>, and ops: <name> (names without @).

Count by the command rules in the README: a message is a command only when it starts with /, and assigning the same role again means handing it over. The bot's echoes, commands written inside sentences, and roles the bot refused are not role changes.

Decisions the IC put hands on

Find the decisions on which the IC typed /doing personally while holding command at that moment, and write their decision numbers (in the form D1) in /root/ic/roles/answers/ic-hands.txt, one per line (if there is no such decision, write none). What the former IC did after handing over IC does not count here.

The channel is in time order. Read from the top while holding "who is IC right now" in a single variable, and compare it with the person who typed /doing.

Decisions without owners and overlapping work

Among the decisions recorded with /decide, find (1) those with no owner set by /own and (2) those where another person typed /doing on the same decision before the previous person typed /done (worked on simultaneously), and write them in /root/ic/roles/answers/work.txt as two lines, unowned: <decisions> and concurrent: <decisions> (leave it empty after the colon if there are none).

For each decision you must track "who is doing it now". Doing it again to double-check after the previous person finished is not an overlap. An associative array in awk or a dict in python is convenient.

Write the first status update

Using the time Comms typed /status as the reference, write the first status update in /root/ic/roles/answers/status.txt as five lines. The first line is [SEV<n>] <title> — status update (the severity and the title from /declare), followed by time: <YYYY-MM-DD HH:MM> KST (the update time, to the minute), impact: failed_orders=<N> affected_customers=<M> since=<HH:MM> KST (the number of failed orders, their customers, and the first alert time), command: ic=@<name> comms=@<name> (IC and Comms at the update time), and next_update: <YYYY-MM-DD HH:MM> KST (within the interval for the severity in the README). Count the impact numbers from orders.csv between the first alert time and the update time. The older format with the keys written in Korean is also accepted.

The times in orders.csv are KST but carry no time zone marker; to compare them with the first alert (UTC), bring them to the same time zone. Count customers without duplicates. For roles, reflect only the /role commands up to the update time.

Incident: three alerts woke nobody for 20 minutes

Running incident ic-slow-page raises the incident. Last night, alerts for three teams (payments, search, identity) each fired once, but for all three, the page reached nobody for the first 20 minutes (page log /var/log/pager/pages.log, configuration /srv/pager/, simulator python3 /opt/fixtures/ic/pagersim.py <alert> <time>). Find the cause for each team and fix it so that whenever it fires, it reaches that team's on-call person at minute 0. Keep the first step of each policy as calling that team's on-call (oncall) at minute 0, do not load more than 60% of a week's on-call onto one person, and do not delete people from people.csv. Assign on-call from among the people listed in people.csv under that team before the incident (do not create new people or bring back people who have left). Contacts must take a form the paging service can use: sms:<number>, voice:<number>, push:<id>, or email:<address>. When fixing the route rules, add to them rather than changing them (alerts with the original names must still go to that team). In /root/incident/ic-slow-page/report.txt, write three lines, payments: <code>, search: <code>, and identity: <code>, plus evidence: <pages.log-line> (one line of the page log). Each team's code is one of the following: ack-timeout (the page arrived but acknowledgment was late), contact-missing (the on-call person's contact method is empty), override-departed (an on-call override points to someone who has left), route-mismatch (the alert name does not match the routing rules and falls to the default policy), schedule-gap (the on-call schedule does not cover that time).

Direction: a page reaches a person by going through the route → the step of the team's policy → the on-call person at that time (overrides take precedence over the schedule) → that person's existence and contact method, in that order. Where it broke differs per team. Tools: the FAILED and route lines in pages.log, the simulator (try any time in advance), grep, awk -F,. Concretely: for each team, read the route line and the level=1 line at the incident time, and find that time's and that person's lines in the file those lines point to (routes, people, overrides, schedule). After fixing, run the simulator not only at that time but at other times of the week as well.