Roles and Declaration

In one sentence

In a large incident, the first thing to collapse is not the servers but "who is doing what right now". Splitting roles, declaring early, and assigning an owner to every decision keep that order.

Why this was needed

Say a payment error alert goes off at dawn. Five people join the channel, all competent, all doing the right thing. One starts a rollback, another restarts the same server at the same moment, and a third asks whether to clear the cache, but nobody answers. Customer support, not knowing what is going on, just keeps repeating "we are looking into it". Chapter 14 of Google's SRE book (Managing Incidents) opens with exactly this kind of "unmanaged incident": what makes the incident grow is not individual skill but the absence of coordination.

Putting off the declaration has the same root. People put it off "because it might be nothing and we don't want to blow it up". But the cost of declaring is one channel and three roles, while the cost of a late declaration is the whole stretch of time in which nobody communicated and nobody commanded.

How it works

Three roles are the basis (as the scale grows, add planning and scribe roles).

Role	What it does	What it does not do
IC (command)	Priorities, decisions, role assignment, situational awareness	Fixing things by hand: when the hands are busy, command goes empty
Ops (operations)	The hands that actually change the system	Widening the scope on its own judgment
Comms (communication)	Status updates to the outside at set intervals	Putting out guessed numbers

A few rules hold this table up. Roles are handed over explicitly: the record must say "handing over IC" for the next person to know. Decisions have owners: a decision without an owner gets done by two people at once or by nobody. Announcing "I am doing this now" before touching anything removes overlapping work. Many teams leave this in the record with chat bot commands (such as /declare, /role, /own).

Set the declaration criteria in advance. User impact is visible, you are not confident you can finish alone within 30 minutes, more than one team is needed: if any one of these holds, declare. And measure the times: first alert → first response (ack) → declaration → mitigation → resolution. These intervals are the starting point for the next improvement.

Status updates have a format: what is affected, how much (in numbers), and since when; who is commanding; and when the next update will be. The numbers must be values counted from data, not "a lot", and the next update time keeps to the interval for the severity. Keeping to the interval can matter more than the numbers: "we don't know the cause yet, we will update again in 30 minutes" is also an update, and silence is not.

The channel record ends up like this. With bot commands, you can later recount mechanically who took on what and when.

13:04:47 jun     결제 오류 경보 받았습니다. 보는 중이에요
13:17:12 jun     /declare sev2 결제 실패 증가        ← 첫 경보에서 13분
13:17:54 alex    /role ic @alex
13:18:56 alex    /role comms @blake
13:27:05 alex    /decide D1 결제 v2 기능 플래그 끄기  ← 주인(/own)이 없다
13:28:05 casey   /doing D1
13:29:04 drew    /doing D1                         ← 앞사람이 끝내기 전 — 겹쳤다
13:35:52 alex    /role ic @erin                    ← IC 를 넘겼다. 이 뒤의 alex 는 손이다

Last comes paging. Command begins only when the page reaches a person. A paging service connects the alert name to a team through routing rules, follows that team's escalation policy step by step, and at each step finds the on-call person at that time (overrides take precedence over the schedule) and calls them through their contact method. If even one link in this chain breaks, the first step fails silently, and the next step (usually the team lead) rings only 15 or 20 minutes later. Escalation is a safety net, not the first response.

What it looks like in the field

The most common failure is the IC putting hands on. The person who knows the rollback best takes IC, and while doing the rollback personally, nobody sends updates for 30 minutes and other decisions stall. So touching things after handing over IC is fine, but touching things while still IC must be visible in the record. The second most common is the page never arriving at all. If a newcomer's contact information is empty, a vacation override points to someone who has left, or an alert name changed and no longer matches the routing rules, command cannot even begin. Command starts where the page arrives.

These defects surface only when an incident happens, so mature teams check them between incidents. They put a week's on-call schedule into a simulator and run "who gets it if it rings at any time?" in advance, and every quarter they count the time to declaration and the number of times the IC put hands on. Only with numbers can you say whether the next quarter got better.

What you will do in the next lab

You unpack the record of a payment incident channel (chat, pages, orders) and find from the data the time to declaration, the flow of role handovers, the decisions the IC put hands on, decisions without owners, and overlapping work, then write a status update in the set format with the numbers you computed. At the end, you find in the paging configuration why three teams' pages reached nobody for 20 minutes, and fix it.