Incident Command and Postmortems
Estimating Impact
In one sentence
"How many people were affected" is not a matter of counting logs but of setting rules. Only after deciding what to exclude (health checks, bots, internal traffic), what to merge into one (retries, several sessions of the same person), and when to count (a window with a time zone attached) do the numbers converge on one.
Why this was needed
When an incident happens, the same question gets three numbers. The status page team says "4,000 requests failed", the customer support team says "172 users", and the SRE team says "80". All three counted from the same log and none of the calculations is wrong, but the rules differ. Only one can go out in the update, and compensation, regulatory reports, and SLO calculations use that number as is. Write it too large and you get unnecessary compensation and damaged trust; write it too small and it becomes a cover-up.
How it works
First, remove the noise. Kubernetes health checks (/healthz, /readyz) actually fail more during an outage, and search engine bots and internal synthetic monitoring are not users either. Separate them by path, User-Agent, and internal range (such as 10.0.0.0/8).
Second, merge retries into one. Clients retry when they fail. Lines with the same request id are one request, and the result the user experienced is the last attempt. If the first attempt failed and the retry succeeded, that request is not a failure (it was only slow). Counting lines inflates failures, and counting "failed at least once" inflates failed users.
Third, count people. The number of failed requests is not the number of users. One person fails several times. A logged-in person is one person even across several devices and sessions, and for visits that are not logged in you have no choice but to count one session as one person (so this number is closer to "at least this many").
Fourth, set the window and the denominator. Exchange windows as times with a time zone attached: reading "from 14:10" as UTC counts a window nine hours off. The error rate is failures ÷ requests under the same rules, and it means something only when set beside the usual error rate. Affected users, too, show their size only when written together with the active users in that window.
Finally, leave the rules as code. A number counted by hand has to be counted by hand again at the next incident, and it differs again from person to person.
Breaking the rules one at a time on the same log splits the numbers like this (an example counted from one set of lab materials).
| Calculation | Affected users | What went wrong |
|---|---|---|
| By the rules | 99 | — |
| Health checks, bots, and the internal range left in | 118 | Requests that are not users mixed in |
| Failed if any retry failed | 172 | Counted even people who succeeded on retry |
| Number of failed requests as is | 117 | One person's several failures as several people |
| +09:00 dropped from the window | 0 | Counted nine hours later |
What is frightening is that, looking at one number alone, none of them looks strange. So many teams write one line in the update alongside the number with the rule that produced it (what was excluded and what was merged), so the next person can count again with the same rule.
What it looks like in the field
When three teams' numbers differ, what to do is not argue about who is right but line up the rules one line at a time. The status page script did not exclude health checks, the customer support script counted a retry's first failure as a failure, and another read the window as UTC: such differences disappear when the scripts follow the same rules, and one single number remains.
Visits that are not logged in have limits. If the same person switches browsers, there are two sessions counted as two people, and conversely a shared device counts several people as one. So services with many anonymous visits write the two separately, as in "at least N affected users (M by logged-in count)". Either way, the key is not hiding the rule.
Read numbers together with their denominator. "99 people" is an entirely different incident when it is 99 out of 372 active in that window versus 99 out of 30,000. The error rate, too, shows its size only beside the usual error rate: if what was usually 0.4% became 9%, that is twenty times, and if what was usually 8% became 9%, it may not be an incident at all. For teams that use SLOs, these numbers become error budget consumption as is. Only by subtracting the budget with requests and failures counted under the same rules can you compare "what % of the budget this incident used" with other incidents at the end of the quarter.
Lastly, the impact number changes several times during an incident. The number in the first update is the value up to that time, and the number in the postmortem after the incident ends is the value for the whole window. It is normal for the two to differ; you only need to keep both coming from the same rules. That is why each update also states "as of what time": a number without a reference time cannot be matched later, no matter who recounts it.
What you will do in the next lab
From the access log of a payment outage, you filter out health checks, bots, and internal traffic, merge retries to count requests, failures, and the error rate, and count affected users by the rules. You turn those rules into a script and have it confirmed against logs it has never seen, and at the end you fix the scripts of three teams whose numbers differ.