The Order of Triage, and Keeping Notes
In one line
In incident response, the difference in skill comes not from tools but from order and records. With an order, you do not look at the same place twice, and with records, the next person does not start from scratch.
Why this exists
When a report comes in at 3 a.m., anyone gets impatient. Impatience breaks the order, and when the order breaks, the question "Did I check that already?" keeps coming back. That is why experienced people mechanically follow an order decided in advance. Judgment comes after the data is collected.
How it works
The order goes from the bottom up. If a lower layer is broken, there is no need to look at the layers above it.
- Name — run
getent hosts <name>anddig +short <name>side by side. If the two results differ, that difference is exactly where the cause is. - Reach —
ping -c 3 <IP>. Even if it fails, do not jump to a conclusion. ICMP may be blocked. - Port —
nc -zv -w 2 <IP> <port>. Always write down whether it was refused or timeout. That one word decides the direction of the next investigation. - Binding — on the target host,
ss -ltnp | grep <port>. If the Local Address is127.0.0.1, that is the end of it. - Response —
curl -o /dev/null -s -w '...'. Look at the status code and the time per phase together.
At each step, keep the command and its output as they are. A record that says only "checked" is worth nothing to the next person. On the other hand, when the command and output are attached, the next person can read that output again and may even reach a different conclusion.
A postmortem must have at least these five items.
| Item | Content |
|---|---|
| Symptom | What the user actually saw (not a guess) |
| Layer | At which layer it stopped |
| Cause | Together with the command output that backs it |
| Fix | What was changed and how |
| Prevention | What keeps the same thing from happening again |
What to probe at each layer
There is an order for narrowing down "it doesn't work". Going from the bottom up, the candidates shrink at every step.
| Layer | Check command | If it fails here |
|---|---|---|
| 1. Name | getent hosts api.example.com |
DNS, /etc/hosts, search domain |
| 2. Route | ip route get 10.0.3.12 |
Routing, default gateway |
| 3. Reach | ping -c1 10.0.3.12 |
ICMP may be blocked (move on even if it fails) |
| 4. Port | nc -zv 10.0.3.12 8080 or curl -v telnet://… |
Firewall, security group, nothing listening |
| 5. Service | curl -sv http://10.0.3.12:8080/healthz |
Application |
It is important not to stop even if step 3 fails. ICMP is often blocked, so "ping doesn't work, so it's a network problem" is often wrong. Step 4 is the real verdict.
The result of nc splits three ways, and each means something different.
- connected — the port is open. The problem is at step 5 or above.
- Connection refused — the packet arrived and nobody is listening. The service did not start or is bound to a different port. It is not a firewall problem.
- timeout — the packet disappeared. Look at the firewall, security group, or routing.
Telling these three apart alone changes the scope of the investigation a great deal.
Checking where it is listening
If you get Connection refused, look from the server side.
ss -ltnp | grep 8080
LISTEN 0 4096 127.0.0.1:8080 0.0.0.0:* users:(("app",pid=42,fd=3))
↑ the trap is here
If it is 127.0.0.1:8080, it is reachable only from the host itself. This is the typical reason curl localhost:8080 works
inside the container but not from outside. It has to be 0.0.0.0:8080 or [::]:8080 to accept connections from outside.
Fix the bind address in the application settings (--host 0.0.0.0, bind = "0.0.0.0").
If this is wrong, no amount of fixing the port mapping will help.
One more layer in Kubernetes
When Pods cannot reach each other, suspect network policies first. In a namespace with default deny, every connection that is not written in a policy is a timeout.
kubectl -n <ns> get networkpolicy
kubectl -n <ns> describe networkpolicy <name> # read from/to
If the symptom is "it works sometimes", look at the endpoints, not the policy. If only one of three Pods is not Ready, only the requests sent to that one fail, which looks like intermittent errors from outside.
kubectl -n <ns> get endpoints <svc> # how many addresses are there?
What it looks like in the field
Assuming there is only one cause. Real outages often have two or more causes overlapping. If you fix one, say "still broken", and head in another direction, you end up doubting even what you already fixed. So it is faster in the end to check each layer all the way through and write down everything you find.
Not verifying after the fix. If you fixed hosts, check again with getent, and if you fixed the binding, knock again from an outside address. Reproducing the failure and confirming the success with the same command is the last step of a fix.
What you will do in the next lab
You get an outage with three defects at the same time. You narrow down the layers one by one to pin down the causes, fix them, and leave a postmortem in the required format.