Turning "It Feels Slow" Into Four Numbers

Goal

Starting from the single line "the system has been slow lately," you answer four questions in numbers: what · since when · how much · is it always. When you finish, the investigation target is narrowed to a single version.

Environment

/opt/data/app.jsonl holds 340 requests, one per line.

{"ts": "2026-08-19T00:00:27Z", "level": "info", "service": "payment",
 "request_id": "req-0163", "path": "/api/pay", "status": 200,
 "latency_ms": 120, "msg": "ok"}

/opt/data/deploy.log holds the deploy and rollback history of the same day. jq and awk are included, so you do not need to install any new tools.

How to compute percentiles by hand

Sort, then pick the position. With 100 samples, p95 is the 95th value.

jq -r 'select(.path=="/login") | .latency_ms' /opt/data/app.jsonl   | sort -n | awk '{v[NR]=$1} END {print v[int(NR*0.95)]}'

Files to create

All are under /root/slow/.

by_path.txt   경로별 p95 — 전부가 느린 게 아니라는 증거
normal.txt    정상 경로의 기준선
when.txt      시간대별로 자른 결과
cause.txt     그 시각에 무슨 일이 있었나
delta.txt     문제 구간과 정상 구간을 나란히
shape.txt     평균 · p50 · p95 로 본 분포의 모양
errors.txt    같은 창에서 오류도 올랐는가
report.md     네 질문에 답한 한 장

Steps

Compute p95 per path and put the five paths side by side.
Write the p95 of the normal paths as the baseline. Without something to compare against, "slow" is not proven.
Pick only the slow path and slice it by time. Put several time slots side by side.
Cross-check what was in deploy.log at that time. Also find the reason the window closed — that is the other half of the correlation.
Write the p50 of the problem interval and the normal interval side by side and state how many times larger it is.
Put the mean, p50, and p95 together and judge whether it is always or sometimes.
Count whether 5xx also rose in the same window. Write the counts for both intervals together.
Write a one-page report that answers the four questions. Including the next step.

Notes

Half of it is already done at step 1. It is rare for everything to be slow. If you look only at the overall average without splitting by path, you get 1,380 ms, and that number applies to none of the paths.

Is everything slow

Compute p95 per path and put the five paths side by side.

Compute p95 separately for each path. Put the five side by side and only one differs in order of magnitude. Start by extracting the list of paths with jq -r '.path'.

What is normal

Write the p95 of the normal paths as the baseline. Without something to compare against, "slow" is not proven.

The rest, excluding the slow path, is the baseline. If the four paths have similar values, that is this system's normal.

Since when

Pick only the slow path and slice it by time. Put several time slots side by side.

Pick only the slow path and group by the time part of ts. .ts[11:13] gives only the hour. You have to put several time slots side by side to see where it spikes.

What happened at that time

Cross-check what was in deploy.log at that time. Also find the reason the window closed — that is the other half of the correlation.

Slice /opt/data/deploy.log to that time slot. Do not look only for the deploy; look for the record of reverting it too — the moment the window closed is as important as the moment it opened.

How slow is it

Write the p50 of the problem interval and the normal interval side by side and state how many times larger it is.

Measure p50 inside the window and outside the window the same way, put them side by side, and write how many times larger it is. Whether it is 10x or 10% decides the nature of the problem.

Always or sometimes

Put the mean, p50, and p95 together and judge whether it is always or sometimes.

Put the mean, p50, and p95 together. The overall average comes out as a value that applies to none of the paths. That is the reason not to trust the average.

Did errors rise too

Count whether 5xx also rose in the same window. Write the counts for both intervals together.

If you look only at latency and do not count status codes, you see only half the cause. Write the 5xx counts inside and outside the window together — the fact that one side is 0 confirms the window.

One page that answers the four questions

Write a one-page report that answers the four questions. Including the next step.

A non-engineer will read it. Answer what · since when · how much · is it always, and finish with the suspect and the next single step.