Tying Scattered Logs Into One Causal Chain

Goal

You will be able to overlay three logs of different formats and tie the scattered facts together into a single causal chain.

Why it matters

Looking at the error log first is natural, but it means reading from the middle of the story. A typical degradation proceeds in the order change → some requests delayed (warning) → resource holding → timeouts (errors) → alarm, and if you look only at the error log, you see only the last two stages. So the question to ask is not "when did the errors start" but "when did it stop being normal," and the time of the first appearance of the warning level is that answer.

Another thing you will confirm in this lab is the powerlessness of the average. The overall average latency does not represent any request, but "how many requests exceed 1000 ms" immediately reveals the problem. If you must pick out just one metric, pick the count over a threshold, not the average.

Three logs

• /opt/data/app.jsonl — one JSON per line. ts, level, service, request_id, path, status, latency_ms, msg
• /opt/data/db-slow.log — ts=<epoch> duration_ms=<n> table=<name> query="..."
• /opt/data/deploy.log — deployment and rollback history

Steps

1. Create the /root/correlate directory.
2. Write the number of lines in app.jsonl whose level is error in /root/correlate/error_count.txt.
3. Write the table name that the error message points to in /root/correlate/table.txt.
4. Round down the average of all latency_ms to an integer and write it in /root/correlate/avg_latency.txt.
5. Write the number of requests whose latency_ms exceeds 1000 in /root/correlate/slow_count.txt.
6. Write the time of the earliest line whose level is warn, as HH:MM, in /root/correlate/first_warn.txt.
7. Write the payment service version deployed right before the incident, from deploy.log, in /root/correlate/version.txt.
8. Summarize the causal chain in /root/correlate/link.md. It must contain the deployment version, the table that slowed down, the time of the first warning, and the time of the error.

Notes

• grep '"level": "error"' /opt/data/app.jsonl | wc -l
• It may be easier to parse with a python3 one-liner: python3 -c "import json,sys; ..."
• If you have jq: jq -r 'select(.level=="warn") | .ts' /opt/data/app.jsonl | sort | head -1
• Common mistake 1: rounding in step 4. It is rounding down.
• Common mistake 2: writing the first time of error in step 6. The warn is earlier, and that gap is the point of this lab.

Create the working directory

Create the /root/correlate directory.

You collect the results under /root/correlate.

Count the error logs

Write the number of lines in app.jsonl whose level is error in /root/correlate/error_count.txt.

In app.jsonl each line is one JSON. Count the lines whose level field is error.

Find the causal table

Write the table name that the error message points to in /root/correlate/table.txt.

The answer is in the error message itself. Properly read just one line of the msg field.

Work out the average latency

Round down the average of all latency_ms to an integer and write it in /root/correlate/avg_latency.txt.

Round down the average of all latency_ms to an integer. This value is contrasted with the later steps.

Count the slow requests

Write the number of requests whose latency_ms exceeds 1000 in /root/correlate/slow_count.txt.

It is the number of requests whose latency_ms is over 1000. What the average alone could not show comes out.

Find the time of the first warning

Write the time of the earliest line whose level is warn, as HH:MM, in /root/correlate/first_warn.txt.

It is the HH:MM of the earliest ts among the lines whose level is warn. It comes before the errors.

Find the version deployed just before

Write the payment service version deployed right before the incident, from deploy.log, in /root/correlate/version.txt.

It is the version of the payment service deployed right before the time of the incident in deploy.log.

Summarize the causal chain

Summarize the causal chain in /root/correlate/link.md. It must contain the deployment version, the table that slowed down, the time of the first warning, and the time of the error.

Tie the deployment version, the table that slowed down, the time of the first warning, and the time of the error together in one document.