TT Lab
Get started
Learn Learning paths Courses

Finding the Cause in Logs

Building a Timeline From Five Events

Continue in TT Lab

Goal

You will be able to gather the five times scattered across several logs to build a timeline, and calculate the impact duration from the user's point of view.

Why it matters

What the customer reads first in an outage report is not the root cause analysis but the timeline. That is because it shows at the same time "how long we did not know" and "how quickly we moved once we knew."

The four intervals between the five times each have a name. Change → impact is the incubation interval, impact → awareness is the detection delay, awareness → action is the response delay, and action → recovery confirmation is the verification interval. Among these, if the detection delay is large, what to fix is not the system but observation. And a report without a verification interval has said only up to "we think it is fixed."

Impact duration is counted from the impact start to the recovery confirmation. That is because users do not know when the deployment was, and they get out of the impact not at the moment the rollback command went in but at the moment things actually became normal.

Three logs: /opt/data/deploy.log, /opt/data/app.jsonl, /opt/data/alert.log

Steps

  1. Create the /root/timeline directory.
  2. Write the time payment 2.7.0 was deployed, as HH:MM, in /root/timeline/t_deploy.txt.
  3. Write the time the first error occurred in /root/timeline/t_error.txt.
  4. Write the time the critical alarm went to firing in /root/timeline/t_alert.txt.
  5. Write the time the rollback was performed in /root/timeline/t_rollback.txt.
  6. Write the time the alarm changed to resolved in /root/timeline/t_resolved.txt.
  7. Write /root/timeline/timeline.md. All five times must be included and must be arranged in time order within the file.
  8. Write the number of minutes from the impact start to the recovery confirmation, as a number only, in /root/timeline/mttr.txt.

Notes

Create the working directory

Create the /root/timeline directory.

You collect the results under /root/timeline.

Find the change time

Write the time payment 2.7.0 was deployed, as HH:MM, in /root/timeline/t_deploy.txt.

It is the time payment 2.7.0 was deployed in deploy.log. Write only HH:MM.

Find the impact start time

Write the time the first error occurred in /root/timeline/t_error.txt.

It is when users actually began to experience failures. Look at the first error time in app.jsonl.

Find the awareness time

Write the time the critical alarm went to firing in /root/timeline/t_alert.txt.

It is the time critical went into the firing state in alert.log.

Find the action time

Write the time the rollback was performed in /root/timeline/t_rollback.txt.

The rollback is recorded in deploy.log.

Find the recovery confirmation time

Write the time the alarm changed to resolved in /root/timeline/t_resolved.txt.

It is the time the same rule changed to resolved in alert.log.

Write the timeline document

Write /root/timeline/timeline.md. All five times must be included and must be arranged in time order within the file.

All five times must be included and arranged in time order within the file.

Calculate the impact duration

Write the number of minutes from the impact start to the recovery confirmation, as a number only, in /root/timeline/mttr.txt.

The honest interval from the user's point of view is from the impact start to the recovery confirmation. Write only the number in minutes.