TT Lab
Get started
Learn Learning paths Courses

In Front of an Unfamiliar System

Answer without touching, and if you touch, put it back

Continue in TT Lab

Goal

You answer five questions about a customer's production server using only read-only means. Then you build a tool that freezes and compares the state of a directory with fingerprints, change one line of configuration that you must change, check it, and actually roll it back to prove that the number of files whose contents differ from the snapshot is 0.

Why it matters

If you change something during an investigation, you can no longer tell whether the values you see afterward are the original values or ones we created. The metrics the customer saw in the meantime also get contaminated, and to the question "since when has it been like this?", we become part of the answer. The smaller the change, the more easily you forget to roll it back. And believing you rolled it back and it being proven that you rolled it back are different. "I put it back as it was" is a sentence that cannot be verified, and "after rolling back, I compared with the snapshot and the number of files with different contents is 0" is a sentence that can be checked. The question to ask even before that is whether you really have to change it to get the answer. In many cases there is a way to answer the same question read-only. The way not to deceive yourself about whether it was read-only is to record the commands you used. This lab is different from a change job that you apply after getting customer approval. It is the stage before that — the first week's discipline for the moment your hand slips during an investigation. The grader does not trust the wording you write down. It remeasures the five answers itself, and for the snapshot tool it builds places where a file was added, deleted, had its content changed, had only its permissions changed, and had only its time changed, and checks whether the five are separated.

Steps

  1. Create and run /root/readonly/gen_site.py to produce /root/readonly/site/. It holds the configuration, an order ledger with 240 rows, a log with 180 lines, and a lookup tool.
  2. Answer the five questions using only read-only means and write them in /root/readonly/answers.json. Leave the command you used together with each answer.
  3. Create /root/readonly/snapshot.py so that it walks the directory and records the fingerprint, size, permissions, and modification time of each file, and freeze the current state in /root/readonly/baseline.json.
  4. Add --baseline <스냅샷> (the placeholder is the snapshot) so that it outputs the comparison result. Sort into five: added, removed, changed, mtime_only, and same.
  5. Change max_retry in the configuration from 3 to 5, and write what you change and why, along with the rollback command, in /root/readonly/change.json.
  6. Check the effect of the change with the lookup tool and write it in /root/readonly/verify.json.
  7. Roll it back and leave the result of comparing with the snapshot in /root/readonly/revert_diff.json. The number of files with different contents must be 0.
  8. Write /root/readonly/readonly_report.md in four sections.

Notes

Get the production server in hand

Create and run /root/readonly/gen_site.py to produce /root/readonly/site/. It holds etc/app.conf, data/app.db (orders 240 rows), var/log/app.log (180 lines), and bin/report.py.

Think of the lookup tool report.py as a tool that already exists on the customer's server. We did not make it; it was already there, and all we can do is run it. After you build it, run the lookup tool once and see what you can ask.

Answer by reading only

In /root/readonly/answers.json, write the answers to the five questions. The ids are app_version, pending_orders, log_errors, backoff_ms, and oldest_pending_days, and for each answer you leave in how the command you actually used.

Open and read the configuration, open the ledger read-only and count, and count the lines of the log. Using the existing lookup tool is reading too. If a write is mixed into a command you put in how, the grader catches it, so check it yourself as you write.

Freeze the state before changing

Create /root/readonly/snapshot.py so that it walks the directory with --dir --out, and freeze the current state of site in /root/readonly/baseline.json. For each file include sha256, bytes, mode, and mtime.

If you do not freeze it, there is nothing to prove later that you rolled it back. Record paths as relative paths based on --dir, and do not include directories or symbolic links. Write mode as only the three permission digits and mtime as an integer in seconds.

Add the comparison feature

Add --baseline <스냅샷> (the placeholder is the snapshot) so that it outputs the comparison result. Sort into five: added, removed, changed, mtime_only, and same, and in changed put only the names of the items that differ in fields.

If you put the case where the content, the size, and the permissions are the same and only the modification time differs into changed, the result after a rollback is always red and gives no information. Counting that case separately is the heart of this tool. The grader builds all five cases and feeds them in.

Just one thing, with the rollback ready

Change max_retry in the configuration from 3 to 5, and write target_file, key, before, after, sha256_before, sha256_after, why, approved_by, baseline, apply, and rollback in /root/readonly/change.json. rollback must contain the original value.

Write the rollback command before you change. If you try to write it after changing, you end up relying on memory for the original value, and whether it was 3 or 30 becomes astonishingly blurry 30 minutes later. Leave the approval as one line too — without it, that change later becomes something we did in secret.

See by value whether the intended effect occurred

Check the effect of the change with the lookup tool and write question, command, before, after, and confirmed in /root/readonly/verify.json. question is retry_budget, and before and after are the lookup tool's output before and after the change.

The fact that you changed it and the fact that what you changed produced the intended effect are different. The lookup tool can be given another configuration file with --conf, so you can get before by running it once with a temporary configuration holding the value before the change. Write the output as a string as it is.

Roll back and prove it

After rolling back as in the rollback of change.json, leave the result of comparing with the snapshot in /root/readonly/revert_diff.json. added and removed must be empty and changed must be 0. A file that differs only in modification time remains in mtime_only.

Believing you rolled it back and it being proven that you rolled it back are different. The comparison result is the proof. Do not set the time back to hide the difference in modification time — from then on we would have erased the trace.

Write down what you touched and what you rolled back

In /root/readonly/readonly_report.md, write four sections: ## 읽기만으로 답한 것 ## 꼭 바꿔야 했던 한 건 ## 바꾼 뒤 무엇을 확인했나 ## 되돌린 증거 (the Korean headings mean "What we answered by reading only", "The one change we had to make", "What we checked after the change", and "The evidence of the rollback"). The ids of the five questions, the key and value you changed, the change in the lookup tool's output, and the numbers of the comparison result must all appear.

Do not write the report by hand; generate it from the four files answers, change, verify, and revert_diff. In the last section, write the remaining marks (the files that differ only in modification time) as they are — if you hide them, from then on we have erased the trace.