The moment your hand slips while investigating

In one line

If you change something during the investigation, from then on you can no longer tell whether every value we see is the original value or one we created. If you must change something, first freeze the state before the change, then change it, check, and roll back.

Why this was needed

An investigation at a customer site usually starts with read permission. But after about half a day, a moment like this certainly comes — "if I just raised this one setting for a moment, I'd know right away." It is true. Raise that value and you get the answer in 5 minutes. The problem is the world after those 5 minutes.

After you change it, three things blur at the same time. First, you cannot tell whether the symptom you see now is the original symptom or one we created. Second, the logs and metrics the customer saw in the meantime get contaminated. Third, later, to the question "since when has it been like this?", we become part of the answer.

There is one common misconception here. It is "it's a small change, so it's fine." The smaller the change, the more easily you forget to roll it back. And believing you rolled it back and it being proven that you rolled it back are different.

How it works

The order is four steps.

1  굳힌다   바꾸기 전 상태를 지문으로 기록한다 (파일 해시·크기·권한)
2  바꾼다   한 건만, 무엇을 왜 바꾸는지와 되돌리는 명령을 함께 적고
3  확인한다 바꾼 것이 의도한 효과를 냈는지 값으로 본다
4  되돌린다 되돌린 뒤 1번의 기록과 견주어 다르지 않음을 보인다

Step 1 is the one most often missing. If you do not freeze it, there is nothing to prove at step 4. "I rolled it back to the original" is a sentence that cannot be verified, and the customer can neither believe it nor disbelieve it. On the other hand, "after rolling back, I compared with the snapshot and the number of files with different contents is 0" is a sentence that can be checked.

What should the frozen record contain? Always include the file's content fingerprint (sha256), size, and permissions. Include the modification time too, but handle it separately when comparing. This is because a file that was rolled back has the same content but a different modification time — that is not a failure but a mark of our having touched it. Do not try to erase the mark; write it in the report.

But there is something you have to ask even before that. Do you really have to change it to get the answer? In many cases there is a way to answer the same question read-only. Open the configuration file and read the value, open the database read-only and count, count the log, run an existing lookup tool. You can spend about 30 minutes looking for this way and then decide to change it, and it is not too late.

The way not to deceive yourself about whether it was read-only is to record the commands you used.

You also have to decide the scope of what you freeze. If you fingerprint the whole server, it takes a long time, and because the logs keep growing in the meantime, the next comparison turns entirely red. Conversely, if you fingerprint only the one file you will change, you miss what changed alongside it. In practice you take one level of the directory the target belongs to — if you freeze the configuration directory as a whole, even a backup file or temporary file left by an editor shows up as it is. For places that keep growing, like logs, either include them in the scope but handle them separately when comparing, or leave them out of the scope from the start and write down that fact.

Keep the frozen files themselves in a safe place too. If you create the snapshot inside the directory under investigation, that file gets caught as a 'newly created file' in the next comparison, and worse, it means we left a file on the customer's server. Keep the snapshot and the records in our own working directory and hand them over together when you hand over. If you write down the command that produced each answer, you can see whether an UPDATE or a redirection is mixed in among them. If you do not write it down, all that is left is the memory of "I only queried".

What it looks like in the field

First, change one thing at a time. If you change two things together, you do not know which side the effect came from. And when rolling back too, you must roll back one at a time to know what is left.

Second, write the rollback command before you change. If you try to write it after changing, you end up relying on memory for the original value. Whether it was 3 or 30 becomes astonishingly blurry 30 minutes later.

Third, leave the approval as a sentence. A temporary change during an investigation also needs someone's consent. One line such as "verbal approval from section chief Choi of the operations team, 13:40" is enough. Without this line, that change later becomes something we did in secret.

Fourth, when proving the rollback, do not hide the difference in modification time. If the content is the same and only the time differs, write it as it is: "content identical, only the modification time changed". If you set even the time back to the original, from then on we have erased the trace.

Fifth, this is a different matter from an approved change job. A change that you plan, set abort criteria for, and apply after getting customer approval has a separate discipline. What is covered here is the stage before that — the first week's discipline for the moment your hand slips during an investigation.

What really matters in practice

First look for a way to answer read-only. If you write down the commands you used, you will not deceive yourself.
Freeze before you change. If you do not freeze it, you cannot prove that you rolled it back.
One thing at a time, with the rollback command written in advance.
Deliver the rollback not as words but as a comparison result. Write the remaining marks instead of erasing them.

What you will do in the next lab

You hold the production server of (fictional) Suwon Pay. First you answer five questions using only read-only means and leave the commands you used alongside. Then you build a tool that freezes the state of a directory as fingerprints and add a comparison feature to it — the grader builds places where a file was added, deleted, had its content changed, had only its permissions changed, and had only its time changed, and checks whether your tool separates the five. Then you change one line of configuration that you must change, together with the approval and the rollback command, check the effect with the lookup tool, and actually roll it back to prove that the number of files whose contents differ from the snapshot is 0.