See For Yourself What Rotation Loses

Goal

Build the two log rotation methods yourself and see what each one loses. Reading an explanation is different from seeing a file left at 0 bytes.

Why it matters

If you set up rotation wrongly, one of two things happens.

• create without sending SIGHUP → the new log file stays at 0 bytes forever, and the real logs keep flowing into the .1 file. Nothing is lost, but nobody can find it
• copytruncate → logs written between the copy and the truncate disappear

If you discover either during an incident investigation, it is already too late.

Steps

1. A process that keeps writing → /root/log/01-pid.txt
2. Space that doesn't come back after deletion → /root/log/02-deleted.txt
3. Cleaning up with a restart → /root/log/03-restart.txt
4. Simulate the rename method → /root/log/04-rename.txt
5. Simulate the copytruncate method → /root/log/05-truncate.txt
6. Write a logrotate configuration → /root/log/06-app.conf
7. What is lost → /root/log/07-tradeoff.md
8. Clean up → /root/log/08-clean.txt

Notes

• You get a background process's PID with echo $!.
• : > 파일 truncates a file to 0 bytes (the placeholder is the file name).
• Containers usually don't use logrotate. If you write to stdout, the container runtime does the rotation for you — that is also a concept you confirm in this lab.

Create a process that keeps writing

A process that keeps writing → /root/log/01-pid.txt

Start a process in the background that keeps writing logs. It has to write with the file kept open — every phenomenon this lab is meant to show comes from an "open file."

mkdir -p /root/log
( i=0; while true; do i=$((i+1)); echo "line $i"; sleep 0.05; done ) >> /root/log/app.log &
echo $! > /root/log/01-pid.txt

The key is putting the redirection outside the loop. If you put it inside, as in echo ... >> 파일 (the placeholder is the file name), it opens and closes the file every time, so no handle stays behind when you delete the file, and after a rename it writes to the new file. A real daemon opens the file once and keeps writing.

See that space doesn't come back after deletion

Space that doesn't come back after deletion → /root/log/02-deleted.txt

After running rm /root/log/app.log, look at ls -l /proc/<PID>/fd | grep deleted. Save the result to /root/log/02-deleted.txt. The file is gone, but the process is still writing, and its blocks are not returned.

Restart to clean up

Cleaning up with a restart → /root/log/03-restart.txt

Kill that process (kill <PID>) and start it again. In /root/log/03-restart.txt, write two lines: the result of ls -l /proc/*/fd 2>/dev/null | grep -c deleted after killing it, and one line on why a restart is the fix.

Simulate the rename method

Simulate the rename method → /root/log/04-rename.txt

Start a new writer, run mv /root/log/app.log /root/log/app.log.1, wait 5 seconds, and then look at the line counts of the two files with wc -l. Write the line counts of the two files in /root/log/04-rename.txt. The new app.log will not even have been created — that is why SIGHUP is needed.

Simulate the copytruncate method

Simulate the copytruncate method → /root/log/05-truncate.txt

Copy and then truncate with cp /root/log/app.log.1 /root/log/app.log.2 && : > /root/log/app.log.1. Then look at wc -l /root/log/app.log.1 5 seconds later. In /root/log/05-truncate.txt, write the line counts right after the truncate and 5 seconds later. The goal is to confirm that the process keeps writing.

Write a logrotate configuration

Write a logrotate configuration → /root/log/06-app.conf

Write a logrotate configuration in /root/log/06-app.conf. What you need: a path pattern, daily, rotate 14, compress, delaycompress, missingok, notifempty, and one of create or copytruncate. It is fine if the logrotate command is not available — the goal is to be able to read and write the configuration.

Write down what is lost

What is lost → /root/log/07-tradeoff.md

Write four lines in /root/log/07-tradeoff.md: what create loses / what copytruncate loses / the approach you chose / why. The conclusion of this lab is that neither is free.

Clean up

Clean up → /root/log/08-clean.txt

Terminate with the PID you recorded: kill $(cat /root/log/01-pid.txt). Then delete the files with rm -f /root/log/app.log*, confirm nothing is left with something like pgrep -f sleep | wc -l, and write that in /root/log/08-clean.txt.

⚠️ Do not use pkill -f 'while true'. The -f option of pkill includes its own command line among the things it checks, so it kills the very shell that contains that string. In real work too, it is common to cut off your own session with pkill -f — it is always safer to record the PID and kill with that.