TT Lab
Blog Get started
Learn Learning paths Courses

Linux Incident Response

Diagnosing Disk and inodes

Continue in TT Lab

Goal

You measure disk capacity problems along two axes, blocks and inodes, deliberately create and recover from the situation "I deleted it but the space didn't shrink", and then, in a real incident mixed with decoys, find where the missing space is and get it back without stopping the service.

Why it matters

No space left on device does not mean "there are no blocks" but "the kernel has no resource to accommodate this write." That resource may be blocks, inodes, or free space excluding the space reserved for root. So you have to look at df -h and df -i together, and when df and du disagree, you must suspect deleted-but-open files. In particular, the situation you create in steps 5–6 is the form you meet most often in practice - and the moment you restart the process in that situation, your chance of recovery disappears with it. The order is the skill.

The incident in step 8 is exactly what happens in the field. Hidden dot directories and thousands of small files both plausibly fit "du doesn't show it". Diagnosis means measuring each candidate's size and keeping only the one that explains the missing share; killing the service to make the numbers match is not a fix.

Steps

Work in the /root/disk directory.

  1. Measure the inode usage (%) of the filesystem that /root belongs to and write the number only to /root/disk/inode_pct.txt. It is a different value from block usage.
  2. Measure the space the /etc directory actually occupies and write it as a number in KB only to /root/disk/etc_kb.txt. It is based on occupied blocks, not the sum of file sizes.
  3. Write the file name of the largest regular file directly under /usr/bin to /root/disk/biggest.txt.
  4. Create the /root/disk/many directory and put exactly 500 files of size 0 in it, then measure the space that directory occupies (du -sk) and write it to /root/disk/many_kb.txt.
  5. Create /root/disk/ghost.log of 10MB or more, and while some process keeps that file open, delete only the file name. Write that process's PID to /root/disk/ghost_pid.txt. That process must be alive at grading time.
  6. Recover the contents of the file that the process from step 5 is holding to /root/disk/recovered.log. The size must be the same as the original.
  7. Create /root/disk/findghosts.sh <최소바이트> (the argument is the minimum number of bytes). Across the whole system, it finds deleted-but-open files of at least that size and prints them one per line in the form pid=<PID> bytes=<크기>.
  8. Incident incident ts-disk-full: tsapp volume usage grew by about 130MB, but adding up with du does not account for that much. Get the space back without stopping or restarting the tsapp service and without deleting the archived logs or session files. Write the cause code (deleted-open, hidden-dotdir, small-files, reserved-blocks, or mount-shadow) and one line of evidence to /root/incident/ts-disk-full/report.txt.

Notes

Read inode usage

Measure the inode usage (%) of the filesystem that /root belongs to and write the number only to /root/disk/inode_pct.txt. It is a different value from block usage.

Blocks and inodes are different resources, so df shows them with different options too (look for inode in man df). Leave out the % sign and write only the number.

Measure the actual usage of a directory

Measure the space the /etc directory actually occupies and write it as a number in KB only to /root/disk/etc_kb.txt. It is based on occupied blocks, not the sum of file sizes.

du walks the path and counts "occupied blocks". There are options for summarizing and for choosing KB units. Also confirm that the result differs from the sum of the file sizes.

Find the largest file

Write the file name of the largest regular file directly under /usr/bin to /root/disk/biggest.txt.

Limit the depth to 1 so it does not descend into subdirectories, pick only regular files (excluding symbolic links), and order them by size. Write only the name, not the path.

Create 500 empty files

Create the /root/disk/many directory and put exactly 500 files of size 0 in it, then measure the space that directory occupies (du -sk) and write it to /root/disk/many_kb.txt. Think about why it is not 0 even though the content is 0 bytes.

To create empty files all at once, use the shell's brace expansion or a tool that lists numbers. Each file has no data block, but look at where the names themselves are stored.

Create a deleted-but-open file

Create /root/disk/ghost.log of 10MB or more, and while some process keeps that file open, delete only the file name. Write that process's PID to /root/disk/ghost_pid.txt. That process must be alive at grading time.

You need a long-lived process that keeps the file open (a command that keeps watching the end of a file is the classic example). Delete only the name after that process has opened the file, and look at what remains in that process's fd table after the deletion.

Recover the ghost file

Recover the contents of the file that the process from step 5 is holding to /root/disk/recovered.log. The size must be the same as the original.

The name is gone, but the file is still open in that process's fd table. If you find that fd under /proc, it reads like a file itself. Recover it before you touch the process.

Ghost file detection script

Create /root/disk/findghosts.sh <최소바이트> (the argument is the minimum number of bytes). Across the whole system, it finds deleted-but-open files of at least that size and prints them one per line in the form pid=<PID> bytes=<크기>.

Scan the fd tables of all processes and pick out deleted files by the marker appended to the end of the link target's name. Measure the size on the file beyond the link, not on the link itself. Quietly skip processes that have already exited; you use this tool in the next step's incident.

Incident: the volume fills up, but adding up with du does not account for it

Running incident ts-disk-full raises an incident in the tsapp service (read the ticket first). Symptom: volume usage grew by about 130MB, but adding up with du does not account for that much. Find where the missing share is and get the space back. However, you must not stop or restart tsapp-writer or tsapp-health (customer requests would be cut off), the writer must keep writing its log, and deleting normal data (the hidden archived logs, session files) to make the numbers match is not allowed either. In /root/incident/ts-disk-full/report.txt, write cause: <코드> and evidence: <관찰한 출력 한 줄> (one line of output you observed). The code is one of deleted-open (a deleted but open file), hidden-dotdir (a hidden dot directory), small-files (thousands of small files), reserved-blocks (root reserved blocks), or mount-shadow (files hidden under a mount).

Direction: the share du cannot count is either "a place du did not walk" or "blocks with no name, so nothing can walk to them". Measure each candidate's size and decide with numbers whether it explains the missing share (about 100MB). Tools: du (the way that counts dot files and the way that leaves them out), find, /proc//fd, stat. Concretely: compare du measured on the directory itself with du measured with *, and in the fd tables of all processes look at the marker appended to the end of the link targets and the size of the file beyond them. How the service reopens its log is described in a comment in /etc/tsapp/tsapp.conf.