See a Process With Your Own Eyes
Goal
You will build for yourself the things you read about in the operating systems book: zombies, orphans, virtual memory, and open files. You can create every one of them in a few lines inside this Pod.
What you learn here applies directly when you later investigate an outage. The answers to "the disk won't free up", "why is it using so much memory", and "the Pod won't die" are all here.
Where to look
Linux shows you kernel state as files.
ps -eo pid,ppid,stat,comm # 프로세스 목록
ls /proc/<PID>/task # 그 프로세스의 스레드들
ls -l /proc/<PID>/fd # 열어 둔 파일들
cat /proc/<PID>/status # 메모리를 포함한 상태 전부
/proc is not a real disk; it is a view the kernel creates for you.
When you start a process in the background
If Python output does not show up, you probably left out flush=True.
print(os.getpid(), flush=True)
Steps
- Process lineage →
01-tree.txt - Threads →
02-threads.txt - Zombie →
03-zombie.txt - Orphan and PID 1 →
04-orphan.txt - VSZ and RSS →
05-memory.txt - Deleted file →
06-deleted.txt - TERM and KILL →
07-signal.txt - Summary →
08-notes.md
Notes
In steps 5 and 6 you must record both numbers, because the before-and-after comparison is the whole point of those steps.
Who gave birth to whom
Extract the parent–child relationships of the processes currently running in the Pod and save them to 01-tree.txt. Also write down the PID of your own shell and its parent.
Use ps -eo pid,ppid,stat,comm. Your own shell is echo $$, and its parent is ps -o ppid= -p $$.
Every process has a parent. The parent creates a child with fork, and when the child finishes, the parent collects the result with wait. These two sentences are all you need for the next three steps.
Where are the threads
Start a program that creates 4 threads, and show that ps displays one process even though there are actually several. Record this in 02-threads.txt.
In Python, 4 threading.Thread objects are enough. Print the PID with print(os.getpid(), flush=True) (without flush=True the output will not appear).
To check, run ls /proc/<PID>/task | wc -l, which gives 5: 1 main thread plus 4 threads. On Linux a thread is just a task that shares memory, and to the kernel it is not very different from a process.
Create a zombie
Create a state where the child has finished but the parent has not collected it, so that ps shows it as Z, and record this in 03-zombie.txt.
Create a child with os.fork(), have the child call os._exit(0), and have the parent sleep without calling wait().
A zombie uses no memory. It is a slot that holds only the fact that the process has ended and its exit code. It does occupy a PID, though, so if they pile up you can no longer create new processes.
When the parent dies first
Make the parent die before its child, and record in 04-orphan.txt what the child's PPID changes to. Also record what PID 1 is in this Pod.
Check with ps -o ppid= -p <자식PID>, putting the child's PID in place of the placeholder. A process that loses its parent is handed over to PID 1 (reparenting).
Then look at ps -o comm= -p 1. PID 1 in a container is usually your application, not init. If it is not init, it does not reap the zombies it inherits. That is why zombies pile up in containers, and why people use --init or tini.
Reserved memory versus memory actually used
Start a program that only reserves 300MB and actually touches just 80MB of it, and write down how VSZ and RSS differ in 05-memory.txt.
Reserve the memory with mmap.mmap(-1, 300*1024*1024), and record the output of ps -o vsz=,rss= -p <PID> both before and after you touch it.
bytearray(300*1024*1024) will not work, because Python fills it with zeros and has already touched all of it.
VSZ is the address space that has been reserved, and RSS is what has actually been backed by physical memory. With 300MB reserved, RSS is well under 10MB.
Deleted, but the space does not come back
Create a 64MB file, delete it while it is still open, show with df that the space does not come back, and record in 06-deleted.txt that it returns once you kill the process.
Create it with dd if=/dev/zero of=big.bin bs=1M count=64, then in Python open() it, os.remove() it, and sleep.
Here is the evidence: ls -l /proc/<PID>/fd/ still lists it as big.bin (deleted).
Deleting a file name is not the same as deleting the data. The blocks are returned only when the last reference (a name or an open fd) is gone. This is exactly the cause of the incident where you delete a log but the disk does not free up.
The difference between TERM and KILL
Write a program that receives SIGTERM and handles it but does not die, and record in 07-signal.txt that it survives TERM and dies on KILL.
Use signal.signal(signal.SIGTERM, 핸들러), where the second argument is your handler function. To check, run kill -TERM <PID>, then kill -0 <PID> (it succeeds if the process is alive), and then kill -KILL <PID>.
TERM is a request and KILL is a notice of execution. TERM gives the program a chance to receive it and clean up, while the kernel simply removes the process on KILL, so it cannot clean up after itself. That is why Kubernetes sends TERM first, waits for terminationGracePeriodSeconds, and then sends KILL.
Summarize the three points
Write at least three lines in 08-notes.md: what a zombie is and why it appears, the difference between VSZ and RSS, and when the space of a deleted file comes back.
The text must contain 좀비 (zombie), RSS, and 참조 (reference).