Diagnosing Memory and OOM

Goal

You read memory metrics accurately, distinguish the memory a process has actually taken hold of from address space that was merely reserved, and confirm what the OOM Killer looks at when it chooses.

Why it matters

Most misdiagnoses of memory problems start with looking at the wrong column. You should look at the available column of free, not the free column, and at RES (RSS) in top, not VIRT. The OOM Killer's score is also RSS-based - a process that merely reserved 64GB with mmap is not even a candidate. And inside a container free shows the whole host, so the real limit must be read from the cgroup files. The reason step 7 of this lab even grades "if it isn't there, write that it isn't" is that in the field too, the first action is to check whether that file exists.

Steps

Work in the /root/mem directory.

1. From /proc/meminfo, write the MemTotal value as a kB number only to /root/mem/total_kb.txt.
2. Write the MemAvailable value, as a kB number only, to /root/mem/available_kb.txt.
3. Start a process whose actual resident memory (VmRSS) is 100MB or more in the background and write its PID to /root/mem/hog_pid.txt. It must stay alive through all the following steps.
4. Write the VmSize and VmRSS of that process to /root/mem/vm.txt as the following two lines.
   • vsize_kb=<값>
   • rss_kb=<값>
5. Set that process's oom_score_adj to 500, and write the oom_score value read after that to /root/mem/oom_score.txt.
6. Create /root/mem/topmem.sh <N>. It prints the top N processes by RSS in the form <rss_kb> <comm> in descending order. It must be exactly N lines.
7. Write the cgroup memory limit of this container, as it is, to /root/mem/limit.txt. If it is an environment where that file cannot be read, write unavailable.
8. Create /root/mem/memguard.sh <임계값kB> (the argument is the threshold in kB). If any process has an RSS above the threshold, print over=<PID> one per line and exit with code 1; if none, print one line ok and exit with code 0. With no argument, exit with a non-zero code.

Notes

• You can really hold memory with python3 -c "import time; b=bytearray(150*1024*1024); time.sleep(9999)" &.
• /proc/<PID>/status has VmSize and VmRSS, and /proc/<PID>/oom_score and oom_score_adj are separate files.
• Kernel threads have no VmRSS entry, so you must filter them out in steps 6 and 8.
• Common mistake 1: in step 3, if you only allocate memory without using it, only VmSize grows and VmRSS stays the same.
• Common mistake 2: in step 5, if you read oom_score first and change adj afterward, the value won't match.

Read total memory

From /proc/meminfo, write the MemTotal value as a kB number only to /root/mem/total_kb.txt.

It is the first entry in /proc/meminfo. Write only the number, in kB as it is.

Distinguish MemAvailable from MemFree

Write the MemAvailable value, as a kB number only, to /root/mem/available_kb.txt.

Linux uses spare memory as page cache. There is a separate value that counts in the reclaimable amount.

Actually hold memory

Start a process whose actual resident memory (VmRSS) is 100MB or more in the background and write its PID to /root/mem/hog_pid.txt. It must stay alive through all the following steps.

If you only allocate and don't touch it, RSS doesn't grow. You have to actually fill it. Create a bytearray in python3 and sleep.

Compare VmSize and VmRSS

Write the VmSize and VmRSS of that process to /root/mem/vm.txt as the following two lines.

• vsize_kb=<값>
• rss_kb=<값>

Both are in /proc//status. Confirm the difference between reserved address space and actual resident memory.

Adjust the OOM score

Set that process's oom_score_adj to 500, and write the oom_score value read after that to /root/mem/oom_score.txt.

Just write the value to /proc//oom_score_adj. Raising it is always allowed. Read oom_score after changing it.

Pick the top RSS processes

Create /root/mem/topmem.sh <N>. It prints the top N processes by RSS in the form <rss_kb> <comm> in descending order. It must be exactly N lines.

Collect VmRSS from /proc/*/status and sort. Kernel threads have no VmRSS entry.

Find the container's real limit

Write the cgroup memory limit of this container, as it is, to /root/mem/limit.txt. If it is an environment where that file cannot be read, write unavailable.

The free command shows the host. The real ceiling is in the cgroup files. If it isn't there, write that it isn't.

Memory watch script

Create /root/mem/memguard.sh <임계값kB> (the argument is the threshold in kB). If any process has an RSS above the threshold, print over=<PID> one per line and exit with code 1; if none, print one line ok and exit with code 0. With no argument, exit with a non-zero code.

Take the threshold as an argument and report processes that exceed it. If there are any, exit code 1; if none, ok and 0.