Diagnosing File Descriptors and Limits

Goal

You check how many layers of limits there are for file descriptors, and build a tool by hand to track fd leaks.

Why it matters

When people see Too many open files, almost everyone runs ulimit -n 65536. But that value is my shell's value and has no effect on a service that is already running. The place to check is /proc/<PID>/limits, and the actual usage is ls /proc/<PID>/fd | wc -l (lsof -p counts even the cwd, executable and mmap'ed libraries, and these are not included in the limit). And raising the limit is rarely the answer - it is usually a leak, and then raising the limit only pushes the outage back a few hours. You tell a leak from load by the growth pattern of the fd count.

Steps

Work in the /root/fd directory.

1. Read the open file limit from /proc/1/limits and write it to /root/fd/limits.txt as the following two lines.
   • soft=<값>
   • hard=<값>
2. Start in the background a process that has /etc/hostname open 50 or more times at once and write its PID to /root/fd/holder_pid.txt. It must stay alive through all the following steps.
3. Of that process's fds, count those that point to /etc/hostname, and count the number of distinct targets they point to, then write both to /root/fd/distinct.txt as the following two lines.
   • count=<개수>
   • distinct=<개수>
4. In an environment with a lowered limit, keep opening files to reproduce the Too many open files error, and save that message to /root/fd/emfile.txt.
5. Write the system-wide fd limit, as a number only, to /root/fd/file_max.txt.
6. Start python3 -m http.server 8099 --bind 127.0.0.1 in the background and write its PID to /root/fd/http_pid.txt.
7. Create /root/fd/fdtop.sh <N>. It prints the top N processes by number of open fds in the form <fd개수> <PID> in descending order. It must be exactly N lines.
8. Write the inode number of /etc/hostname to /root/fd/inode.txt.

Notes

• You can hold fds with python3 -c "fs=[open('/etc/hostname') for _ in range(60)]; import time; time.sleep(9999)" &. If you don't keep them in a list, they are closed immediately.
• You can lower the limit in a subshell like bash -c 'ulimit -n 32; python3 -c "..."' 2> /root/fd/emfile.txt.
• /proc/sys/fs/file-nr has three numbers: 할당된 수 / 미사용 할당분 / 최대값 (allocated / unused allocated / maximum).
• Common mistake 1: in step 7, errors come from /proc/self or processes that have already exited. Skip the entries that fail.
• Common mistake 2: in step 4, if you try to capture the error through standard output, nothing is captured. The error comes out on standard error.

Read the real limit

Read the open file limit from /proc/1/limits and write it to /root/fd/limits.txt as the following two lines.

• soft=<값>
• hard=<값>

You have to look not at your shell's ulimit but at that process's /proc//limits. The 'Max open files' line has soft and hard side by side.

Hold 50 fds

Start in the background a process that has /etc/hostname open 50 or more times at once and write its PID to /root/fd/holder_pid.txt. It must stay alive through all the following steps.

In python3, open the same file several times, keep them in a list and sleep. If you don't keep them in a variable, they are closed right away.

Count the fds and the targets

Of that process's fds, count those that point to /etc/hostname, and count the number of distinct targets they point to, then write both to /root/fd/distinct.txt as the following two lines.

• count=<개수>
• distinct=<개수>

Readlink each entry of /proc//fd to see its target. See what happens when you open the same file several times.

Reproduce EMFILE

In an environment with a lowered limit, keep opening files to reproduce the Too many open files error, and save that message to /root/fd/emfile.txt.

Lower ulimit -n in a subshell and keep opening files inside it. Save the error message to a file.

Read the system-wide limit

Write the system-wide fd limit, as a number only, to /root/fd/file_max.txt.

/proc/sys/fs/file-nr has three numbers. The last one is the maximum.

A socket is an fd too

Start python3 -m http.server 8099 --bind 127.0.0.1 in the background and write its PID to /root/fd/http_pid.txt.

Start python3 -m http.server on 127.0.0.1 and readlink the fds of that process.

Pick the processes with the most fds

Create /root/fd/fdtop.sh <N>. It prints the top N processes by number of open fds in the form <fd개수> <PID> in descending order. It must be exactly N lines.

Count the entries in /proc/*/fd and sort. Skip the entries that give a permission error.

Trace from fd to inode

Write the inode number of /etc/hostname to /root/fd/inode.txt.

If you follow the symlink with stat -L, you can learn the inode of the file that fd points to.