Diagnosing File Descriptors and Limits
Goal
You check how many layers of limits there are for file descriptors, build a tool by hand to track fd leaks, and then separate services that raise the same Too many open files for different reasons according to the cause, and fix them.
Why it matters
When people see Too many open files, almost everyone runs ulimit -n 65536. But that value is my shell's value and has no effect on a service that is already running. The place to check is /proc/<PID>/limits, and the actual usage is ls /proc/<PID>/fd | wc -l (lsof -p counts even the cwd, executable and mmap'ed libraries, and these are not included in the limit). And raising the limit is rarely the answer - it is usually a leak, and then raising the limit only pushes the outage back a few hours. You tell a leak from load by the growth pattern of the fd count.
The incident in step 8 makes you actually draw that distinction. One service uses a constant amount but has a small limit, so it trips on momentary usage (the answer is to raise the limit without a restart); another is leaking, so even if you raise the limit it fills up again a few minutes later (the answer is to roll back to a state that does not leak). The service that uses the most fds is, if anything, healthy.
Steps
Work in the /root/fd directory.
- Read the open file limit from
/proc/1/limitsand write it to/root/fd/limits.txtas the following two lines. It may differ from your shell's value.soft=<값>hard=<값>
- Start in the background a process that has
/etc/hostnameopen 50 or more times at once and write its PID to/root/fd/holder_pid.txt. It must stay alive through all the following steps. - Of that process's fds, count those that point to
/etc/hostname, and count the number of distinct targets they point to, then write both to/root/fd/distinct.txtas the following two lines.count=<개수>distinct=<개수>
- In an environment with a lowered limit, keep opening files to reproduce the
Too many open fileserror, and save that message to/root/fd/emfile.txt. - Write the system-wide fd limit, which is separate from the per-process limits, as a number only, to
/root/fd/file_max.txt. - Start an HTTP server listening on
127.0.0.1:8099in the background and write its PID to/root/fd/http_pid.txt. Also check which entry in that process's fd table is the socket. - Create
/root/fd/fdtop.sh <N>. It prints the top N processes by number of open fds in the form<fd개수> <PID>in descending order. It must be exactly N lines. - Incident
incident ts-fd-emfile: the logs of two services are flooded withToo many open files. Classify tsfd-gateway, worker, and cache as one offd-leak,low-soft-limit,healthy, orfile-max, write three lines<서비스>: <코드>(service: code) and oneevidence:line to/root/incident/ts-fd-emfile/report.txt, and fix them in the way that matches the cause. Bring gateway's soft limit to 1024 or more without a restart, including the configuration file, and do not touch cache.
Notes
- Incident tools:
incident listshows the incidents you can raise, andincident show ts-fd-emfileshows the ticket again. Raising the same incident again returns it to its original state. - Report format: one line
tsfd-<이름>: <코드>(name: code) per service, and at the endevidence: <판단의 근거가 된 출력 한 줄>(one line of output your judgment was based on). - Manage the incident's services with
tsfdctl status|start|stop|restart <서비스>(service); the configuration is in/etc/tsfd/*.conf. You can change the soft limit of another running process without privileges, as long as it does not exceed hard. /proc/sys/fs/file-nrhas three numbers:할당된 수 / 미사용 할당분 / 최대값(allocated / unused allocated / maximum).- Common mistake 1: in step 7, errors come from
/proc/selfor processes that have already exited. Skip the entries that fail. - Common mistake 2: in step 4, if you try to capture the error through standard output, nothing is captured. The error comes out on standard error.
- Common mistake 3: in step 8, judging from an fd count taken only once. Only by counting twice, a few seconds apart, can you tell "constant" from "growing".
Read the real limit
Read the open file limit from /proc/1/limits and write it to /root/fd/limits.txt as the following two lines. It may differ from your shell's value.
soft=<값>hard=<값>
You have to look not at your shell's ulimit but at that process's limits table. Find the open files line in the table, and soft and hard are side by side.
Hold 50 fds
Start in the background a process that has /etc/hostname open 50 or more times at once and write its PID to /root/fd/holder_pid.txt. It must stay alive through all the following steps.
Even for the same file, each open creates one fd. Remember that if you do not hold on to what you opened somewhere, the language closes it right away, and make it a program that sleeps for a long time.
Count the fds and the targets
Of that process's fds, count those that point to /etc/hostname, and count the number of distinct targets they point to, then write both to /root/fd/distinct.txt as the following two lines.
count=<개수>distinct=<개수>
Each entry in the fd table is a link. Collect the targets the links point to, and count the total and the count without duplicates separately. The point is what becomes how many when you open the same file several times.
Reproduce EMFILE
In an environment with a lowered limit, keep opening files to reproduce the Too many open files error, and save that message to /root/fd/emfile.txt.
Limits are inherited from parent to child. Lower the limit only in a subshell so you do not break your whole shell, and keep opening files inside it without closing them. Also note that the error comes out on standard error, not standard output.
Read the system-wide limit
Write the system-wide fd limit, which is separate from the per-process limits, as a number only, to /root/fd/file_max.txt.
The kernel-wide limit is under /proc/sys/fs. There is a file that shows only the limit, and one that shows it as three numbers together with the current allocation.
A socket is an fd too
Start an HTTP server listening on 127.0.0.1:8099 in the background and write its PID to /root/fd/http_pid.txt. Also check which entry in that process's fd table is the socket.
Python has an HTTP server module you can start with one line (it also has an option to bind the address to 127.0.0.1). In the fd table, a socket shows up as a target that does not look like a file path.
Pick the processes with the most fds
Create /root/fd/fdtop.sh <N>. It prints the top N processes by number of open fds in the form <fd개수> <PID> in descending order. It must be exactly N lines.
Count the entries in each process's fd table and sort numerically. Quietly skip processes that exit while you scan and entries you cannot read; in the next step's incident you use this tool to narrow down the suspects.
Incident: two services raise the same Too many open files
Running incident ts-fd-emfile floods the logs of two services with Too many open files. Classify each of the three services tsfd-gateway, tsfd-worker, and tsfd-cache as one of fd-leak (fds keep growing), low-soft-limit (usage is constant but the soft limit is small), healthy (uses a lot but stays constant within the limit), or file-max (reached the system-wide limit), and fix them in the way that matches the cause. End state: no more EMFILE is appended to the two logs, gateway has a soft limit of 1024 or more as the very same process, without a restart, and /etc/tsfd/*.conf is updated too so the same outage does not come back at the next restart. Do not touch cache. Manage services with tsfdctl. In /root/incident/ts-fd-emfile/report.txt, write three lines <서비스>: <코드> (service: code) and evidence: <관찰한 출력 한 줄> (one line of output you observed).
Direction: even for the same EMFILE, "the limit is small" and "it is leaking" have different prescriptions. Raising the limit is the answer only for the former; for the latter it just buys a little time. Your shell's ulimit has nothing to do with services that are already running. Tools: /proc//limits, /proc//fd, the service logs (/var/log/tsfd), the util-linux tool that changes the limits of an already running process from outside (look for it in man), tsfdctl. Concretely: for each service, count the fds twice, a few seconds apart, and look side by side at whether the count is growing and how close it is to the soft limit. The stats lines in the logs keep the fd count over time, and the deployment record is in /etc/tsfd/worker.conf and /opt/tsfd.