TT Lab
Blog Get started
Learn Learning paths Courses

Linux Incident Response

Diagnosing File Descriptors and Limits

Continue in TT Lab

Goal

You check how many layers of limits there are for file descriptors, build a tool by hand to track fd leaks, and then separate services that raise the same Too many open files for different reasons according to the cause, and fix them.

Why it matters

When people see Too many open files, almost everyone runs ulimit -n 65536. But that value is my shell's value and has no effect on a service that is already running. The place to check is /proc/<PID>/limits, and the actual usage is ls /proc/<PID>/fd | wc -l (lsof -p counts even the cwd, executable and mmap'ed libraries, and these are not included in the limit). And raising the limit is rarely the answer - it is usually a leak, and then raising the limit only pushes the outage back a few hours. You tell a leak from load by the growth pattern of the fd count.

The incident in step 8 makes you actually draw that distinction. One service uses a constant amount but has a small limit, so it trips on momentary usage (the answer is to raise the limit without a restart); another is leaking, so even if you raise the limit it fills up again a few minutes later (the answer is to roll back to a state that does not leak). The service that uses the most fds is, if anything, healthy.

Steps

Work in the /root/fd directory.

  1. Read the open file limit from /proc/1/limits and write it to /root/fd/limits.txt as the following two lines. It may differ from your shell's value.
    • soft=<값>
    • hard=<값>
  2. Start in the background a process that has /etc/hostname open 50 or more times at once and write its PID to /root/fd/holder_pid.txt. It must stay alive through all the following steps.
  3. Of that process's fds, count those that point to /etc/hostname, and count the number of distinct targets they point to, then write both to /root/fd/distinct.txt as the following two lines.
    • count=<개수>
    • distinct=<개수>
  4. In an environment with a lowered limit, keep opening files to reproduce the Too many open files error, and save that message to /root/fd/emfile.txt.
  5. Write the system-wide fd limit, which is separate from the per-process limits, as a number only, to /root/fd/file_max.txt.
  6. Start an HTTP server listening on 127.0.0.1:8099 in the background and write its PID to /root/fd/http_pid.txt. Also check which entry in that process's fd table is the socket.
  7. Create /root/fd/fdtop.sh <N>. It prints the top N processes by number of open fds in the form <fd개수> <PID> in descending order. It must be exactly N lines.
  8. Incident incident ts-fd-emfile: the logs of two services are flooded with Too many open files. Classify tsfd-gateway, worker, and cache as one of fd-leak, low-soft-limit, healthy, or file-max, write three lines <서비스>: <코드> (service: code) and one evidence: line to /root/incident/ts-fd-emfile/report.txt, and fix them in the way that matches the cause. Bring gateway's soft limit to 1024 or more without a restart, including the configuration file, and do not touch cache.

Notes

Read the real limit

Read the open file limit from /proc/1/limits and write it to /root/fd/limits.txt as the following two lines. It may differ from your shell's value.

You have to look not at your shell's ulimit but at that process's limits table. Find the open files line in the table, and soft and hard are side by side.

Hold 50 fds

Start in the background a process that has /etc/hostname open 50 or more times at once and write its PID to /root/fd/holder_pid.txt. It must stay alive through all the following steps.

Even for the same file, each open creates one fd. Remember that if you do not hold on to what you opened somewhere, the language closes it right away, and make it a program that sleeps for a long time.

Count the fds and the targets

Of that process's fds, count those that point to /etc/hostname, and count the number of distinct targets they point to, then write both to /root/fd/distinct.txt as the following two lines.

Each entry in the fd table is a link. Collect the targets the links point to, and count the total and the count without duplicates separately. The point is what becomes how many when you open the same file several times.

Reproduce EMFILE

In an environment with a lowered limit, keep opening files to reproduce the Too many open files error, and save that message to /root/fd/emfile.txt.

Limits are inherited from parent to child. Lower the limit only in a subshell so you do not break your whole shell, and keep opening files inside it without closing them. Also note that the error comes out on standard error, not standard output.

Read the system-wide limit

Write the system-wide fd limit, which is separate from the per-process limits, as a number only, to /root/fd/file_max.txt.

The kernel-wide limit is under /proc/sys/fs. There is a file that shows only the limit, and one that shows it as three numbers together with the current allocation.

A socket is an fd too

Start an HTTP server listening on 127.0.0.1:8099 in the background and write its PID to /root/fd/http_pid.txt. Also check which entry in that process's fd table is the socket.

Python has an HTTP server module you can start with one line (it also has an option to bind the address to 127.0.0.1). In the fd table, a socket shows up as a target that does not look like a file path.

Pick the processes with the most fds

Create /root/fd/fdtop.sh <N>. It prints the top N processes by number of open fds in the form <fd개수> <PID> in descending order. It must be exactly N lines.

Count the entries in each process's fd table and sort numerically. Quietly skip processes that exit while you scan and entries you cannot read; in the next step's incident you use this tool to narrow down the suspects.

Incident: two services raise the same Too many open files

Running incident ts-fd-emfile floods the logs of two services with Too many open files. Classify each of the three services tsfd-gateway, tsfd-worker, and tsfd-cache as one of fd-leak (fds keep growing), low-soft-limit (usage is constant but the soft limit is small), healthy (uses a lot but stays constant within the limit), or file-max (reached the system-wide limit), and fix them in the way that matches the cause. End state: no more EMFILE is appended to the two logs, gateway has a soft limit of 1024 or more as the very same process, without a restart, and /etc/tsfd/*.conf is updated too so the same outage does not come back at the next restart. Do not touch cache. Manage services with tsfdctl. In /root/incident/ts-fd-emfile/report.txt, write three lines <서비스>: <코드> (service: code) and evidence: <관찰한 출력 한 줄> (one line of output you observed).

Direction: even for the same EMFILE, "the limit is small" and "it is leaking" have different prescriptions. Raising the limit is the answer only for the former; for the latter it just buys a little time. Your shell's ulimit has nothing to do with services that are already running. Tools: /proc//limits, /proc//fd, the service logs (/var/log/tsfd), the util-linux tool that changes the limits of an already running process from outside (look for it in man), tsfdctl. Concretely: for each service, count the fds twice, a few seconds apart, and look side by side at whether the count is growing and how close it is to the soft limit. The stats lines in the logs keep the fd count over time, and the deployment record is in /etc/tsfd/worker.conf and /opt/tsfd.