One byte offset is not enough
In one line
A collector that follows a file has to write how far it has read to disk, and a single byte position is not enough to remember. It also has to know whether the file has become a different file with the same name (rotation) and whether it has become smaller than the place it read up to (truncation).
Why this was needed
After an overnight deployment, the logs on the dashboard suddenly stopped. The collector Pod was alive and there were no error logs. After a restart, they flowed again.
The cause was log rotation. At midnight, logrotate moved app.log to app.log.1 and created a new app.log. The collector held only the memory "I have read up to byte 320 million," and the new file was only a few kilobytes. When it tried to read from that position there was nothing to read, and for several hours, until the new file grew past that position, nothing was collected.
How it works
A collector that follows a file has to handle three kinds of state change.
| Situation | What you see in the file | What to do |
|---|---|---|
| Append | Only the size grows, the inode stays the same | Keep reading from where you were |
| Rotation (rename) | A different inode under the same name | Read the new file from the beginning |
| Truncation | The inode stays the same but the size shrinks | Read that file from the beginning |
So the position record includes at least the inode and the offset together. Fluent Bit puts this state in a SQLite database (the DB setting), Fluentd puts it in pos_file, and Promtail puts it in positions.yaml. The names differ, but the job is the same.
One more thing remains here — a rotated file may still have an unread tail. If rotation happens while the collector is paused for a moment, it misses the last few lines of the old file. Real collectors don't drop the open file descriptor right away but hold on to it a little longer to read this tail (settings such as Rotate_Wait are that). In container environments, the rotation interval is short and files are small, so this window is more dangerous.
You also have to decide the policy for when the position file is missing. If you read from the beginning, you resend lines already sent and get duplicates, and if you read from the end, you lose all the lines that piled up until then. Neither is free. So the position file should be kept somewhere that survives a Pod restart (a volume), and if that is not possible, reading from the beginning and accepting duplicates is usually safer — duplicates can be collapsed later, but lost lines cannot be recovered.
What it looks like in the field
The most common incident is the "quiet stop after rotation" above. The symptom is quiet, so if nobody is watching the dashboard it can last for days. The only defense is to measure the collector's own metrics (bytes read, number of open files) and alert when they drop to 0.
The second is keeping the position file in an emptyDir. When the Pod restarts, the memory is gone, and depending on the policy, either duplicates pour in or gaps appear.
The third is the copytruncate method. It copies the original and then truncates the original to 0, so the inode stays the same while the size shrinks. A collector that looks only at the inode misses this change, and reads nothing until the file grows back past the old position.
What you will do in the next lab
You run an "application" that writes logs with deterministic sequence numbers, and build a collector that keeps reading from where it left off. You cause an append, a rotation, and a truncation in turn, check by sequence number each time that loss and duplication are both 0, and experiment on a copy to see what happens when the position file is missing, recording the numbers. At the end, you get a situation that mixes rotation and truncation through in one go.