The Day Auto-Update Halted the GPU: New Libraries, Old Kernel Module

Status: Resolved (2026-10-13 ~ 14), recorded as of this writing. The update timestamp and package list were verified on 2026-10-16 by reading unattended-upgrades.log on the node, and the error message of that day was retrieved from operational notes. Why the pod error looked that way is an assumption. Exact driver version numbers are omitted as they are tied to specific nodes.

Table of Contents

• One-Line Summary
• 1. Symptoms: Errors That Do Not Point to the Root Cause
• 2. Cause: Dawn Auto-Update Three Days Ago
• 3. How to Verify
• 4. The Fix: Rebooting, and Why It Requires Human Judgment
• 5. Prevention: Excluding Drivers from Auto-Updates
• 6. Remaining Issues and Limitations
• References

One-Line Summary

1. Symptoms: Errors That Do Not Point to the Root Cause

The pod requesting a GPU failed to create its container, and the event error ended with the following:

… open /run/nvidia-persistenced/socket: no such file or directory

Looking solely at the pod error leads you to suspect the GPU Operator, container toolkit, or persistenced daemon configuration first. Indeed, that node's nvidia-persistenced was not running. However, that was merely a symptom. Querying the node directly revealed the real answer.

$ nvidia-smi
Failed to initialize NVML: Driver/library version mismatch

We were unable to verify why the pod error specifically mentioned the socket. Our assumptionis as follows: because the versions mismatched and NVML failed to initialize, persistenced could not start and thus the socket was never created, but the component attaching the GPU to the container expected the socket to be there and failed while trying to open it. The lesson is clear: If a GPU pod behaves strangely, run nvidia-smi on the node first.

2. Cause: Dawn Auto-Update Three Days Ago

The node's unattended-upgrades.log held the answer. On 2026-10-10 at 06:08, the auto-update included 15 NVIDIA-related packages in its update list — user libraries (libnvidia-*), utilities, driver meta-packages, kernel module sources, and dkms packages.

This is where the versions split into two.

• Disk: New libraries are installed, and DKMS builds a new kernel module and places it on disk.
• Memory: The already loaded kernel module remains unchanged. Linux package installations do not replace active modules. That node had been up for 19 days.

However, nvidia-smi or any process using the GPU opens the new libraries on diskevery time it starts. When the new libraries encounter the old module, they reject it due to a version mismatch.

Why didn't a reboot follow? The Automatic-Reboot of unattended-upgrades is disabled by default(according to the official README). The node's configuration also retained the commented-out default value. Automatic updates combined with manual reboots create this "half-upgraded" state.

And no one noticed for three days. That was because no new pods had claimed the GPU on that node during that time. Processes that had already acquired and were running on the GPU continued using the libraries they opened previously, making them appear fine (assumption — it was unverified whether any GPU-holding processes actually ran during those three days).

3. How to Verify

# 적재된 모듈의 판
cat /proc/driver/nvidia/version
# 디스크에 깔린 패키지의 판
dpkg -l | grep -E 'nvidia-(driver|utils)-'
# 언제 누가 올렸나
grep -i nvidia /var/log/unattended-upgrades/unattended-upgrades.log
# 노드가 언제부터 켜져 있었나
uptime -s

This incident has occurred if the library version and driver version differ, update logs exist in the third place, and the fourth condition precedes them.

4. The Fix: Rebooting, and Why It Requires Human Judgment

The surest way to load the new module is a reboot. While it is possible to unload and reload just the module, it will refuse to unload if even a single process is holding the GPU.

This reboot was performed by a human. That node hosted not only GPU pods but also several other production services and experimental VMs. Rebooting would momentarily disconnect these services or migrate them to other nodes. The exact timing and impact are matters for human judgment. The agent compiled and handed over the cause, commands, and list of impacts.

We rebooted that node on 2026-10-13, and another GPU node that received the same auto-update on 2026-10-14, aligning the module and library versions on both nodes.

5. Prevention: Excluding Drivers from Auto-Updates

Both nodes were configured as follows (verified by reading the file on 2026-10-16).

// /etc/apt/apt.conf.d/51-no-nvidia-autoupgrade
Unattended-Upgrade::Package-Blacklist {
  "nvidia-";
  "libnvidia-";
  "xserver-xorg-video-nvidia";
};

Package-Blacklist entries are regular expressions. Matching packages are not upgraded, nor are packages depending on them (according to the official README). You could also accomplish this using apt-mark hold . The difference with hold is that it also blocks manual apt upgrade .

We should also note the trade-off. Driver security updates will no longer arrive automatically. Therefore, driver updates are planned and applied manually.

1. Drain the node (kubectl drain).
2. Upgrade the driver packages.
3. Reboot.
4. nvidia-smi Verify with a single GPU pod and uncordon the node (kubectl uncordon).

The platform support table in the NVIDIA GPU Operator documentation already includes a recommendation to "disable automatic kernel updates in unattended-upgrades to prevent upgrading to unsupported kernels." Since there was no mention that pre-installed drivers (driver.enabled=false) experience the same issue with driver packages, I submitted a PR adding this case to the troubleshooting documentation — NVIDIA/cloud-native-docs#524(open as of 2026-10-16).

6. Remaining Issues and Limitations

• There is one more GPU node running a different driver family. We have not checked the auto-update settings on that node (unverified). Assuming the same risk applies and applying the same configuration is our next task.
• Because these configurations were applied manually to each node, newly provisioned nodes might miss them. They must be incorporated into the node provisioning procedure.
• There is currently no alert that proactively notifies us of a "version mismatch." Right now, we only find out when a GPU pod fails.

A quick tip from Titi. Auto-updates are only half-automatic. They replace files, but memory only changes when you reboot. And if a pod complains about a missing file, run nvidia-smi on the node before hunting for that file! The real reason is usually right there.

References

All materials were accessed on 2026-10-16.

• unattended-upgrades README (Package-Blacklist, Automatic-Reboot defaults): https://github.com/mvo5/unattended-upgrades/blob/master/README.md
• NVIDIA GPU Operator Platform Support (Kernel auto-update recommendation): https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/platform-support.html
• Documentation PR submitted for this issue: https://github.com/NVIDIA/cloud-native-docs/pull/524