GPU Passthrough and vfio-pci: The Principle of Handing Over a GPU Entirely to a VM

Status: Draft (2026-10-05), reflecting actual measurements. The values in the "Actual Measurement Example" section were directly verified using read-only commands on a single GPU server, and identification information such as device IDs has been replaced with placeholders. The explanation of the principles is based on kernel and KubeVirt documentation, which was not re-opened and cross-checked at the time of writing this post (unverified). Unverified assumptions, such as the relationship between memory pinning and crash loops, are marked as such. Writing rule: Actual measurement data is the default, and anything that could not be measured is marked as "Unmeasured".

Table of Contents

• 1. Why Use Passthrough
• 2. Principles: Four Components
  • 2.1 Driver Binding
  • 2.2 IOMMU: The Device That Traps Device Memory Access
  • 2.3 VFIO Interface
  • 2.4 Reset
• 3. Actual Measurement Example: A Server with One 6GB-class GPU Passed Through
  • 3.1 Configuration File
• 4. Why nvidia-smi Fails on the Same Device
• 5. How KubeVirt Connects Devices to VMs
• 6. Common Pitfalls: Memory Pinning and Host OOM
• 7. Reverting to a Container GPU (Example Procedure)
• 8. Likely Interview Questions
• 9. Common Misconceptions
• External Reference: Performance Overhead of Passthrough
• Unverified Items
• References
• Verified Versions and Dates

1. Why Use Passthrough

There are broadly four ways to provide a GPU to a workload.

The scenarios requiring passthrough narrow down to three: when the guest needs a different kernel or driver version than the host, when the guest OS is not Linux, and when isolation stronger than what containers can offer is required. Instead, that GPU cannot be used simultaneously by the host and containers. Later in this post, we will actually see the same device nvidia-smi failing on one side.

2. Principles: Four Components

Passthrough is the process of transferring device ownership from a general host kernel driver to a user-space process (QEMU). Four components work together to achieve this.

2.1 Driver Binding

PCI devices have a vendor ID and a device ID (for example, the GPU has 10de:XXXX), and the kernel attaches a single driver matching that ID. Normally, the nvidia driver is attached, and then nvidia-smi runs on the host. In pass-through, vfio-pci is attached instead. vfio-pci is not a driver that uses the device, but an intermediary driver that hands the device over to user space. A device with this driver attached is not used by the host kernel.

There are two ways to attach it. One is to switch it at runtime using driver_override and bind/unbind, and the other is to have vfio-pci claim the corresponding ID first at boot time. The example server uses the latter method (Section 3).

2.2 IOMMU: A Device That Restricts Device Memory Access

The GPU reads from and writes to system memory directly via DMA. When the device is given to the guest, the memory addresses used by the guest differ from the actual addresses of the host, so the device could touch unintended host memory. IOMMU translates the DMA addresses generated by the device and restricts the allowed rangeto prevent this issue. This is done by opening only the guest memory region to the device.

The minimum unit of isolation is the IOMMU group. Since devices belonging to the same group cannot isolate each other's DMA, all devices in the group must be handed over to the same owner . Therefore, the GPU's audio function (the 01:00.1in the example) is also bound together to vfio-pci .

2.3 VFIO Interface

VFIO is a kernel feature that safely exposes devices to user space. A user-space process opens /dev/vfio/vfio(container) and /dev/vfio/<그룹 번호> to do the following: map the device's BAR (memory region) into its own address space, register guest memory with the IOMMU, and receive the device's interrupts via event file descriptors to inject them into the guest. QEMU performs these tasks.

Since guest memory is the target of DMA, it must be pinned so that the host cannot reclaim or swap it out. A VM with 12GiB of memory means that 12GiB of host memory is pinned upon startup (based on documentation, not directly measured).

2.4 Reset

When a VM terminates, the device must be returned to a clean state for the safety of the next user. The best method is FLR (Function Level Reset) supported by the device, and if not supported, a bus reset is used. GPUs that do not reset properly cause an issue where the host must be rebooted after using the VM once.

3. Measurement Example: A server passing through a 6GB-class GPU

These values were all checked using read-only commands and are example valuesthat vary by equipment. The host kernel is 7.0.0-34-generic. The device IDs are masked as XXXX(GPU) and YYYY(audio).

The cause for the link width dropping to x8 could not be verified. Since it could be due to board wiring or power states, no definitive conclusion is made.

3.1 Configuration files

In this example, the setup that creates the passthrough consists of two files.

# /etc/modprobe.d/vfio-gpu.conf
options vfio-pci ids=10de:XXXX,10de:YYYY
softdep nvidia pre: vfio-pci
softdep nouveau pre: vfio-pci
softdep snd_hda_intel pre: vfio-pci

# /etc/modules-load.d/vfio-pci.conf
vfio-pci

• options vfio-pci ids=… makes vfio-pci claim these two IDs as its own.
• softdep … pre: vfio-pci loads nvidia, nouveau, snd_hda_intel before vfio-pci is loaded, preventing those drivers from taking the device first.
• modules-load.d loads the vfio-pci module itself at boot time.

The example server's boot image (initramfs) did not contain a single vfio entry. Therefore, this configuration works purely with files without requiring an image regeneration, and it can be reverted for the same reason by removing the files (though it has not yet been tested whether it actually reverts after a reboot).

4. Why nvidia-smi fails on the same device

Running nvidia-smi on this host outputs a message saying "Failed to initialize NVML: Driver/library version mismatch" or that it cannot communicate with the NVIDIA driver. This does not mean the driver is not installed. The host has nvidia-driver-595-open(595.91.07) installed, along with the module package for the current kernel.

The kernel logs show the cause clearly.

NVRM: GPU 0000:01:00.0 is already bound to vfio-pci.

nvidia Every time the module attempts to claim the device, a message repeatedly appears stating that it is already owned by vfio-pci . Since there is only one driver attached to the device, the host's nvidia-smi has nothing to communicate with while ownership is held by VFIO. This is not a malfunction, but normal operation as configured.

The Kubernetes side status tells the same story.

5. How KubeVirt connects a device to a VM

1. In the KubeVirt configuration, permittedHostDevices and pciVendorSelector: 10DE:XXXX are registered under resourceName: nvidia.com/<모델별 자원 이름> , and the feature gate HostDevices is enabled.
2. The node's virt-handler finds the device with that ID and advertises it as a Kubernetes extended resource. The result is the nvidia.com/<모델별 자원 이름>: 1 in the table above.
3. Write the devices.gpus in the VM definition to deviceName: nvidia.com/<모델별 자원 이름> . The scheduler places the VM only on nodes that have this resource.
4. When the VM starts, QEMU opens the corresponding IOMMU group via VFIO and attaches the device to the guest.

The important point here is that a separate name is used for the resource name, nvidia.com/gpu not. GPUs for containers and GPUs for VMs are different resources, and a single GPU is advertised as only one of the two.

6. Common Pitfall: Memory Pinning and Host OOM

A VM passed through a GPU pins its guest memory to the host, as described in Section 2.3. Since pinned memory cannot be absorbed by swap, the host needs an amount of memory equal to the guest size all at once the moment the VM starts. If other workloads and node daemons are running together on the same host, the total easily exceeds the physical memory.

If the host as a whole runs out of memory, the kernel's OOM killer activates. At this time, the kernel log records a host-wide out-of-memory error (Memory cgroup out of memory), not a container limit exceeded error (Out of memory: Killed process). The victim may not only be the VM process (qemu-kvm) but also a system daemon of the node. If the VM dies and restarts, it becomes a crash loop repeating the same thing, while the daemons also continue to die along with it.

The host OOM logs observed in the author's experimental environment matched this flow. However, since the actual pinned memory size was not measured when the VM started, this interpretation is an estimate(unmeasured). There are two lessons: it is safer not to place multiple GPU VMs on a server where host memory is not ample, and before bringing up a VM, you should first add up the sum of the guest memory and the memory already used by the host.

7. Reverting to Container GPUs (Example Procedure)

The simplest way to unbind passthrough and revert the device for containers is to cordon and drain the node, clear the configuration files, and reboot. However, if other services are running on that node, a reboot becomes a major operation that takes everything down. Below is an example of transferring only the device without a reboot . The basis for determining that this was possible was first verified entirely in read-only mode.

First, we backed up two configuration files, removed them, and executed the following two steps.

# 1) 이 장치는 nvidia 드라이버로만 붙도록 지정한다.
echo nvidia | sudo tee /sys/bus/pci/devices/0000:01:00.0/driver_override
# 2) vfio-pci 에서 푼다. 풀리는 즉시 nvidia 가 장치를 가져간다.
echo 0000:01:00.0 | sudo tee /sys/bus/pci/drivers/vfio-pci/unbind

Result. unbind Immediately afterwards, the device driver changed to nvidia and nvidia-smi operated.

Points to note together.

• The audio function (01:00.1) remains as it is in vfio-pci . We assume there is no impact on GPU usage for containers, but this was not verified.
• Two lines of NVRM: … NV_ERR_INVALID_DATA(PlatformRequest) warnings were printed to the kernel log, and power.draw was read as approximately 752W. This is an impossible number for the power value of this GPU, suggesting an issue with reading sensor or platform information, the cause of which was not identified. Do not trust the power value immediately after switching.
• We did not test whether nvidia automatically attaches even after a reboot. Since we removed the configuration file, we expect it to attach, but this is still an assumption.
• The Kubernetes side remains unchanged. The node's nvidia.com/gpu is still 0, and KubeVirt's nvidia.com/<모델별 자원 이름> also remains 1 (this appears to be a stale value untilvirt-handler reads it again). For containers to use the GPU, you must change the node label nvidia.com/gpu.deploy.operands to true to enable the GPU Operator components.

To revert, change driver_override to vfio-pci and release nvidia . After this switch, GPU VMs cannot use this GPU. If a single GPU must be shared alternately between containers and VMs, this switch must be performed every time,so in environments with a small number of GPUs, the use case must be fixed to one.

8. Potential Interview Questions

• How do passthrough, vGPU, and MIG differ? They are distinguished by their isolation unit (entire device, software partitioning, hardware partitioning), licensing, and supported hardware.
• Why are IOMMU groups important? Devices in the same group cannot isolate DMA from each other and must be passed through together. The example server's group was clean as it contained only the GPU and audio.
• Why is host memory pinned when a VM starts? Because the device performs DMA on guest memory, ensuring that region cannot be moved or swapped.
• If nvidia-smi does not work on the host even though the driver is installed? First check the driver attached to the device (iflspci -nnk). vfio-pci is attached, it is operating according to the configuration).
• Can this GPU be shared simultaneously by containers and VMs? Not with the passthrough method. Time-sharing, MIG, or vGPU must be considered.

9. Common Misconceptions

• "The driver is broken": The ownership may simply belong to a different driver.
• "Passthrough results in no performance loss": The device's own performance remains almost identical, but operational overheads such as link width, resets, and memory pinning follow. Performance loss was not measured on the example server (unmeasured; since it is a service-running server, we did not switch the GPU back to passthrough to test). External measurements are summarized in "External Evidence" below.
• "IOMMU must be enabled separately": It was enabled by default without arguments on the example kernel. Since it varies depending on hardware and kernel configurations, first verify whether the groups are visible.

External Evidence: Performance Overhead of Passthrough

Since we could not directly measure the performance of the passthrough VM, we checked external materials that measured the same question. The accessible scope includes abstracts, vendor articles, and some original texts.

• Verdict: Moderately supported. "Device performance is almost identical" is consistent with the 0-5% range reported by external measurements. However, since the verified figures are from abstracts, vendor articles, and non-saturating workloads, we could not obtain figures under the exact same conditions as our GPU. Primary data measured with link width, huge pages, and IOMMU settings as variables was not found.
• Considering that the KubeVirt documentation guides to enable kernel arguments (intel_iommu=on, amd_iommu=on) while distribution defaults differ, "first verify whether groups are visible" from this article is a safe statement.

Unverified Items

• The actual memory size pinned when the GPU VM starts (the interpretation in Section 6 is an assumption consistent with logs).
• The cause of the PCIe link dropping to x8.
• GPU performance inside the passthrough VM (throughput compared to containers was unmeasured; since production services run on the same server, tests to hand the GPU back to the VM were omitted, and external measurements were used instead).
• Whether nvidia automatically attaches after a reboot (we switched without a reboot, and reboots were not tested).
• This article's explanation of principles did not re-open and cross-reference the kernel VFIO documentation and KubeVirt host device documentation this time.

References

• Kernel VFIO Documentation: https://www.kernel.org/doc/html/latest/driver-api/vfio.html
• KubeVirt Host Devices User Guide: https://kubevirt.io/user-guide/compute/host-devices/

Verified Version and Date

As of 2026-10-05. Host kernel 7.0.0-34-generic, NVIDIA driver package 595.91.07 (installed on host), KubeVirt HostDevices feature gate enabled.