GPU Operator Pod Components and Roles, and Do Hosts Need Drivers?

Status: Draft (2026-10-07). Pod names, types, init container orders, node labels, and installation option combinations are read-only observations from an actually running environment. The method of the operator mounting drivers as containers was not directly tested because this environment does not use it (unmeasured), and was instead verified through official documentation descriptions. Writing rule: Measured data is the default, and anything unmeasured is marked as "Unmeasured".

Table of Contents

• 1. Pods Created Upon Installation
• 2. Pods Wait for Each Other: Validation Chain
• 3. Does the Host Need an NVIDIA Driver?
• 4. Until a Pod Receives a GPU
• 5. Read-Only Commands for Direct Verification
• 6. Common Misconceptions
• Verified Versions, Dates, and Unverified Items
• References

When you first use GPU Operator, gpu-operator more than ten pods run in the namespace, making it difficult to tell what is what by names alone. This article answers two questions.

1. What each pod does, on which node it runs, and what symptoms occur if it is missing.
2. Can you use GPUs with only the GPU Operator without installing an NVIDIA driver on the host?

Short answer. Pods are broadly divided into controllers (Deployment), node labelers (NFD), and a bundle of DaemonSets running one per GPU node (driver, toolkit, device plugin, GFD, DCGM Exporter, validator). The host does not need a driver. This is because the operator mounts the driver as a container by default. Conversely, if you have already installed a driver on the host, a configuration that disables the operator's driver (driver.enabled=false) is possible and actually common. The two approaches differ in "who manages the driver's lifecycle".

1. Pods Created Upon Installation

This is the result of reading the gpu-operator namespace in an environment where GPU Operator (v26.3.3) is actually running. Since numbers (node counts) vary by environment, only the types and roles are summarized.

In NVSwitch systems (such as HGX), Fabric Manager is additionally required. The operator's driver container installs this together when it detects an NVSwitch, and you must install it directly if you use a host driver (this is stated in official documentation and was not tested because this environment lacks NVSwitch, unmeasured).

2. Pods Wait for Each Other: Validation Chain

DaemonSets start all at once on a GPU node, but there is a dependency order. This is the order confirmed by the actual init container names of the pods.

Therefore, if one stage is blocked, the subsequent pods remain at Init:0/1 , and the root of the problem lies in the earliest blocked stage. Diagnosis is performed from the front in the order of driver -> toolkit -> plugin .

The operator determines which DaemonSet to run on which node using node labels. If you check, every GPU node has the nvidia.com/gpu.deploy.driver, .container-toolkit, .device-plugin, .gpu-feature-discovery, .dcgm, .dcgm-exporter, .operator-validator, .node-status-exporter label attached as "true" , and each DaemonSet reads this label as a nodeSelector. If you want to turn off only one component on a specific node, you can change that node's corresponding label to false .

3. Is it required to have NVIDIA drivers on the host?

Yes, it is possible. Official documentation states that the default value of driver.enabled is true, toolkit.enabled and the default value of true and the operator installs the driver via a container (nvidia-driver-daemonset). Therefore, as long as the GPU-equipped node has an OS and a container runtime, the operator brings up the rest.

There are three configurations.

In the verified environment, ClusterPolicy is driver.enabled=false, toolkit.enabled=true, Device Plugin, GFD, and DCGM Exporter are enabled, MIG Manager is disabled, and as a result, nvidia-driver-daemonset was not created at all. Still, all validators passed and plugins on each GPU node are nvidia.com/gpu was advertised. In other words, the combination of "host driver + operator's toolkit, plugins, and metrics" actually works. At this time, the toolkit pod mounts the host root (/) and the driver installation path (/run/nvidia/driver), and before starting, checks whether the host driver is usable with driver-validation (Section 2).

When the host driver already exists and driver.enabled is not turned off, there is a description of automatic detection in the official documentation. It states that if the driver pod's init container detects a pre-installed driver, it labels the node to terminate the driver pod and does not schedule it again (this is based on reading the documentation and was not tested in this environment, unmeasured). Still, to make the intent clear, it is better to explicitly specify the configuration.

Cost of the driver container approach.

• A driver image matching the OS version and kernel is required for each node. When the kernel is updated, the driver must be re-matched, and pre-compiled images are in the format <브랜치>-<커널>-<OS> , so that combination must exist.
• Since the driver is a kernel module, all pods using GPUs must be stopped before replacement. Because the operator manages this as a state machine using cordon, waiting, deleting GPU pods, and optional drain, the GPU on that node cannot be used during the upgrade.
• If automatic kernel upgrade (unattended-upgrades) is turned on, the driver may not match after the kernel changes, so the official documentation recommends turning it off.

Cost of the host driver approach. The operator does not manage the lifecycle (upgrade, replacement) of the host driver. Instead, node OSes can be mixed, and the driver replacement procedure can be controlled using the team's existing methods (images, package management).

4. Until a pod receives a GPU

This is the flow after installation is complete.

1. The device plugin registers with kubeletand passes the list of devices.
2. Kubelet updates the node status capacity and allocatable .). The scheduler only looks at this number, not the device. It is an integer.nvidia.com/gpu: N).
3. The scheduler only looks at this number, not the device. It is an integer.). The scheduler only looks at this number, not the device. It is an integer.NodeResourcesFit. The scheduler only looks at this number, not the device. It is an integer. limits . The scheduler only looks at this number, not the device. It is an integer.requests . The scheduler only looks at this number, not the device. It is an integer.
4. kubelet calls Allocate , and receives device nodes, environment variables, mounts, and CDI names in response., and receives device nodes, environment variables, mounts, and CDI names in response.
5. The container runtime injects the devices.. Starting with Operator 25.10.0, CDI (Container Device Interface) is the default, and regular workloads do not need runtimeClassName . Starting with Operator 25.10.0, CDI (Container Device Interface) is the default, and regular workloads do not need

If the connection to the plugin is lost, kubelet marks the device as unhealthy, drops allocatable to 0, and clears the resource name after a 5-minute grace period (described in kubelet source code). Therefore, even if the " " events are the same, the cause could be anywhere from the plugin, toolkit, driver, to scheduling constraints.Insufficient nvidia.com/gpu" events are the same, the cause could be anywhere from the plugin, toolkit, driver, to scheduling constraints.

5. Read-only commands for direct verification

# 오퍼레이터가 올린 파드와 노드별 분포
kubectl -n gpu-operator get deploy,ds
kubectl -n gpu-operator get po -o wide

# 오퍼레이터가 어떤 구성요소를 켰는가(ClusterPolicy)
kubectl get clusterpolicy -o jsonpath='{.items[0].spec.driver.enabled} {.items[0].spec.toolkit.enabled}{"\n"}'

# 노드별 GPU 광고량과 GPU 라벨
kubectl get nodes -o custom-columns=N:.metadata.name,CAP:.status.capacity.nvidia\.com/gpu,ALLOC:.status.allocatable.nvidia\.com/gpu
kubectl get nodes -L nvidia.com/gpu.present,nvidia.com/gpu.product

# 어떤 노드에 어떤 구성요소를 띄우도록 라벨이 붙었는가
kubectl get node <노드> -o json | grep -o 'nvidia.com/gpu.deploy[^,]*'

# 검증기가 막힌 곳(초기화 컨테이너 이름이 막힌 단계)
kubectl -n gpu-operator describe po -l app=nvidia-operator-validator | grep -A2 -i "init containers"

6. Common Misconceptions

• "GPU Operator is just a driver installer." Drivers are just one component. Even in environments using host drivers, the operator remains useful for toolkit installation, plugin registration, label management, metrics, and validation.
• "If you have host drivers, you don't need the operator." Drivers alone do not allow Kubernetes to recognize GPUs as a resource. Device plugins and container runtime configurations are required for nvidia.com/gpu to be created. The operator automatically configures both.
• "Deploying the operator makes containerd configuration safe." The toolkit modifies the host's containerd configuration file (/etc/containerd) directly. If it overlaps with other configuration files, the CRI settings may be replaced entirely (in containerd 1.7, configuration merging replaces the entire plugins section, as verified from the source code). See Section 5 of Part 03.
• "When the toolkit sends SIGHUP to containerd, only the configuration is reloaded." When I sent SIGHUP to containerd 2.2.1 on a disposable VM, the process terminated and systemd restarted it 5 seconds later (Restart=always). This is why you should apply changes one node at a time rather than all nodes at once (refer to Part 03).
• 「nvidia.com/gpu.present 」 alone means the installation is complete. This label only indicates that NFD and the operator have detected the GPU hardware, not that the drivers and plugins are operational. You must check the node's allocatable and the validator status.

Verified Versions and Dates, and Unverified Items

The environment reviewed on 2026-10-07 consists of GPU Operator v26.3.3, Container Toolkit v1.19.1, Device Plugin v0.19.3, NFD v0.18.3, and DCGM Exporter 4.5.3-4.8.2 (distroless image). The official documentation was reviewed by opening the installation guide and overview pages.

• The operator's driver container method was not tested directly (unmeasured). This is because the verified environment uses host drivers. The behavior and cost of this method are based on the official documentation description.
• Automatic detection when a host driver is present (terminating the driver pod) was read from the documentation and not tested.
• Fabric Manager operation on NVSwitch systems, actual application of MIG Manager, and MPS were not verified as they were not used in this environment.
• The pod list is a snapshot at a single point in time, and components may increase or decrease depending on the configuration (KubeVirt device plugins for sandbox workloads, vGPU managers, etc., were not present in this environment).

References

• NVIDIA GPU Operator Overview: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/overview.html
• Getting Started with NVIDIA GPU Operator (Pre-installed Drivers and Helm Options): https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html
• Part 3 (GPU Operator Stack Anatomy and Troubleshooting), Part 17 (DCGM Exporter Metrics), and Part 22 (Knowledge Map) of this series