Reading NVIDIA GPU Operator: How a single ClusterPolicy prepares GPU nodes and how dcgm-exporter finds the owner of a GPU

Status: Draft (2026-10-17), partially measured. Reference versions: NVIDIA/gpu-operator v26.3.3 (commit b0a49c0e7b2e), NVIDIA/k8s-device-plugin v0.19.3 (570e776d6813), NVIDIA/dcgm-exporter 4.5.3-4.8.2 (691c92762eb5). Since a small cluster is running these exact versions, measurements were obtained on 2026-10-17 from that cluster by read-only methods (reading ClusterPolicy, DaemonSets, node labels, pod configurations, reading /metrics via API server proxy, and reading files inside running pods). Records measuring the same phenomenon at greater length on 2026-10-16 are also cited. Writing rules: Measured data is the default; unmeasured items are marked as "Unmeasured," and unverified items are marked as "Unverified."

Table of Contents

• 1. One-line summary: The "Installation technician + Checklist" for GPU nodes
• 2. Architecture diagram: State list and the preparation chain inside a node
• 3. Code map
• 4. Following the core path
  • 4.1 Reconcile loop: Taking states "one step at a time"
  • 4.2 In-node preparation chain: Validators and "-ready" files
  • 4.3 Device plugin: Time-sharing replication and CDI specs
  • 4.4 dcgm-exporter: "Whose GPU is this?"
• 5. Author's contributions and next steps
• 6. Measurements: Observations from a small cluster
• 7. Where to start reading
• 8. Good places to make your first contribution
• 9. Limitations
• Note

1. One-line summary: "Installation technician + Checklist" for GPU nodes

To use GPUs in Kubernetes, multiple layers must align on each node: kernel drivers, a container toolkit that allows containers to see GPU devices, a device plugin that reports nvidia.com/gpu resources to kubelet, GPU Feature Discovery (GFD) which attaches GPU specifications as node labels, and dcgm-exporter which exports metrics. If even one thing is out of order, pods will get stuck in Init or fail to find the GPU.

The GPU Operator installs these layers sequentially like an installation technician, checks whether the previous step is complete like a checklist, and then moves on to the next step. Users only need to interact with a single CR called ClusterPolicy .

2. Architecture Diagram: State List and Readiness Chain within Nodes

How to read the diagram:

• The upper part is the operator side. A single "state" corresponds to one folder under the repository assets/ , and the YAMLs in that folder (ServiceAccount, RBAC, ConfigMap, DaemonSet, etc.) are applied as a bundle. The numbers represent the actual application order, and dashed boxes indicate states that are disabled in small cluster configurations.
• The lower part is inside the GPU node. Even if the operator determines the order, DaemonSets start simultaneously, so the order inside the node is synchronized using files. The validator leaves driver-ready, toolkit-ready … , and the initContainers of other DaemonSets wait until that file is created.
• The operator only checks whether "the DaemonSet is ready," while the validator inside the node checks whether "the GPU actually works." This separation of roles is the core of this design.

3. Code Map

Looking at the three repositories together, focusing on the most important top-level locations.

controllers/object_controls.go is over 5,000 lines long. Rather than reading it from the beginning, it is better to look up the respective function when its name appears in the paths below.

4. Following the Core Path

4.1 The Reconcile Loop: Taking "One Step" at a Time

1. When the operator receives a ClusterPolicy, controllers/clusterpolicy_controller.go:93 Reconcile runs. Since only one ClusterPolicy is allowed per cluster (subsequent ones are Ignored), if it is the first one, it calls init .
2. controllers/state_manager.go:854 init finds GPU nodes, labels them (state_manager.go:521 labelGPUNodes), and registers the state list. Whether a node is a GPU node is determined by the NFD label feature.node.kubernetes.io/pci-10de.present=true(where 10de is NVIDIA's PCI vendor ID), and such nodes are labeled with nvidia.com/gpu.present=true and nvidia.com/gpu.deploy.<구성요소>=true . The nodeSelector for each DaemonSet corresponds to these labels.
3. Registration is performed as state_manager.go:175 addState loads controllers/resource_manager.go:102 addResourcesControls for each folder path, looks at the kind in the YAML, and builds a list of "functions that apply this object." One state equals one function list.
4. Reconcile repeats state_manager.go:1051 step up to the final state. step runs all functions for that state, and if even one is not ready, marks the state as NotReady and moves on to the next state. After running through to the end, if any state remains unready, it runs again after 5 seconds.

for _, fs := range n.controls[n.idx] {
    stat, err := fs(*n)
    if err != nil {
        return stat, err
    }
    // successfully deployed resource, now check if its ready
    if stat != gpuv1.Ready {
        result = stat
    }
}
// move to next state
n.idx++

Source: Part of NVIDIA/gpu-operator v26.3.3 controllers/state_manager.go(step ), Apache-2.0.

A design choice is visible here. It does not block waiting for the preceding state to be ready. The DaemonSets are created upfront, and the actual sequencing is handled by the readiness files inside the nodes (4.2). If the operator forced each node to wait, a delay in a single node would halt the application across the entire cluster.

5. The function that applies DaemonSets is controllers/object_controls.go:4621 DaemonSet . If a state is disabled, it deletes the object and returns Disabled . If enabled, object_controls.go:728 preProcessDaemonSet selects a transformation functionbased on the DaemonSet name (nvidia-device-plugin-daemonset → TransformDevicePlugin, nvidia-dcgm-exporter → TransformDCGMExporter ...). These transformation functions are where the images, environment variables, and configurations of the ClusterPolicy are injected into the YAML. Readiness is checked by object_controls.go:3989 isDaemonSetReady .
6. When a new GPU node joins, clusterpolicy_controller.go:246 addWatchNewGPUNode 's node watcher wakes up reconcile again.

4.2 Node-level Readiness Chain: Validators and the "-ready" File

Validators are located in a single file: cmd/nvidia-validator/main.go . The directory where state files are kept is /run/nvidia/validations(line 146 defaultStatusPath), and the filenames are gathered as constants on lines 161–189 (driver-ready, toolkit-ready, cuda-ready, plugin-ready …).

• operator-validator The initContainer of the DaemonSet validates in the order of driver → toolkit → cuda → plugin and leaves files behind (assets/state-operator-validation/0500_daemonset.yaml).
• Other DaemonSets only wait for the files. The initContainer of the device plugin toolkit-validation is a single line of shell script that checks every 5 seconds until toolkit-ready appears (assets/state-device-plugin/0500_daemonset.yaml:29).
• Driver validation starts at main.go:810 runValidation . It first checks the driver installed on the host (main.go:745 validateHostDriver), and if that fails, waits on the driver container side (main.go:765 validateDriverContainer).

The file-based chaining approach is simple yet highly advantageous. When a node reboots, /run is cleared, so validation naturally reruns from the beginning, and an administrator can see exactly where it stopped with a single ls /run/nvidia/validations command on the node.

4.3 Device Plugin: Time-Slicing Replication and CDI Specs

The device plugin is a gRPC server implementing kubelet's device plugin API. The flow can be summarized in three functions.

1. internal/plugin/server.go:229 Register — Registers to the kubelet socket stating "nvidia.com/gpu is managed by me."
2. server.go:267 ListAndWatch — Streams the device list (including health status) to the kubelet.
3. server.go:306 Allocate — Returns the environment variables, devices, and CDI names to inject into the container when allocating devices to a pod.

Time-slicing occurs in step 2. If replicas: 5 is present in the configuration, internal/rm/device_map.go:286 updateDeviceMapWithReplicas scales up a single GPU to be advertised as GPU-<uuid>::0 … ::4 five "devices." Attaching and detaching ::N is handled by a small type around internal/rm/devices.go:243 NewAnnotatedID . Kubelet distributes these to five pods one by one, unaware that they are the same physical GPU. There is no memory isolation. This ::N connects to the issues discussed in Section 5.

CDI spec is a file telling the container runtime, "To provide this device, include these device files and libraries." internal/cdi/cdi.go:187 CreateSpecFile saves the spec to /var/run/cdi(line 41 cdiRoot), and this function is called once when the plugin starts (cmd/nvidia-device-plugin/plugin-manager.go:84). The spec includes library filenames including the driver version . Therefore, if you update only the host driver without restarting the plugin, the library version pointed to by the spec and the actually loaded kernel module version may mismatch. Since the plugin restarts upon receiving kubelet restarts, configuration changes, or signals (main.go start restart loop in ), a node reboot or plugin pod restart is practically necessary after a driver update.

4.4 dcgm-exporter: "Whose GPU Is This?"

dcgm-exporter reads per-GPU metrics from DCGM and /metrics exports them. DCGM_EXPORTER_KUBERNETES=true If so, each line is pod·namespace·container labeled, and that information comes not from the Kubernetes API, but from kubelet's pod-resources gRPC(default /var/lib/kubelet/pod-resources/kubelet.sock). This is because kubelet is the one that knows 「which device ID this container received」.

1. /metrics When a request arrives, internal/pkg/server/server.go:249 Metrics → render runsserver.go:309 transformation.Process). per scrape the conversion list.
2. internal/pkg/transformation/kubernetes.go:296 getMappings connects to the socket and calls kubernetes.go:533 listPods via List() .
3. In the default(--kubernetes-virtual-gpus disabled) mode, kubernetes.go:712 toDeviceToPod builds a map of 「device ID → single pod」. Time-sliced device IDs GPU-<uuid>::N use :: the part before it, using the GPU UUID as the key.

} else if strings.Contains(deviceID, "::") {
    gpuInstanceID := strings.Split(deviceID, "::")[0]
    // (디버그 로그 생략)
    deviceToPodMap[gpuInstanceID] = podInfo
}

Source: NVIDIA/dcgm-exporter 4.5.3-4.8.2 internal/pkg/transformation/kubernetes.go lines 887–899 summary, Apache-2.0.

If three pods share the same GPU, the three pods take turns overwriting the same key(kubernetes.go:898). Ultimately, the last pod in the response wins. And the order of that response changes on every call because kubelet builds the pod list by popping it from a Go map (based on kubelet v1.34 pkg/kubelet/pod/pod_manager.go map iteration source). When the 「last one wins」 behavior of a single line of code meets the randomness of response order, labels change on every scrape.

--kubernetes-virtual-gpus(Enabling the environment variable KUBERNETES_VIRTUAL_GPUS) makes kubernetes.go:662 toDeviceToSharingPods group a list of podsper GPU and duplicate device metrics one line per pod (with some fields split into per-process values). The GPU Operator default deployment does not enable this value.

5. Author's Contributions and Where to Follow Up

• NVIDIA/gpu-operator#3086(Issue #3085, Pending Review): This fixes the problem where the validator logs say 「no driver on host」 and the retry warning has no reason when host drivers and library versions mismatch. It fixes line 810 of 4.2 runValidation(which records validateHostDriver failure as 「none」) and the retry warning on validateDriverContainer (line 802, where a literal \n is printed at the end). The incident story is at The Day Auto-Update Paused GPUs .
• NVIDIA/cloud-native-docs#524(Pending Review): A PR documenting the same 「driver/library version mismatch」 in the troubleshooting guide.
• dcgm-exporter Time-Slicing Label Flapping(Drafting, Unpublished): A proposal to make 4.4 toDeviceToPod iterate in alphabetical order of pod names so the same pod is always consistent. Since this changes default behavior, it was decided to ask via an issue first, and only a draft exists.

6. Measurements: Readings from a Small Cluster

2026-10-17, all read commands (kubectl get, inside running pod ls·grep, API server proxy kubectl get --raw …/proxy/metrics). Nodes are masked by role, and pods are masked as P, Q, and R.

ClusterPolicy and DaemonSets.

Out of eight nodes, five are GPU nodes, and all five had nvidia.com/gpu.present=true and gpu.deploy.* eight labels attached (Step 2 in 4.1).

Validation chain. operator-validator pod's initContainer was in the driver-validation → toolkit-validation → cuda-validation → plugin-validation order, and the device plugin pod was toolkit-validation → config-manager-init, and the dcgm-exporter pod was toolkit-validation just one. Exactly like the chain at the bottom of the diagram.

Time-slicing. Only one GPU node (GPU Node B) nvidia.com/device-plugin.config=shared-5 was using a different configuration chosen via labels. That configuration was replicas: 5, failRequestsGreaterThanOne: true(preventing a single pod from taking two or more slots), renameByDefault: false and the result is as follows.

One GPU (24GB class) appears as five to kubelet. The label gpu.replicas·gpu.sharing-strategy was attached by GFD.

CDI spec. Inside the device plugin pod, looking at /var/run/cdi , there were two files: the k8s.device-plugin.nvidia.com-gpu.json created by the plugin and the management.nvidia.com-gpu.yaml created by the toolkit. The modification time of the plugin spec across all five nodes was the time the plugin pod started(as mentioned in section 4.3, "Once at startup"), and within the spec, the version of libnvidia-ml.so.<판> and the kernel module version of /proc/driver/nvidia/version were identical across all five nodes today. Since the spec dates on each node were scattered from early September to mid-October, we can see that nodes where the driver was updated also had their plugins restarted afterward.

pod labels for dcgm-exporter. The exporter on GPU Node B /metrics was scraped six times at 10-second intervals. In each scrape, 15 metrics (configuration file dcp-metrics-included.csv) were output as one line per GPU, and there were three running pods requesting GPUs on that node.

changed all five times over the course of five scrapes. When scraping the same node 30 times at 2-second intervals on 2026-10-16, the three pods appeared 19, 8, and 3 times (19, 10, and 1 in the second run; changes between two consecutive runs occurred 12 times), and one hour's worth of data accumulated in Prometheus (at 15-second intervals) split a single GPU into three time series with 129, 76, and 35 samples, and over 24 hours, the number of distinct exported_pod values was 7. This is why lines appear sporadically broken when grouped by pod on the dashboard.

7. Where to start reading

1. controllers/clusterpolicy_controller.go:93 Reconcile — The entire flow is visible within 150 lines.
2. controllers/state_manager.go:854 init and 1051 lines of step — State list and a step.
3. assets/state-device-plugin/ folder — Visually check what kind of YAML bundle a "single state" is.
4. object_controls.go:1520 TransformDevicePlugin — An actual example of how ClusterPolicy values go into YAML.
5. internal/plugin/server.go 's Register·ListAndWatch·Allocate — Three functions of the device plugin API.
6. internal/pkg/transformation/kubernetes.go:296 to Process — The path of pods attaching to metrics.

8. Good First Contribution Areas

• The dcgm-exporter README lacks explanations for Kubernetes flags. pkg/cmd/app.go has 38 flags along with their environment variable names, but the README does not cover --kubernetes-virtual-gpus or the pod-resources socket path. Moreover, environment variable names are mixed. Most of them start with DCGM_EXPORTER_ , but 7 do not follow this prefix, and among them, four including KUBERNETES_VIRTUAL_GPUS, KUBERNETES_ENABLE_DRA, NVIDIA_RESOURCE_NAMES do not even have DCGM_ (pkg/cmd/app.go:303). A documentation PR organizing this into a single table is a great first step.
• The device plugin's CDI spec is unaware of driver updates. Like v4.3, the spec is generated only when the plugin starts. Adding "The plugin must be restarted after a driver update" to the documentation, or implementing a mechanism to detect driver version changes and regenerate the spec, is worth asking about via an issue first.
• TODOs in gpu-operator. step header's TODO (Check OwnerRef before touching objects owned by another controller), transformValidationInitContainer 'sTransformValidatorComponent Duplicate with "object_controls.go:3901" TODO (transforms_test.go) have a small scope and tight tests (
• ), making them easy to work on.Signed-off-byRule: gpu-operator requires only commit signing (CONTRIBUTING.md) instead of a CLA (

). External PRs face long wait times for NVIDIA-side CI validation.

• 9. Limitations
• --kubernetes-virtual-gpus Driver, MIG, vGPU, and sandbox-related states were turned off in the small cluster, so only the code was read (unmeasured). The NVIDIADriver CRD path was also not covered.
• Labels after enabling
• were not measured (unmeasured). The cost of metric lines increasing proportionally to the number of pods when enabled is also unmeasured.

How pods fail when the CDI spec is actually misaligned was not reproduced this time (only recorded in Part 2 Incident Notes). The working pods spawned by the validator's CUDA check and the MPS path were not covered.

Titi's Word.

Five people take turns using a single ping-pong table, and the scorekeeper's nameplate always shows the name of "the person who stood in line last." Since the queue order changes every time, the owner changes every time, too. Organizing them just alphabetically would quiet things down!

• References controllers/, assets/, cmd/nvidia-validator/main.go, CONTRIBUTING.md
• All access dates are 2026-10-17. internal/plugin/server.go, internal/rm/device_map.go, internal/cdi/cdi.go, api/config/v1/replicas.go
• NVIDIA/gpu-operator v26.3.3: internal/pkg/transformation/kubernetes.go, internal/pkg/server/server.go, pkg/cmd/app.go
• NVIDIA/k8s-device-plugin v0.19.3: pkg/kubelet/apis/podresources/server_v1.go, pkg/kubelet/pod/pod_manager.go
• NVIDIA/dcgm-exporter 4.5.3-4.8.2:
• kubernetes/kubernetes v1.34.1: