An Anatomy and Troubleshooting Guide for the GPU Operator Stack

Status: Draft (Updated Oct 7, 2026), partially reflecting actual measurements. The descriptions are based on official documentation, with direct verification limited to the path where containerd is restarted via SIGHUP in Section 5 (verified directly on a disposable virtual machine) and reading the node's containerd version and unit settings. Everything else is based on documentation. Please read "Versions and Dates Verified" and "Things Not Verified" at the end of the post first. Writing rule: actual measurement data is the baseline, and unmeasured items are marked as "Unmeasured".

Table of Contents

• Summary
• 1. Components: What Does What
• 2. Until a Pod Receives a GPU
• 3. Choosing an Installation Method
• 4. Diagnostic Order by Symptom
• 5. Common Pitfall: Drop-in Overwrote CRI Settings Entirely
• 6. Capacity Planning
• 7. Try It Yourself (Read-Only)
• 8. Potential Interview Questions
• 9. Common Misconceptions
• 10. References and Verified Versions

Summary

The GPU Operator is an operator that allows you to declaratively install drivers, container runtime configurations, device plugins, and metrics collectors—tasks previously done manually on each GPU node—using a single object called ClusterPolicy . This post builds sequentially upon the roles of components, failure modes, the two gateways a Pod passes through to receive a GPU, the diagnostic order by symptom, and capacity planning pitfalls.

What You Will Be Able to Explain in an Interview After Reading This Post

1. The role and sequential dependencies of each DaemonSet launched by the operator, and where failures manifest
2. The two gateways: extended resources viewed by the scheduler, and handlers/CDI viewed by the runtime, along with the current path changed to default to CDI
3. Insufficient nvidia.com/gpu Why

1. Components: What Does What

The operator uses node labels to determine which components to deploy on which nodes. When Node Feature Discovery (NFD) attaches PCI vendor labels (such asfeature.node.kubernetes.io/pci-10de.present ), the operator recognizes the node as a GPU node, attaches nvidia.com/gpu.present and nvidia.com/gpu.deploy.<이름>=true , and each DaemonSet reads those labels via nodeSelector (confirmed in the operator source code). Sequential dependencies are handled by init containers. The toolkit must pass driver-validation, and the device plugin must pass toolkit-validation before they can start.

Fabric Manager is a separate service that configures the inter-GPU memory fabric in NVSwitch systems (such as HGX). The operator's driver container installs this when it detects an NVSwitch, and it must be installed manually when using host drivers.

2. Until a Pod Receives a GPU

flowchart TD
  P["디바이스 플러그인<br/>kubelet 소켓에 등록, 장치 목록 전달"] --> N["kubelet<br/>노드 status 의 capacity·allocatable 갱신"]
  N --> S["kube-scheduler (NodeResourcesFit)<br/>요청 ≤ allocatable − 이미 배정 ?"]
  S --> A["kubelet: Allocate 호출<br/>응답: 장치 노드·환경변수·마운트·CDI 이름"]
  A --> R["containerd<br/>CDI 스펙 적용 또는 nvidia 런타임이 OCI 스펙 수정"]
  R --> C["runc 가 컨테이너 시작"]

Gateway 1 is the number. The scheduler looks at numbers in node status, not devices. GPUs are used only in limits (must match values if usingrequests ) and are integers. If connection to the plugin is lost, kubelet marks the device as unhealthy, drops allocatable to 0, and removes the resource name after a 5-minute grace period (kubelet source). Even if a single device becomes unhealthy due to health monitoring, capacity remains unchanged while only allocatable decreases (official documentation).

Gateway 2 is the injection method. In the past, nvidia-container-runtime wrapped runc to invoke nvidia-container-cli via a prestart hook, and starting from operator version 25.10.0, Container Device Interface (CDI) is the default. General workloads do not need runtimeClassName . However, the operator's own operand pods have controllers attach runtimeClassName: nvidia (unless using NRI plugins, source verified), so if the handler disappears, the plugin fails to start first and advertising ceases.

3. Choosing an Installation Method

Since drivers are kernel modules, all GPU clients must be stopped before replacement. The documentation (Ubuntu 22.04 footnote) recommends turning off automatic kernel upgrades for unattended-upgrades , and once upgrades are stopped, asks to find nodes labeled nvidia.com/gpu-driver-upgrade-state=upgrade-failed . The author's experimental environment uses the host driver (595 series) and toolkit.enabled=true with v26.3.3. On Secure Boot nodes, DKMS modules may fail to load due to signature rejection, and if you use a distribution-signed module package, that module is tied to the kernel version, so the kernel version must match the driver. 26.3.x is in "Deprecated" status while 26.7.x is "Supported". Since consumer GPUs are not on the document's supported products list, the author's environment should be read as out of support.

4. Troubleshooting Order by Symptom

GPU 파드가 안 된다
├─ 파드 오브젝트가 없다 ........ 어드미션: ResourceQuota, 없는 RuntimeClass (apply 때 에러)
└─ 있다
   ├─ spec.nodeName 이 비어 있다 (Pending)
   │   ├─ 이벤트에 taint/selector ... 톨러레이션, 라벨
   │   └─ Insufficient nvidia.com/gpu → 노드별로 capacity / allocatable / 배정 합을 대조
   │        ├─ 자원 이름이 없다 ........ 플러그인 미등록 (5분 유예 경과 포함)
   │        ├─ capacity > allocatable .. unhealthy 장치 (XID), 또는 플러그인 연결 끊김 직후
   │        └─ 배정 합 = allocatable ... 진짜 자리 소진
   └─ 노드에 붙었다
       ├─ no runtime for "nvidia" ..... containerd config dump 의 런타임 목록
       ├─ operand 가 Init 에서 정지 .... 선행 단계(드라이버, 툴킷) 로그
       └─ Running 인데 nvidia-smi 실패 . limits 누락, 드라이버/NVML, 장치 접근 상실
                                        (노드에서 dmesg | grep -iE 'NVRM|Xid')

If it is 요청 > allocatable − 이미 배정 , the scheduler writes Insufficient <자원 이름> regardless of the cause (kube-scheduler source). Therefore, you should not jump to conclusions about the cause based on a single event line.

5. Common Pitfall: Drop-in Overwrote the Entire CRI Configuration

In the author's experimental environment, 4 GPUs were not advertised for a week. To summarize:

• Because of a private registry accessed via HTTP, a registry configuration drop-in was added to conf.d . The content consisted of just three lines of the CRI plugin registry.config_path .
• The last file touching the same CRI plugin takes over the entire configuration. Actual measurements on containerd 1.7.27 showed 99-nvidia.toml alone 6, registry drop-in alone 0, and both 0.
• cat 99-nvidia.toml looks fine, and config dump also showed runc and plausible default values, leading to two misdiagnoses. Removing drop-ins one by one and dumping again revealed the cause.
• According to records, when time-slicing was applied to four nodes at once, the toolkit ran again and the configuration took effect. Afterward, the procedure was changed to apply it one node at a time.

What was confirmed through external evidence is this: mergeConfig in the containerd 1.7 source "completely replaces the values of Plugins" as stated in the comment, and the official man page phrase "map is append" does not explain this behavior. The NVIDIA toolkit source has a comment stating that in older versions like 1.7, the plugin section is merged only by key, so the entire CRI configuration must be duplicated in the drop-in. This replacement code is not visible in the 2.x source, but whether the same reproduction occurs was not verified. The toolkit deletes its own drop-in file upon exit. One explanation the author saw states that "handlers are not registered via SIGHUP," but since SIGHUP is not in the processing signal list of the containerd source (1.7, main) and the toolkit's default restart method is SIGHUP, my interpretation is that the process terminates and systemd revives it (the upstream unit is Restart=always). This path was directly verified in a disposable VM. Running Ubuntu package's containerd 2.2.1 as a separate service and sending SIGHUP three times via systemctl kill -s HUP resulted in the process terminating and being replaced by a new processall three times (PID changed, status went through activating to active). The journal recorded Sent signal SIGHUP to main process followed by Scheduled restart job, Started containerd.service , and the unit was Restart=always, with a 5-second restart wait. Even when SIGHUP was sent with a configuration file where the log level was changed to debug , a new process started and read the new configuration, so this is reflection through restart, not configuration reload. Reading the nodes in the author's environment, some containerd instances were 1.7.27 and others were 2.3.5, and all units were Restart=alwayswith a 5-second restart wait (read only, SIGHUP was not sent). Limitation: The tested version was 2.2.1, and SIGHUP in 1.7 was not tested (confirmed only up to the absence of SIGHUP in the source's processing signal list).

6. Capacity Planning

• capacity and allocatable: Unhealthy devices are excluded only from allocatable. Capacity is counted based on allocatable.
• allocatable and actual usage: The scheduler looks at allocation while DCGM looks at usage. In the author's 3-hour measurement, a 24GB consumer GPU had 0 utilization, held 13.7 GiB of framebuffer, and drew 21 W. It is a card that is idle but not empty.
• Time-slicing: replicas: N is registering the same device N times, and memory and errors are not isolated. For example, if a 24 GB GPU is advertised as 5, the average 4.8 GB per slot is merely explanatory arithmetic with a guaranteed value of 0. failRequestsGreaterThanOne: true rejects requests of 2 or more. The operator does not watch for ConfigMap configuration changes, and DCGM cannot attribute containers during time-slicing (official documentation).
• MIG: Since the A100 40GB has eight 5 GB memory slices and seven SM slices, 1g.5gb is at most 7 with 5 GB remaining (calculated from document structure). Since consumer GPUs are not on the support list, it cannot be used in the author's experimental environment.

7. Trying It Out (Read-Only)

# 노드별 광고량
kubectl get nodes -o custom-columns=N:.metadata.name,CAP:.status.capacity.nvidia\.com/gpu,ALLOC:.status.allocatable.nvidia\.com/gpu
# operand 현황과 NFD 라벨
kubectl -n gpu-operator get ds,po -o wide
kubectl get nodes -L nvidia.com/gpu.present,nvidia.com/device-plugin.config
# 각 GPU 노드에서: 파일이 아니라 로드된 것을 본다 (0 이면 사고)
containerd config dump | grep -c runtimes.nvidia; ls /etc/containerd/conf.d
# 툴킷이 SIGHUP 을 보낸 시각에 containerd 가 어떻게 되었나
sudo journalctl -u containerd --since "-7d" | grep -E "Started|Stopped|signal" | tail

Confirmation ending with a temporary pod is a single line of nvidia-smi -L (kubectl -n gpu-operator run gputest --rm -it --restart=Never ... --limits=nvidia.com/gpu=1 -- nvidia-smi -L). This pod temporarily uses one slot and is cleaned up by --rm . No other modification commands were used.

8. Potential Interview Questions

1. A GPU pod is Pending. Where do you look first? Compare the sum of capacity, allocatable, and allocation in the node status rather than the event sentence. The same Insufficient appears for unregistered plugins, unhealthy devices, and exhausted slots.
2. Explain the path by which a pod receives a GPU. Plugin registration, node status advertisement, scheduler number judgment, Allocate, CDI or nvidia runtime injection, and then the runc order. The front part is the gateway of the scheduler, and the back part is the gateway of the runtime.
3. What is the difference between placing drivers in a container versus on the host? Containers offer automated upgrades and OS consistency constraints, while hosts offer flexibility and direct management burden. A common risk is that drivers must keep pace with kernel changes.
4. How are the capacities of timeslicing and MIG counted? Timeslicing simply multiplies the advertisement N times with zero memory guarantee. MIG consists of hardware slices, so the sum can be smaller than the physical capacity.
5. What did you learn from the containerd drop-in incident? You must understand the merging rule where the last file touching the same plugin takes over the entire configuration. The decision is made by containerd config dump rather than by file, and changes are applied one node at a time.

9. Common Misconceptions

• 「nvidia-smi "If it works on the host, it works in the pod": Runtime handlers and advertisements are separate things.
• 「requests "Writing 0.5 to yields half a card": GPUs are integer-extended resources.
• 「replicas: 5 " divides memory into 5 parts": They share the same VRAM.
• "Running the operator makes containerd configuration safe": The toolkit writes to host files.

Article 23 (GPU Operator Pod Components and Roles, Do We Really Need Host Drivers?) is an elaboration of the component table in this post broken down by pod units.

10. References and Verified Versions

All were directly accessed on 2026-10-05.

• release-notes, platform-support, getting-started, troubleshooting, cdi, gpu-driver-upgrades, gpu-sharing, and gpu-operator-mig pages of https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/
• https://github.com/NVIDIA/gpu-operator (controllers/object_controls.go, state_manager.go, assets/)
• https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/ (install-guide, arch-overview, cdi-support)
• https://github.com/NVIDIA/nvidia-container-toolkit (cmd/nvidia-ctk-installer, pkg/config/engine/containerd)
• https://github.com/NVIDIA/k8s-device-plugin (README.md)
• https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/
• https://github.com/kubernetes/kubernetes (noderesources/fit.go, pkg/kubelet/cm/devicemanager, master)
• https://github.com/containerd/containerd (config.go 1.7, main_unix.go, docs/man/containerd-config.toml.5.md)
• supported-gpus and concepts pages of https://docs.nvidia.com/datacenter/tesla/mig-user-guide/

Verified Versions: GPU Operator 26.7.1 (2026-08-21, toolkit 1.20.1, device plugin 0.20.1, DCGM Exporter 4.6.1-4.8.4), Kubernetes stable v1.37.1, containerd 2.4.1 and 1.7.36. The author's experimental environment was GPU Operator v26.3.3.

Unverified Items: The reason why the minimum containerd version for 26.7.0 is documented as "1.8 to 2.0" (1.8 appears to be a typo), observation of SIGHUP in containerd 1.7 (tested only in 2.2.1), merge behavior in 2.x, the installation guide's statement that "if no driver is present, the plugin quietly reports 0" (the plugin README states that FAIL_ON_INIT_ERROR defaults to true and thus fails), and the operand pod status logs at the time of the incident.