GPU Hardware and Driver Failure Diagnosis

Status: Draft (Updated on 2026-10-06). Descriptions confirmed by official documentation serve as the basis, and only the parts confirmed as read-only on three consumer GPU models (the "Actual Measurements" at the end of Section 10) are values directly verified. Data center GPUs and actual failure events were not measured. Please read "Verified Versions and Dates" and "Unverified Items" at the end of the post first. Writing rule: Actual measured data is the baseline, and unmeasured items are marked as "Unmeasured".

Table of Contents

• Summary
• 1. Path of Signal Flow
• 2. Consumer GPUs vs. Data Center GPUs
• 3. How to Read XIDs
• 4. ECC, Row Remapping, and Retirement
• 5. NVLink, NVSwitch, and Fabric Manager
• 6. Throttling and PCIe Degradation
• 7. A Single Slow GPU Slows Down the Entire Workload
• 8. Kubernetes Response: Isolation and Automatic Recovery
• 9. Alert Design
• 10. Try It Yourself (Read-Only)
• External Evidence: Findings Confirmed via Papers and Official Materials
• 11. Interview Questions
• 12. Common Misconceptions
• 13. References and Verified Versions
  • Actual Measurement: Values Read from Three Consumer GPU Models (Read-Only)

Summary

GPU failure diagnosis is the process of reading three streams of signals: XID messages in kernel logs, DCGM (Data Center GPU Manager) and nvidia-smi metrics, and Kubernetes node states.

What you can explain in an interview after reading this article:

1. How to read XIDs in the order of "number, catalog immediate action bucket, and whether it occurred solo" to distinguish between application errors and hardware errors
2. Which metrics and codes reveal ECC, NVLink, throttling, and PCIe degradation, and how to find a single slow GPU
3. The architecture of an automatic isolation and recovery pipeline, and criteria for deciding when to page versus when to leave a ticket

1. Path of Signal Flow

GPU + 드라이버(NVML)
 ├─ 커널 로그 "NVRM: Xid (PCI:0000:ca:00): 79, ..." ─► 로그 수집기(syslog 모니터)
 ├─ NVML 이벤트(XID) ─► 디바이스 플러그인 ─► 장치 unhealthy ─► kubelet allocatable 감소
 └─ DCGM ─┬─ dcgm-exporter :9400/metrics ─► Prometheus ─► 알림
          └─ health watch, dcgmi diag (능동 점검)

These are used for diagnosis from the dcgm-exporter default metrics file (default-counters.csv, main). Parentheses indicate default states.

Since DCGM API documentation field names (e.g., ..._TEMP_CELSIUS) may differ from exporter metric names, base your work on the CSV of the exporter you are using.

2. Consumer GPUs vs. Data Center GPUs

This is an actual measurement in an experimental environment. In the raw dcgm-exporter output for an 8GB laptop GPU (driver 595.91.07), the XID_ERRORS and PROF_* lines were absent, and the MEMORY_TEMP and remapping row metrics were 0. We could not verify whether 0 meant "supported and healthy" or "unsupported thus 0", nor whether the absence of the XID line was due to the exporter version or consumer GPU constraints. The "Applicable Products" column in the XID catalog also lists only A100 and newer data center products.

3. How to Read XIDs

The format is NVRM: Xid (PCI 주소): 번호, 부가 정보 . The reading order is: ① immediate action bucket from the catalog, ② whether it is solo or accompanied by other XIDs (45 can also be logged as a result of other errors), and ③ whether it repeats on the same GPU. The buckets are RESTART_APP(restart app only), RESET_GPU, RESTART_BM(restart node), and IGNORE, CONTACT_SUPPORT . Xid 154 indicates the required recovery action (None, Drain P2P, Drain and Reset, GPU Reset Required, Node Reboot Required).

Since the catalog states that 31 is frequently app-related while the GPU debug guide terms it "Suspected Hardware Problems", one must explicitly branch: single occurrences are app issues, while repeated occurrences are hardware issues. The device plugin hardcodes 13, 31, 43, 45, 68, and 109 as "application errors" to be ignored, and marks devices as unhealthy on the remaining XIDs that NVML reports as critical (source). Since a FIXME in the source notes there is no recovery path from unhealthy, we infer that resetting followed by restarting the plugin is required (unconfirmed).

4. ECC, Row Remapping, and Retirement

Single-bit errors are corrected by hardware, while double-bit errors are only detected. Counters are divided into volatile (since driver load) and aggregate (lifetime cumulative). Since Ampere, row remapping has replaced page retirement (older models max 64), supporting up to 512 remappings, and application requires a GPU reset. The remapping failure flag is triggered when attempting to remap a bank that already has 8 rows remapped, attempting to remap an already remapped row, or reaching a total of 512 remappings, serving as the basis for RMA decisions. The reference documentation is based on A100 and later.

nvidia-smi -q -d ECC,ROW_REMAPPER,PAGE_RETIREMENT

5. NVLink, NVSwitch, and Fabric Manager

Fabric Manager (FM) unifies all GPUs into a single memory fabric in NVSwitch systems and monitors links. On HGX A100 systems, starting CUDA before FM initialization results in cudaErrorSystemNotReady . Starting from Hopper, GPUs and switches perform self-link training and do not require FM for resets. FM must share the same version as the driver. When 74 occurs, use nvidia-smi nvlink -e to check CRC, replay, and recovery error counters. Fatal SXids abort CUDA tasks and leave both 74 and 45. Blackwell series uses XIDs 144 through 150 for NVLink 5, and SXids do not apply. Ampere NVSwitch systems can reset GPUs individually only while FM is running; otherwise, all GPUs bound via NVLink must be reset together.

6. Throttling and PCIe Degradation

clocks_event_reasons is a bitmask. The values are GPU idle 0x1, application clocks 0x2, SW power cap 0x4, HW slowdown 0x8, sync boost 0x10, SW thermal 0x20, HW thermal 0x40, HW power brake 0x80, display clock 0x100, board limit 0x200, and reliability 0x400 (go-nvml). 0x8 and 0x40 drop core clocks to half or lower, and 0x80 is significant as an external power brake signal. The Llama 3 paper notes that throughput varied by 1 to 2% on a daily cycle due to midday ambient temperatures. For PCIe, compare pcie.link.gen.current and pcie.link.gen.max, pcie.link.width.current , but since the manual explains that current values may drop during idle times, measurements must be taken under load. Retries are counted in PCIE_REPLAY_COUNTER , and the DCGM health example output warns when exceeding 8 times per minute.

7. A Single Slow GPU Slows Down the Entire Workload

Meta reported 466 outages (419 unexpected) over a 54-day period for Llama 3 405B, with about 78% of unexpected outages attributed to hardware causes (GPU 58.7%). Opening and verifying the original paper, this 58.7% is stated in the text but is the sum of printed percentages in Table 5; recounting by counts from the same table yields 268 GPU-related cases, which is 64.0% out of 419 (the table's "Faulty GPU" figure of 30.1% appears to be a print error inconsistent with the 148 count, which by count is 35.3%). This is summarized in "External Evidence" below. The same article notes that stragglers—which remain operational yet slow—are hard to find, a single one can slow down thousands of GPUs, and they manifest as slow communication. The troubleshooting order is my summary: ① Compare step and collective communication times per rank (PyTorch flight recorder). ② Place the slow rank GPU's throttle reasons, temperature, power, PCIe generation and width, and NVLink errors side by side. ③ After draining the node, verify with dcgmi diag -r 3(including targeted_stress, nvbandwidth, and nccl_tests). DCGM documentation states that this does not substitute for comprehensive hardware diagnostics and RMA evidence.

8. Kubernetes Response: Isolation and Automatic Recovery

flowchart LR
  D["탐지<br/>XID, DCGM, 플러그인"] --> C["분류<br/>카탈로그 버킷, 반복 여부"]
  C --> Q["cordon·taint"]
  Q --> R["drain<br/>PDB 존중"]
  R --> X["복구<br/>GPU 리셋, 노드 재시작, 교체"]
  X --> V["검증<br/>dcgmi diag -r 1 또는 2"]
  V --> U["uncordon"]
  X -. 반복 실패 .-> H["사람 호출"]

Even if a device becomes unhealthy, already assigned pods remain bound to that device (Kubernetes documentation), so draining is required. Kubernetes 1.36 ResourceHealthStatus(beta) exposes device health via container status allocatedResourcesStatus . NVSentinel (v1.25.0, Beta/Stable, Kubernetes 1.34+, requires standalone DCGM) classifies DCGM and syslog XIDs using a built-in catalog to drive cordon, drain, reset, and reboot actions. It features a circuit breaker that halts if 50% of nodes are cordoned within 5 minutes, and defaults to monitoring-only. Amazon EKS's node monitoring agent maps well-known XIDs to node conditions AcceleratedHardwareReady=False to be escalated so that 48 is logged as a reboot, 74 and 79 as a replacement, and unknown XIDs are left only as events. Before contacting the vendor, nvidia-bug-report.sh outputs, FM logs, and DCGM diagnostic logs are collected (GPU Debug Guide).

9. Alert Design

The Google SRE book states, "If a response is mechanical, it is not a page." Below is my proposal applying this.

10. Hands-On (Read-Only)

# GPU 가 있는 노드에서 실행
journalctl -k --no-pager | grep -E "NVRM: Xid" | tail   # dmesg 가 막힌 계정에서도 읽힘, 아래 「실측」 참고
nvidia-smi --query-gpu=index,name,pcie.link.gen.current,pcie.link.gen.max,pcie.link.width.current,temperature.gpu,power.draw,clocks_event_reasons.active --format=csv
nvidia-smi -q -d PERFORMANCE,ECC,ROW_REMAPPER | head -80

# 쿠버네티스에서 dcgm-exporter 지표 확인
kubectl -n gpu-operator get po -l app=nvidia-dcgm-exporter -o wide
curl -s http://<파드IP>:9400/metrics | grep -E "XID|REMAP|PCIE_REPLAY|VIOLATION"

Dashboards that only monitor utilization, power, temperature, VRAM, and clock speeds easily miss GPU-specific alert rules. I observed on an 8GB card that a server process's allocator cache grew from 6.4 GB to 7.8 GB in just 6 minutes, and such VRAM trends make great material for practicing alerts.

External Evidence: Verified by Papers and Official Documentation

Since I lack a datacenter GPU and large-scale cluster, I could not measure actual failure incidents. Instead, I opened large-scale operations reports and NVIDIA official documentation to cross-check 11 descriptions. Because external metrics involved different equipment and conditions, I only evaluated whether they aligned in direction.

Vendor figures were transcribed only when their conditions were stated in the text, and materials whose original papers could not be accessed were omitted.

11. Potential Interview Questions

1. You encountered XID 79. What do you do? It means the GPU is inaccessible via PCIe, and the catalog bucket is node reboot. Reboot after cordon and drain, and if it recurs, inspect links, power, and system event logs before replacement.
2. How do you handle XID 13 and 48? 13 is usually an application error so investigate with Compute Sanitizer, and suspect hardware if it repeats on the same TPC/GPC. 48 is an uncorrectable ECC error requiring a reset or reboot, and if accompanied by 63/64, drain and reset.
3. How do you find a single slow GPU? Narrow down the suspected rank using per-rank communication times and compare that GPU's throttle reasons, temperature, PCIe, and NVLink errors. After clearing the node, verify with dcgmi diag -r 3 .
4. What should you be careful about in auto-remediation? Prevent mass isolation using concurrency limits and circuit breakers, and escalate repeated failures to humans. Resets require that no processes are using the GPU.
5. When do you trigger a page? Automate what is handled mechanically and log it only. Page when auto-remediation fails or repeats, or when available GPUs fall below the threshold.

12. Common Misconceptions

• "An XID entry means the GPU is broken": 13, 31, 43, and 45 are often application errors.
• 「XID_ERRORS " is the error count": It is the latest XID value.
• "High utilization means healthy": Slow GPUs also have high utilization.

13. References and Verified Versions

All accessed on 2026-10-05.

• https://docs.nvidia.com/deploy/xid-errors/ (Catalog, last updated 2026-09-09)
• https://docs.nvidia.com/deploy/a100-gpu-mem-error-mgmt/ , https://docs.nvidia.com/deploy/gpu-debug-guidelines/
• https://docs.nvidia.com/datacenter/dcgm/latest/ (feature-overview, dcgm-diagnostics, getting-started, dcgm-api-field-ids)
• https://github.com/NVIDIA/dcgm-exporter (default-counters.csv), https://github.com/NVIDIA/k8s-device-plugin (health.go), https://github.com/NVIDIA/go-nvml (const.go)
• https://docs.nvidia.com/deploy/nvidia-smi/index.html , https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index.html
• https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/troubleshooting.html
• https://github.com/NVIDIA/NVSentinel , https://docs.nvidia.com/nvsentinel/ , https://docs.aws.amazon.com/eks/latest/userguide/node-health-nma.html
• https://arxiv.org/abs/2407.21783 , https://arxiv.org/abs/2505.05713 , https://sre.google/sre-book/monitoring-distributed-systems/
• https://docs.pytorch.org/tutorials/unstable/flight_recorder_tutorial.html , https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/

Measurements: Values read from three consumer GPU models (read-only)

Read fields and kernel logs from a 24GB-class desktop (Ampere generation, driver 570.195.03), an 8GB-class laptop (Ada generation, 595.91.07), and a 6GB-class laptop (Ampere generation, 595.91.07). nvidia-smi -q -d ECC,ROW_REMAPPER,PAGE_RETIREMENT, --query-gpu All three were idle and nothing was modified.

• The statement in Section 2's table that "Row remapping is displayed in the GA10x and Ada columns" varied by product. The two consumer Ampere-generation models returned N/A for the driver, while only the Ada-generation laptop returned values. On this GPU, 0 meant "supported and healthy," whereas on unsupported GPUs the same query returned N/A instead of 0. Therefore, dashboards must distinguish between N/A and 0, and a value of 0 should not be read as healthy simply because it is always 0. The remapping value of 0 shown by the 8GB laptop GPU's exporter in Section 2 is consistent with this.
• ECC counters were not exposed across these three models (GDDR6 generation). Since all three models returned N/A, memory errors cannot be caught with hardware counters, and we must rely on Xid logs and application misbehavior. We do not generalize that \u201cconsumer GPUs have no ECC.\u201d NVIDIA's RTX Blackwell whitepaper explains that GDDR7 ECC is built into the DRAM die, always on, and cannot be disabled by software (whether the counter is visible is a separate matter), and I could not find an official statement explicitly writing that the RTX 30 and 40 generations of consumer product lines lack ECC (indirect evidence includes the nvidia-smi documentation stating \u201cconsumer product lines have very limited information\u201d and that the ECC mode entry requires an InfoROM object).
• The query field names were valid in both driver 570 and 595. remapped_rows.correctable, remapped_rows.uncorrectable, remapped_rows.pending, remapped_rows.failure, retired_pages.pending, ecc.errors.corrected.volatile.total, inforom.ecc, clocks_event_reasons.hw_thermal_slowdown, clocks_event_reasons.sw_power_cap, pcie.link.gen.gpucurrent returned a value or [N/A] without errors. In other words, because there was no error stating that the field name was wrong, N/A does not mean the field does not exist, but rather that the GPU does not provide the value.
• Kernel logs may block dmesg . All three nodes had kernel.dmesg_restrict=1 , so regular accounts\u2019 dmesg failed with read kernel buffer failed: Operation not permitted , whereas journalctl -k was readable. Since commands instructed to require sudo are blocked on passwordless accounts, operations check scripts are better off using journalctl -k first.
• grep -iE "NVRM|Xid" catches incorrect lines. In logs like this, this pattern caught both (1) the line where the NIC driver printed the chip revision XID 541 and (2) non-Xid NVRM: messages (allocation failure NV_ERR_NO_MEMORY). Since neither are GPU failures, alert rules and check commands must be narrowed down to NVRM: Xid . The actual NVRM: Xid lines totaled 0 across all three nodes.
• The PCIe generation had dropped to 1 during idle. All three nodes had gpucurrent as 1, which looks different from the maximum (4). However, since this may be due to idle power saving, the demotion determination in Section 6 must be re-read under load. One laptop GPU came out with a width of x8 (maximum x16), which might be due to the design, so it was not judged.
• Re-read with a load applied (one 8GB-class laptop GPU). Loaded as a serving server for about 80 seconds while reading 38 samples every 2 seconds. The PCIe generation was at its maximum of 4 (37 samples, with the remaining 1 being 2), and the width was consistently 8. The SM clock had a median of 2,640MHz (falling short of the maximum 3,105MHz, with a sample dropping down to 375MHz when the load started), power had a median of 78W and a maximum of 91.8W, temperature reached a maximum of 63\u00b0C, and utility had a median of 100%. **The clock throttling reason was 0 (none) in all 37 load samples**, and only a single idle sample was 0x01 (idle). This means that no power cap or thermal throttling was applied under this load, and falling short of the max clock seems to be due to load and power policies without a throttling reason (estimated). Even in an idle state, if a server process holds the GPU, the PCIe generation is maintained at 4 (6 samples, SM clock 1,890MHz, power 8.7W), and generation 1 read from nodes without processes was very likely a power-saving state without context (estimated). Hot and enclosed environments where thermal throttling and power caps are actually triggered were not tested (15-minute continuous load was checked below and did not trigger clock throttling).
• No clock throttling reasons were triggered even under a 15-minute continuous load (one 8GB-class laptop GPU). Loaded continuously with 32 concurrent loads for 15 minutes on the same server while reading 443 samples at 2-second intervals. PCIe remained constant at gen 4, width 8; the SM clock had a median of 2,625MHz (range 1,890\u20132,640), power had a median of 78W and a maximum of 93.2W, and temperature reached a maximum of 70\u00b0C. Clock throttling reasons were all 0 excluding one idle bit sample (no samples hit power caps or thermal throttling). Throughput during the same time was 829\u2013832 tok/s, within \u00b10.2% across 23 measurements. Although the temperature increased by 7\u00b0C compared to the 80-second test\u2019s maximum of 63\u00b0C, heat did not degrade performance in this environment. This differs from long-term datacenter training or hotter environments.
• I actually encountered a GPU whose PCIe generation does not increase even when a load is applied (one 24GB-class desktop GPU). Although this GPU supports a maximum link of 16GT/s (gen 4) and a width of x16, even after running matrix multiplication and host-to-GPU copies concurrently for 25 seconds to reach the P0 state, 100% utilization, and an SM clock of 1,950MHz, the current generation in nvidia-smi was 1, and values read from the host side using /sys/bus/pci/devices/<장치>/current_link_speed 20 times every 3 seconds were also all 2.5GT/s (gen 1). Host-to-GPU transfers using 1GiB of pinned memory were 1.7~1.9GB/s, and GPU-to-host transfers were 1.6~2.2GB/s, remaining identical across three repetitions. This is about 6% of the theoretical bandwidth of PCIe 4.0 x16 (approx. 31.5GB/s). Since the same GPU's memory bandwidth (905GB/s read) and compute throughput (fp16 matrix multiplication 63~71 TFLOPS) were normal, it appears to be an issue where only the link is locked to a low generation. The root cause (motherboard settings, power-saving policies, riser, slot) was not identified (unconfirmed). Because host-to-GPU transfers become the bottleneck for model loading and KV offloading on this GPU, loading a 7GB model takes about 4 seconds for transfers alone (7 \u00f7 1.8), and the benefits of loading 8 volumes of KV are also reduced (calculated). There are two inspection methods. Read nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.gen.max --format=csv during load, and compare the host's current_link_speed and max_link_speed . Having a low generation at idle is normal, but being low even under load indicates a demotion.

Limitations: Limited to only three consumer models; actual failures such as datacenter GPU ECC, row remapping failures, and XID 79 were not encountered. This test also could not determine why DCGM's XID_ERRORS metrics do not appear on consumer GPUs (whether it is an exporter version issue or a GPU limitation) (unconfirmed).

Verified Versions: DCGM 4.6.1-1 bundled in GPU Operator 26.7.1, DCGM Exporter 4.6.1-4.8.4, Kubernetes stable v1.37.1.

Unverified Items: Actual output of consumer GPU ECC, row remapping, and XID metrics; exporter version in the experimental environment; whether query field names (based on third-party published --help-query-gpu output) are valid in driver 595; and applicability of consumer GPUs in the catalog. The alert severity table and investigation order are my own judgment, not official regulations.