Anatomy of all DCGM exporter metrics: 18 that actually appear and ones that do not

Status: Draft (2026-10-05, updated 2026-10-09), reflects actual measurements. Metric lists, values, labels, and logs were received as read-only from dcgm-exporter in the experimental environment. Supplementary explanations of metric meanings are based on NVIDIA DCGM and dcgm-exporter documentation; the documentation was not reopened and cross-checked at the time of writing this article (unconfirmed). PromQL examples were not executed (unexecuted). Writing rule: Actual measurement data is the default, and unmeasured items are marked as "Unmeasured".

Table of Contents

• 1. Architecture and Experimental Environment Configuration
• 2. 18 Metrics That Actually Appeared
  • 2.1 Clocks, Temperature, and Power
  • 2.2 Utilization
  • 2.3 Memory
  • 2.4 Errors and Reliability
• 3. Labels
• 4. Differences and Observations Revealed by Labels
• 5. What Does Not Appear and Why
• 6. Query Examples (Unexecuted)
• 7. Potential Interview Questions
• 8. Common Misconceptions
• External Evidence: Verified with Official Documentation and Sources
• Unverified Items
• References
• Verified Versions and Dates

1. Architecture and Experimental Environment Configuration

DCGM (Data Center GPU Manager) is NVIDIA's GPU management and monitoring library, and dcgm-exporter is a program that exports values read by DCGM in Prometheus format. The flow is "GPU Driver (NVML) → DCGM → dcgm-exporter (HTTP 9400) → Prometheus".

The configuration of the experimental environment is as follows.

2. 18 Metrics That Actually Appeared

There were 18 metrics in 54 lines of output from a single 8GB-class laptop GPU, and they were the same 18 metrics on another laptop GPU of the same generation. The values in the table below are example values when this GPU is idle. The meanings are based on the metric HELP text and DCGM documentation.

2.1 Clocks, Temperature, and Power

• Memory temperature is 0 on all observed GPUs. This metric does not seem to mean "0°C"; rather, these GPUs appear not to provide the value, though the cause has not been confirmed. 0 should not be treated as a normal value when creating alert conditions.
• Cumulative energy is a counter. You should look at the increment rather than the raw value. Since it is in mJ units, rate() divided by 1000 gives the average watts. For example, the cumulative 8,447,476,600mJ in the above example is approximately 2.3kWh (since boot).
• Idle clocks are low. The SM clock was 210MHz when all observed GPUs were idle.

2.2 Utilization

GPU_UTIL The biggest pitfall of is the impression given by its name. This value is not "how full the SM is" but "the ratio of time during which at least one kernel was running" (per documentation). If a single small kernel runs continuously, it will show 100% even if most of the SM is idle. Therefore, you cannot tell whether a GPU is actually saturated in LLM serving using this value alone. As seen in Section 5 below, the GPUs in this experimental environment cannot receive more accurate metrics (profiling).

2.3 Memory

Adding the three values gives the total memory of the GPU. The above example is 12 + 7,795 + 380 = 8,187MiB, and on other cards of the same class it is 8,186MiB, and on a 24GB-class consumer GPU it is 24,575MiB, matching 8GiB-class and 24GiB-class cards respectively. Therefore, when calculating utilization, the denominator should use USED + FREE rather than USED + FREE + RESERVED to be accurate. The reserved area is space that the model cannot use.

2.4 Errors and Reliability

• If PCIe retransmissions increase, suspect link quality issues. A typical inspection target is when the PCIe link width has decreased.
• Retired row metrics varied by GPU. On Ada-generation laptop GPUs, all three metrics were output, but on the 24GB-class consumer GPU, these three were omitted from the output, leaving only 15 metrics. It appears that if a GPU does not support the corresponding value, the exporter does not emit the metric, but the exact rule was not verified.
• ROW_REMAP_FAILURE If is 1, it is a signal to consider replacement on datacenter GPUs (per documentation). All observed GPUs were 0.

3. Labels

The same labels are attached to all metrics.

If there is a Pod using the GPU, container, namespace, pod labels are added. We verified that these three labels are attached on GPUs that have pods using them. These three are obtained from kubelet's pod-resources socket, and were not attached when the GPU was idle.

To attach custom labels of pods, the exporter needs permission to read the API server. In the experimental environment, the exporter was unable to access the API server within the cluster, and Failed to get in-cluster config, pod labels will not be available remained in the logs. This means that to aggregate costs by team or project using pod labels, the exporter must be granted permission to read the API server, but we did not test changing it that way (untested).

Since the variety of label values is proportional to the number of GPUs, the cardinality of this metric is small. This contrasts with Hubble in the next post (Cilium metrics).

4. Differences and Observations Revealed by Labels

• Driver versions can be mixed in the same cluster, and that fact is immediately revealed by the DCGM_FI_DRIVER_VERSION label. In the experimental environment, two versions, 595.91.07 and 570.195.03, coexisted. This does not mean they are the cause of a problem, but rather that they are variables to filter out first during upgrades or failure investigations.
• Idle power varies by GPU model. The 8GB-class laptop GPU consumed 1.22W at idle, while the 24GB-class consumer GPU consumed 21.1W even at idle.

5. What Does Not Appear and Why

The reasons remain intact in the exporter startup logs (8GB-class laptop GPU, 2026-09-23).

Not collecting DCP metrics: Profiling is not supported for this group of GPUs or GPU
Skipping line 21 ('DCGM_FI_PROF_GR_ENGINE_ACTIVE'): metric not enabled
Skipping line 22 ('DCGM_FI_PROF_PIPE_TENSOR_ACTIVE'): metric not enabled
Skipping line 23 ('DCGM_FI_PROF_DRAM_ACTIVE'): metric not enabled
Skipping line 24 ('DCGM_FI_PROF_PCIE_TX_BYTES'): metric not enabled
Skipping line 25 ('DCGM_FI_PROF_PCIE_RX_BYTES'): metric not enabled
Not collecting NvSwitch metrics; no switches to monitor
Not collecting CPU metrics; ... DCGM module that is not currently loaded

The most disappointing thing in LLM serving operations is profiling metrics. You need to know the tensor core active ratio and DRAM active ratio to determine whether computation is the bottleneck or memory bandwidth is the bottleneck, but these consumer laptop GPUs cannot receive them. We re-verified up to a desktop 24GB-class GPU on 2026-10-09. We simply read the collector output of five GPUs in the cluster (four models: two generations of laptops and one desktop) via API proxy, and the number of lines starting with DCGM_FI_PROF_ was 0 across all five cards. They did not appear simply because it was a desktop GPU. The metric types were also not identical; the latest generation laptop GPU had 18 types, while the previous generation laptop and desktop GPUs had 15 types, with the three missing being row remapping metrics (DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS, ..._UNCORRECTABLE_REMAPPED_ROWS, DCGM_FI_DEV_ROW_REMAP_FAILURE). NVIDIA documentation only notes "limited DCGM functionality" for non-datacenter GPUs such as consumer and workstation products, and the feature table (Features Supported on Different GPU Products) also lacks a profiling line, so we could not distinguish from documentation which products output them. Whether they actually appear on datacenter GPUs remains unmeasured since we lack the equipment.

XID error metrics and power/thermal limit violation metrics are absent from this output. They are not even in the skipped list in the logs. We initially presumed that XID was omitted from the collection list in the first place, but verification with external materials proved that presumption wrong. This Pod's configuration is DCGM_EXPORTER_COLLECTORS=/etc/dcgm-exporter/dcp-metrics-included.csv, the image tag is 4.5.3-4.8.2-distroless , and the same tag file in the upstream repository (original text checked) contains DCGM_FI_DEV_XID_ERRORS as active lines (sequence 11 excluding comments). The five PROF lines in the same file correspond exactly to "Skipping line 21" through "line 25" in the log, and the 18 metrics we observed are exactly the 26 active lines in this file minus XID, NVLink bandwidth, and the five PROF lines. Therefore, it appears that they are in the list but values do not come out on this GPU(presumption. The exporter source skips a metric if DCGM returns a blank value. We could not open the running file (distroless), and currently, lines in the DCGM_FI_DEV_XID_ERRORS family are 0). Power and thermal limit violation metrics are commented out in the upstream file, so they are not in the list to begin with. Currently, XID can only be checked via kernel logs (NVRM: Xid), and this workflow will be covered further in part 11.

6. Query Examples (Unexecuted)

These are examples that were not actually executed in Prometheus. Metric names and units match the actual measurements above.

# 평균 전력(W): 누적 에너지(mJ)의 증가율을 1000 으로 나눈다
rate(DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION[5m]) / 1000

# 메모리 사용률: 분모에 예약 영역까지 포함한다
DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE + DCGM_FI_DEV_FB_RESERVED)

# 알림 후보
DCGM_FI_DEV_ROW_REMAP_FAILURE > 0
increase(DCGM_FI_DEV_PCIE_REPLAY_COUNTER[1h]) > 0

7. Potential Interview Questions

• GPU_UTIL If is 100%, is the GPU saturated? No. Since it is the ratio of time during which at least one kernel was running, it is separate from SM occupancy or tensor core usage.
• What metrics are used to distinguish between compute bottlenecks and memory bottlenecks? Profiling metrics (tensor active, DRAM active). On GPUs that do not support them, you must indirectly estimate them using serving metrics (time per token, batch size).
• Where do you view XIDs? Kernel logs are the default, and you can also view them as metrics by adding fields to the collection list.
• What are the issues when driver versions differ in the same cluster? CUDA compatibility, the container toolkit, and error behavior may vary, so this must be filtered out first during troubleshooting.

8. Common Misconceptions

• "Memory temperature 0 is normal": There might be no value.
• 「FB_USED + FB_FREE is the total memory": There is a separate reserved area.
• "Just running the exporter provides all metrics": This depends on the GPU family and the collection list. In this experimental environment, there were 18.

External Evidence: Verified via official documentation and source code

Things Not Verified

• Time variations in time series. All values in this post are snapshots at a single point in time, and values under load were not measured.
• The cause of the 0 memory temperature and the exact rule for why row-remapping metrics are omitted on 24GB-class consumer GPUs.
• The running contents of the dcp-metrics-included.csv being used (could not open as the image lacked a shell. Checked with upstream tag files instead, and the exact reason why XID values do not appear was unverified).
• Supplemental explanations of metric meanings were not cross-referenced by reopening the cited DCGM documentation this time.
• Whether pod labels are actually attached when granting the exporter API server read permissions.

References

• https://github.com/NVIDIA/dcgm-exporter
• https://docs.nvidia.com/datacenter/dcgm/latest/

Versions and Dates Verified

As of 2026-10-05. dcgm-exporter 4.5.3-4.8.2, drivers 595.91.07 and 570.195.03.