Dissecting All Cilium Metrics: Agent, Operator, Envoy, and Hubble

Status: Draft (2026-10-05), reflecting actual measurements. The metric list and values are a read-only snapshot taken at a single point in time from an agent, the same node's Envoy and Hubble, and a single operator in an experimental environment. Metric meanings are based on HELP strings and Cilium documentation; the docs were not re-opened and cross-checked at the time of writing (unverified). PromQL examples were not executed (unexecuted). Writing rule: Actual measured data is the baseline; unmeasured items are marked as "Unmeasured".

Table of Contents

• 1. Four Places That Emit Metrics
• 2. Agent Metrics (cilium_* approx. 100 families)
  • 2.1 Datapath: Forwarding, Drops, and Connection Tracking
  • 2.2 BPF Resources
  • 2.3 Endpoints, Identities, and Policies
  • 2.4 Kubernetes Integration and Control Plane
  • 2.5 Nodes and Cluster
  • 2.6 Feature Flags (cilium_feature_* 32 items)
  • 2.7 Process and Runtime
• 3. Operator Metrics (96 families)
• 4. Envoy Metrics (450 families)
• 5. Agent Status Read Through Actual Measurements (Single Node, ~11.8 Days Uptime)
• 6. The Cardinality Problem of Hubble Metrics
• 7. Alert and Dashboard Candidates (Unexecuted)
• 8. Likely Interview Questions
• 9. Common Misconceptions
• Unverified Items
• References
• Verified Versions and Dates

1. Four Places That Emit Metrics

Cilium 1.20.1 (kube-proxy-replacement=true, VXLAN tunnel, Hubble active) emits metrics separately from four locations. All of them are exposed as headless services within the cluster.

The configuration is enable-metrics=true, enable-hubble-open-metrics=true . The metrics to be exported by Hubble are determined by the hubble-metrics setting.

dns drop tcp flow port-distribution icmp
httpV2:exemplars=true;labelsContext=source_ip,source_namespace,source_workload,destination_ip,destination_namespace,destination_workload,traffic_direction

The fact that the vast majority of the numbers come from a single Hubble location is an important observation of this article, so it is covered separately in Section 6.

2. Agent Metrics (cilium_* Approx. 100 Families)

Out of 169 metric families, the agent has about 100 cilium_* , while the rest are Go runtime (go_*) and process metrics. The meanings are translated from HELP strings into Korean.

2.1 Datapath: Forwarding, Drops, and Connection Tracking

2.2 BPF Resources

2.3 Endpoints, Identities, and Policies

2.4 Kubernetes Integration and Control Plane

2.5 Nodes and Cluster

2.6 Feature Flags (cilium_feature_* 32 items)

This is a collection of gauges indicating that a value of 1 means enabled. It is convenient for configuration auditing. The enabled states read from a single node are as follows.

• Enabled (1): Bandwidth manager, Envoy configuration and Envoy proxy, kube-proxy replacement, node port configuration, host firewall, policy other than default deny. (Items like identity allocation, IPAM, and datapath mode are distinguished by mode labels rather than values, so they were not interpreted here.)
• Disabled (0): BGP, egress gateway, L2 load balancer and L2 pod advertisement, SCTP, VTEP, local redirect policy, mutual authentication, endpoint slices, endpoint routes.
• Cumulative Count (counter): Shows the cumulative value by policy type loaded since the agent started.

There are no encryption-related metrics. wireguard and encrypt lines were 0. This matches the state where transparent encryption (WireGuard) is disabled, and metrics generated after enabling were not directly tested (unmeasured, to avoid changing production settings). Instead, reading the official Cilium metrics documentation (Feature Metrics table), it states that there is a gauge indicating the transparent encryption mode (transparent_encryption, with the label mode, node2node_enabled, strict_mode_enabled) and that the mode value can take wireguard and ipsec . Since the prefix of the actually exposed name was not verified, after enabling it should be searched as grep -i encryption .

2.7 Process and Runtime

cilium_process_*(CPU time, open files, resident memory, network transmit/receive bytes, etc., 9 families) and go_*(memory stats 22 families, GC, scheduler) exist. Agent resident memory on a single node was approximately 270 MiB (283,545,600 B).

3. Operator Metrics (96 families)

The fact that Gateway API and Ingress controller are enabled in the experimental environment matches these feature flag metrics.

4. Envoy Metrics (450 families)

Cilium's Envoy handles the Gateway API and L7 policies, and emits standard Envoy metrics as-is. The number of families based on the prefix envoy_cluster_* 191, envoy_http_* 89, envoy_listener_* 78, envoy_server_* 28, envoy_cilium_* is around 14.

Envoy on a single node had a server uptime of 1,019,397 seconds (~11.8 days) and memory allocation of about 10 MB. Requests passing through the gateway numbered 14,270 on a single listener, broken down by response code family as 12,916 for 2xx, 1,217 for 3xx, 51 for 4xx, and 4 for 5xx. 5xx accounts for about 0.03% of the total. However, this is the share handled by that single node's Envoy, and Envoys on other nodes were not aggregated.

5. Agent Status Read Through Actual Measurements (Single Node, ~11.8 Days Uptime)

• Unsupported L3 protocol accounts for almost all drops. We suspect they might be non-IPv4 packets (such as IPv6), but we could not open and verify the packets (unverified). Since this reason is not a policy rejection, it should not be interpreted as an intrusion sign, and filtering out this reason when creating alerts should reduce noise.
• BPF map fill ratios are all low. This means there is plenty of headroom for a while even if the number of services increases.
• Service backend not found 159 instances are packets sent to a service whose endpoints have disappeared. This may be a normal value occurring during a short window of a rolling deployment, but if it continuously increases, it is subject to inspection.

6. The Cardinality Problem of Hubble Metrics

For example, on a single node, Hubble had only 14 metric families but about 80,000 time series. The main culprits are three families.

The cause is the labelsContext setting. Because this setting instructs source_ip and destination_ip to be used as labels, every single incoming client IP becomes a separate time series. In the largest series, there were thousands of different values for source_ip , and many entries had empty source namespaces and workloads, appearing to be requests coming from outside the cluster. This value continues to increase as visitors increase (source categorization was not verified).

This configuration has two problems:

1. Prometheus memory and storage costs increase. There are 80,000 time series on just one node, and costs increase as the number of nodes receiving external requests grows.
2. Visitors' IP addresses remain as they are in the metrics storage. This is an area that needs review from a privacy perspective as well.

The direction for improvement seems to be removing labelsContext and source_ip from destination_ip and leaving only the namespace and workload. However, this is just a proposal; the reduction in time series upon actually making this change was not verified (untested).

The configuration includes dns , but hubble_dns_* metrics were not present in the output. The 14 series are HTTP, port distribution, flow, drop, TCP flags, ICMP, lost events, metric handler, and gRPC. DNS metrics are known to require separate DNS visibility settings, but this was not verified in this experimental environment, and the cause is unknown (unconfirmed).

7. Candidate Alerts and Dashboards (Not Executed)

This is an unexecuted example. Metric names match the actual measurements above.

# 연결할 수 없는 노드, 실패한 컨트롤러
cilium_unreachable_nodes > 0
cilium_controllers_failing > 0

# BPF 맵이 가득 차기 전에
cilium_bpf_map_pressure > 0.9

# 정책 거절과 서비스 백엔드 없음: 노이즈인 L2/L3 프로토콜 사유는 제외한다
sum by (reason) (rate(cilium_drop_count_total{reason!~"Unsupported.*"}[5m])) > 0

# 준비되지 않은 엔드포인트
cilium_endpoint_state{endpoint_state!="ready"} > 0

# 쿠버네티스 API 오류
sum by (return_code) (rate(cilium_k8s_client_api_calls_total{return_code=~"5.."}[5m])) > 0

8. Potential Interview Questions

• What metrics are used to check the state of service load balancing when kube-proxy is replaced? cilium_bpf_map_pressure Map fill of lb4_* , and drop reason Service backend not found .
• Where do you look for traffic blocked by policies? Agent's cilium_drop_count_total{reason="Policy denied"} and Hubble's hubble_drop_total .
• What if Prometheus slows down after turning on Hubble metrics? labelsContext Suspect
• first. Labels with many values, such as IPs, cause time series to explode. How can you check at a glance whether the Cilium state is healthy?

Controller failure count, unreachable node count, endpoint status, and map pressure.

• "If there are drops, it's an attack": Most drops in this experimental environment were unsupported protocols, not policy rejections.
• "Fewer metric series mean fewer time series": Hubble produced about 80,000 time series across 14 series. The variety of label values determines this.
• "Just writing dns in Hubble outputs DNS metrics": They did not appear in this experimental environment.

Unverified Items

• Changes over time. All values are a snapshot at a single point in time, and rates or trends were not calculated.
• Agents on nodes other than the measured node, Envoy, and values from any other operator pods, if present.
• Unsupported L3 protocol Actual protocols of drops.
• How much time series decrease when Hubble source_ip is removed.
• Why DNS metrics do not appear.
• The Cilium documentation referenced for supplemental explanations of metric meanings was not reopened and cross-checked this time.

References

• https://docs.cilium.io/en/stable/observability/metrics/
• https://docs.cilium.io/en/stable/observability/hubble/

Verified Version and Date

As of 2026-10-05. Cilium 1.20.1, uptime approx. 11.8 days (based on agent start time).