gRPC Debugging and Observability: grpcurl, Reflection, channelz, and Metric Pitfalls

Status: Draft (2026-10-17), partially measured. Reference versions: grpc/grpc-go v1.83.2, grpc/grpc v1.84.0 documentation, kubernetes v1.34.10, go-grpc-prometheus v1.2.0. Metrics were obtained solely by reading Prometheus around 03:00 KST on 2026-10-17. grpcurl and channelz were not executed(unmeasured). What is required is the grpcurl binary (or downloading the module to build with Go) and a test server with reflection and channelz enabled. Therefore, the commands in sections 3 and 4 are usages verified through documentation and code, and their outputs are not included.

Table of Contents

• 1. Intuition: Four Ways to Read Binary Letters
• 2. Reflection and grpcurl
• 3. channelz: Peering Inside Channels
• 4. Metrics: Two Families
• 5. Measurements: Two Pitfalls
  • 5.1 Received Message Metric in kube-apiserver 1.34 Counts Failures
  • 5.2 Argo CD Server: 13 out of 1,938 Time Series
• 6. Limitations
• 7. Contribution Candidates: Where Documentation and Code Diverge
• 8. Connection to Small Clusters
• References

1. Intuition: Four Ways to Read Binary Letters

JSON APIs can be read by humans in curl a single line. Because gRPC has no field names in its bytes (Part 2), you need a contract to read it. Debugging tools ultimately solve the problems of where to get the contractand where to view the connection state.

2. Reflection and grpcurl

Reflection is an optional feature where the server exposes its own proto definitions so thatserver-reflection.md). A representative use case is command-line debugging tools, and grpcurl is such a tool.

In grpc-go, it is enabled with a single line. reflection.Register registers both v1 and the legacy v1alpha. This is because many clients still only understand v1alpha, as explained by a comment in RegisterV1 of the same file.

What you do with grpcurl is roughly as follows (not executed, unmeasured).

grpcurl -plaintext 서버:포트 list                                   # 서비스 목록(리플렉션 필요)
grpcurl -plaintext 서버:포트 describe tt.v1.Scoreboard              # 메서드와 메시지 정의
grpcurl -plaintext -d '{"rallyId":150}' 서버:포트 tt.v1.Scoreboard/Hit   # JSON 으로 요청, JSON 으로 응답
grpcurl -plaintext -import-path . -proto rally.proto 서버:포트 list   # 리플렉션 없이 .proto 를 직접 줌

• When you write a request in JSON, the tool looks at the contract, converts it into the binary of Part 2, and sends it. Responses are converted back and displayed.
• Reflection exposes the service structure as-is. It is usually disabled on ports open to the outside and enabled only on internal or management ports (similar to how hubble-relay in Part 5 Section 6 separates health on 4222 from the main service on 4245).
• Health checks work without reflection. Since the health proto is not embedded in grpcurl, you can pass health.proto from the grpc repository as -proto (document usage, unmeasured).

3. channelz: Peering Inside Channels

channelz (gRFC A14) presents the layers of Part 7 directly as statistics. From the top-down tree of channel to subchannel to socket, it provides the status of each node (such as READY), number of started, successful, and failed calls, last call time, and the number of streams, messages, and keepalives for sockets.

In grpc-go, you enable it directly with channelz/service.RegisterChannelzServiceToServer , or enable channelz and CSDS (if using xDS) together with admin.Register . Questions like "247 etcd connections in apiserver" from Part 6 are normally answered using this window. kube-apiserver does not expose channelz, so this time we used ss instead (channelz registration could not be found in kube-apiserver source, unconfirmed).

Logs can also serve as a window. grpc-go enables internal logs via environment variables GRPC_GO_LOG_SEVERITY_LEVEL(info, warning, error) and GRPC_GO_LOG_VERBOSITY_LEVEL(number) (grpclog/loggerv2.go:65). Resolver updates, picker replacements, and connection failures are logged here.

4. Metrics: Two Families

Three useful combinations:

• started_total − handled_total is the number of currently open streams(242 Watches in Part 4, Section 6.2).
• handled_total Dividing grpc_code by
• Latency histograms are usually disabled by default. In the small cluster, only istiod produced grpc_server_handling_seconds , so the latency tables in this series were replaced with apiserver's etcd_request_duration_seconds, and kubelet/Tempo's own histograms.

5. Measurements: Pitfall Two

5.1 kube-apiserver 1.34 Received Message Metric Counts "Failures"

While looking at apiserver's etcd client metrics, I encountered a strange number. It stated that Range was called over 160 million times since startup, but the number of received messages was 132.

sum by (grpc_service, grpc_method, grpc_type) (grpc_client_msg_received_total{job="apiserver"})
sum by (grpc_method, grpc_code) (grpc_client_handled_total{job="apiserver", grpc_type="unary"})

The number of received messages is exactly equal to the error count. apiserver 1.34 attaches github.com/grpc-ecosystem/go-grpc-prometheus to its etcd client (etcd3.go:316), and the version in go.mod is v1.2.0. The unary client interceptor in that version looks like this:

func (m *ClientMetrics) UnaryClientInterceptor() func(ctx context.Context, method string, req, reply interface{}, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
	return func(ctx context.Context, method string, req, reply interface{}, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
		monitor := newClientReporter(m, Unary, method)
		monitor.SentMessage()
		err := invoker(ctx, method, req, reply, cc, opts...)
		if err != nil {
			monitor.ReceivedMessage()
		}
		st, _ := status.FromError(err)
		monitor.Handled(st.Code())
		return err
	}
}

Source: grpc-ecosystem/go-grpc-prometheus v1.2.0 client_metrics.go lines 108–120, Apache-2.0

err != nil It counts as "received" when . The condition is inverted. While master in the same repository has been fixed to err == nil , there have been no releases since, and the repository is archived. Streams (Watches) are counted via a different path (wrapped inRecvMsg ), so they are normal. On the etcd server side of the same cluster, grpc_server_msg_received_total was normal because it uses a different library (Compact 10,268 = Startup 10,268).

• Kubernetes replaced this dependency with go-grpc-middleware/providers/prometheus (PR #136227, merged 2026-01-20), included starting from v1.36.0 (the same file in v1.35.0 still uses the old library). The unary interceptor in the new library counts response reception once regardless of errors (interceptors/client.go 's UnaryClientInterceptor, providers/prometheus/reporter.go 's PostMsgReceive).
• Therefore, during the 1.35 → 1.36 upgrade, the value of this metric jumps by several orders of magnitude(from hundreds to hundreds of millions). Dashboards or alerts built on this metric might be startled. The PR body only mentions the dependency update and does not mention the change in metric semantics.

5.2 Argo CD Server: 13 out of 1,938 Time Series

count(grpc_server_handled_total{job="argocd-server-metrics"})          → 1,938
count(grpc_server_handled_total{job="argocd-server-metrics"} > 0)      → 13
count(count by (grpc_service,grpc_method) (grpc_server_started_total{job="argocd-server-metrics"}))  → 114

114 methods × 17 status codes = 1,938. Interceptor-based libraries have a feature that pre-creates all method and code combinations registered on the server as 0 (meant to make "methods never called yet" appear in ratio calculations), and it appears the Argo CD server enabled this (estimated, did not open Argo CD source). As a result, this single job's grpc_* time series reached 2,280, accounting for 95% of the small cluster's total grpc_* time series of 2,402 . Only 13 have values. If your time series budget is tight, metric_relabel_configs dropping zero-value combinations with is worth considering.

6. Limitations

• grpcurl, channelz, and reflection were not executed (unmeasured). Required downloads: grpcurl release binary, or a Go module to build the grpc-go example server.
• The judgment in Section 5.1 is based on a comparison of metric numbers and library sources. I did not trace the interceptor chain to the end to verify whether apiserver calls ReceivedMessage through other paths. However, 132 = 132 and 39 = 39 match precisely, so I assume there are no other paths (estimated).
• I did not verify which Argo CD configuration enables the pre-creation feature in Section 5.2 (unverified).

7. Contribution Candidates: Discrepancies Between Documentation and Code

Found while writing this series. Not posted anywhere yet, and must be re-verified against the latest main before posting.

• Option 1 is the smallest and most certain. In the proto block of the document, List add these two messages, and summarize the server behavior paragraph (full list, or RESOURCE_EXHAUSTED if there are many services) to match A90.
• grpc-go and the grpc organization have a generative AI policy. Issues, PRs, and comments must be written by humans, and submitters must fully understand and review any AI-generated content (Section 7, Paragraph 7). Kubernetes also has separate rules, such as requiring notices in PR bodies.

8. Connection with small clusters

• If it is a custom gRPC server, place health and reflection on the management port, and attach channelz to the same management port using admin.Register so that investigations like those in Part 6 can be done ss without.
• If you are using grpc_client_msg_received_total on your dashboard, check Section 5.1 before upgrading to Kubernetes 1.36.
• When looking at time-series budgets, start with count by (job) ({__name__=~"grpc_.*"}) . In small clusters, a single job accounted for 95%.

A word from TT. The record of "132 catches" was actually "132 dropped balls." Scoreboards are sometimes written backward, so if the numbers look strange, check the calculation formula first!

References

All materials were accessed on 2026-10-17.

• grpc/grpc v1.84.0: doc/server-reflection.md, doc/health-checking.md, doc/statuscodes.md; grpc-proto grpc/health/v1/health.proto
• gRFC: A14 channelz, A66 OpenTelemetry Metrics, A90 Health Service List Method
• grpc/grpc-go v1.83.2: reflection/serverreflection.go, channelz/service/service.go, admin/admin.go, grpclog/loggerv2.go, stats/opentelemetry/client_metrics.go, health/server.go, internal/transport/http2_server.go; PR grpc-go#8155
• go-grpc-prometheus: v1.2.0 client_metrics.go, master client_metrics.go
• kubernetes: v1.34.10 etcd3.go, v1.36.0 same file, PR #136227, issue #111476
• Cited code: go-grpc-prometheus v1.2.0, Apache License 2.0.