gRPC in Action on a Small Cluster: etcd, CRI, Hubble, OTLP

Status: Draft (2026-10-17), measured. Measurements were taken via read-only access from 2026-10-17 03:00 to 03:05 (KST) using Prometheus instant queries (GET via the API server's service proxy), kubectl get·logs, as a regular user on control node A, and inside the Cilium agent ss·ps·curl -s http://127.0.0.1:2381/metrics, inside the Cilium agent cilium-dbg status·hubble observe --last 3 . Target versions: kubernetes v1.34.10, etcd v3.6.5, containerd 1.7.27 (4 nodes) and 2.3.5 (4 nodes), Cilium/Hubble v1.20.1, OTel Collector 0.159.0. Raw data is in the working directory measure/grpc/ and only masked values have been transcribed into this article.

Table of Contents

• 1. Intuition: Intercoms in a Building
• 2. Diagram
• 3. apiserver → etcd: One Line per Resource
  • 3.1 Number of Connections
  • 3.2 Calls and Bytes
• 4. kubelet → containerd: Polling Lists Every Second
  • 4.1 Connection
  • 4.2 Call Distribution
• 5. Hubble and hubble-relay: Collecting and Re-streaming Streams
• 6. OTLP: Collector → Tempo
  • 6.1 Configuration and Flow
  • 6.2 Measurements
• 7. Summary: Four Connection Topologies
• 8. Limitations
• 9. Next Up
• References

1. Intuition: Intercoms in a Building

A cluster is like a building with many intercoms. The management office (apiserver) talks to the safe (etcd) all day, the floor manager (kubelet) asks the kitchen (containerd) "Who is here now?" every second, the security office (hubble-relay) tunes into the CCTV (Hubble) for every floor at once, and the audit team (Collector) sends collected logs to the archive (Tempo). All of these lines are gRPC, but how many lines are laid and how often they ring are completely different from one another. how many lines are laid and how often they ringare completely different from one another.

2. Diagram

3. apiserver → etcd: One Line per Resource

3.1 Number of Connections

Counting etcd client connections from control node A using ss -Htn state established "( sport = :2379 )" , we found there were 247, and all counterparts were 127.0.0.1. Because of the kubeadm-style setup, apiserver only connects to etcd on the same node (processes on that node checked withps are just kube-apiserver and etcd).

In Part 1, we mentioned "one connection per channel, multiplexing calls over it," so why are there 247 connections? The Kubernetes source code provides the answer. etcd3.go The comment above the logger variable in states, "Currently a separate etcd client is created for each CRD. The correct fix is not to do this," and points to kubernetes#111476 . In reality, newETCD3Storage creates a new client (= new grpc-go channel = new TCP) via newETCD3Client every time it is called.

The numbers show a pattern where each resource gets its own client, and a Watch stream for the watch cache is attached over that client (the remaining 5 out of 247 appear to be clients for health checks and DB size monitoring, though unconfirmed and estimated). Unary Range and Txn also flow over the same connections. While etcd clients can group multiple watches of the same context into a single gRPC stream (theclient/v3/watch.go map in streams ), this grouping does not happen because clients are separated per resource.

The more CRDs a cluster has, the more connections, goroutines, and memory scale in proportion to the number of resources. Although Kubernetes main is currently replacing the metrics collection libraries for these clients (Part 8), the "client per resource" architecture itself remains unchanged in v1.34.10, which we verified here.

3.2 Calls and Bytes

Past 1-hour ratio (rate(grpc_client_handled_total{job="apiserver"}[1h]), sum of three nodes).

etcd request latency measured by apiserver (etcd_request_duration_seconds, 1 hour) was: get p50 2.7ms, p99 22ms; list p50 2.8ms, p99 42ms; update p50 13.6ms, p99 25ms. Writes are about 5 times slower than reads because they go through Raft consensus and disk synchronization.

gRPC bytes self-measured by etcd (etcd_network_client_grpc_sent_bytes_total·received_bytes_total, Node A, 58.4 days since startup) were 883 GiB sent and 50 GiB received, averaging 183 KiB vs 10 KiB per second. etcd is predominantly a "sending" serverbecause responses and watch events are much larger than requests.

4. kubelet → containerd: Polling Lists Every Second

4.1 Connection

kubelet's CRI client connects to the runtime service and image service. create a separate channel (NewRemoteRuntimeService, NewRemoteImageService). Plaintext, :authority is localhost, and the receive limit is 16MiB instead of the default gRPC 4MiB (maxMsgSize, because the container list can grow large). Both go over Unix sockets /run/containerd/containerd.sock .

When checking from control node A to ss -Hx , the number of established sockets on the containerd.sock side was 4. Two of them are the kubelet channels above, and the owners of the other two could not be verified because processes cannot be viewed with regular user privileges (unverified). On the same node, the number of sockets on the containerd.sock.ttrpc side was 22and there were 22 containerd-shim processes as well. It matches the numbers that shim connects to containerd via lightweight ttrpc instead of gRPC (the structure is Reading containerd and CRI).

4.2 Call Distribution

Total for 8 nodes, last 1 hour(sum by (operation_type) (rate(kubelet_runtime_operations_total[1h]))*3600).

• Out of about 110,000 times per hour, 80% are list queries. Even if pods do not change, PLEG asks every second. The goal of reducing this waste is Evented PLEG (CRI server streaming) where the runtime pushes events, but in the v1.34.10 feature gate table, GetContainerEvents is still alpha and disabled by default( EventedPLEG ).kube_features.go:1277).
• Actual work (creating pods and containers) happens dozens of times per hour. Looking together with the latency table in Section 4.1 of Part 4, what is slow are these rare calls, while frequent list queries finish within 5ms.
• Cumulative errors (kubelet_runtime_operations_errors_total) were in the order of exec_sync 1,420, run_podsandbox 140, container_status 139, remove_container 123, list_podsandbox 114. Because the error labels did not contain status codes, it was not possible to distinguish whether they were DEADLINE_EXCEEDED or NOT_FOUND (unconfirmed).

5. Hubble and hubble-relay: Collecting and restreaming streams

The Cilium agent runs a Hubble gRPC server on each node. The configuration (cilium-config, read) is hubble-listen-address: :4244, uses server TLS, and has a local Unix socket /var/run/cilium/hubble.sock . Hubble on node A was recording 58.5 flows per second with its 4,095-slot flow ring buffer full.

hubble-relay aggregates these per-node servers into one. The sequence seen from logs and configuration is as follows.

1. Upon startup, it opens a gRPC health server on port 4222 and the main server on 4245 (server-side TLS disabled, client-side TLS enabled).
2. hubble-peer service (443 → 4244) to open a Peer/Notify server stream. This service is a internalTrafficPolicy: Local , so the agent on the node with the relay responds. The first attempt was connection refused(UNAVAILABLE) and it reconnected 30 seconds later.
3. As soon as it received the PEER_ADDED of the 8 nodes via the stream, it opened an mTLS connection to 4244 on each node (with server names in the format 노드이름.default.hubble-grpc.cilium.io ). All eight connections were Connected within 0.3ms.
4. When a user calls the relay's hubble observe using GetFlows , the relay opens GetFlows streams to each node (retrieveFlowsFromPeer), sorts the received flows chronologically (sortFlows), and returns them as a single stream.

Connecting directly to each node without using load balancing contrasts with Part 5. Because the relay needs "every node" rather than "any node," the service is used only to provide the list of nodes, and actual connections are opened directly to the node addresses. As nodes scale up, the Notify stream informs the relay, eliminating DNS re-resolution issues.

6. OTLP: Collector → Tempo

6.1 Configuration and Flow

The trace pipeline in the Collector configuration (ConfigMap, read) is the otlp receiver (gRPC 4317, HTTP 4318) → memory_limiter → k8s_attributes → resource → batch(512 or 5s) → otlp_grpc/tempo exporter (plaintext, queue 2,000, max retry 5 min). OTLP/gRPC has only a single method service.

service TraceService {
  rpc Export(ExportTraceServiceRequest) returns (ExportTraceServiceResponse) {}
}

Source: open-telemetry/opentelemetry-proto opentelemetry/proto/collector/trace/v1/trace_service.proto(main, accessed 2026-10-17; latest release v1.11.1), Apache-2.0

of the response partial_success to signal "partially received," and if it fails, it determines whether to retry according to the table defined by the OTLP specification. The specification treats CANCELLED, DEADLINE_EXCEEDED, ABORTED, OUT_OF_RANGE, UNAVAILABLE, and DATA_LOSS as retryable, and RESOURCE_EX RetryInfo HETERO_EXHAUSTED only when the server returnsOTLP specification). This is a great example of part 3's "The status code chosen by the app determines the client's action."

6.2 Measured Data

• 50.1 spans per second ÷ 222.3 batch size = 0.225 batches per secondand the internal Push received by Tempo is 0.226 per second . A single batch from the Collector becomes one OTLP Export call, which maps directly to a single Push inside Tempo, aligning the numbers perfectly.
• 69% of the batches did not reach 512 items and were flushed due to the 5-second timeout. Since this is a low-traffic cluster, it trades latency (up to 5 seconds) for batching.
• Inside Tempo as well, /grpc.health.v1.Health/Check comes in at 0.133 times per second (once every 7.5 seconds). This appears to be the client pool inside Tempo performing periodic health checks on ring members (estimated, as Tempo and dskit sources were not opened). Tempo's tempo_grpc_concurrent_streams_limit was 100, and judging by the metric name, grpc_concurrent_streams_by_conn_max , which represents the maximum concurrent streams per connection, was 20 (metric definitions were not verified).
• Failure logs can be found in section 4.2 of Part 4 (restarts are UNAVAILABLE, slowness is DEADLINE_EXCEEDED).

7. Summary: Four Connection Shapes

8. Limitations

• Observations were limited to a single node (control plane node A). ss·ps Connections for containerd 2.3 on workers were not measured (unmeasured).
• The number of TCP connections for Collector → Tempo was not measured because it required running ss inside the pod (unmeasured).
• Hubble's gRPC metrics (hubble-metrics-server :9965) were not scraped by Prometheus, so only flow counts could be observed.
• Etcd metrics are from a single read on node A. The other two members were not read.

9. Next Up

Part 7 explores how grpc-go's channels, subchannels, and transports are implemented in the source code, and Part 8 examines metric pitfalls to watch out for when looking at these numbers.

TT's Word. The management office ran 247 telephone lines into the safe! It's like having a separate phone in every single room. Calls go through great, but the phone bill (memory) scales with the number of rooms!

References

All materials were accessed on 2026-10-17.

• kubernetes v1.34.10: storagebackend/factory/etcd3.go, cri-client/pkg/remote_runtime.go, cri-client/pkg/remote_image.go, cri-client/pkg/utils.go, pkg/kubelet/kubelet.go; Issue #111476
• etcd v3.6.5: client/v3/watch.go, api/etcdserverpb/rpc.proto
• cilium v1.20.1: api/v1/observer/observer.proto, api/v1/peer/peer.proto, pkg/hubble/relay/observer/server.go, pkg/hubble/relay/observer/observer.go
• OpenTelemetry: trace_service.proto, OTLP specification
• Related Posts: Reading containerd and CRI, Reading Cilium
• Quoted Code: opentelemetry-proto, Apache License 2.0.