gRPC in Kubernetes: Why It Floods a Single Pod and Health Checks

Status: Draft (2026-10-17), partially measured. Based on grpc/grpc-go v1.83.2, kubernetes v1.34.10, cilium v1.20.1, and grpc/grpc v1.84.0 documentation. Measurements were obtained during the 03:00 hour (KST) on 2026-10-17 from Prometheus andkubectl getinside the Cilium agent cilium-dbg status by reading only. Load skew was not directly reproduced(unmeasured). To reproduce it, a test gRPC server image and deployment would need to be deployed to the cluster, but this task decided to read-only. Requirements are noted in Section 8.

Table of Contents

• 1. Intuition: Taxi Stand
• 2. Diagram
• 3. Why Sizing Skews
• 4. Three Solutions
  • 4.1 Headless Service + Client-side round_robin
  • 4.2 Periodic Rotation via Server MaxConnectionAge
  • 4.3 L7 Proxy: Envoy, Mesh, Cilium
  • 4.4 xDS: Proxy-free Mesh
  • 4.5 Selection Criteria
• 5. Health Checks: grpc.health.v1
• 6. Kubelet gRPC Probes
  • 6.1 Measurement: Who Uses gRPC Probes
• 7. Limitations
• 8. Connection to Small Clusters
• References

1. Intuition: Taxi Stand

A Service (ClusterIP) is a dispatcher at a taxi stand. When a customer (connection) arrives, it assigns an empty taxi (pod). HTTP/1.1 clients make short trips frequently and get off, giving the dispatcher plenty of opportunities to evenly distribute them. A gRPC client charters a single taxi all day long charters a single taxi all day long . Even if thousands of trips (calls) are made inside that taxi, the dispatcher was only involved the very first time. Even if the number of taxis is increased to ten, the customer who already chartered one continues to ride the first taxi.

2. Diagram

3. Why Sizing Skews

• Whether it is kube-proxy's iptables/IPVS or Cilium's kube-proxy replacement, ClusterIP chooses backends on a connection basis. Cilium's socket-level load balancing replaces the service IP with the backend address when the app calls connect() and binds the socket directly to that backend (kube-proxy free documentation). The agent in the small cluster was also KubeProxyReplacement: True (cilium-dbg status, read). The architecture is in Cilium Read.
• A gRPC channel maintains a single long-lived connection to a target and multiplexes calls over HTTP/2 streams (Part 1). The gRPC load balancing documentation states that load balancing was designed from the start to be done on a per-call rather than per-connection basis(load-balancing.md). However, that distribution must be handled by the LB policy inside the client library, and since ClusterIP provides only a single address, the library has no targets to split across.
• The default LB policy pick_first attaches to the first one in the address list and sends all calls there (same document). Even if there are multiple addresses, the default does not split them.

This issue was covered by the Kubernetes blog in 2018 in "gRPC Load Balancing on Kubernetes without Tears" (by Buoyant, kubernetes.io/blog).

4. Three Solutions

4.1 Headless Service + Client-side round_robin

clusterIP: None A headless service returns pod IPs directly via DNS. When a client creates a channel with dns:///scoreboard.tt.svc.cluster.local:50051 and selects round_robin in the service config, the channel creates a subchannel for each pod and rotates them per call .

{"loadBalancingConfig": [{"round_robin": {}}]}

In grpc-go, it can be passed via grpc.WithDefaultServiceConfig(...) or received via a DNS TXT record (Kubernetes DNS does not provide TXT, so the former method is usual).

There is a catch: when is DNS re-queried. The grpc-go DNS resolver does not periodically re-query once successful, but waits until someone calls ResolveNow (though at least 30 seconds must have passed since the last query).

var nextResolutionTime time.Time
if err == nil {
	// Success resolving, wait for the next ResolveNow. However, also wait 30
	// seconds at the very least to prevent constantly re-resolving.
	backoffIndex = 1
	nextResolutionTime = internal.TimeNowFunc().Add(MinResolutionInterval)
	select {
	case <-d.ctx.Done():
		return
	case <-d.rn:
	}
}

Source: grpc/grpc-go v1.83.2 internal/resolver/dns/dns_resolver.go excerpt from lines 218-229 (watcher), Apache-2.0

ResolveNow is mainly called when a connection drops or fails (clientconn.go:1366·1515). Thus, when pods decrease (because connections drop), it notices quickly, but when they increase, no one knows. This is because if existing pods are healthy, connections have no reason to drop.

4.2 Periodic Rotation via Server MaxConnectionAge

The solution is for the server to give connections a lifespan. If you configure keepalive.ServerParameters.MaxConnectionAge on the grpc-go server, it sends GOAWAY to connections that exceed that time (±10% jitter) and closes them after MaxConnectionAgeGrace (comment inkeepalive.go ). The defaults for both are infinity. Clients receiving GOAWAY finish ongoing calls and send new calls over a new connection, re-resolving DNS in the process of dropping the old connection. Even behind ClusterIP, dispatching happens again when opening new connections, gradually resolving the skew. The reason for jitter is to prevent all clients from reconnecting at the exact same moment.

4.3 L7 Proxy: Envoy, Mesh, Cilium

Proxies that understand HTTP/2 can load balance on a per-stream (call) basis backends. Sidecar meshes (Istio and Linkerd) or gateways handle this, so there is no need to change client code. The costs are an extra hop and the proxy's CPU and memory.

Cilium can handle gRPC at L7 using the node's Envoy. Taking advantage of the fact that gRPC methods are POST /패키지.서비스/메서드 , it makes allowlists by path, and denied calls return a PERMISSION_DENIED status instead of dropping packets (Cilium gRPC policy guide).

rules:
  http:
  - method: "POST"
    path: "/tt.v1.Scoreboard/Hit"

The agent in the small cluster was Proxy Status: OK … 0 redirects active (cilium-dbg status). This means that because no L7 policies were applied anywhere, the path inspecting gRPC at L7 was not used.

4.4 xDS: Proxyless Mesh

This is an approach where gRPC libraries directly speak xDS (Envoy's configuration API) (gRFC A27). Clients create channels with xds:///서비스:포트 , receive listeners, routes, clusters, and endpoints from a control plane (e.g., istiod), and perform load balancing per call. Istio provides this as "proxyless gRPC," where an agent writes bootstrap files and relays xDS on its behalf (Istio Blog 2021).

Note that gRPC implements only part of xDS. Unsupported configurations are ignored without logs, and if an unsupported value is provided for a supported field, the entire configuration is rejected (NACK) (grpc_xds_features.md). Settings that worked in an Envoy sidecar may silently have no effect in proxyless gRPC.

The istiod in the small cluster was connected to three Envoy proxies via xDS (pilot_xds 3), and after startup, it sent 44,798 EDS pushes, 21,150 RDS pushes, and 21,102 each for CDS and LDS (sum by (type) (pilot_xds_pushes)). Endpoints (EDS) change most frequently because pods come and go. There were no proxyless gRPC clients.

4.5 Selection Criteria

5. Health Checking: grpc.health.v1

gRPC has a standard health service (health-checking.md).

• Check(service) returns SERVING or NOT_SERVING. If the service name is an empty string, it represents the overall server status, and unknown names return NOT_FOUND. grpc-go's health.Server.Check behaves exactly like this, and newly created instances have an empty name set to SERVING.
• Watch(service) uses server streaming to notify whenever the status changes.
• List() provides the status of all registered services at once (gRFC A90). It is already present in grpc-go 1.83.2 and the proto files, but not yet in the grpc/grpc documentation (candidate for a contribution in Part 8).
• Enabling client-side health checking (gRFC A17) causes the channel to open a Watch for each subchannel, preventing calls from being sent to backends that are NOT_SERVING (addrConn.startHealthCheck). When used with round_robin, this helps avoid pods that are "alive but should not receive traffic."

6. gRPC Probes in kubelet

Kubernetes has supported stable grpc probes since version 1.27. By providing a port and an optional service name, kubelet calls grpc.health.v1.Health/Check . The constraints noted in the documentation are (Probe Configuration Documentation): Named ports cannot be used, plaintext connections are the default, and TLS has been added as a GRPCContainerProbeTLS feature gate in newer documentation (not present in 1.34, and certificates are not verified).

The implementation in kubelet v1.34.10 is concise.

opts := []grpc.DialOption{
	grpc.WithUserAgent(fmt.Sprintf("kube-probe/%s.%s", v.Major, v.Minor)),
	grpc.WithBlock(),
	grpc.WithTransportCredentials(insecure.NewCredentials()), //credentials are currently not supported
	...
}
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
addr := net.JoinHostPort(host, fmt.Sprintf("%d", port))
conn, err := grpc.DialContext(ctx, addr, opts...)
...
resp, err := client.Check(metadata.NewOutgoingContext(ctx, make(metadata.MD)), &grpchealth.HealthCheckRequest{
	Service: service,
})

Source: Kubernetes v1.34.10 pkg/probe/grpc/grpc.go excerpt from lines 56–90 (Probe), Apache-2.0

• It creates a single channel per probeand closes it afterwards. This contradicts the principle of "reusing channels" from Section 6 of Part 1, but it is appropriate here because probes check whether a pod can accept new connections right now. Instead, a TCP and HTTP/2 handshake occurs on every probe interval.
• timeoutSeconds handles both connection and calls. If the server has not implemented the health service, it receives UNIMPLEMENTED and is marked as a failure.
• Kubernetes main has changed the same function to grpc.NewClient("passthrough:///"+addr, …) . Because NewClient 's default resolver is dns , the old DialContext to write the address as is like passthrough , you need to prepend it (Part 1, Section 6).

6.1 Measurement: Who Uses gRPC Probes

kubectl get pods -o json counted the probe types in the cluster (excluding one user business namespace, 271 pods).

All 3 gRPC probes belong to the startup, liveness, and readiness probes of hubble-relay (port 4222, empty service name, period 10s for liveness/readiness, 3s for startup). The relay logs also show that upon startup, it opens the "gRPC health server" on port 4222. The resulting metrics were as follows.

sum by (probe_type, result) (prober_probe_total{namespace="kube-system", container="hubble-relay"})

• The startup probe succeeded after failing 8 times. The relay experienced a connection refused error to the hubble-peer service providing the node list right after startup and retried 30 seconds later (in the logs, connection refused, Part 6), and it appears that health was not SERVING in the meantime (estimated).
• The 6-hour latency was within the first bucket (5ms) for both p50 and p99. This means the cost of opening a new connection for each probe within the same node is around this level.
• Since both probes open and close connections every 10 seconds, this single pod generates about 17,280 short TCP connections per day (calculated).

7. Limitations

• Traffic skew reproduction (unmeasured). What is needed: a test gRPC server image (e.g., built from grpc-go's examples/helloworld ), a deployment with 3 replicas, one ClusterIP and one headless service each, and a load generator (such as ghz). You can compare grpc_server_handled_total per pod as a metric.
• No place in the small cluster ran a gRPC server with multiple replicas (even the Argo CD repo-server had only 1). Therefore, actual skew cases could not be measured.
• Linkerd and Envoy's gRPC retry and timeout policies, as well as Gateway API's GRPCRoute, were not covered.

8. Connection to Small Clusters

• If you deploy a custom gRPC server with multiple replicas, adding MaxConnectionAge (e.g., a few minutes) to the server first is the cheapest starting point. If you can modify the client, add headless + round_robin.
• Registering the health service and using kubelet's grpc probe achieves the same thing without exec probes (grpc_health_probe binary). Exec probes create CRI ExecSync calls, putting a load on containerd (about 11,000 exec_sync calls per hour in Part 4, Section 4.1).

Titi's Note. Renting a taxi for the whole day does not help no matter how smart the dispatcher is. You occasionally need to get out (MaxConnectionAge) and queue up again to ride a new taxi!

References

All materials were accessed on 2026-10-17.

• grpc/grpc v1.84.0: doc/load-balancing.md, doc/health-checking.md, doc/grpc_xds_features.md, doc/naming.md
• gRFC: A17 Client-side Health Checking, A27 xDS-Based Global Load Balancing, A90 Health Service List Method
• grpc/grpc-go v1.83.2: internal/resolver/dns/dns_resolver.go, clientconn.go, health/server.go, keepalive/keepalive.go
• kubernetes v1.34.10 pkg/probe/grpc/grpc.go, from the same file main
• Kubernetes documentation Configure Liveness, Readiness and Startup Probes, blog gRPC Load Balancing on Kubernetes without Tears
• Cilium: Kubernetes Without kube-proxy, Securing gRPC
• Istio blog gRPC Proxyless Service Mesh
• Cited code: grpc-go v1.83.2 and kubernetes v1.34.10, Apache License 2.0.