Streaming, deadline, retry, and keepalive: When calls run long or fail

Status: Draft (2026-10-17), partially measured. Baseline versions: grpc/grpc-go v1.83.2, gRFC A6 Client Retries, grpc/grpc v1.84.0 doc/keepalive.md. Measurements were obtained by reading Prometheus andkubectl logsnode's etcd /metrics during the 03:00 KST hour on 2026-10-17. Experiments involving actual expansion of flow control windows, triggered retries, or keepalives were not performed (unmeasured; requirements include downloading Go modules to build the grpc-go example server and client).

Table of Contents

• 1. Intuition: Phone Calls and Radio
• 2. Diagram: Four Approaches
• 3. Flow Control: The Receiver Sets the Pace
• 4. Propagation of Deadlines and Cancellations
  • 4.1 Measurement: kubelet CRI Deadlines and Actual Latency
  • 4.2 Measurement: OTel Collector 5-Second Timeout
• 5. Retries and Hedging
• 6. Keepalive: Detecting Dead Connections Quickly
  • 6.1 Measurement: How apiserver and etcd Aligned
  • 6.2 Measurement: How Long Bidirectional Streams Live
• 7. Limitations
• 8. Connection to Small Clusters
• References

1. Intuition: Phone Calls and Radio

• Unaryis a single text message and a single reply.
• Server streamingis a radio broadcast. Once you tune in to a frequency (request), the broadcast streams continuously. Hubble flow observation (GetFlows --follow) works this way.
• Client streamingis sending multiple photos in succession and receiving a single "Received" confirmation at the end.
• Bidirectional streamingis a phone call. Both parties speak and listen independently. etcd's Watch and LeaseKeepAlive operate this way.

All four methods use the exact same HTTP/2 stream on the wire; the only differences are how many messages are exchangedand who signals the end first.

2. Diagram: Four Approaches

(etcd declarations in rpc.proto, Hubble in observer.proto·peer.proto)

Streams are not free. Official performance guidelines recommend using them only when benefits are clear, noting that open streams cannot be migrated for load balancing and are difficult to debug (Performance Best Practices). This is why the concentration issues in Part 5 are more severe with streams.

3. Flow Control: The Receiver Sets the Pace

What happens if the server sends 10,000 messages per second in a server stream, but the client processes only 1,000? HTTP/2 prevents overload through the receiver opening a window indicating "you may send this much more" . There is one window per stream and one for the entire connection, with a default size of 65,535 bytes (RFC 9113). When the receiver consumes data, it reopens the window using WINDOW_UPDATE. If the window is 0, the sender's Send is blocked and waits. This is the backpressure mechanism in gRPC streams.

Two key features of grpc-go:

• It automatically grows the window size. It measures round-trip time and bandwidth via pings to estimate the bandwidth-delay product (BDP), expanding the window up to 16MiB (bdp_estimator.go:30 of bdpLimit). This prevents large, long-distance streams from being bottlenecked by the 64KiB window.
• WithInitialWindowSize·WithInitialConnWindowSize Explicitly setting the window with disables this automatic adjustment(the option also enables StaticWindowSize = true , dialoptions.go:211). This is a point where manual tuning can inadvertently degrade performance.

4. Propagation of Deadlines and Cancellations

1. When a client invokes a call with a 0.5-second deadline, a grpc-timeout: 500m is sent (Part 3, Section 3.1).
2. The server sets a timeout on its own context using that value. If it consumes 120ms and invokes the next service with the same context , grpc-go recalculates and sends the remaining 380ms as the new grpc-timeout . In effect, the timeout ticks down as it traverses the call chain.
3. When the deadline expires, the client receives DEADLINE_EXCEEDED and sends a RST_STREAM(CANCEL) to the stream. Because the server-side context is cancelled, downstream calls invoked by that server using the inherited context are also stopped in a cascading fashion.

The trap lies in "with the same context" mentioned in step 2. In Go, if a server handler makes a downstream call using ctx instead of context.Background() , propagation is broken, causing downstream work to continue even after the upper layers have abandoned it. Since the specification dictates that requests without deadlines should run indefinitely (PROTOCOL-HTTP2.md Requests section), relying on defaults is dangerous.

4.1 Measurement: kubelet CRI Deadlines and Actual Latency

Kubelet's CRI client sets a timeout for every call. Defaults are runtimeRequestTimeout 2 minutes, RunPodSandbox twice that (4 minutes), and StopContainer is 2 minutes + termination grace period (remote_runtime.go:204, line 379). Conversely, PullImage has no deadline and only uses cancellation (remote_image.go of PullImage, context.WithCancel). This is because image pulls may legitimately take several minutes depending on size.

Calculated as a 6-hour histogram sum across 8 nodes.

histogram_quantile(0.99, sum by (operation_type, le) (rate(kubelet_runtime_operations_duration_seconds_bucket[6h])))

stop_container 's p50 of 24 seconds is not due to a slow runtime, but rather because it waits for the app to finish within the grace period (usually 30s) after SIGTERM (estimated: pods with slow shutdowns are mixed in). Since the bucket boundaries are coarse at 0.005, 0.0125 ..., "Under 5ms" for list operations means it is within the first bucket. Whether there were CRI deadline exceeded errors in the last 6 hours was kubelet_runtime_operations_errors_total unconfirmed as the error type label was missing, making them indistinguishable.

4.2 Measurement: OTel Collector's 5-second deadline

Collector exports enforce a default 5-second deadline per request (opentelemetry-collector v0.159.0 timeout_sender.go:31). We counted export failures to Tempo by status code in the Collector logs (all logs since startup on 2026-09-29, reading kubectl logs ).

• Restarts were left as UNAVAILABLE(connection refused), and slowness as DEADLINE_EXCEEDED . This matches "When created by libraries" from the table in Part 3.
• Failed requests were retried by the Collector itself (retry_on_failure, up to 5 minutes), and 179 batches that ultimately failed to send (169 on 10-05, 10 on 10-13) were dropped. In terms of metrics, this accounts for 11,624 failed spans, which is 0.012% of the 93.89 million spans received during the same period.
• This retry is an app-level retry, not a gRPC library retry (Section 5). The OTLP specification defines separately which codes to retry (Part 6).

5. Retry and Hedging

gRPC allows libraries to retry per method via service config policies retryPolicy . For example, it looks like this (A6 format).

{"methodConfig": [{"name": [{"service": "tt.v1.Scoreboard"}],
  "retryPolicy": {"maxAttempts": 4, "initialBackoff": "0.1s", "maxBackoff": "1s",
                  "backoffMultiplier": 2, "retryableStatusCodes": ["UNAVAILABLE"]}}]}

The function that actually configures retries in grpc-go is csAttempt.shouldRetry . The rules are clear once you read it.

• Calls that have already been committed are not retried. If the server has sent response headers (unless the response is Trailers-Only), it means the server application has started something, so it stops.
• The status code must be in retryableStatusCodes . Which codes are safe depends on the application, so there is no default list (see the final paragraph of statuscodes.md).
• maxAttempts is the count including the original call, and if it exceeds 5, it is capped at 5 (dialoptions.go:43 defaultMaxCallAttempts).
• The backoff delay is calculated by scaling initialBackoff × multiplier^n by maxBackoff and multiplying by a random factor of 0.8 to 1.2 . If the server provides grpc-retry-pushback-ms in the trailers, that value is used as-is; if it is negative, no retries are performed.
• Retry throttling. Each server name has a token bucket, subtracting 1 for each failure and adding tokenRatio for each success. Retries are stopped if the bucket drops to half or below (clientconn.go:1733 retryThrottler). This prevents retries from multiplying the load when the server is unhealthy.
• To resend a message, the sent message must be retained. If it exceeds 256KiB per call, it is committed at that moment and can no longer be retried (rpc_util.go:205, stream.go:940).
• Transparent retries. If it is certain that the request did not reach the server application (pre-send failure, REFUSED_STREAM, or a stream with a number greater than the last stream number in GOAWAY), it is resent regardless of the policy (A6).

if a.transportStream != nil {
	if !a.transportStream.TrailersOnly() {
		return false, err
	}
	sps := a.transportStream.Trailer()["grpc-retry-pushback-ms"]
	...
}
rp := cs.methodConfig.RetryPolicy
if rp == nil || !rp.RetryableStatusCodes[code] {
	return false, err
}
if cs.retryThrottler.throttle() {
	return false, err
}
if cs.numRetries+1 >= rp.MaxAttempts {
	return false, fmt.Errorf("max retries exhausted: failed after %d attempts: %w", cs.numRetries+1, err)
}

Source: grpc/grpc-go v1.83.2 stream.go excerpt from lines 727–768, Apache-2.0

Hedging is a method of proactively sending the same request to multiple servers at regular intervals (hedgingDelay) without waiting for a response, using the fastest answer (A6). It reduces tail latency but should only be used for idempotent methods. In grpc-go 1.83.2, there are only TODO(hedging) comments (stream.go:634) and no code that reads hedgingPolicy from the service config. Essentially, it is not yet usable in Go (other language implementations have not been verified).

6. keepalive: Detecting dead connections quickly

TCP connections appear to stay alive for a while even if the peer disappears (e.g., power cutoff, eviction from NAT table). gRPC sends an HTTP/2 PING and drops the connection if there is no response.

The most common incident is a mismatch in configurations between both sides. If the client sends a ping every 30 seconds while the server insists on its default (5 minutes), the server considers the pings "too frequent" and GOAWAY(ENHANCE_YOUR_CALM, "too_many_pings") closes the connection with (handlePing). The client sees this as RESOURCE_EXHAUSTED or UNAVAILABLE, and grpc-go clients subsequently double their ping intervals (keepalive.go comment).

6.1 Measurement: How apiserver and etcd are aligned

• The etcd client in kube-apiserver is configured with ping every 30 seconds, disconnect if no response within 10 seconds(etcd3.go:63). Comments note that this is an aggressive value intended to quickly detect when etcd goes down.
• If it were the grpc-go server default (at least 5 minutes), this ping would be a violation. etcd lowers its server-side minimum interval to 5 seconds(etcd v3.6.5 server/embed/config.go:69 DefaultGRPCKeepAliveMinTime, server ping interval 2 hours, response timeout 20 seconds).
• Thus, the two components mesh with a 30-second ping > 5-second lower bound. When enabling client keepalive on a custom gRPC server, the server's MinTime must also be lowered.

6.2 Measurement: How long do bidirectional streams live?

sum by (instance) (grpc_client_started_total{job="apiserver",grpc_method="Watch"})
  - sum by (instance) (grpc_client_handled_total{job="apiserver",grpc_method="Watch"})

There were 242, 239, and 239started yet unfinished Watch streams across three apiserver instances, respectively. At the same time, etcd on control node A etcd_debugging_mvcc_watch_stream_total 242, etcd_debugging_mvcc_watcher_total had 242, matching the client-side count exactly. Over the past hour, only 0.002 Watches ended per second (about 8 per hour), and the total Watch messages received across the three instances were about 181 per second. It appears that a single stream lives for days while flowing change events. Why there are precisely around 240 streams will be examined alongside the number of TCP connections in Part 6.

7. Limitations

• We did not reproduce the flow control window expansion process via BDP estimation, the GOAWAY generation due to keepalive violations, or the exact retry count of retry policies (unmeasured).
• The causes of the 10-05 and 10-09 delays in Section 4.2 (such as Tempo disk or memory issues) were not verified (unverified).
• The relationship between the etcd client's internal retry interceptor and gRPC service config retries was not covered.

8. Connecting to Small Clusters

• If a custom gRPC service calls another service, first check whether it passes the incoming ctx as-is. That alone brings in the cascaded cancellation from Section 4 for free.
• If client keepalives are enabled, keep the server's EnforcementPolicy.MinTime shorter than that. etcd does this.
• If there is application-level retries like in the Collector, enabling gRPC service config retries as well will multiply the attempt counts. Keep only one of them.

Titi's note. When rallies get long, the referee yells "380ms remaining!" and passes it to the next referee. If you reset the clock (Background), you end up swinging alone even after the match is over!

References

All materials were accessed on 2026-10-17.

• gRFC A6 Client Retries, A8 Client-side Keepalive
• grpc/grpc v1.84.0: doc/keepalive.md, doc/PROTOCOL-HTTP2.md, doc/statuscodes.md
• grpc/grpc-go v1.83.2: stream.go, clientconn.go, dialoptions.go, rpc_util.go, keepalive/keepalive.go, internal/transport/defaults.go, internal/transport/bdp_estimator.go
• RFC 9113 HTTP/2: https://www.rfc-editor.org/rfc/rfc9113 (Flow control, initial window 65,535)
• kubernetes v1.34.10: staging/src/k8s.io/cri-client/pkg/remote_runtime.go, remote_image.go, storagebackend/factory/etcd3.go
• etcd v3.6.5: server/embed/config.go, api/etcdserverpb/rpc.proto
• opentelemetry-collector v0.159.0: exporter/exporterhelper/internal/timeout_sender.go
• Cited code: grpc-go v1.83.2, Apache License 2.0.