gRPC over HTTP/2: Headers, 5-Byte Prefix, and Trailer Status Codes

Status: Draft (2026-10-17), partially measured. Reference versions: grpc/grpc v1.84.0 doc/PROTOCOL-HTTP2.md·doc/statuscodes.md, grpc/grpc-go v1.83.2. Status code distributions were obtained by only reading etcd from Prometheus and control node A in a small cluster during the 03:00 hour (KST) on 2026-10-17. /metrics Actual frames were not captured (unmeasured; prerequisites: grpcio or grpc-go example server, capture tools like Wireshark, etc.). Therefore, the bytes in sections 3, 4, and 5 are values constructed and calculated from specifications and code.

Table of Contents

• 1. Intuition: Envelope, Letter, and Result Stamp
• 2. Diagram
• 3. Request Headers
  • 3.1 Choosing the Unit for grpc-timeout
• 4. 5-Byte Message Prefix
• 5. Response and Trailers
• 6. 17 Status Codes
• 7. Measurement: Counting the Same Call from Both Sides
• 8. Limitations
• 9. Connection to a Small Cluster
• References

1. Intuition: Envelope, Letter, and Result Stamp

It is easy to think of registered mail. The envelope(request headers) specifies the recipient department, task name, and processing deadline. Inside are multiple letters(messages), and each sheet has a label attached stating "this sheet is X bytes." The processing result is the result stamp(trailer) applied at the very end. The post office (HTTP) returns 200 indicating successful delivery, but whether the task succeeded must be checked via the stamp.

Thanks to this structure, gRPC can stream responses piece by piece and attach the status at the end. The status line in HTTP.1 must come at the very beginning, making this difficult.

2. Diagram

• A stream number (stream=1) represents a single call. Streams opened by the client in a single connection increase in odd numbers.
• When END_STREAM is attached to the final DATA of a request, it means "nothing more to send." Responses follow the order of HEADERS → DATA → HEADERS (trailer, END_STREAM).
• When an error is returned immediately, it ends with a single HEADERS frame (Trailers-Only). Retries in Part 4 make decisions by observing this difference.

3. Request Headers

Summarizing the Call-Definition from the specification (PROTOCOL-HTTP2.md Requests section).

In grpc-go, http2Client.createHeaderFields builds this list.

headerFields = append(headerFields, hpack.HeaderField{Name: ":method", Value: "POST"})
headerFields = append(headerFields, hpack.HeaderField{Name: ":scheme", Value: t.scheme})
headerFields = append(headerFields, hpack.HeaderField{Name: ":path", Value: callHdr.Method})
headerFields = append(headerFields, hpack.HeaderField{Name: ":authority", Value: callHdr.Host})
headerFields = append(headerFields, hpack.HeaderField{Name: "content-type", Value: grpcutil.ContentType(callHdr.ContentSubtype)})
headerFields = append(headerFields, hpack.HeaderField{Name: "user-agent", Value: t.userAgent})
headerFields = append(headerFields, hpack.HeaderField{Name: "te", Value: "trailers"})
...
if dl, ok := ctx.Deadline(); ok {
	timeout := time.Until(dl)
	if timeout <= 0 {
		return nil, status.Error(codes.DeadlineExceeded, context.DeadlineExceeded.Error())
	}
	headerFields = append(headerFields, hpack.HeaderField{Name: "grpc-timeout", Value: grpcutil.EncodeDuration(timeout)})
}

Source: grpc/grpc-go v1.83.2 internal/transport/http2_client.go excerpt from lines 578–614, Apache-2.0

• The deadline is sent as remaining time rather than an absolute time. This is a choice to ensure it works even if clocks differ across machines. If it has already passed, it is not even sent and is terminated with DEADLINE_EXCEEDED.
• If it is a retried call, grpc-previous-rpc-attempts is attached additionally (Part 4).

3.1 Choosing the Unit for grpc-timeout

EncodeDuration starts from nanoseconds and selects the smallest unit that fits within 8 digitsand rounds up. Values calculated by porting the same rule to Python (pb_by_hand.py).

Even if a 5-second deadline is set, a few microseconds elapse by the moment the header is created, so it goes out as 5S rather than 4999987u . The server reads it back as decodeTimeout and sets a timeout in its own context. If the value is malformed, it is rejected with INTERNAL (aroundhttp2_server.go:453 ).

4. 5-Byte Message Prefix

What is carried in DATA frames is a sequence of Length-Prefixed-Message instances: 1 compression flag byte (0 or 1) + 4 length bytes (big-endian) + message.

The receiving side in grpc-go works like this.

func (p *parser) recvMsg(maxReceiveMessageSize int) (payloadFormat, mem.BufferSlice, error) {
	err := p.r.ReadMessageHeader(p.header[:])
	if err != nil {
		return 0, nil, err
	}
	pf := payloadFormat(p.header[0])
	length := binary.BigEndian.Uint32(p.header[1:])
	...
	if int(length) > maxReceiveMessageSize {
		return 0, nil, status.Errorf(codes.ResourceExhausted, "grpc: received message larger than max (%d vs. %d)", length, maxReceiveMessageSize)
	}
	data, err := p.r.Read(int(length))

Source: grpc/grpc-go v1.83.2 rpc_util.go excerpt from lines 785–800, Apache-2.0

• Message boundaries and DATA frame boundaries have no relation. Large messages can be split across multiple DATA frames, and multiple small messages can fit into a single DATA frame (Data Frames in the spec). Therefore, the receiving side cuts and reads based on the 5-byte header rather than frames.
• It reads the length first and checks against the upper limit. The default receive limit is 4MiB for both client and server (clientconn.go:139, server.go:61). If exceeded, it results in RESOURCE_EXHAUSTED. The send limit is unlimited by default (max int32).
• A compression flag of 1 means grpc-encoding it was compressed using the method specified in . The compression context starts fresh for each message. The sending side msgHeader writes these 5 bytes.

5. Response and Trailers

The response consists of headers containing :status 200 and content-type , messages, and trailers. The trailer contains grpc-status(decimal) and optionally grpc-message, grpc-status-details-bin(containing google.rpc.Statusin base64). Even for success (0), the status is always carried in the trailer.

grpc-message encodes UTF-8 with percent-encoding. It leaves only spaces and tildes (excluding% ) as is and replaces the rest with %XX . This value is calculated by porting grpc-go's encodeGrpcMessage rules.

If a response with a non-200 HTTP status is received (such as 502 generated by an intermediate proxy), the client fabricatesthe status. grpc-go's HTTPStatusConvTab mapping is 400→INTERNAL, 401→UNAUTHENTICATED, 403→PERMISSION_DENIED, 404→UNIMPLEMENTED, and 429·502·503·504→UNAVAILABLE. If you see unmotivated UNAVAILABLE errors behind a proxy, it means they might actually have been HTTP 503. Other HTTP errors become UNKNOWN.

Conversely, if a stream is cut off with RST_STREAM , HTTP/2 error codes are translated into statuses: REFUSED_STREAM→UNAVAILABLE (retryable as it occurs before processing), CANCEL→CANCELLED, ENHANCE_YOUR_CALM→RESOURCE_EXHAUSTED (keepalive ping violation from Part 4), and most others are INTERNAL (http2ErrConvTab, specification Errors section).

6. The 17 Status Codes

statuscodes.md is summarized from an operational perspective. The "Library" column indicates cases where the gRPC library can generate them itself.

The documentation states that INVALID_ARGUMENT, NOT_FOUND, ALREADY_EXISTS, FAILED_PRECONDITION, ABORTED, OUT_OF_RANGE, and DATA_LOSS are never generated by the library. Therefore, seeing these codes can be interpreted as "the app intentionally returned them." However, grpc-go sends requests with an invalid content-type along with HTTP 415 carrying grpc-status: 3(INVALID_ARGUMENT) (http2_server.go:504). Because this point contradicts the documentation, I noted it as a contribution candidate in Part 8.

7. Measurement: Counting the same calls from both sides

On control node A, kube-apiserver connects only to the local etcd (127.0.0.1:2379). Since both processes started at the same second (2026-08-19 07:42:56 UTC), their cumulative counters cover the same period. The client side is apiserver's grpc_client_handled_total(Prometheus), and the server side is etcd's grpc_server_handled_total(on the node, curl -s http://127.0.0.1:2381/metrics, reads only). Only non-OK results are collected.

sum by (instance, grpc_code, grpc_method) (grpc_client_handled_total{job="apiserver", grpc_code!="OK"})

During the same period, OK calls were 83.33 million for Range and 38.67 million for Txn on the server side, making the error ratio very small. More interesting than the ratio is how the names mismatch.

• OUT_OF_RANGE 94 = 94. Since this is a code not generated by the library, both sides match exactly. It was returned directly by etcd to mean "tried to read an already compacted revision." etcd defines both "compacted revision" and "future revision that does not exist yet" as OUT_OF_RANGE (api/v3rpc/rpctypes/error.go:32). Whether the 94 cases are one or the other cannot be distinguished by metrics (unconfirmed).
• Watch's CANCELLED is a normal termination. This increases by one every time the apiserver closes the stream to restart a watch. This is a prime example of why you should not trigger alerts based solely on metrics as errors.
• CANCELLED and DEADLINE_EXCEEDED exist only on the client side. Both are determined using their respective local clocks and contexts. By the time a client times out and gives up, the server might still be processing or may have already finished. It is possible that Txn UNKNOWN 4 on the server and Txn DEADLINE_EXCEEDED 4 on the client represent the same event, but this has not been verified (unconfirmed).
• Txn UNAVAILABLE was 14 on the server and 9 on the client . Because the etcd client (clientv3) has a policy of retrying only in safe situations, such as UNAVAILABLE when no connection exists yet (client/v3/retry.go), it is likely that only the final result after those retries remained in the client metrics (estimated, interceptor order unverified).

To sum it up in one line, "status codes are recorded differently depending on who made the determination". Use caller-side metrics for alerts and place both sides side-by-side for root cause analysis.

One more bonus. If you run hubble list nodes inside the Cilium agent, you get rpc error: code = Unimplemented desc = GetNodes not implemented . The Hubble server for a single node deliberately does not implement GetNodes (local_observer.go:250), meaning you should direct that question to hubble-relay, which is aware of multiple nodes. This is an example where UNIMPLEMENTED is used not only when "the method does not exist at all" but also when "it is not supported here."

8. Limitations

• Actual bytes of HPACK (header compression) and the flow of SETTINGS, WINDOW_UPDATE, and PING frames were not captured (unmeasured).
• grpc-status-details-bin Rich error details carried viagoogle.rpc.ErrorInfo·RetryInfo (such as ) are only briefly looked at in Part 6 on OTLP retries.
• The table in Section 7 is a single read of cumulative counters. Since times were not aligned and compared per event, pairing them as the "same event" is an estimation.

9. Connection with Small Clusters

If you are creating apiserver-to-etcd alerts, do not use grpc_code!="OK" as-is; instead, you need to check Watch CANCELLED and OUT_OF_RANGE separately. The former is normal, while the latter indicates that the watch requested a revision that is too old, giving you a reason to check the compaction interval and watch cache.

A word from TT. Even if the post office says "Delivered (200)," you shouldn't let your guard down. Be sure to check the final stamp at the end of the letter (grpc-status)!

References

All materials were accessed on 2026-10-17.

• grpc/grpc v1.84.0: doc/PROTOCOL-HTTP2.md, doc/statuscodes.md
• grpc/grpc-go v1.83.2: internal/transport/http2_client.go, internal/transport/http2_server.go, internal/transport/http_util.go, internal/grpcutil/encode_duration.go, rpc_util.go
• etcd v3.6.5: api/v3rpc/rpctypes/error.go, client/v3/retry.go, api/etcdserverpb/rpc.proto
• cilium v1.20.1: pkg/hubble/observer/local_observer.go
• Calculation code: Working directory measure/grpc/pb_by_hand.py
• Cited code: grpc-go v1.83.2, Apache License 2.0.