Status: Draft (2026-10-17), partially measured. Baseline versions: grpc/grpc v1.84.0 (docs), grpc/grpc-go v1.83.2 (commit 030ee8becb20). Measurements were obtained by only reading Prometheus on a small cluster during the 03:00 hour (KST) on 2026-10-17 (query details in Section 5). Writing rules: Measured data is the default; unmeasured items are marked as "Unmeasured", and unverified items are marked as "Unverified".
Table of Contents
- 1. Intuition: Forms Instead of Phone Calls
- 2. Diagram
- 3. Why Not JSON over HTTP/1.1?
- 4. What Code Generation Produces
- 5. Measurements: What Communicates via gRPC in a Small Cluster
- 6. Channel Lifespan: Long-Lived Connections, Short-Lived Calls
- 7. Limitations
- 8. Connection to a Small Cluster
- References
1. Intuition: Forms Instead of Phone Calls
When asking a neighboring team to handle a task, making a phone call is fast but leads to misunderstandings. Therefore, companies create forms. When the field names, order, and which fields are numeric are standardized, the recipient can process requests just by looking at the form. Even if the form changes slightly (e.g., adding a field), they can continue working with people using the old form.
gRPC writes this form as a .proto file, and automatically generates code in both languagesfrom it. The caller simply invokes an ordinary function like client.Hit(ctx, req) , and beneath it, the library converts the request into bytes, sends it over an HTTP/2 connection, and returns the response back as an object.
It all started when Google decided to open-source the successor to its internal RPC framework (Stubby), which had been used for over 10 years, in March 2015. It entered the CNCF incubating stage on 2017-02-16 (grpc.io/about, CNCF Project Page).
| Component | Responsibility | In This Series |
|---|---|---|
| IDL (.proto) | Contract for services, methods, and messages. Field numbers are the key to backward compatibility. | Part 2 |
| Code Generation (protoc + plugins) | Language-specific message types, client stubs, server interfaces | Part 1, Section 4 |
| Channel (ClientConn) | Name resolution, load balancing, connection maintenance, retries, deadlines | Parts 4, 5, 7 |
| HTTP/2 Transport | One call = one stream, 5-byte message prefix, trailer status codes | Part 3 |
2. Diagram
- The top row represents build-time activities. Passing
.protointoprotocand language-specific plugins produces message types, stubs, and server interfaces. - The bottom row represents runtime activities. The client stub simply tells the channel, "Send these bytes to this method path." Within the channel, a name resolver creates an address list, a load balancing policy selects a subchannel (a connection to a single address), and a new stream is opened on that subchannel's HTTP/2 connection.
- A single thick line represents a single TCP connection. Multiple calls flow concurrently over it. This is the root cause of the "load skewing to a single pod" issue discussed in Part 5.
3. Why Not JSON over HTTP/1.1?
Remote calls are certainly possible with REST and JSON. Looking at why gRPC exists separately across four dimensions—contracts, bytes, connections, and failures—reveals the following:
| Axis | Common REST/JSON | gRPC | Trade-off |
|---|---|---|---|
| Contract | OpenAPI is optional and prone to drifting from code | .proto serves as the source of truth for code | A code generation step is introduced into the build process |
| Bytes | Field names are transmitted as text every time | Field numbers + binary values. 21 bytes in Part 2 example vs. 72 bytes for JSON | Not human-readable directly (requires tools from Part 8) |
| Connection | HTTP/1.1 allows one request per connection | Many calls over a single connection via HTTP/2 streams, bidirectional streaming | L4 load balancing operates at the connection level, causing skew (Part 5) |
| Failure | HTTP statuses and body formats vary across services | 17 status codes, deadline propagation, and retry policies handled in the library | The responsibility of choosing appropriate status codes remains with the app (Part 3) |
Browser JavaScript APIs do not allow direct manipulation of HTTP/2 frames and trailers, making it difficult to use gRPC as-is. Therefore, separate bridges like gRPC-Web or HTTP/JSON transcoding gateways are used (not covered in this series).
4. What Code Generation Produces
This is an example contract. This service is used throughout the series (a fictional service recording a single table tennis rally).
syntax = "proto3";
package tt.v1;
message Rally { int32 rally_id = 1; string server = 2; repeated int32 hits = 3; bool won = 4; sint32 delta = 5; }
message Score { int32 home = 1; int32 away = 2; }
service Scoreboard {
rpc Hit(Rally) returns (Score); // 단항
rpc Watch(Rally) returns (stream Score); // 서버 스트리밍
}
In this contract, the method path for Hit is /tt.v1.Scoreboard/Hit . The frame diagram in Part 3 also uses this path.
The generated client stub does surprisingly little work. Without running protoc directly, looking at the actual generated file (health check service) included in the grpc-go repository, a single method looks like this:
const Health_Check_FullMethodName = "/grpc.health.v1.Health/Check"
func (c *healthClient) Check(ctx context.Context, in *HealthCheckRequest, opts ...grpc.CallOption) (*HealthCheckResponse, error) {
cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
out := new(HealthCheckResponse)
err := c.cc.Invoke(ctx, Health_Check_FullMethodName, in, out, cOpts...)
if err != nil {
return nil, err
}
return out, nil
}
Source: grpc/grpc-go v1.83.2 health/grpc_health_v1/health_grpc.pb.go(generated by protoc-gen-go-grpc v1.6.2), Apache-2.0
- The stub simply passes the method path stringand the request and response objects to the channel's
Invoke. The path consists of/+ package.service +/+ method, and this string becomes the HTTP/2:pathas-is (Specification PROTOCOL-HTTP2.md Appendix A). - The server-side generated code is a table named
ServiceDesc. Under the service name, there is a list of unary methods (Methods) and streams (Streams, indicating whether it is server or client streaming), with each entry pointing to a handler function. The server uses the incoming:pathto look up this table and invoke the handler (Part 7). - Generated code varies by language, but the wire format remains the same. This is why a Go client can call a C++ server.
5. Measurements: What Communicates via gRPC in a Small Cluster
From 2026-10-17 03:00 to 03:04 (KST), the following query was sent to Prometheus (via GET through the Kubernetes API server's service proxy, read-only).
count by (__name__, job) ({__name__=~"grpc_.*|.*_grpc_.*|rpc_.*"})
sum by (grpc_service, grpc_method, grpc_type) (grpc_client_started_total{job="apiserver"})
sum by (operation_type) (kubelet_runtime_operations_total)
Collecting the gRPC service names found in the metrics yields the following (values detailed in Part 6):
| Who → To Whom | gRPC Service (Metric Label) | Basis |
|---|---|---|
| kube-apiserver → etcd | etcdserverpb.KV·Watch·Lease·Maintenance |
apiserver's grpc_client_* metric |
| kubelet → containerd | runtime.v1.RuntimeService·ImageService |
kubelet_runtime_operations_total 's 21 operation_type variants |
| App → OTel Collector → Tempo | OTLP TraceService/Export |
Collector's transport="grpc", export target :4317 |
| Tempo internal | /tempopb.Pusher/PushBytesV2, /grpc.health.v1.Health/Check |
tempo_request_duration_seconds 's route label |
| Argo CD UI/CLI → argocd-server | application.ApplicationService and 113 other methods |
grpc_server_started_total |
| Sidecar → istiod | istio.v1.auth.IstioCertificateService/CreateCertificate |
istiod's grpc_server_* |
| hubble-relay → Hubble per node | observer.Observer·peer.Peer |
Relay logs, gRPC probes (Parts 5 and 6) |
Names matching the pattern etcdserverpb.KV like proto package.service occur because the path conventions from Section 4 are passed down directly to metric labels. Conversely, kubelet does not use common gRPC metrics and counts things under its own name like list_containers . Even with gRPC, observation methods vary by implementation.
6. Channel Lifespan: Long-Lived Connections, Short-Lived Calls
- Channels do not connect upon creation. grpc-go's
NewClientstarts in an IDLE state and resolves names and connects on the first call. If there are no calls, it returns to IDLE after 30 minutes (default valuedialoptions.go:729). - Channel states are five: IDLE, CONNECTING, READY, TRANSIENT_FAILURE, and SHUTDOWN (connectivity-semantics-and-api.md).
- Therefore, the rule of thumb is to reuse channels. Official performance guides also recommend reusing channels and stubs (Performance Best Practices). Creating a new channel for every call incurs TCP, TLS, and HTTP/2 handshake overhead each time. As seen in Part 5, kubelet's gRPC probes operate precisely this way.
- Default name resolution also varies by version.
NewClientusesdns, while the olderDial·DialContextusespassthrough(dialoptions.go:730,clientconn.gocomments). This is whyDialis appended when Kubernetes source code migrates fromNewClienttopassthrough:///(Part 5, Section 6).
7. Limitations
- Internal structures of other language implementations (C++ Core, Java) were not examined. Code reading in this series is limited to grpc-go.
- Authentication mechanisms such as gRPC-Web, JSON transcoding gateways, and ALTS are not covered.
- The list in Section 5 only counts targets scraped by Prometheus. gRPC that does not emit metrics or is not scraped (e.g., CSI driver Unix sockets, kubelet plugin registration) is omitted (Unmeasured).
8. Connection to a Small Cluster
Even if you handle clusters daily without ever writing gRPC directly, it is helpful to realize that you are already using it hundreds of millions of times a day. Control node A's etcd has processed Ranges approximately 83.33 million times since startup, and the 8 kubelets combined have invoked CRI list lookups about 69 million times (Part 6). The next part examines what kind of bytes each of those calls transmits.
Titi's Note: As long as you make a good form, it doesn't matter if the other party is Go or C++. Just don't change the field numbers! The numbers are the agreement!
References
All materials were accessed on 2026-10-17.
- grpc.io About, Performance Best Practices
- CNCF Project Page: gRPC
- grpc/grpc v1.84.0:
doc/PROTOCOL-HTTP2.md,doc/connectivity-semantics-and-api.md - grpc/grpc-go v1.83.2:
clientconn.go,dialoptions.go,health/grpc_health_v1/health_grpc.pb.go - Cited Code: grpc-go v1.83.2, Apache License 2.0.