Protobuf Wire Format: 21 Bytes One by One

Status: Draft (2026-10-17), empirically measured. All bytes were directly encoded using python3 (3.9, standard library only). Cross-checking by serializing the same message with the official protobuf library was skipped because it requires downloading a package (unmeasured, requirements: protobuf Python package and protoc). The rules were verified using protobuf.dev Encoding documentationand the proto3 Language Guide(accessed 2026-10-17).

Table of Contents

• 1. Intuition: Field Numbers Instead of Field Names
• 2. Diagram: 21 Bytes of a Rally
• 3. Four Rules
  • 3.1 varint: 7 bits at a time, starting from the least significant bits
  • 3.2 Tag = (Field Number << 3) | Wire Type
  • 3.3 LEN: Length prepended
  • 3.4 Default values are not sent
• 4. Try It Yourself: Hand Encoding vs. JSON
• 5. Backward Compatibility: How to Skip Unknown Fields
• 6. Limitations
• 7. Connection to Small Clusters
• References

1. Intuition: Field Numbers Instead of Field Names

JSON writes {"rallyId":150} like field names every time . It is easy to read, but the same names repeat across messages. Protobuf assumes that both sides already have the contract (.proto) and only writes field numbers and valuesinto the bytes. 「Field 1: 150」. Names are in the contract, so they don't need to be sent.

Two things follow from this:

• It is small. There are no names, and small integers are written compactly.
• Numbers are the contract. Field names can be changed (since they aren't in the bytes), but changing or reusing numbers will cause old data to be read into the wrong fields.

2. Diagram: 21 Bytes of a Rally

We populate one message from the contract in Part 1. rally_id=150, server="티티", hits=[3,270,1], won=true, delta=-2.

• Each color represents a single field. Every field starts with a tag byte(indicated as 「Tag」 in the diagram).
• The string "티티" is 6 bytes in UTF-8 (ed 8b b0 twice), preceded by the length 06 .
• hits 's 270 takes two bytes (8e 02), so the bundle length is 4.

3. Four Rules

3.1 varint: 7 bits at a time, starting from the least significant bits

Integers are sliced into 7-bit chunks, written starting from the lowest bits , and if there are more bytes following, the most significant bit (MSB) is set to 1. 150 is binary 1001 0110 so the lower 7 bits 001 0110 with the continuation bit set becomes 0x96, and the remaining 1 becomes 0x01 yielding 96 01 . This is a manually calculated value.

The last line is a trap. int32 's negative numbers are extended to 64-bit two's complement and take 10 bytes (see the "Signed Integers" section of the Encoding documentation). Fields where negative numbers appear frequently use sint32/sint64 . ZigZag maps 0, -1, 1, -2, 2... to 0, 1, 2, 3, 4... and then writes them as varints.

3.2 Tag = (Field Number << 3) | Wire Type

The wire type only indicates the shapeof the value. 0 is VARINT (int32, int64, uint, bool, enum, sint), 1 is I64 (fixed64, double), 2 is LEN (string, bytes, message, packed bundle), 5 is I32 (fixed32, float), while 3 and 4 are legacy group types and are not used. Tags are also varints, so larger numbers make them longer.

Numbers 1-15 take 1 byte for the tag, and numbers 16-2047 take 2 bytes. This is why the guide recommends giving smaller numbers to frequently populated fields. Numbers can range from 1 to 536,870,911 (2²⁹-1), with 19,000 to 19,999 reserved for implementation use (proto3 guide).

3.3 LEN: Length Prepended

Strings, bytes, submessages, and packed bundles are 태그 + 길이(varint) + 내용 . In proto3, numeric repeated are packed by default , so hits=[3,270,1] is a single tag 1a(number 3, LEN) + length 04 + 03 8e 02 01 . If it were not packed, each value would have a tag attached, taking 7 bytes (calculated).

3.4 Default Values Are Not Sent

Standard scalar fields in proto3 (implicit presence) are not serializedif their value is 0, empty string, or false. A Rally where all fields are default values is 0 bytes. The receiving side cannot distinguish between "0 was sent" and "not sent." If distinction is necessary, optional is added to explicitly specify presence (Field Presence in proto3 guide). The health check request HealthCheckRequest{service:""} in Part 5 being actually 0 bytesis also due to this rule.

4. Try It Yourself: Hand Encoding vs. JSON

• Environment: Laptop, Python 3.9.6, standard library only (struct, json)
• Code and raw data: Working directory measure/grpc/pb_by_hand.py, output out-pb_by_hand.txt
• Method: Following the rules in Section 3, fields were converted to bytes one by one and concatenated.

def varint(n):
    if n < 0:
        n &= (1 << 64) - 1          # int32 음수는 64비트로 늘려 10바이트
    out = bytearray()
    while True:
        b, n = n & 0x7F, n >> 7
        out.append(b | 0x80 if n else b)
        if not n:
            return bytes(out)

def tag(field, wire):
    return varint((field << 3) | wire)

Result (Measured):

Writing the same content in JSON (with field names following the proto3 JSON convention of lowerCamelCase) was 72 bytes, indent=2 for a single unspaced line and 111 byteswhen indented. In this example, protobuf is 29% the size of a single JSON line. The difference grows larger when messages are mostly numeric and shrinks when they are mostly long strings. Comparison after compression (gzip) was not performed (unmeasured).

delta If int32 was declared as

5. Backward Compatibility: How to Skip Unknown Fields

Suppose a new version adds field number 6 string note = 6; and includes "hello" . 32 05 68 65 6c 6c 6f is appended to the end of the bytes. An old-version parser that does not know about field 6 reads wire type 2 (LEN) from tag 32 and skipsby the length of 5. Indeed, the script's simulation of an "old parser" (skip_fields) (6, 2, 6) finished reading to the end while printing . This is possible because tags indicate the shape of the value.

Instead of discarding unknown fields, proto3 preserves them and writes them back out during re-serialization(Unknown Fields in proto3 guide). However, converting to JSON or copying fields individually may cause them to disappear.

Here is a summary of what to keep in mind (proto3 guide summary).

The last line can be verified with the example in Section 4. Reading 28 03 as int32 gives 3, and reading it as sint32 gives -2.

6. Limitations

• Message size limit: Serialized messages must be less than 2GiB (Encoding documentation). gRPC sets a much smaller default limit (4MiB receive) separately (Part 3).
• This article only covered proto3. Features of Editions (2023~), proto2 required/group, and map wire representation (repeated submessages) were not covered.
• Hand encoding is for understanding the rules, not for verifying an implementation. Byte-level comparison with official library output was unmeasured.
• Field order: The above example used numerical order, but the specification does not guarantee serialization order. You must not assume that the same message always produces the same bytes (be careful when using for hashing or signing).

7. Connection to Small Clusters

CRI messages between kubelet and containerd also use this format. Therefore, even if kubelet is newer than containerd (even if new fields are added to CRI), older containerd can work by skipping unknown fields. Conversely, this is also why the CRI side does not arbitrarily change field numbers. Raising the limit on kubelet CRI responses from the gRPC default of 4MiB to 16MiB (Part 6) is because lists of hundreds of containers are contained in a single message.

TT's Note. Simply writing 「Field 1: 150」 on the scoreboard is enough as long as both sides share the same format. However, writing negative scores in an int32 field takes 10 spaces, so use an sint32 field instead!

References

All materials were accessed on 2026-10-17.

• protobuf.dev: Encoding(wire types, varint, ZigZag, packed, size limits), Language Guide (proto3)(field number ranges, reserved, unknown fields, type compatibility, field presence)
• Byte calculation code and output: Working directory measure/grpc/pb_by_hand.py, out-pb_by_hand.txt