Read and write the bytes by hand

Goal

In a Pod with neither protoc nor the protobuf package, you build a protobuf wire-format encoder and decoder yourself using only the Python standard library. You stack them up in the order varint → tag → signed integer → field → message, and finally decode a submessage and a packed repeated field. You will use the tool you build in the next lab as well.

Why it matters

Half of gRPC outages come from not knowing that "bytes have no names". The wire carries only the field number and the wire type, and the names and declared types are attached by the reader's .proto. If you encode by hand once, the calculations show why you must not change a number, why a negative int32 takes as many as 10 bytes, and why an old client skips a new field without error. Someone who has opened up once what the library hides sees different things in a schema review.

Steps

1. Create /root/grpc/pb.py and implement encode_varint(n) -> bytes and decode_varint(data, pos=0) -> (값, 다음 위치) (returning the value and the next position). Write the hex encodings of 1, 150, 300, and 16384 to /root/grpc/01-varint.txt, one per line, as 값=16진수 (value=hex).
2. Add encode_tag(field, wire_type) -> bytes and decode_tag(data, pos=0) -> (번호, 와이어타입, 다음 위치) (field number, wire type, next position). Write the hex of the four tags 1:0, 2:2, 3:2, and 16:0 to /root/grpc/02-tag.txt as 번호:와이어타입=16진수 (number:wire type=hex).
3. Add zigzag_encode(n) -> int and zigzag_decode(u) -> int, actually encode int32 -1, sint32 -1, int32 -150, and sint32 -150, and write them to /root/grpc/03-signed.txt as 타입 값 = 16진수 (N bytes) (type, value, hex). int32 sends the 64-bit two's complement as a varint, and sint32 sends the value that went through ZigZag as a varint.
4. Add encode_field(field, kind, value) -> bytes. kind is one of int32 int64 uint32 uint64 sint32 sint64 bool string bytes. Using that function, create an order with id(1)=4242, qty(2)=17, note(3)="한글 메모" by writing /root/grpc/04-make.py, run it, and leave /root/grpc/04-order.bin.
5. Add decode_message(data) -> [(번호, 와이어타입, 원시값), ...] (a list of field number, wire type, raw value). VARINT is an int, LEN is bytes, and I32/I64 are the 4- and 8-byte bytes as they are. Decode /opt/app/grpc/order_v1.bin and write it to /root/grpc/05-decode.txt, one line each, as field=N wire=W value=V (LEN values in hex).
6. Add encode_records(records) -> bytes (the inverse function of decode_message). Decode /opt/app/grpc/order_v2.bin and, for each record, record whether the number is known to the v1 schema (/opt/app/grpc/order_v1.proto) in /root/grpc/06-unknown.txt, as field=N wire=W known or unknown.
7. Field 6 of /opt/app/grpc/nested.bin is a Customer submessage. Decode its payload again with decode_message and write three lines to /root/grpc/07-nested.txt: customer.bytes=16진수, customer.name=이름, and customer.tier=숫자 (hex, name, and number).
8. Add decode_packed_varints(raw) -> [int, ...], unpack field 7 (packed repeated int32) of /opt/app/grpc/packed.bin, and write three lines to /root/grpc/08-packed.txt: tags=쉼표목록 (a comma-separated list), packed_record=16진수(필드 7 레코드 전체) (hex of the whole field 7 record), and expanded_record=16진수(원소마다 태그를 붙인 형태) (hex of the form with a tag on every element).

Notes

• The source of the rules is the Protocol Buffers encoding document. The tag is (field_number << 3) | wire_type, and the wire types are 0 VARINT · 1 I64 · 2 LEN · 5 I32.
• There is a reference implementation, /opt/app/grpc/pbmini.py. Read it if you get stuck, but this lab's grader imports your /root/grpc/pb.py and runs it with random values that are not in the instructions as well. If you hard-code the answers, you fail on the spot.
• You can view the fixture bytes with python3 -c "print(open('/opt/app/grpc/order_v1.bin','rb').read().hex())".
• Common mistakes: decode_varint ignoring the pos argument, counting the string length in characters (it must be the number of UTF-8 bytes), and truncating a negative int32 to 32 bits (it is 64 bits).

Build and read a varint

In /root/grpc/pb.py, implement encode_varint(n) and decode_varint(data, pos=0), and write the hex of 1, 150, 300, and 16384 to /root/grpc/01-varint.txt as 값=16진수 (value=hex).

Cut the value into 7-bit groups and emit them from the lowest, and if there is more after, set the most significant bit (0x80) of that byte. 150 in binary is 10010110 → putting the continuation mark on the lower 7 bits 0010110 gives 0x96, and the remaining 1 is 0x01 — so it is 96 01.

decode_varint must read from pos and return (값, 다음 위치) (the value and the next position). The grader also calls it from the middle, such as pos=3.

Check: python3 -c "import sys; sys.path.insert(0,'/root/grpc'); import pb; print(pb.encode_varint(300).hex())" must give ac02.

Put the number and wire type in a tag

Add encode_tag(field, wire_type) and decode_tag(data, pos=0) to /root/grpc/pb.py, and write the hex of 1:0, 2:2, 3:2, and 16:0 to /root/grpc/02-tag.txt as 번호:와이어타입=16진수 (number:wire type=hex).

A tag is just a varint — put (field << 3) | wire_type into encode_varint and you are done. When reading, after reading the varint, the lower 3 bits are the wire type, and the rest shifted right by 3 bits is the number.

Numbers 1–15 take a one-byte tag, and from 16 on it becomes two bytes. The document's recommendation to give small numbers to frequently used fields comes from here.

(1 << 3) | 0 is 8, so 08, and (2 << 3) | 2 is 18, so 12.

A negative int32 is 10 bytes, a sint32 is one or two

Add zigzag_encode(n) and zigzag_decode(u), actually encode int32 -1, sint32 -1, int32 -150, and sint32 -150, and write them to /root/grpc/03-signed.txt as 타입 값 = 16진수 (N bytes) (type, value, hex).

A negative int32 is sent by treating the 64-bit two's complement as unsigned and then sending it as a varint — in Python, put n & 0xFFFFFFFFFFFFFFFF into encode_varint. The most significant bit is set, so it uses all ten bytes.

sint32 goes through ZigZag first. The document's formula is (n << 1) ^ (n >> 31), and Python integers use an arithmetic shift, so using this formula as is gives -1 → 1, 1 → 2, -2 → 3. The formula to undo it is (u >> 1) ^ -(u & 1).

The line format is, for example, sint32 -1 = 01 (1 bytes).

Turn one field into a record

Add encode_field(field, kind, value) (kind: int32 int64 uint32 uint64 sint32 sint64 bool string bytes). In /root/grpc/04-make.py, use that function to concatenate id(1)=4242, qty(2)=17, note(3)="한글 메모" and save it as /root/grpc/04-order.bin.

A record is the tag plus the value. For the varint family, it is encode_tag(f, 0) + encode_varint(값), and for string it is encode_tag(f, 2) + encode_varint(len(raw)) + raw, where raw is bytes encoded as UTF-8 and the length is also that byte count. The Korean memo in the task is 5 characters but 13 bytes.

For int32/int64 negatives, use the 64-bit mask as in step 3; for the sint family, ZigZag (for 64 bits, n >> 63); and a bool is 0 or 1.

A message is just records concatenated — there is no delimiter and no header.

Turn the bytes back into a list of records

Add decode_message(data) so that it returns [(번호, 와이어타입, 원시값), ...] (a list of field number, wire type, raw value). Decode /opt/app/grpc/order_v1.bin and write it to /root/grpc/05-decode.txt as field=N wire=W value=V (LEN values in hex).

Repeat until the end: read the tag → read the value according to the wire type. For 0 it is a varint, for 2 it is a length varint followed by that many bytes, for 5 it is 4 bytes, and for 1 it is 8 bytes. Do not interpret the raw value — here you know neither the name nor the declared type.

If this function does not advance pos correctly, it reads the next tag from the wrong place. The grader runs it with 12 random messages.

Fixture check: python3 -c "print(open('/opt/app/grpc/order_v1.bin','rb').read().hex())".

Skip unknown fields without losing them

Add encode_records(records) (the inverse function of decode_message). Decode /opt/app/grpc/order_v2.bin and, for each record, record whether the number is known to the v1 schema (/opt/app/grpc/order_v1.proto) in /root/grpc/06-unknown.txt, as field=N wire=W known or unknown.

An old parser can skip a new field without error because the wire type tells it the length. And a proto3 parser preserves that unknown field and includes it when serializing again — encode_records does that job: for each record it writes the tag again, then for VARINT a varint, for LEN the length plus the bytes, and for I32/I64 the bytes as they are.

The grader round-trips random messages through your decode_message → encode_records and checks whether the bytes equal the original.

The numbers v1 knows are only 1, 2, and 3.

A submessage is another message inside a LEN

In /opt/app/grpc/nested.bin, decode the payload of field 6 (the Customer submessage) again with decode_message, and write three lines to /root/grpc/07-nested.txt: customer.bytes=16진수, customer.name=이름, and customer.tier=숫자 (hex, name, and number).

A submessage is a LEN record, exactly like a string — only those bytes are another message, so you just decode them once more with the same function. If you look at the schema (/opt/app/grpc/nested.proto), Customer is string name = 1; int32 tier = 2;.

customer.bytes is the .hex() of the raw value of field 6 (the payload without the tag and length).

Unpack a packed repeated field

Add decode_packed_varints(raw), unpack field 7 (packed repeated int32) of /opt/app/grpc/packed.bin, and write three lines to /root/grpc/08-packed.txt: tags=쉼표목록, packed_record=16진수, and expanded_record=16진수 (a comma-separated list, and two hex strings).

Packed means varints are concatenated inside a single-tag LEN record. If you repeat decode_varint until the payload ends, the list comes out.

packed_record is the hex of the whole field 7 record (tag + length + payload) in packed.bin, and expanded_record is the hex of the same list made with encode_field(7, "int32", v) for each element and concatenated. Compare the lengths of the two — you can see how much the tag cost went down.