TT Lab
Blog Get started
Learn Learning paths Courses

I renumbered a field and the old client silently read the wrong value

Change the schema in front of an old client

Continue in TT Lab

Goal

You leave the "old client", which has the v1 schema hard-coded in its code, as it is and change the schema to v2. You add new fields with new numbers, seal the deleted field with reserved, and reproduce how the old client wrongly reads, without any error, bytes whose numbers were deliberately changed. Then you measure the traps of default values and presence, and leave a final schema with optional and reserved and a compatibility table.

Why it matters

Clients and servers are never deployed at the same moment. There are rollbacks, and old bytes remain in logs. So the safety of a schema change must be judged not by "does it compile" but by "does an old binary read new bytes, and a new binary old bytes, correctly". The wire has no names and only numbers, so changing a number is the same as deleting a field and creating a new one — yet the parser raises no error at all and reads the value of the changed number under the old name. The purpose of this lab is to reproduce that incident and engrave it in memory.

Steps

  1. Run python3 /opt/app/grpc/old_client.py /opt/app/grpc/order_v1.bin and save the four lines of output as they are to /root/grpc/evolve/01-old.txt.
  2. Write /root/grpc/evolve/v2.proto. Start with syntax = "proto3";, in message Order keep v1's int32 id = 1; int32 qty = 2; string note = 3; unchanged, and add int64 unit_price and string currency with new numbers.
  3. In v2.proto, delete note and leave reserved 3; and reserved "note";. The rest stays the same.
  4. In /root/grpc/evolve/make_v2.py, use /opt/app/grpc/pbmini.py to encode id=9001, qty=2, unit_price=15000, currency="KRW" with the numbers of v2.proto and save it to /root/grpc/evolve/order_v2.bin, then save the output of reading that file with the old client to /root/grpc/evolve/04-old-reads-v2.txt.
  5. /opt/app/grpc/renumbered.bin is the same order made with /opt/app/grpc/renumbered.proto (a schema with the numbers reassigned). Save the output of reading it with the old client to /root/grpc/evolve/05-renumbered.txt, and below it write one line truth: id=<true-id> qty=<true-qty> and one line why: <why-no-error>.
  6. With pbmini, make an order with id=9002, currency="KRW" and qty=0 in two ways. /root/grpc/evolve/06-zero.bin uses implicit presence (0 is not sent), and /root/grpc/evolve/06-zero-explicit.bin uses explicit presence (0 is sent too). In /root/grpc/evolve/06-presence.txt, write four lines: implicit=<hex> explicit=<hex> implicit_qty=<qty-read-by-old-client> explicit_qty=<qty-read-by-old-client>.
  7. Write the final /root/grpc/evolve/Order.proto. It includes optional int32 qty = 2;, reserved 3; + reserved "note";, unit_price and currency, and message Customer { string name = 1; int32 tier = 2; } and Order's Customer customer = <new-number>;.
  8. Write a markdown table in /root/grpc/evolve/compat.md. The first column is the six keys add-new-number remove-and-reserve reuse-number renumber int32-to-string rename-only, the second column is safe or unsafe, and the third column is a one-line reason.
  9. Start the incident with incident gc-zero-dropped — at three warehouses, only the updates that set stock to 0 are not applied. Determine the cause for each warehouse, write <warehouse>: <code> and evidence: <tap-line> to /root/incident/gc-zero-dropped/report.txt, fix it so that 0 reaches the merger, and merge again with gz-sync.

Notes

Read the v1 bytes with the old client

Save the four lines of output of python3 /opt/app/grpc/old_client.py /opt/app/grpc/order_v1.bin as they are to /root/grpc/evolve/01-old.txt.

Direction: the old client does not reread the .proto; it reads believing numbers 1, 2, and 3 are id, qty, and note. You keep its output as a record.

Tools: creating a directory and the shell redirection >.

Specifics: do not copy it by hand; save the output as it is — the grader runs the same command again and compares.

New fields get new numbers

In /root/grpc/evolve/v2.proto, keep v1's three fields as they are and add int64 unit_price and string currency with new numbers.

Direction: do not change a single character of the number, name, or type of an existing field, and give a new field a number that has never been used so far.

Tools: proto3 syntax — syntax = "proto3";, message Order { <type> <name> = <number>; }.

Specifics: 19,000–19,999 is a range reserved by the implementation and cannot be used, and 1–15 take a one-byte tag, which is good for frequently used fields. unit_price is int64 and currency is string.

Seal the deleted number

In v2.proto, delete note and leave reserved 3; and reserved "note";.

Direction: deleting itself is safe on the wire. The danger is someone using that number again later, and reserved is the device that makes protoc reject that.

Tools: reserved <number>; and reserved "<name>";.

Specifics: a number and a name cannot be mixed in one statement, so you write two lines. They may be anywhere inside the message block.

The old client reads the v2 bytes

In /root/grpc/evolve/make_v2.py, use pbmini to encode id=9001, qty=2, unit_price=15000, currency="KRW" with the numbers of v2.proto and save it to /root/grpc/evolve/order_v2.bin, then save the output of reading it with the old client to /root/grpc/evolve/04-old-reads-v2.txt.

Direction: fields added with new numbers are unknown fields to the old client — it learns their length from the wire type and skips them, and reads the fields it knows correctly.

Tools: pbmini's encode_message([(field, kind, value), ...]) and the old client.

Specifics: use the numbers exactly as you chose them in the step 2 v2.proto, and do not include the deleted note. The two new numbers must appear on the unknown= line of the old client's output.

The old client reads the bytes with swapped numbers

Save the output of reading /opt/app/grpc/renumbered.bin with the old client to /root/grpc/evolve/05-renumbered.txt, and append two lines, truth: id=<true-id> qty=<true-qty> and why: <why>.

Direction: someone renumbered the fields. The wire has only numbers, so the old client still reads number 1 as id.

Tools: /opt/app/grpc/renumbered.proto, the old client, and the values you read yourself according to the numbers in renumbered.proto.

Specifics: renumbered.bin is the same order serialized with that schema. The wire type is the same VARINT, so there is no error or warning — the truth line is the real values read according to renumbered.proto.

Was qty=0 sent or not sent

Make an order with id=9002, currency="KRW", qty=0 as /root/grpc/evolve/06-zero.bin (implicit presence, 0 is not sent) and /root/grpc/evolve/06-zero-explicit.bin (explicit presence, 0 is sent too), and in /root/grpc/evolve/06-presence.txt write four lines: implicit= explicit= implicit_qty= explicit_qty=.

Direction: a proto3 unlabeled scalar does not serialize its default value (0, the empty string). With optional, 0 is sent too.

Tools: make two messages with pbmini, look at the bytes with .hex(), and have the old client read each.

Specifics: the explicit side carries the two extra bytes 10 00 of a field 2 record. Yet the old client reads qty=0 in both cases — that inability to tell them apart is why optional is needed.

The final schema with optional and reserved

Write /root/grpc/evolve/Order.proto — optional int32 qty = 2;, reserved 3; + reserved "note";, int64 unit_price and string currency, message Customer { string name = 1; int32 tier = 2; } and Order's Customer customer = <new-number>;.

Direction: optional separates 0 from "absent", reserved seals deleted numbers, and a submessage uses a separate message as its type.

Tools: the optional label, reserved, message Customer { ... } and a field in Order that uses that type.

Specifics: the number of customer must be one that has never been used and is not reserved. On the binary wire, attaching the label itself does not change the bytes (it only means that 0 gets sent).

Write the compatibility table

Write a markdown table in /root/grpc/evolve/compat.md. The first column is add-new-number remove-and-reserve reuse-number renumber int32-to-string rename-only, the second column is safe/unsafe, and the third column is the reason.

Direction: this step sums up what you saw first-hand in this lab — the evidence for each row is one of the earlier steps.

Tools: the results of step 4 (adding with a new number), step 3 (delete and reserve), and step 5 (renumbering), and the wire type table.

Specifics: for a type change, if the wire type changes (VARINT → LEN), an old parser reads it with a different meaning; changing only a name leaves the bytes the same, because there are no names on the binary wire. Use only one verdict word, safe or unsafe — the grader looks at the first one that appears in a row.

Incident: only updates that set the value to 0 disappear

Run incident gc-zero-dropped. At three warehouses (icn, pus, gmp), only the updates that set stock to 0 are not applied — other updates are fine and there are no errors. Determine the cause for each warehouse and write three <warehouse>: <code> lines to /root/incident/gc-zero-dropped/report.txt. The code is one of implicit-presence · merge-skips-zero · omit-empty · relay-reserialize · wrong-number. For each warehouse, attach one line from the gz-tap output before fixing that shows where the 0 disappeared (the stage=sent or stage=recv line of the update that set the value to 0) as evidence: <tap-line>. If you grade with a wrong code three times, the incident is locked and you must start it again (the causes are chosen anew). Then fix the sender side (/srv/gz/<warehouse>/producer.*) so that 0 reaches the merger, and merge again with gz-sync.

Direction: the symptom is the same for all three, but the hop where 0 disappears differs — does the sender never put it in at all (no 10 00 in the sent bytes), or is it put in and lost along the way (present in what was sent, absent in what was received).

Tools: the two lines stage=sent and stage=recv of gz-tap <warehouse> side by side, the qty label in producer.proto, and omit_empty and route in producer.conf. The schemas of the two relay versions are in /opt/fixtures/grpcc/gz/.

Specifics: if it was not put in, check whether it is the schema (the label) or the configuration (the omission list); if it disappeared, check which relay version it passed through. After fixing, you must run gz-sync again for stock.json to change, and also check that an update that changed only the price does not overwrite the quantity with 0.