I renumbered a field and the old client silently read the wrong value
Change the schema in front of an old client
Goal
You leave the "old client", which has the v1 schema hard-coded in its code, as it is and change the schema to v2.
You add new fields with new numbers, seal the deleted field with reserved, and reproduce how the old client
wrongly reads, without any error, bytes whose numbers were deliberately
changed. Then you measure the traps of default values and presence, and leave a final
schema with optional and reserved and a compatibility table.
Why it matters
Clients and servers are never deployed at the same moment. There are rollbacks, and old bytes remain in logs. So the safety of a schema change must be judged not by "does it compile" but by "does an old binary read new bytes, and a new binary old bytes, correctly". The wire has no names and only numbers, so changing a number is the same as deleting a field and creating a new one — yet the parser raises no error at all and reads the value of the changed number under the old name. The purpose of this lab is to reproduce that incident and engrave it in memory.
Steps
- Run
python3 /opt/app/grpc/old_client.py /opt/app/grpc/order_v1.binand save the four lines of output as they are to/root/grpc/evolve/01-old.txt. - Write
/root/grpc/evolve/v2.proto. Start withsyntax = "proto3";, inmessage Orderkeep v1'sint32 id = 1; int32 qty = 2; string note = 3;unchanged, and addint64 unit_priceandstring currencywith new numbers. - In v2.proto, delete
noteand leavereserved 3;andreserved "note";. The rest stays the same. - In
/root/grpc/evolve/make_v2.py, use/opt/app/grpc/pbmini.pyto encodeid=9001, qty=2, unit_price=15000, currency="KRW"with the numbers of v2.proto and save it to/root/grpc/evolve/order_v2.bin, then save the output of reading that file with the old client to/root/grpc/evolve/04-old-reads-v2.txt. /opt/app/grpc/renumbered.binis the same order made with/opt/app/grpc/renumbered.proto(a schema with the numbers reassigned). Save the output of reading it with the old client to/root/grpc/evolve/05-renumbered.txt, and below it write one linetruth: id=<true-id> qty=<true-qty>and one linewhy: <why-no-error>.- With pbmini, make an order with
id=9002, currency="KRW"andqty=0in two ways./root/grpc/evolve/06-zero.binuses implicit presence (0 is not sent), and/root/grpc/evolve/06-zero-explicit.binuses explicit presence (0 is sent too). In/root/grpc/evolve/06-presence.txt, write four lines:implicit=<hex>explicit=<hex>implicit_qty=<qty-read-by-old-client>explicit_qty=<qty-read-by-old-client>. - Write the final
/root/grpc/evolve/Order.proto. It includesoptional int32 qty = 2;,reserved 3;+reserved "note";,unit_priceandcurrency, andmessage Customer { string name = 1; int32 tier = 2; }and Order'sCustomer customer = <new-number>;. - Write a markdown table in
/root/grpc/evolve/compat.md. The first column is the six keysadd-new-numberremove-and-reservereuse-numberrenumberint32-to-stringrename-only, the second column issafeorunsafe, and the third column is a one-line reason. - Start the incident with
incident gc-zero-dropped— at three warehouses, only the updates that set stock to 0 are not applied. Determine the cause for each warehouse, write<warehouse>: <code>andevidence: <tap-line>to/root/incident/gc-zero-dropped/report.txt, fix it so that 0 reaches the merger, and merge again withgz-sync.
Notes
- Source of the rules: proto3 language guide — updating a message type, field presence.
old_client.pyis a program with the v1 schema hard-coded. Its output is four lines:id=qty=note=unknown=. In step 6, have it read each of the two files.- Using pbmini:
import sys; sys.path.insert(0, "/opt/app/grpc"); import pbmini as pb; pb.encode_message([(1, "int32", 9001), (2, "int32", 2)]). - The grader reads .proto leniently (comments and whitespace are free). But number reuse, a missing reserved, and type changes always fail.
- The
rename-onlyin step 8 is on the basis of the binary wire — there are no names on the wire, so changing only a name has no effect on the bytes (it is different for JSON and text formats). - Step 9 is an incident. The "sent 0 versus did not send it" you saw in step 6 splits, in the field, anywhere among the schema, the sending code, and an intermediate hop. The merger and relays are programs that are already deployed and cannot be changed, and aligning
patches.json,base.json, orstock.jsonby hand is not a fix. An update that changes only the price must not overwrite the quantity with 0 either. This lab takes about 60 minutes, and your work in/rootdisappears when the session ends, so extend the session if it runs long.
Read the v1 bytes with the old client
Save the four lines of output of python3 /opt/app/grpc/old_client.py /opt/app/grpc/order_v1.bin as they are to /root/grpc/evolve/01-old.txt.
Direction: the old client does not reread the .proto; it reads believing numbers 1, 2, and 3 are id, qty, and note. You keep its output as a record.
Tools: creating a directory and the shell redirection >.
Specifics: do not copy it by hand; save the output as it is — the grader runs the same command again and compares.
New fields get new numbers
In /root/grpc/evolve/v2.proto, keep v1's three fields as they are and add int64 unit_price and string currency with new numbers.
Direction: do not change a single character of the number, name, or type of an existing field, and give a new field a number that has never been used so far.
Tools: proto3 syntax — syntax = "proto3";, message Order { <type> <name> = <number>; }.
Specifics: 19,000–19,999 is a range reserved by the implementation and cannot be used, and 1–15 take a one-byte tag, which is good for frequently used fields. unit_price is int64 and currency is string.
Seal the deleted number
In v2.proto, delete note and leave reserved 3; and reserved "note";.
Direction: deleting itself is safe on the wire. The danger is someone using that number again later, and reserved is the device that makes protoc reject that.
Tools: reserved <number>; and reserved "<name>";.
Specifics: a number and a name cannot be mixed in one statement, so you write two lines. They may be anywhere inside the message block.
The old client reads the v2 bytes
In /root/grpc/evolve/make_v2.py, use pbmini to encode id=9001, qty=2, unit_price=15000, currency="KRW" with the numbers of v2.proto and save it to /root/grpc/evolve/order_v2.bin, then save the output of reading it with the old client to /root/grpc/evolve/04-old-reads-v2.txt.
Direction: fields added with new numbers are unknown fields to the old client — it learns their length from the wire type and skips them, and reads the fields it knows correctly.
Tools: pbmini's encode_message([(field, kind, value), ...]) and the old client.
Specifics: use the numbers exactly as you chose them in the step 2 v2.proto, and do not include the deleted note. The two new numbers must appear on the unknown= line of the old client's output.
The old client reads the bytes with swapped numbers
Save the output of reading /opt/app/grpc/renumbered.bin with the old client to /root/grpc/evolve/05-renumbered.txt, and append two lines, truth: id=<true-id> qty=<true-qty> and why: <why>.
Direction: someone renumbered the fields. The wire has only numbers, so the old client still reads number 1 as id.
Tools: /opt/app/grpc/renumbered.proto, the old client, and the values you read yourself according to the numbers in renumbered.proto.
Specifics: renumbered.bin is the same order serialized with that schema. The wire type is the same VARINT, so there is no error or warning — the truth line is the real values read according to renumbered.proto.
Was qty=0 sent or not sent
Make an order with id=9002, currency="KRW", qty=0 as /root/grpc/evolve/06-zero.bin (implicit presence, 0 is not sent) and /root/grpc/evolve/06-zero-explicit.bin (explicit presence, 0 is sent too), and in /root/grpc/evolve/06-presence.txt write four lines: implicit= explicit= implicit_qty= explicit_qty=.
Direction: a proto3 unlabeled scalar does not serialize its default value (0, the empty string). With optional, 0 is sent too.
Tools: make two messages with pbmini, look at the bytes with .hex(), and have the old client read each.
Specifics: the explicit side carries the two extra bytes 10 00 of a field 2 record. Yet the old client reads qty=0 in both cases — that inability to tell them apart is why optional is needed.
The final schema with optional and reserved
Write /root/grpc/evolve/Order.proto — optional int32 qty = 2;, reserved 3; + reserved "note";, int64 unit_price and string currency, message Customer { string name = 1; int32 tier = 2; } and Order's Customer customer = <new-number>;.
Direction: optional separates 0 from "absent", reserved seals deleted numbers, and a submessage uses a separate message as its type.
Tools: the optional label, reserved, message Customer { ... } and a field in Order that uses that type.
Specifics: the number of customer must be one that has never been used and is not reserved. On the binary wire, attaching the label itself does not change the bytes (it only means that 0 gets sent).
Write the compatibility table
Write a markdown table in /root/grpc/evolve/compat.md. The first column is add-new-number remove-and-reserve reuse-number renumber int32-to-string rename-only, the second column is safe/unsafe, and the third column is the reason.
Direction: this step sums up what you saw first-hand in this lab — the evidence for each row is one of the earlier steps.
Tools: the results of step 4 (adding with a new number), step 3 (delete and reserve), and step 5 (renumbering), and the wire type table.
Specifics: for a type change, if the wire type changes (VARINT → LEN), an old parser reads it with a different meaning; changing only a name leaves the bytes the same, because there are no names on the binary wire. Use only one verdict word, safe or unsafe — the grader looks at the first one that appears in a row.
Incident: only updates that set the value to 0 disappear
Run incident gc-zero-dropped. At three warehouses (icn, pus, gmp), only the updates that set stock to 0 are not applied — other updates are fine and there are no errors. Determine the cause for each warehouse and write three <warehouse>: <code> lines to /root/incident/gc-zero-dropped/report.txt. The code is one of implicit-presence · merge-skips-zero · omit-empty · relay-reserialize · wrong-number. For each warehouse, attach one line from the gz-tap output before fixing that shows where the 0 disappeared (the stage=sent or stage=recv line of the update that set the value to 0) as evidence: <tap-line>. If you grade with a wrong code three times, the incident is locked and you must start it again (the causes are chosen anew). Then fix the sender side (/srv/gz/<warehouse>/producer.*) so that 0 reaches the merger, and merge again with gz-sync.
Direction: the symptom is the same for all three, but the hop where 0 disappears differs — does the sender never put it in at all (no 10 00 in the sent bytes), or is it put in and lost along the way (present in what was sent, absent in what was received).
Tools: the two lines stage=sent and stage=recv of gz-tap <warehouse> side by side, the qty label in producer.proto, and omit_empty and route in producer.conf. The schemas of the two relay versions are in /opt/fixtures/grpcc/gz/.
Specifics: if it was not put in, check whether it is the schema (the label) or the configuration (the omission list); if it disappeared, check which relay version it passed through. After fixing, you must run gz-sync again for stock.json to change, and also check that an update that changed only the price does not overwrite the quantity with 0.