TT Lab
Blog Get started
Learn Learning paths Courses

I renumbered a field and the old client silently read the wrong value

Old and New Versions Running Together — Tracing Silently Wrong Values in the Bytes

Continue in TT Lab

Goal

Have two processes, an old-version (v1) client and a new-version client, each read with its own .proto the same bytes sent by a single new-version (v2) server, and narrow down the cause in the order mismatched name → field number in the bytes → difference between the two .proto files. After hardening that process into a test script, you take on an incident in which three services produce the same symptom from different causes.

Why it matters

In the earlier labs you learned the rule that if you change a number, the old client reads it wrong. In the field you start not from the rule but from the symptom — the settlement numbers are off, yet there is not a single line in the error log. At that point, if you run the two versions side by side and read the same response with two pairs of eyes, the mismatched name is immediately visible, and if you know which number that name is to the old version, you can pick out that one record in the byte dump. Confirming with tools that names exist only in each .proto and that the bytes carry only numbers is itself the diagnostic procedure. And the fix is always on the new version's side — because in most cases the old version is already deployed and cannot be rebuilt.

Steps

  1. Start the demo v1 server on 7100 (python3 /opt/fixtures/grpcc/gcsvc.py --proto /opt/fixtures/grpcc/demo/v1.proto --message Order --data /opt/fixtures/grpcc/demo/data.json --port 7100 &), and write /root/grpc/skew/tap.py <port> <id> <out-file> using only the standard library socket and struct. It puts a GetRequest (int32 id = 1) in a gRPC message frame, sends it, and saves the response frame whole (including the 5-byte header). Save the response for id 2 to /root/grpc/skew/01-resp.bin.
  2. Start the demo v2 server on 7101 (--proto /opt/fixtures/grpcc/demo/v2.proto), and read the same id 2 with the old version (/opt/fixtures/grpcc/gcclient.py --proto /opt/fixtures/grpcc/demo/v1.proto ...) and the new version (--proto /opt/fixtures/grpcc/demo/v2.proto). For each name the two versions read differently, write one line <name> old=<old-value> new=<new-value> to /root/grpc/skew/02-diff.txt (if the new version has no such name, new=(none)).
  3. Spread the bytes into record lines with gcclient.py --dump demo --port 7101 --id 2, and for each name from step 2 write <name> field=<number-in-v1> wire=<wire-type-on-the-wire> cause=<code> to /root/grpc/skew/03-cause.txt. The code is one of enum-renumbered field-renumbered number-reused sint-changed wire-type-changed.
  4. Write a schema that keeps the new features of demo v2 (PENDING, coupon, currency) but is fixed so the old version reads correctly, as /root/grpc/skew/04-v2.proto. Start one more server with that schema and check with the two clients.
  5. Write /root/grpc/skew/skewtest.py <port> <old.proto> <new.proto> <message> <id,id,...>. It reads the same response with two schemas and, for each mismatched name, prints MISMATCH id=<id> <name> old=<value> new=<value> and exits 1; if there are none, OK on the last line · exit 0. For a name the new version deleted, it is normal if the old version reads the default value, and it is also normal for a new enum value the old version does not know to appear as a number.
  6. Start the incident with incident gc-silent-skew and read the ticket. Before fixing, determine the cause for each service and write three <service>: <code> lines to /root/incident/gc-silent-skew/report.txt, and for each service one line of gc-tap output that shows its cause, as evidence: <tap-line> (three lines).
  7. Fix the three services — the settlement batch (old version) must read every record correctly, and the new-version client must also read the v2 slots as they are. The place to fix is /srv/gc/<service>/v2.proto, and after fixing, restart with gc-svc restart <service>.

Notes

Send and receive the gRPC message frame by hand

Write /root/grpc/skew/tap.py <port> <id> <out-file> using only socket and struct, and save the response frame for id 2 (including the 5-byte header) from the demo v1 server started on 7100 to /root/grpc/skew/01-resp.bin.

Direction: both the request and the response go in the frame "1-byte compression flag + 4-byte length (big-endian) + message". The request message is a single GetRequest — the id goes in field 1 as a varint.

Tools: struct.pack(">I", n) and struct.unpack, and a small function that repeats sock.recv() until the desired length is filled. The server deliberately sends the response in pieces, so a single recv() returns only the 5-byte header.

Specifics: after receiving the full 5-byte header, receive the body for the length inside it, then join the two and write them to the file. For a nonexistent id the server sends a frame of length 0, so if the length is 0 you must not wait for a body. If you import and use skewlib or gcclient, this step fails.

The same bytes, two worldviews

Read the id 2 response of the demo v2 server (7101) with an old-version (v1.proto) client and a new-version (v2.proto) client, and for each name read differently write <name> old=<old-value> new=<new-value> to /root/grpc/skew/02-diff.txt.

Direction: there is one server and one set of bytes. The only thing that differs is the .proto each client carries, and neither raises an error.

Tools: run python3 /opt/fixtures/grpcc/gcclient.py --proto <v1-or-v2.proto> --message Order --port 7101 --id 2 twice, capture the output to files, and compare by name.

Specifics: a name the new version lacks is new=(none), but if the old version read the default value (0, the empty string) in that case, it is not a mismatch — a deleted field correctly appears to the old version as its default. Do not write names whose values are the same.

From the mismatched name to the number in the bytes

Look at the record lines of gcclient.py --dump demo --port 7101 --id 2 and the two .proto files, and for each name from step 2 write <name> field=<number> wire=<wire> cause=<code> to /root/grpc/skew/03-cause.txt.

Direction: the bytes have no names. Find "the number the old version gives that name" in v1.proto, see in the dump which wire type the record with that number was actually sent as, and finally see how the declarations of v1 and v2 differ at that number.

Tools: field=, wire=, and raw= of the dump lines, the lines with the same number in the two .proto files, and the value numbers in enum blocks.

Specifics: if the old name remains in v2 with a different number (only the number moved), suspect field-renumbered; if the old name is gone and another name uses that number, number-reused; if it is the same name but the enum value numbers shifted, enum-renumbered; if the wire type itself differs, wire-type-changed; and if it is the same VARINT but the number is about double, sint-changed (the ZigZag of 150 is 300).

Keep the new features and make the old version correct too

Write a schema that keeps demo v2's PENDING, coupon, and currency but is fixed so the old version reads correctly, as /root/grpc/skew/04-v2.proto.

Direction: change none of the meanings of the numbers the old version knows (names, types compatible on the wire, enum value numbers), and put new things on new numbers. If you revert wholesale to v1, the new version's slots disappear and it fails.

Tools: the table from step 3 — each cause has one place to revert. Start one more server with the fixed file and check with gcclient.py twice.

Specifics: put the new enum value at the end number; take the new field off the old number another name used, move it to a new number, and seal that old number with reserved. sint32 and int32, and fixed32 and int32, are not compatible on the wire.

Build a test that reads with two pairs of eyes

Write /root/grpc/skew/skewtest.py <port> <old.proto> <new.proto> <message> <id,id,...> — for each mismatched name, MISMATCH id=<id> <name> old=<value> new=<value> and exit 1; if there are none, OK on the last line and exit 0.

Direction: ask the server once, read that single set of bytes with two schemas, and compare name by name. This is the only way to catch a compatibility accident that raises no error.

Tools: call, load, decode_view, default_of, and show from /opt/fixtures/grpcc/skewlib.py (sys.path.insert(0, "/opt/fixtures/grpcc")).

Specifics: you must avoid the two false alarms — for a name the new version lacks, it is normal if the old version's value is that type's default, and it is also normal for a new enum value the old version does not know (the new version's name is not in the old enum) to appear as a number. The grader starts five never-before-seen harmful pairs (one per cause) and one compatible pair, and also mixes in nonexistent ids (empty responses) — if you compare only the two .proto files or look only at the first id, you fail.

Incident: three services, the same symptom, different causes

Run incident gc-silent-skew. After the new version was deployed, the settlement batch (old version) reads wrong values without errors from three services: orders, ledger, and stock. Before fixing, determine the cause for each service and write three <service>: <code> lines to /root/incident/gc-silent-skew/report.txt. The code is one of enum-renumbered · field-renumbered · number-reused · sint-changed · wire-type-changed. Then, for each service, attach one line of gc-tap <service> output that shows the cause, as evidence: <tap-line> (exactly one line per service). If you grade with a wrong code three times, the incident is locked and you must start it again (the causes are chosen anew).

Direction: the symptom is the same for all three, and the cause differs per service. Each v2 also mixes in safe changes (new fields, new values at the end of an enum, deprecated, declaration order), so rather than looking at every difference, you must pick out the one difference at the number the old version read wrong.

Tools: find the mismatched names with gc-old and gc-new (or the skewtest from step 5), then look up that name's number in v1.proto and that number's record in gc-tap.

Specifics: paste the evidence line whole, without altering it, from the gc-tap output before fixing (<service> id=… field=… wire=… raw=… bytes=…). It must be the line for the field number that caused it.

Incident: leave the old version as is, make the new version compatible

Fix the three services. The settlement batch (old version, /srv/gc/<service>/v1.proto) cannot be redeployed, and the new version cannot be reverted wholesale either. Fix /srv/gc/<service>/v2.proto and restart it with gc-svc restart <service>, so that the old version reads every record correctly and the new-version client also reads the v2 slots as they are. Seal the numbers of deleted old fields with reserved.

Direction: the fix is to revert the one difference you picked in step 6. If you revert the safe changes too, the new version breaks.

Tools: gc-svc restart <service> (the server reads the .proto only at startup), gc-old and gc-new, and the skewtest from step 5 run against the three services.

Specifics: old names go back to their old numbers, old types, and old enum numbers; a new field that took an old number moves to a free number, and that old number goes to reserved. Do not touch v1.proto or data.json.