When Old and New Versions Run Together

In one line

Compatibility accidents that occur while the old and new versions run side by side raise no errors. So diagnosis starts not from the logs but from reading the same bytes with two schemas and comparing them, and it goes down in this order: the mismatched name → that name's number → that record in the bytes → that number in the two .proto files. The fix is always on the new version's side.

Why this was needed

In the earlier modules you learned the rules — do not change numbers, seal deleted numbers with reserved, and change types only among those compatible on the wire. But what comes looking for you in the field is not a rule violation but a symptom. "The settlement numbers have been off since yesterday, and the batch finished successfully." There is no exception and no warning, so no amount of digging through logs turns it up. As the best practices document says, clients and servers are not deployed at the same moment, and a batch built months ago still runs today. That batch carries the .proto from when it was built and does not know what the server has changed into.

How it works

Two processes, one set of bytes. The new-version server looks up database values by field name and writes them with the numbers and types its own .proto defines. The old-version client reads those bytes by number and attaches the names from its own .proto. Names exist only in the two .proto files, not on the wire. So if you read the same response with two clients and compare by name, the mismatched names are exactly where the accident is. This lab's server puts the gRPC message frame — a 1-byte compression flag, a 4-byte big-endian length, the message — directly on TCP. Real gRPC puts this frame in HTTP/2 DATA frames (gRPC over HTTP/2). TCP does not know message boundaries, so it is the client's job to keep reading until the full length has arrived.

Five ways to be wrong without an error. You meet four of them in the lab's demo (field-renumbered you saw in an earlier module), and all five in the incident.

Code	What v2 did	What the old version sees
enum-renumbered	Inserted a new value at 1 in the enum and pushed the rest back	Names shift by one (PAID → SHIPPED)
field-renumbered	Swapped the numbers of two fields of the same type	The two values are swapped
number-reused	Deleted a string field and gave its number to a new string field	Someone else's value appears in the deleted slot
sint-changed	int32 to sint32	The same VARINT, so a ZigZagged number — 150 becomes 300, -40 becomes 79
wire-type-changed	int32 to fixed32	The wire type differs, so it skips it as an unknown field and gets 0

The difference between the fourth and fifth is interesting. The proto3 language guide states that sint32 is compatible only with sint64, not with the other integer types. Since the wire type is the same, the parser reads a changed value without any suspicion. For fixed32 the wire type itself is I32, so this lab's decoder skips that record as an unknown field — depending on the implementation it may become a parse error, which is actually the better case.

Two false alarms. Not every difference between the two versions is an accident. A field the new version deleted correctly appears to the old version as its default value, and a new value added at the end of an enum correctly appears to the old version as an unknown number. proto3 enums are open, so they hold unknown numbers as values too — what that looks like differs by language. If a test flags these two as accidents, nobody will trust that test.

From bytes to cause. Once you find a mismatched name, look up that name's number in the old version's .proto. Find that number in the dump that spreads the bytes into records without a schema, and the wire type and raw value point to the cause. Then compare the declarations at that number in the two .proto files, and it narrows down to one of the five. The new version also contains safe changes, so you do not revert every difference; you fix that one.

What it looks like in the field

The most common picture is "the new versions work fine with each other". The new-version client and the new-version server use the same .proto, so whatever the change, they read each other correctly. So the new-version team's integration tests are all green, and the problem shows up only in another team's old batch. You will want to fix the old version to match, but usually that is impossible — it is another team's binary, and if it is a mobile app, it is installed on users' devices. So the fix is to revert the new version's schema so it is compatible with the old numbers, and to keep it from happening again you put a test before deployment that runs both versions together and compares the same bytes. This test also catches what a static contract checker misses (places the tools do not compare, such as enum value numbers).

One more thing — it is common for someone to save the fixed .proto but not restart the server, so "I fixed it but nothing changed". The server compiles the schema at startup and holds onto it. Real gRPC servers are compiled with generated code, so all the more so.

What you do in the next lab

You start with a small tool that sends and receives the gRPC message frame directly over a socket. Next, you read the same response of a demo v2 server, which mixes in four harmful changes (the table above minus field-renumbered), with two clients — old version and new version — to pinpoint the mismatched names, find the numbers and wire types in the dump to attach causes, and fix it to be compatible while keeping the new features. After hardening that procedure into a test script, you diagnose and fix an incident in which three services produce the same symptom from different causes.