Numbers outlive names

Summary

A field number outlives its name. Add a new field with a new number, seal a deleted number with reserved, and never renumber. And since proto3 scalars do not send default values (0, the empty string), you need optional to tell "sent 0" from "sent nothing".

Why this was needed

The first sentence of the best practices document is the premise of this module — clients and servers are never updated at exactly the same moment. Even if you try to deploy them together, one side can be rolled back, and somewhere in the logs there remain bytes serialized with the old schema. So the assumption "both sides have the same schema right now, so it is fine" has never held.

As we saw in the previous module, the wire has only numbers. So the safety of a schema change must be judged not by whether it compiles but by whether the meaning is preserved when an old binary reads new bytes and a new binary reads old bytes. The proto3 language guide divides changes into three categories by this criterion — wire-safe, not safe, and conditionally compatible.

How it works

Safe changes. Adding a new field is safe. Bytes made by old code are read as is by new code (the new field takes its default value), and bytes made by new code are read by old code, which passes the unknown number on as an unknown field. proto3 preserves unknown fields and includes them when serializing again — meaning an old service sitting in the middle does not erase the new fields. However, that preservation breaks if you convert to JSON or copy fields over one by one.

Deleting a field is also safe. There is one condition — do not reuse that number. The document says to put deleted numbers in a reserved list. You can reserve numbers and names together, and you cannot mix the two in a single statement.

message Order {
  reserved 3;          // 지운 note 의 번호. 9 to 11 처럼 범위도 된다
  reserved "note";     // 이름 예약은 별도 문장으로

  int32 id = 1;
  int32 qty = 2;
  int64 unit_price = 4;
  string currency = 5;
}

reserved is a promise enforced by the compiler. If someone later writes string memo = 3;, protoc rejects it. Reserving a name has no effect on the binary and is for formats such as TextProto and JSON in which names are serialized.

Unsafe changes. Changing the number of an existing field. The document defines this as "the same as deleting that field and creating a new field of the same type". The problem is that the parser has no way of knowing that. The consequences the document lists are these — time lost debugging, parsing and merging errors (and this is the best case), privacy leaks, and data corruption. Two common causes of number reuse are also listed: renumbering to make things look nice, and not reserving a deleted number.

Conditionally compatible. int32, uint32, int64, uint64, and bool can be read as one another, but values can be truncated (reading a 64-bit value as an int32 truncates it to 32 bits). sint32 and sint64 are compatible only with each other and not with other integer types — because they go through ZigZag. string and bytes only when the bytes are valid UTF-8. fixed32 with sfixed32, and fixed64 with sfixed64, as pairs. The document states firmly that this category should be used only when you can control the deployment order, and the best practices document says outright "almost never change a type". Changing int32 to string does not even fall into this category, because the wire type changes from VARINT to LEN.

Default values and presence. In proto3, an unlabeled scalar follows implicit presence — if it is the default value, it is not serialized. The default is 0 for numbers, empty for string and bytes, and false for bool. So if you send qty = 0, there is no field 2 record at all on the wire, and the receiver cannot tell "sent 0" from "did not send". The document notes that in this state there is no has_ method either.

If you attach the optional label, it becomes explicit presence. A value that was explicitly set is serialized even if it is the default (the two bytes 10 00), and you can ask whether it was set. The document recommends always attaching optional to proto3 basic types — because the path to Editions is smoother, and because in a partial update (patch) you can express "change it to 0". With implicit presence, default values are not merged, so an external mechanism such as FieldMask becomes necessary. Changing the label itself is binary compatible, but the document shows by example that if one side relies on has_, that information can disappear in a round trip through the other side.

Range of numbers. It is 1 through 536,870,911, and 19,000–19,999 is reserved for the implementation, so the compiler rejects it. Because 3 bits of the tag are used by the wire type, it is 29 bits, not 32.

What you meet in the field

The incident in the title happens like this. The order service team, while tidying the schema, changed qty to number 1 and id to number 2. The new server serialized with the new schema, and the deployment finished without trouble. But the settlement batch is a build from three months ago. That batch still read number 1 believing it to be id, and the quantity of order 7788 was counted as 7788 items instead of 3. The wire types were both VARINT, so there was not even a place for an error to occur. The lab's renumbered.bin is exactly these bytes.

The second type is the incident of "0 disappears". The inventory service sent an update with qty = 0, but because of implicit presence the field was not carried at all, and the receiver's merge logic was "overwrite only what arrived". As a result, the quantity did not change from its previous value. This is exactly the case the document warns about as "default values cannot be expressed in a partial update". If optional had been attached, the two bytes 10 00 would have been carried and the 0 would have been delivered.

The third is reuse after deletion. A team deleted note without reserving number 3, and half a year later someone else put string memo at number 3. During log reprocessing, the note of an old order was read as memo. Although the names differed, the types were the same, so again there was no error. This is why the best practices document writes "if that change was ever live, a serialized version exists somewhere in the logs".

What you will do in the next lab

You leave the old client, which has the v1 schema hard-coded in its code, as it is and change the schema to v2. You add new fields with new numbers and watch the old client skip them, and delete note while leaving reserved. Then you have the old client read renumbered.bin, in which the numbers were swapped, and record that id and qty come out switched — the incident in the title. You make qty = 0 with implicit and explicit presence and compare the bytes, and leave the final schema with optional and reserved and a compatibility table.