Wire Formats That Survive Change
Data outlives the code that wrote it, so the bytes you design today have to be readable by a version of your program that does not exist yet.
The idea
Inside your program, data lives in structs and objects. The moment it leaves for a file, a socket, or another process, it has to become a flat run of bytes that some other program turns back into meaning. That other program is often just a future or past version of yours, which is what makes encoding harder than it looks.
Compact binary formats like Protocol Buffers get small by leaving field names out of the bytes entirely. The schema assigns each field a small number, its tag, and the wire carries only tag, type, and value. Numbers shrink with varints, where small values take one byte, and strings and nested records carry a length prefix so a decoder can walk the buffer without delimiters.
Avro's sharper idea is that the encoder and decoder need compatible schemas, not identical ones. The decoder holds the writer's schema and its own reader schema and resolves between them: shared fields translate, writer-only fields get skipped, reader-only fields come from defaults, renames heal through aliases. Schema change stops being an outage and becomes a lookup.
You need compatibility in both directions because of how software actually ships. During a rolling upgrade you replace servers one at a time, so old and new versions run side by side reading each other's bytes, sometimes for years when old rows and queued messages are involved. That gives you mechanical rules, and since humans forget rules under deadline, you end up putting a machine at the gate that rejects a breaking schema before it deploys.
The bench — 4 exercises
Bytes by Hand
Write an encoder and decoder for a small binary format from the byte level up, with no serialization libraries, so the wire stops being a black box.
- Define a tiny JSON schema file format: each field has a name, a tag number, a type (bool, int64, double, string, bytes, nested record, array), and an optional default.
- Implement encoding: varints for integers with zigzag for signed values, length prefixes for strings, bytes and nested records, and fields emitted in ascending tag order so output is deterministic.
- Implement decoding by reading tag and wire type, then the value, reconstructing the record as JSON.
- Round-trip a few dozen hand-written records including 0, 127, 128, negatives, empty strings, multibyte UTF-8, three-deep nesting and arrays.
- Print a hex dump of one encoded record and annotate every byte in a comment.
hint
If negative numbers blow up to ten bytes each you forgot zigzag: map signed to unsigned first with (n << 1) ^ (n >> 63).
hint
Real protobuf does not promise deterministic bytes; you are choosing to, which is what makes byte-for-byte comparison possible.
DONE WHEN
· Every test record round-trips encode then decode with identical values.
· Encoding the same record twice produces byte-identical output.
· Your binary output is under 40% the size of the minified JSON for the same corpus.
· You can point at each byte of a hex dump and say what it is.
Two Schemas Walk Into a Decoder
Teach your decoder to take a writer schema and a reader schema and resolve between them, then build the compatibility matrix as a test grid.
- Write five successor schemas to your v1: field added with a default, optional field removed, field renamed via alias, int64 promoted to double, and a tag reused with a different type.
- Extend decode to accept both schemas: match by tag, skip writer-only fields using the wire type to know how many bytes to jump, fill reader-only fields from defaults, honor aliases, apply legal promotions.
- On an unresolvable pair, exit non-zero with a message naming the offending tag instead of emitting garbage.
- Run every writer-by-reader pairing in both directions and print the results as a grid of OK or REJECT.
- Add a fixture where an unknown field sits between two known fields and confirm it still decodes correctly.
hint
Resolution matches on tags; aliases exist to heal names. If your alias code compares tags you have built a no-op.
hint
Skipping unknown fields by wire type is exactly what makes forward compatibility possible later.
DONE WHEN
· The grid shows the verdict you predicted for all twelve pairings before you ran it.
· The reused-tag pairing is a clean rejection, never a silent misparse.
· Rejection messages name the field and tag at fault.
The Rolling Upgrade
Freeze a v1 binary, evolve the protocol to v2, and prove all four client-server version pairings interoperate through a live mid-run swap.
- Build a tiny localhost TCP key-value server and client whose get, put and delete requests and responses are themselves schema-encoded records.
- Copy the built v1 binaries and their schemas into a frozen directory and never touch them again.
- Evolve the protocol: add ttl_ms to put with a default of no expiry and error_code to responses, and update your live code to speak v2.
- Run frozen v1 and live v2 servers on two ports and drive all four pairings with the same functional suite.
- Kill the v1 server mid-run and start v2 on the same port while a client keeps sending, retrying connection errors for a few seconds.
hint
If v1 to v2 works but v2 to v1 does not, your frozen old server is choking on the unknown ttl_ms field. That is forward compatibility, and you cannot patch a frozen binary.
hint
Transient connection errors during the swap are fine; corrupted or wrong answers are not.
DONE WHEN
· All four version pairings pass the same functional suite: puts are visible to gets, deletes delete.
· v2-only features degrade silently against v1 peers instead of crashing.
· Across the mid-run swap no client ever receives a wrong or garbled response.
· Diffing frozen and live schemas shows the protocol genuinely changed.
The Gatekeeper
Break your own format on purpose to watch a silent misread happen, then write the schema-versus-schema checker that would have blocked it before deploy.
- Change ttl_ms from varint to double without changing its tag, rerun the mixed-version suite, and capture the hex dump and the garbage value v1 decoded.
- Write compatcheck old.json new.json that reads only schemas, never data, and prints FULL, BACKWARD_ONLY, FORWARD_ONLY or BREAKING plus one reason line per violation.
- Encode the rules: added field without a default is forward-only, removing a field that had no default is backward-only, tag type change or tag reuse is breaking, alias-covered rename and legal promotions are fine.
- Assemble roughly twenty old-and-new schema pairs, label each verdict yourself first, then score your checker against your labels.
- Wire compatcheck into your build so the interop suite refuses to start when a breaking change is present.
hint
If you are tempted to encode sample records and try decoding them, you have written a test, not a checker. The point is rejecting a change when no data exists yet.
hint
Every rule here is a pure structural predicate over two schemas walked tag by tag.
DONE WHEN
· compatcheck verdicts match your own labels on every pair in the corpus.
· Running it on the frozen v1 schema versus the sabotaged one reports BREAKING and names the tag.
· Feeding a breaking schema to your build stops the interop suite from running at all.
Go deeper (after the bench)
Read DDIA Ch. 4, Encoding and Evolution (pp. 111-149): the Thrift, Protobuf and Avro byte-layout figures are exercise 01's ancestors, best read with your own hex dumps open, and the modes-of-dataflow section is the rolling upgrade you just executed. For one free companion, Martin Kleppmann's blog post "Schema evolution in Avro, Protocol Buffers and Thrift" puts the three formats' bytes and evolution rules side by side in about fifteen minutes.