STAGE 6 · THE STORM ENCLOSURE

The Storm Enclosure

Put a real key-value store on top, then try to break it with partitions, crashes, and duplicated packets.

What you're wiring up

Time to make the engine drive something real: a linearizable key-value store with Put, Append, and Get, where every operation goes through the Raft log. Reads included. Because all replicas apply the same log in the same order, they all land in the same state. Serving a read from leader-local memory feels obviously fine, and it is exactly the stale read the checker is built to catch.

Clients bring a second problem. A client whose request times out will retry, possibly at a different server, so the same operation can legitimately reach the log twice. The fix is a client ID plus a monotonic sequence number, and a per-client table of the last sequence applied, consulted in the apply loop. Exactly-once semantics built on top of at-least-once delivery, which is the only way anyone ever actually gets them.

Then the storm: rolling partitions, dropped and duplicated and reordered messages, crash reboots, five clients hammering the store, and a linearizability checker auditing the whole operation history afterwards. Survive that with a clean audit and you have built Raft.

# every op goes through the log, reads included
client → Op{ClientId: c7, Seq: 42, Type: Append, Key: k, Value: x}
server: raft.Start(op)  →  promised index 118

# wait for 118 to come back out of applyCh
118 holds a different op  →  leadership changed, tell client to retry
118 holds c7/42, dedup table says new  →  apply, then reply

Assembly steps

[ 01 ]

Build the KV server: turn every client RPC into raft.Start(Op{...}) and reply only once that op comes back out of applyCh at the index you were promised.

hint

Keep a map from log index to a waiting RPC's channel. When an index arrives, verify it is your op by ClientId and Seq — a different op there means leadership changed and the client must retry elsewhere.

[ 02 ]

Build the client: remember the last known leader, rotate through servers on timeout or wrong-leader, and stamp every op with a client ID and an increasing sequence number.

hint

Get goes through the log too. Skipping that is the single most tempting shortcut in the whole track, and the checker exists to catch it during partitions.

[ 03 ]

Add the duplicate table: per client, the last applied sequence number and its result, checked at apply time on every replica so retried ops are answered but never re-executed.

hint

Dedup belongs in the apply loop, not the RPC handler. The same op arrives via the log on servers that never saw the client's request.

[ 04 ]

Flip on the harness's chaos transport and fix whatever Sheets 1 through 5 got wrong under reordering, duplication, and delay jitter. You implement no new features here.

hint

The usual casualties are Sheet 3's truncate-only-on-conflict rule and any place you assumed a reply arrives at most once.

hint

The checker's counterexample names the exact pair of client operations that violated linearizability. Start reading there, not at the top of your code.

Go deeper (after it passes)

The blueprint goes solid ink. Read DDIA's Consistency and Consensus chapter one more time — it reads completely differently now — and open trace/linearizability.html even on a passing run, just to see the shape of your machine's history. Then go read etcd's raft package and notice how much of it you recognize.