Replicate It
Copies of your data are never updated at the same instant, and everything hard about replication lives in that gap.
The idea
You keep copies of your data on several machines for three unglamorous reasons: so it survives a machine dying, so reads can be spread across many machines, and so it can sit near its users. If data never changed, replication would be a file copy. The whole subject exists because data does change, and the copies are updated at different times.
The simplest scheme is one boss and many scribes. One replica is the leader and takes every write, records it in an append-only log, and streams that log to followers who apply the same writes in the same order. Same starting state plus same ordered changes equals same ending state. Followers serve reads. This is Postgres, MySQL, MongoDB.
Then the questions start. Does the leader wait for followers before saying done? Wait for all and one slow follower stalls everything; wait for none and a leader crash can erase writes it already acknowledged. Meanwhile a client can write to the leader, read from a lagging follower, and see the past. Three cheap routing promises tame most of that: read-your-writes, monotonic reads, and consistent prefix. None of them need locks or consensus.
Drop the leader entirely and you get the Dynamo shape: write to all replicas, count acknowledgements, and if writes get w acks and reads ask r replicas with w plus r greater than n, every read overlaps a replica holding the newest value. But now two clients can write the same key with nobody to order them. Keeping the latest timestamp silently throws data away. The honest answers are version vectors, which detect concurrency instead of pretending it away, and CRDTs, whose merge is designed so replicas converge no matter what order updates arrive in.
The bench — 4 exercises
The Follower Assembly
Turn your single-node store into one leader and two followers, all separate localhost processes, with the leader shipping its write log over TCP. Replication is crash recovery pointed at another machine.
- Give your store a simple TCP line protocol (SET, GET, DEL) and a role flag so the same binary runs as leader or follower
- Have the leader append each write to its log and stream records to connected followers; followers apply strictly in log order and reject writes with a redirect to the leader
- Have followers announce the log offset they already hold on connect, so a restarted follower resumes from there instead of recopying the database
- Add a sync-replicas K flag: the leader acknowledges a write only after K followers confirm it, with K=0 meaning pure async
- Write a driver that pushes 10k keys at speed, then kill -9 a follower mid-storm and restart it
hint
If followers diverge under a SET/DEL storm, you are applying in receipt order rather than log order. The offset is the truth; a replication log without a single order is just gossip.
hint
Expose the follower's replay offset as a protocol verb now — the next exercise needs it.
DONE WHEN
· After the write storm, both followers answer every GET identically to the leader
· A killed and restarted follower rejoins and transfers bytes proportional to missed records, not to database size
· With sync-replicas 1 and one follower paused, writes still complete; with sync-replicas 2 they block
The Lag Lab
Inject delay on one follower's link, reproduce stale-read anomalies on demand, then eliminate them with routing alone. Breaking it reliably is the first half of the exercise.
- Put a small delay proxy in front of one follower's replication link and add 2 to 5 seconds of lag
- Write a detector client that scripts write-then-read and read-read sequences and reports each anomaly it observes
- Reproduce a read-your-writes violation and a monotonic-reads violation repeatedly with a fixed seed
- Fix read-your-writes by tracking each client's last-write offset and only serving its reads from a replica that has replayed at least that far, falling back to the leader
- Fix monotonic reads by hashing each client ID to a sticky replica, then rerun the detector on the same seeds
hint
If your fix routes everything to the leader you passed the anomalies and failed the point. Compare offsets — recency is a number, not a vibe.
hint
Consistent prefix comes free here because a single leader already orders writes; it returns for real once you shard.
DONE WHEN
· Both anomalies appear on demand in at least 9 of 10 seeded runs before the fix
· After the fix, zero read-your-writes and zero monotonic-reads violations across hundreds of client sessions
· Per-node read counters show followers still serving at least 40 percent of reads
Kill the Leader
Implement timeout-based failover and then count, honestly, exactly how many acknowledged writes async replication ate.
- Add heartbeats; when a follower sees none for the timeout, promote the follower with the highest replicated offset, ties broken by lowest port
- Have the new leader announce itself with an incrementing epoch; the other follower repoints its stream and clients retry redirected writes
- Make your write driver record every write it issued and every write the cluster acknowledged
- Kill -9 the leader mid-storm across several seeded schedules and diff acknowledged writes against surviving state
- SIGSTOP the leader instead of killing it, let failover happen, then SIGCONT it and make the zombie demote itself and truncate its unreplicated log tail
hint
If the zombie corrupts state, your old leader is trusting a boolean it set for itself in happier times. Authority comes from the newest announcement epoch.
hint
Write one sentence in your README explaining why an async ack is not durability. That sentence is the module.
DONE WHEN
· A new leader serves writes within the timeout budget and the cluster reconverges, on every seeded schedule
· Every write that was replicated before the crash survives; the count of acked-but-unreplicated losses is printed, not hidden
· A woken zombie leader never merges writes it accepted after the failover
No Boss, No Problem
Three peer nodes, quorum reads and writes, and the moment you watch last-write-wins destroy data and replace it with a CRDT that merges instead of choosing.
- Run three identical peers with no roles; write every key to all three succeeding at w=2 acks, read all three succeeding at r=2, and version values with a version vector rather than a timestamp
- Implement read repair: when a read sees a stale replica, write the newer value back on the read path
- Partition one node with the proxy, verify reads and writes keep succeeding, then heal it and confirm repair brings it current
- Switch conflict handling to last-write-wins timestamps and run two concurrent writers incrementing a counter through a partition until increments are provably lost
- Implement a G-counter CRDT (one increment slot per node, merge by element-wise max, value by sum) and an LWW-element-set, exchanged on an anti-entropy tick
hint
If your G-counter over-counts, replicas are summing each other's totals. Merge is max per slot; the sum happens only when someone reads.
hint
With version vectors, concurrent writes should surface as siblings — both values returned and flagged — not one silently chosen.
DONE WHEN
· Reads and writes keep succeeding at w=2/r=2 with one node fully partitioned, always returning the newest acknowledged value
· Concurrent writers on one key produce siblings under version vectors, and measurable lost increments under LWW
· Under the same storm, all three nodes converge to the exactly correct G-counter value and identical LWW-set contents regardless of delivery order
Go deeper (after the bench)
Read DDIA Chapter 5 alongside this: "Leaders and Followers" and "Problems with Replication Lag" before exercises 1 and 2, "Handling Node Outages" before exercise 3, and "Leaderless Replication" before exercise 4 — including the multi-leader section this module only brushes past. Then watch the replication and CRDT lectures from Martin Kleppmann's free Cambridge Distributed Systems series on YouTube; the version-vector rules click immediately once you have built the toy.