books / ddia / ch-08SHEET 8 / 11 · REV ASIGN IN
MODULE 8 · DESIGNING DATA-INTENSIVE APPLICATIONS CH 8

Everything Fails: Clocks, Partitions, and Lies

Build the hostile network yourself, then fix every lie it tells your cluster.

The idea

On one machine, things either work or crash. Across machines, they lie. A node can be up but unreachable, a request can be received and executed even though the reply never came back, and "no answer" is the only failure signal you ever get. That is partial failure: something is broken and you cannot tell what. Every hard problem here is a costume worn by that one fact.

A timeout is not a detector, it is a guess with a deadline. There is no correct number to wait, because the network gives you no bound on delay. Too short and you declare healthy-but-slow nodes dead, promote a second leader, and get split brain. Too long and every real failure stalls you. The honest move is to measure real round-trip times and their jitter, and set the threshold from data.

The clock on the wall is not a fact either. Cheap crystals plus NTP means two nodes can disagree by hundreds of milliseconds, and clocks get stepped backwards during sync. Last-write-wins conflict resolution then quietly throws away the genuinely newer write, with no error anywhere. The fix is to stop asking what time it is and start asking what happened before what, using a counter that rides along on every message.

And a process can stop dead for twenty seconds and never notice. A GC pause, a VM migration, a SIGSTOP: from outside the process vanished, from inside no time passed. A leader that wakes from such a pause still believes it leads and keeps writing. You cannot fix that from inside the zombie, so you fix it at the receiver: every leadership epoch gets a number, storage remembers the highest it has seen, and older writes bounce.

The bench — 4 exercises

EX 01

The Chaos Proxy

Write a small TCP relay that all cluster traffic routes through, with a control port that can delay, drop, duplicate, reorder and partition links. Owning the fault injector is what makes the rest of the module possible.

  1. Write a relay that listens on one local port per cluster link and forwards to the configured upstream
  2. Add a control port speaking a line protocol: delay, drop, dup, reorder, partition, heal, seed
  3. Buffer whole application messages (your length-prefixed frames), not raw bytes, and hold them in a per-link queue with release timestamps
  4. Point every inter-node connection in your cluster at the proxy instead of at each other
  5. Seed the randomness and print the seed on every run so any failure is reproducible
hint

One queue with release timestamps gives you delay and reorder from a single mechanism.

hint

Never try to reorder inside a TCP stream byte-by-byte; the frame is the unit.

DONE WHEN

· Echo traffic through a delayed link shows the added latency you asked for

· A 20% drop setting loses roughly 20% of 10,000 sequence-tagged messages

· Partition stops traffic entirely; heal resumes it with byte-identical payloads

· The same seed reproduces the exact same drop and reorder pattern

EX 02

The Timeout Lab

Cause split brain on purpose with a fixed failure-detection timeout, then earn a timeout that is derived from measurement instead of guessed.

  1. Set your cluster's leader-death timeout to a fixed 150ms and inject 300-800ms of jittery delay on the leader's links
  2. Drive writes to both the old and new leader and record the divergence you just created
  3. Replace the fixed timeout with a sliding window of measured heartbeat round-trip times per peer
  4. Set the threshold from observed mean plus a multiple of the jitter, and re-run the same seeds
  5. Separately kill the leader for real and confirm the adaptive detector still notices quickly
hint

Measure RTT through the proxy, not on a side channel; the delayed path is the one you are defending against.

hint

A burst of delayed-then-released heartbeats can poison the window, so think about how you weight recent samples.

DONE WHEN

· With the fixed timeout you can show a window where two nodes both accept writes

· With adaptive timeouts, dozens of seeded sub-second delay schedules cause zero false failovers

· A real kill -9 of the leader is still detected within an order of magnitude of the observed RTT ceiling

EX 03

The Clock-Skew Lab

Watch last-write-wins silently eat a newer write on a node with a skewed clock, then replace wall-clock ordering with Lamport timestamps.

  1. Route all time reads in your cluster through a single time provider, then start node B with a -200ms offset
  2. Write x=1 through node A, then strictly later write x=2 through skewed node B, and read x everywhere
  3. Confirm the newer write vanished with no error logged anywhere
  4. Implement Lamport timestamps: increment a counter on every local event, stamp every message, and on receive set counter to max(local, received) + 1
  5. Compare (lamport, node_id) instead of wall time in conflict resolution and re-run the poison sequence
hint

The classic bug is forgetting to advance your counter on receive; without it the fastest writer always wins.

hint

Keep the wall-clock path behind a flag; the side-by-side comparison is the lesson.

DONE WHEN

· With wall-clock LWW and skew injected, you can reliably demonstrate a silently lost write

· With Lamport ordering, the causally later write wins regardless of clock offset

· Randomized interleavings across skewed nodes leave every node converged on the same value

· Truly concurrent writes pick the same winner everywhere via a deterministic node-id tiebreak

EX 04

The Zombie Leader

Pause your leader mid-write-storm, let the cluster elect a replacement, then wake the zombie and stop its writes with fencing tokens enforced at the receiver.

  1. Give each elected leader a numbered lease epoch that increments on every leadership change
  2. SIGSTOP the leader during a write storm, wait for failover, then SIGCONT it and watch it keep replicating
  3. Attach the leader's epoch to every replicated write as a fencing token
  4. Have every storage node record the highest token it has accepted and reject anything lower, atomically with the write
  5. Re-run many randomized pause schedules varying pause target, duration and timing relative to failover
hint

Followers must check tokens too; a zombie replicating to an unchecked follower is the leak.

hint

Check-then-write with a gap is a race; the token check and the write must be one atomic step.

hint

Stretch: turn on duplicate delivery for client acks, show a double-applied withdrawal, and kill it with idempotent request IDs.

DONE WHEN

· With fencing off, you can catch at least one zombie write applied after the new leader's conflicting write

· With fencing on, no acknowledged write is ever lost across the randomized pause schedules

· No stale-epoch write is applied on any node

· Every schedule ends with exactly one live leader, the zombie having stepped down

Go deeper (after the bench)

Read DDIA chapter 8 now that you have caused these failures yourself; the fencing-token figures are exercise four in two pictures, and "Knowledge, Truth, and Lies" argues the majority-is-truth point at full strength. Then read one write-up from Kyle Kingsbury's free Jepsen analyses (jepsen.io/analyses) and watch a real production database lose data in exactly the ways your proxy just did.