Measure Before You Build
You don't get to say a system is fast — you get to read the number off a histogram you built yourself.
The idea
Every system description promises "reliable, scalable, maintainable" the way menus promise "delicious." Reliable means it keeps working correctly when parts of it go wrong. Scalable means you have a plan, with numbers, for what happens when load grows. Maintainable means someone other than you can change it later without fear. This module is where the first two stop being adjectives and start being measurements.
Before you can talk about scaling you have to say what load means for your system: requests per second, read/write ratio, or the famous Twitter one, fan-out. Twitter's problem was never total tweet volume; it was that one celebrity tweet has to land in millions of home timelines. You can do that work when the tweet is written or when the timeline is read, and which choice wins depends entirely on the numbers, not on taste. So you build both and race them.
Averages lie. A mean of 40ms is compatible with everyone getting 40ms, or with most people getting 15ms while one in a hundred waits three seconds. Percentiles can tell those worlds apart: p50 is a typical user, p99 is your unluckiest one percent, who are often your heaviest and most valuable users. That is why service level objectives are written as "p99 under 200ms" and never as "average under 50ms."
Tails also amplify. A server handles a limited number of requests at once, so one slow request makes the fast ones behind it queue, and if a single page needs a hundred backend calls that are each one percent slow, the page is slow about sixty-three percent of the time. Rare stops being rare exactly when you scale out. The fix is not hope: it is injecting the fault on purpose, watching the collapse on your own instrument, and then engineering the response with timeouts and hedged requests.
The bench — 4 exercises
Naive on Purpose
Build a single-process, in-memory twitter-timeline service that implements the book's two fan-out strategies behind one flag, so you have a specimen worth measuring.
- Expose POST /tweet, GET /timeline/:user (newest first, limit 30) and GET /healthz over localhost HTTP, all state in memory
- Load a follow graph at boot with skewed follower counts, so a handful of celebrities have thousands of followers
- Implement read-fanout: store each tweet once per author, merge the followee lists at read time
- Implement write-fanout behind a startup flag: on every tweet, append it into a precomputed per-follower timeline list
- Write a fixed script of tweets, follows and reads, and run it against both modes
hint
Resist making the write path smart. Tweet arrives, loop over followers, append to each list. The celebrity tweet doing 5,000 appends is not a bug, it is the load parameter you are about to measure.
hint
No database, no persistence, no caching. Naive is the assignment; the point of the sheet is the instruments around it.
DONE WHEN
· Both modes return identical timeline JSON for the same fixed script, after normalizing key order
· Ordering is strictly newest-first and the limit of 30 is respected in both modes
· A user following 2,000 accounts and a tweet fanning out to 5,000 followers are both correct in both modes
Build the Stopwatch
Write your own open-loop load generator and hand-rolled histogram, then prove it is honest against a server that deliberately stalls.
- Fire requests on a fixed schedule at a target rate, never waiting for the previous response before sending the next
- Measure each request's latency from its scheduled send time, not from when your sender actually got around to sending it
- Record every response time into logarithmic HdrHistogram-style buckets of fixed memory, not a list you sort at the end
- Emit a JSON report with achieved throughput, error count and p50/p95/p99/p999 read off your buckets
- Point it at a throwaway local server that sleeps completely for 5 of 60 seconds and check what your p99 says
hint
If your sender thread falls behind schedule, that lateness IS queueing delay and must be counted. Never derive the next send time from the previous response.
hint
Sanity-check the histogram first: feed it a known stream of durations and compare your percentiles against ones computed by sorting the same list.
DONE WHEN
· Percentiles from your bucketed histogram match sorted-array percentiles within a couple of percent
· Against the 5-second stall, your report shows p99 of roughly 5 seconds rather than a healthy number
· Memory use of the generator stays flat as the run length grows
Race the Fan-outs
Sweep both modes across a grid of rates and workload mixes with your own generator, and locate the crossover point in your own numbers.
- Run a campaign over rates from 50 to 2,000 req/s for both modes, with a 90/10 read-heavy mix and a 50/50 write-heavy mix
- Discard warm-up, keep durations fixed, and save one report per cell
- Assemble the reports into a table of rate versus p99, one line per mode
- Write FINDINGS.md naming which mode wins where, the crossover rate, and the max sustainable rate before p99 crosses a 250ms objective
- Explain the mechanism in one paragraph: read-time merge cost versus write-time celebrity fan-out cost
hint
If both modes look identical, your load is too gentle. The interesting behavior lives near saturation.
hint
If nothing ever saturates below 2,000 req/s, check that your generator is genuinely open-loop and that the mode flag is not inert.
DONE WHEN
· Every grid cell has a report whose achieved throughput is within 5% of target, or is explicitly marked saturated
· Under the read-heavy mix, write-fanout's read p99 beats read-fanout's at high rate; the write-heavy mix pulls the other way
· FINDINGS.md states a crossover rate and cites at least two specific cells from your own table
One Percent Poison
Inject a 1% slow sub-operation, watch p99 collapse while p50 barely moves, then kill the tail with timeouts and hedged requests.
- Add a chaos switch that makes 1% of internal per-followee fetches sleep 1 second in read-fanout mode
- Run the generator with chaos on and record p50 and p99 before any mitigation
- In FINDINGS.md, compute why a timeline touching ~100 followees is slow roughly 63% of the time
- Add a per-sub-operation timeout with fallback: skip the straggler followee and mark the response degraded
- Add hedging: if a sub-operation has not returned by your measured p95, fire a duplicate and take the first answer
hint
Hedge at p95, not p50. Too early and you double your traffic for nothing; too late and the tail already happened. Your exercise-03 histogram tells you where p95 is.
hint
Fix the tail without touching the chaos code — the pathology has to stay reproducible on demand.
DONE WHEN
· With chaos on and mitigation off, p99 is at least ~1s while p50 is roughly unchanged from baseline
· With mitigation on, p99 comes back under 3x baseline and no response is silently wrong
· Degraded responses are explicitly flagged and stay under 2% of requests; duplicate hedge requests stay under 10%
Go deeper (after the bench)
Read DDIA chapter 1 (pp. 1-22) now — do exercise 01 first, then read before 03 and 04, because the Twitter fan-out story (pp. 11-13) and the percentile discussion (pp. 13-16) land differently when both modes are already running on your machine. Then watch Gil Tene's free talk "How NOT to Measure Latency", the definitive treatment of coordinated omission and the origin of the HdrHistogram design you hand-rolled.