Turn the Database Inside-Out
Once the log is the source of truth, every cache, index, and dashboard stops drifting and starts following.
The idea
Batch processing is honest because its input is a sealed file. Real data never seals: events keep arriving forever. Streaming is what happens when you refuse to wait for the file to close, processing events as they come and keeping your outputs continuously fresh. The hard part is keeping everything you loved about batch, determinism and repeatability, while the input never ends.
The load-bearing idea is a log. Not a queue that hands each message to one lucky consumer and forgets it, but an append-only file that every consumer reads at its own pace, remembering only its position. Messages survive being read, so a new consumer can start at zero and see all of history, a crashed one resumes from its last offset, and slow consumers never block fast ones. Split the log by key into partitions and you have parallelism with per-key ordering. That is the core of Kafka, and it is your storage engine wearing your wire protocol.
Your database has been producing a stream all along. The write-ahead log you built for crash recovery is, read forward, a perfect ordered feed of everything that ever happened. Publish it and the database stops being a silo and becomes a source: caches, search indexes, and aggregate tables all maintain themselves from the changelog, with no dual writes to keep in sync. Current state is simply the integral of the change stream, which turns "is my cache correct?" into a test you can actually run.
Two things still bite. Every stream has two clocks, the time an event happened and the time you saw it, and counting by the wrong one makes a restart look like a traffic spike. You need an explicit watermark policy for how long to wait for stragglers and what to do with the ones that arrive later still. And crashes force redelivery, so the cure is not fancier delivery but idempotent effects: commit the result and the offset in one local transaction, and a duplicate changes nothing.
The bench — 4 exercises
Build the Broker
Write a mini-Kafka as one localhost server: partitioned append-only logs with consumer groups and committed offsets. You will discover the broker is your storage engine plus your wire format, with offsets promoted to the public API.
- Implement topics as N partitions, each an append-only segment file, reusing your storage-engine segment and offset code
- Expose a length-prefixed TCP protocol with produce, consume, commit, and seek, partitioning by key hash so one key always lands in one partition
- Store per-consumer-group committed offsets durably in the broker, and assign each partition to exactly one live group member
- Reassign partitions when a member dies, detected by connection loss or heartbeat timeout
- Kill a consumer mid-batch after processing but before commit, and confirm it receives those records again on rejoin
hint
Do not build per-message acks or delete-on-read; the broker only appends and reads ranges, consumers own progress
hint
If rebalancing fights you, recompute a static round-robin assignment over live members on every membership change
DONE WHEN
· Consuming a partition from offset 0 returns records in append order, with same-key records in one partition
· Two groups consume the same topic at different paces, and a brand-new group sees full history
· A crashed uncommitted consumer visibly gets duplicate delivery, and no record is skipped after a rebalance
CDC and Materialized Views
Tail your store's WAL, publish it as a changelog, and maintain three derived views from it. The acceptance check is the chapter's slogan: delete the state, replay from zero, get the same bytes.
- Write a walpub process that tails your WAL and publishes each committed change as (key, op, value, wal-position), keyed by record key
- Make walpub resume from its recorded WAL position after a restart without dropping or double-publishing, deduping by wal-position
- Run three independent consumers off that topic: a secondary index, a top-N hot-key cache, and a per-category count/sum table
- Have each view persist locally and record the changelog offset it has applied
- Delete one view's state entirely, restart it from offset 0, and byte-compare the rebuilt file against a twin that was never killed
hint
The app writes only to the store; the log is derived from the WAL, never a second write from the app
hint
If replay is not deterministic, look for a now() call, hash-map iteration order in the serialized state, or eviction driven by wall clock instead of stream position
DONE WHEN
· Every view reflects a new write within a few seconds of it landing in the store
· The changelog contains exactly the committed WAL entries, once each, in order, including entries written while walpub was down
· A view rebuilt from offset 0 is byte-identical to the continuously running twin
Two Clocks and Exactly-Once
Compute per-minute window counts by event time with a watermark, then make the results survive twenty kill-nines by committing the result and the offset in one transaction.
- Consume a shuffled, delayed click stream and compute per-minute tumbling counts keyed by page using the embedded event timestamps
- Maintain a watermark of max event time seen minus allowed lateness, per partition, taking the minimum across partitions before emitting a window
- Implement both late-event policies behind a flag: drop, or emit a marked correction
- Write each emitted window result and the consumed offset into SQLite inside the same transaction, and resume from the SQLite offset on restart
- Run the aggregator under a seeded kill -9 loop at least twenty times and compare the final table against one clean uninterrupted run
hint
First run a naive processing-time aggregator through a forced consumer stall and watch the fake spike, so you know what you are fixing
hint
Key result inserts on (page, window_start) and use INSERT OR REPLACE so a replayed window overwrites instead of appending
hint
Committing the offset to the broker separately from the result write is the control case; run it too and watch it double-count
DONE WHEN
· Event-time counts match an offline count computed straight from the raw events, while processing-time counts spike on a stall
· Under drop, totals equal the oracle minus stragglers; under correct, final totals equal the full oracle and every correction is marked
· SQLite contents after twenty crashes are exactly equal to a single clean run
Capstone: the Unbundled Database
Wire the whole track into one running system and prove it stays correct under chaos. This is Chapter 12 executed small, with every box something you built.
- Route writes through your replicated store, feed walpub into your broker, and bootstrap a search index with a batch job over a store snapshot
- Record the changelog offset the snapshot corresponds to, then start the streaming indexer at exactly that offset
- Drive realistic writes and index queries with your load generator while your chaos proxy injects delays, partitions, and a leader pause
- Delete the search index entirely, re-run bootstrap plus catch-up from the retained log, and byte-compare against the pre-deletion index at the same offset
hint
The splice is where capstones die: starting the consumer at roughly now double-indexes, starting at 0 re-applies history
hint
If chaos runs blow the freshness budget, your consumers should reconnect and resume, not restart from scratch
DONE WHEN
· An acknowledged write is visible in search results within the lag budget under continuous load
· No acknowledged writes are lost across a leader pause and failover, and the index converges after each fault
· No record is indexed twice or missed across the batch-to-stream handoff
· The deleted-and-rebuilt index matches the original byte for byte
Go deeper (after the bench)
Read DDIA chapter 11 now, especially "Databases and Streams", which is exercise 02 in prose, and save chapter 12 for after the capstone, where it lands as a description of what you just built. Then watch Martin Kleppmann's free Strange Loop 2014 talk "Turning the Database Inside-Out" and recognize every diagram as something you ran.