builds / load-balancer / stage-3SHEET 3 / 6 · REV ASIGN IN
ASSEMBLY DIAGRAM — YOUR LOAD BALANCERSCALE: LEARNING
browserthe outside worldFRONT DOOR✓ builtROTOR✓ builtPULSE MONITOR⚙ buildingSCALESstage 4INSTRUMENT PANELstage 5RELIEF VALVEstage 6
BUILTUNDER CONSTRUCTIONNOT YET IMAGINED INTO EXISTENCE
STAGE 3 · THE PULSE MONITOR

The Pulse

Real backends die, and worse, they die quietly.

What you're wiring up

Waiting for a live request to fail is detecting failure with your customers. Active health checking gets there first: every backend gets its own prober hitting GET /healthz on a timer, so the pool knows a server is gone before any user does.

The subtle part is hysteresis. One failed probe means a blip. Three in a row means dead. One good probe on a corpse means nothing; two in a row means it's back. Without thresholds the pool flaps — backends bouncing in and out of rotation on every network hiccup, which is worse than either state alone.

This is failure detection in its smallest honest form. You can never know that a remote process is dead. You can only stop hearing from it and decide, on a threshold you chose, to act as if. Note the status codes too: 502 means my backend failed me, 503 means I have nobody left to give this to.

Assembly steps

[ 01 ]
Give each backend an up/down state plus consecutive-success and consecutive-failure counters, and start one health-check goroutine per backend probing GET /healthz every --health-interval (default 500ms).
hint

Non-2xx counts as a failure, same as a connection error or a timeout.

hint

Give each probe its own short timeout — a hung probe must not stall the checker.

[ 02 ]
Apply thresholds: down after --unhealthy-after consecutive failures (default 3), up after --healthy-after consecutive successes (default 2). Log every transition.
hint

Reset the opposite counter on each probe result — these are consecutive counts, not lifetime totals.

[ 03 ]
Make the rotor skip down backends: if a slot is dead, advance to the next rather than burning a rotation turn on a corpse.
hint

Loop at most pool-size times inside Next(), then give up.

hint

Guard state reads with the same lock or atomics the checker uses for writes.

[ 04 ]
When every backend is down, the front door answers 503 Service Unavailable immediately — no queuing, no waiting for a miracle.

Go deeper (after it passes)

Up / down / (soon) draining is a state machine — HFDP M9 (The Gumball State Machine). The deeper version of what you just built is DDIA M8 (Everything Fails: Clocks, Partitions, and Lies), and it's the exact itch the Raft track scratches.