Building Resilient HTTP Clients: A Deep Dive into Retry Logic

Most distributed-system incidents aren't caused by hard failures. They're caused by transient ones handled badly.

The Problem

Any backend system that talks to the network faces the same set of small daily disruptions: a connection reset when a load balancer rotates a backend, a 429 because you exceeded a rate limit, a 503 during a rolling deployment, a DNS hiccup that clears on the next lookup, a timeout because of a noisy neighbor on a shared link.

None of these mean the request is wrong. They mean “ask again in a moment.” A system that treats them as terminal failures is choosing to break itself in response to conditions that would have resolved on their own. The right default for HTTP calls in a distributed system isn’t “fail fast.” It’s “retry intelligently, fail only when you have to.”

The hard part is making intelligently do real work.

Step 1: Classifying Errors

Most retry guides skip straight to wait strategies. That’s backwards. Before deciding how long to wait, you have to decide what to retry on. This is where naive retry logic goes wrong most often.

Errors fall into three buckets:

Retryable transient errors

Connection resets, DNS failures, socket timeouts, and a specific set of HTTP status codes:

502 Bad Gateway and 503 Service Unavailable: a proxy couldn’t reach the upstream, or the service is explicitly signaling temporary unavailability (deploys, scaling events). These are the clearest retry candidates.
429 Too Many Requests: rate limited. Honor the Retry-After header if the server sends one.
504 Gateway Timeout: the upstream didn’t respond in time. Retryable, but with a caveat: if you’re retrying 504s, your own timeout should be short (under 5 seconds, sometimes under 500ms for latency-sensitive paths). Otherwise your retry chain can stack up to minutes of total wait time.
408 Request Timeout: the server gave up waiting for the client to finish sending the request. Less common in API contexts, but worth retrying when it does appear.

A note on 500 Internal Server Error: many retry implementations include 500 by default, but a recurring 500 usually signals a bug in the upstream service, not a transient condition. If a service is consistently returning 500s, retrying just delays the inevitable failure while adding load. Include 500 in your retry set if you want, but monitor the retry success rate on 500s separately. If retries on 500 rarely succeed, remove it from the list.

Non-retryable errors

400 Bad Request, 401/403 auth failures, 404, 422 validation errors. The remote side is telling you the request itself is wrong. Retrying hammers the server with the same bad request until your retry budget runs out, adding latency to your error responses and, on systems with anomaly detection, making you look like an attacker.

Ambiguous cases

409 Conflict, 410 Gone, 425 Too Early. These need per-endpoint judgment. A 409 on a write might mean “retry with fresh state”; a 409 on an idempotency-key collision means “the request already succeeded.”

The implementation takeaway: be deliberate about what triggers a retry. Don’t retry on a bare Exception. That’s how a permanent 400 turns into a six-attempt cascade that fixes nothing.

Step 2: Checking Idempotency

Even with the right error classification, there’s a deeper question: is the operation safe to retry in the first place?

Retries are only safe when the operation is idempotent, when doing it twice produces the same result as doing it once. The HTTP spec provides a starting point: GET, HEAD, PUT, and DELETE are defined as idempotent; POST and PATCH are not. But actual idempotency depends on what the server does.

The practical split:

Reads and deterministic updates: retry freely.
Writes with an idempotency key (like Stripe’s Idempotency-Key header): the server deduplicates, so retries are safe by design.
Non-idempotent writes without a key: restrict retries to errors that are unambiguously pre-flight (DNS failure, connection refused, certificate errors). Don’t retry on read timeouts or 5xx, where the server might have processed the request before things went wrong.

A timeout is the trickiest case. When your client times out, three things might have happened: the request never reached the server, the server crashed before processing it, or it processed successfully and the response was lost. These are indistinguishable from the client’s perspective. For non-idempotent operations, this ambiguity means retrying a timeout can cause a duplicate.

The long-term answer: design all writes around idempotency keys. Then the question disappears.

Step 3: The Wait Strategy

Once you’ve decided what to retry, the question of when shapes whether your retry logic helps or hurts.

A bad strategy makes things worse than no retries at all. If every client retries immediately, you add load at exactly the wrong moment. If every client uses the same deterministic backoff, they retry in synchronized waves and prevent the recovering service from stabilizing. This is the thundering herd problem, and it has taken down more services than the original failures ever did.

The three-phase approach

We’ve found a three-phase chain works well, with each phase targeting a different failure mode:

Phase 1: Quick fixed retries (0.5s, then 1s). Most transient failures resolve well under a second: a brief socket error, a connection that needed re-establishing. Short fixed waits catch these cheaply. Why 0.5s and not 0s? A zero-delay retry usually just confirms the failure. Half a second gives the network or upstream service the minimum breathing room it needs.

Phase 2: Exponential backoff with jitter (min 2s, max 60s, 4 attempts). If two quick retries didn’t fix it, the failure isn’t a blip. It’s a sustained upstream problem. Exponential backoff gives the service real time to recover. The intervals roughly double each time: ~2s, ~4s, ~8s, ~16s. The 60s cap prevents any single wait from becoming unreasonable.

The jitter component (0-5 seconds of randomness on each attempt) deserves explanation. When a thousand clients all back off deterministically during an outage, they all retry at t+2s, then t+4s, then t+8s, creating coordinated spikes. Adding randomness breaks the synchronization. The exact form of jitter matters less than having some. (The AWS architecture blog post on this topic is the canonical reference.)

Phase 3: Give up. After the chain exhausts, raise the error and let the caller decide. A retry loop that runs forever is a denial-of-service attack on your own infrastructure.

Why this shape?

The numbers aren’t arbitrary:

Phase 1 catches ~70-80% of transient failures in under 2 seconds of total wait. Network blips, connection pool exhaustion, brief upstream restarts. These are the failures you want to absorb silently.
Phase 2 handles the remaining 20-30%: rolling deployments (typically 10-30s), autoscaling events, brief dependency outages. The exponential curve means you’re patient but not endlessly so.
Total wall time for the full chain is roughly 30-90 seconds (depending on jitter). That’s long enough to ride out most operational events, short enough that a genuinely broken dependency doesn’t leave requests hanging for minutes.

The alternative, a flat “retry N times with fixed waits,” is slower on easy cases and more aggressive on hard ones. Exactly backwards.

Step 4: Implementation

Here’s what the strategy above looks like in Python using Tenacity, though the concepts apply to any language or library:

retryer = Retrying(
    wait=wait_chain(
        # Phase 1: quick fixed retries for true transient blips
        *[wait_fixed(0.5) for _ in range(1)]
        + [wait_fixed(1.0) for _ in range(1)]
        # Phase 2: exponential backoff with jitter for sustained issues
        + [wait_exponential(multiplier=1, min=2, max=60)
           + wait_random(0, 5) for _ in range(4)]
    ),
    retry=(
        retry_if_exception_type((ConnectionError, Timeout))
        | retry_if_result(lambda r: r is not None and r.status_code in {408, 429, 502, 503, 504})
    ),
    stop=stop_after_attempt(max_retries),
    reraise=True,  # raise the original exception, not a wrapper
)

The implementation details worth noting:

wait_chain composes a sequence of waits where each attempt uses the next entry. That’s how you get the multi-phase strategy in one expression.
Combining exception and result filters with | lets you layer arbitrary retry conditions.
reraise=True preserves the original exception and stack trace instead of wrapping it. Engineers debugging a failure at 3 AM need the real traceback, not a generic retry wrapper.
Closures over decorators: if different callers need different max_retries (fewer for latency-sensitive paths, more for batch jobs), construct the Retrying instance at runtime rather than using @retry at definition time.

The specific library doesn’t matter much. The same three-phase strategy can be implemented with urllib3.util.Retry, Go’s hashicorp/go-retryablehttp, Java’s Resilience4j, or a hand-written loop. What matters is the shape of the policy: quick attempts first, exponential backoff with jitter second, hard stop third.

Step 5: Limiting Configurability

We made most of the retry strategy non-configurable from the outside. The only knob exposed is max_retries.

This is deliberate. Retry strategies sound like they should be configurable, but configurability is where retry logic goes to die. Every team tweaks the parameters slightly; six months later, every service has subtly different behavior under load, no one remembers what their config does, and the system’s aggregate retry behavior becomes unpredictable. A single well-tuned policy applied everywhere is easier to reason about, easier to monitor, and easier to change centrally.

The cases that genuinely need different behavior (a latency-sensitive endpoint that should fail fast, a batch job that should retry harder) are well-served by tuning max_retries rather than rewriting the whole strategy.

The principle: expose the knobs that should vary by use case, hide the ones that shouldn’t.

Step 6: Monitoring

When retry logic works, individual failures become invisible. That’s the point. But it also means the old alerting model (“alert on any HTTP error”) is now wrong. It fires constantly on transient errors the retry layer is silently fixing, drowning out real problems.

The correct signal is retry exhaustion: cases where all attempts failed and the error escaped the client. That’s the true error rate.

Useful metrics:

Retry exhaustion rate: the real error rate your callers experience.
Retry success rate: calls that initially failed but succeeded on retry. A healthy system has this well above zero; it means the retry layer is doing real work.
Retry attempt distribution: if the average successful call takes more attempts over time, dependencies are degrading even though the surface error rate looks stable. This is an early warning signal.
Per-status-code retry success: which status codes actually recover on retry? If 500s almost never succeed on retry in your environment, stop retrying them and save the load.

The Bigger Picture

Retry logic looks like an implementation detail. It isn’t. It’s a load-bearing piece of any system that talks to the network, which is almost every system worth writing. The difference between a fragile system and a resilient one usually isn’t one big design choice. It’s a hundred small decisions about how to handle the ordinary failures the network throws at you every day. Error classification, idempotency, backoff shape, jitter, alerting. None is exciting on its own. Together, they separate a system that quietly absorbs disruption from one that turns every upstream hiccup into a user-visible incident.