Backoff Strategy #

The retry strategy determines whether an operation is worth retrying. The backoff strategy determines when and how long to wait before trying again. The two can’t be separated — retry without backoff almost always makes things worse. Imagine a service that’s overloaded: every instance that gets a timeout immediately resends its request, and a hundred instances all retrying at nearly the same time produce a traffic wave far larger than the already-struggling service can handle. This is the thundering herd problem — not a sudden failure, but a failure accelerated by the retries themselves. A backoff strategy is the mechanism that prevents this in a mathematical way: slowing down, spreading out, and randomizing retry timing so the downstream has room to recover. This article covers four backoff types with concrete formulas, three jitter variants with effectiveness comparisons, complete Go and Dart implementations, and integration with circuit breakers and DLQs.

The Real Problem Without Backoff #

Before getting into backoff types, it’s important to concretely understand what happens without good backoff.

sequenceDiagram
    participant C as 100 Clients
    participant B as Service B\n(overloaded)

    Note over B: Response time rises to 2000ms
    C->>B: 100 request timeouts
    Note over C: All retry IMMEDIATELY\nwithout delay
    C->>B: 100 simultaneous retries (200 total)
    Note over B: 💥 Total crash
    Note over C: On-call team woken up\nat 3 AM

sequenceDiagram
    participant C as 100 Clients
    participant B as Service B\n(overloaded)

    Note over B: Response time rises to 2000ms
    C->>B: 100 request timeouts
    Note over C: Retries scheduled\nwith jitter (100ms-600ms)
    C->>B: Requests spread out gradually
    Note over B: Has time to recover
    Note over C: All retries succeed\nwithout a crash

The thundering herd isn’t only about retries — it can also happen when many cache entries expire at the same time, when many scheduled jobs run at the same second, or when a circuit breaker reopens after cooldown and all clients try at once. Jitter is the universal solution for all these scenarios.

Four Backoff Types #

flowchart TD
    A[Choose a Backoff Strategy] --> B{"How many\nconcurrent clients?"}
    B -- "1-2 clients\nlow load" --> C["Fixed Backoff\nConstant delay"]
    B -- "A few clients\nrare failures" --> D["Linear Backoff\nLinearly increasing delay"]
    B -- "Many clients\nproduction" --> E["Exponential Backoff\nDoubling delay"]
    E --> F{"Many concurrent\nclients?"}
    F -- Yes --> G["Exponential + Jitter\n✅ Best Practice"]
    F -- Not that many, still safe --> G

1. Fixed Backoff #

The same delay every retry, no matter how many times it has already failed.

// Fixed backoff — simple but not adaptive
func calculateFixedBackoff(delay time.Duration) time.Duration {
    return delay
}

// Delay pattern: 1s → 1s → 1s → 1s → 1s
// No adaptation to downstream conditions

When it’s acceptable: small systems with one or two clients, low load, and very rare failures. Not suitable for systems with many concurrent clients because the thundering herd problem remains.

2. Linear Backoff #

The delay grows linearly — each retry adds one unit of time to the previous delay.

// Linear backoff
// Formula: delay = base_delay × attempt_number
func calculateLinearBackoff(attempt int, baseDelay time.Duration) time.Duration {
    return baseDelay * time.Duration(attempt)
}

// base = 1s: 1s → 2s → 3s → 4s → 5s
// Better than fixed, but slow growth in the early attempts
// and still prone to thundering herds without jitter

Better than fixed because it gives more recovery time after many failures, but still not aggressive enough for systems with many clients without jitter.

3. Exponential Backoff #

The delay doubles with each retry. This is the most effective algorithm for giving a downstream system time to recover — exponential growth means that after a few failures, the delay becomes long enough for real recovery.

// Exponential backoff
// Formula: delay = base_delay × 2^(attempt-1)
func calculateExponentialBackoff(attempt int, baseDelay, maxDelay time.Duration) time.Duration {
    if attempt <= 0 {
        return baseDelay
    }
    // Prevent overflow for very large attempts
    if attempt > 30 {
        return maxDelay
    }
    delay := baseDelay * time.Duration(1<<uint(attempt-1))
    if delay > maxDelay || delay < 0 { // < 0 means overflow
        return maxDelay
    }
    return delay
}

Delay pattern for base=100ms, max=30s:

Attempt	Delay
1	100ms
2	200ms
3	400ms
4	800ms
5	1,600ms
6	3,200ms
7	6,400ms
8	12,800ms
9	25,600ms
10	30,000ms (capped)

The remaining problem: all clients that fail at the same time will retry at exactly the same delays (100ms, 200ms, 400ms…). This can still cause a thundering herd even though the delays are exponential.

Exponential backoff without jitter can still cause a thundering herd. If 200 instances fail at the same time, they’ll all retry at 100ms — then at 200ms — then at 400ms — staying in lockstep. Jitter is a mandatory component, not optional.

4. Exponential Backoff + Jitter — The Best Practice #

Adding randomization to the delay naturally spreads retry timing. There are three common jitter variants, each with different characteristics.

flowchart LR
    subgraph FJ["Full Jitter"]
        F1["delay = random(0, cap)"]
        F2["Distribution: 0ms - cap\nVery spread out"]
    end
    subgraph EJ["Equal Jitter"]
        E1["delay = cap/2 + random(0, cap/2)"]
        E2["Distribution: cap/2 - cap\nPredictable floor"]
    end
    subgraph DJ["Decorrelated Jitter"]
        D1["delay = random(base, prev × 3)"]
        D2["Distribution: natural/organic\ndepends on the previous delay"]
    end

Full Jitter — the delay is completely random between 0 and the exponential value:

// Full Jitter — the most even distribution
// delay = random(0, base × 2^attempt)
func fullJitter(attempt int, baseDelay, maxDelay time.Duration) time.Duration {
    cap := calculateExponentialBackoff(attempt, baseDelay, maxDelay)
    return time.Duration(rand.Int63n(int64(cap) + 1))
}

// Example distribution for attempt 3 (base=100ms, max=30s → cap=400ms):
// could be: 12ms, 387ms, 203ms, 51ms, 341ms
// → very spread out, a thundering herd is impossible

Equal Jitter — the delay is half the exponential value plus a random half:

// Equal Jitter — more minimal than full jitter
// delay = (base × 2^attempt) / 2 + random(0, (base × 2^attempt) / 2)
func equalJitter(attempt int, baseDelay, maxDelay time.Duration) time.Duration {
    cap := calculateExponentialBackoff(attempt, baseDelay, maxDelay)
    half := cap / 2
    return half + time.Duration(rand.Int63n(int64(half)+1))
}

// Example distribution for attempt 3 (cap=400ms):
// could be: 200ms-400ms (always at least 200ms)
// → spread out but with a predictable floor

Decorrelated Jitter — the delay depends on the previous delay, producing a more natural distribution:

// Decorrelated Jitter — recommended by AWS for distributed systems
// delay = random(base_delay, min(max_delay, prev_delay × 3))
func decorrelatedJitter(prevDelay, baseDelay, maxDelay time.Duration) time.Duration {
    minDelay := baseDelay
    maxJitter := prevDelay * 3
    if maxJitter > maxDelay {
        maxJitter = maxDelay
    }
    if maxJitter < minDelay {
        maxJitter = minDelay
    }
    return minDelay + time.Duration(rand.Int63n(int64(maxJitter-minDelay)+1))
}

Comparing the three variants:

Variant	Distribution	Minimum Delay	Best For
Full Jitter	Very spread out	0ms	Large distributed systems
Equal Jitter	Moderately spread	cap/2	Systems needing predictability
Decorrelated	Natural/organic	baseDelay	AWS-style workloads

AWS and Google recommend full jitter as the default for most cases because it produces the most even distribution and most effectively prevents thundering herds.

Complete Go Implementation #

type BackoffConfig struct {
    InitialDelay time.Duration
    MaxDelay     time.Duration
    Multiplier   float64 // usually 2.0 for exponential
    Jitter       JitterType
    MaxAttempts  int
}

type JitterType int

const (
    NoJitter          JitterType = iota
    FullJitter
    EqualJitter
    DecorrelatedJitter
)

type Backoff struct {
    config  BackoffConfig
    attempt int
    prev    time.Duration
    rng     *rand.Rand
}

func NewBackoff(cfg BackoffConfig) *Backoff {
    return &Backoff{
        config: cfg,
        prev:   cfg.InitialDelay,
        rng:    rand.New(rand.NewSource(time.Now().UnixNano())),
    }
}

// NextDelay calculates the delay for the next attempt
func (b *Backoff) NextDelay() (time.Duration, bool) {
    b.attempt++
    if b.config.MaxAttempts > 0 && b.attempt > b.config.MaxAttempts {
        return 0, false // no more attempts
    }

    // Calculate the base exponential delay
    base := float64(b.config.InitialDelay) *
        math.Pow(b.config.Multiplier, float64(b.attempt-1))
    if base > float64(b.config.MaxDelay) {
        base = float64(b.config.MaxDelay)
    }

    var delay time.Duration

    switch b.config.Jitter {
    case FullJitter:
        delay = time.Duration(b.rng.Int63n(int64(base) + 1))

    case EqualJitter:
        half := base / 2
        delay = time.Duration(half) + time.Duration(b.rng.Int63n(int64(half)+1))

    case DecorrelatedJitter:
        maxJitter := float64(b.prev) * 3
        if maxJitter > float64(b.config.MaxDelay) {
            maxJitter = float64(b.config.MaxDelay)
        }
        min := float64(b.config.InitialDelay)
        delay = time.Duration(min) +
            time.Duration(b.rng.Int63n(int64(maxJitter-min)+1))

    default: // NoJitter
        delay = time.Duration(base)
    }

    b.prev = delay
    return delay, true
}

// Reset for reuse
func (b *Backoff) Reset() {
    b.attempt = 0
    b.prev = b.config.InitialDelay
}

// Usage in a retry loop
func callWithBackoff(ctx context.Context, fn func() error) error {
    bo := NewBackoff(BackoffConfig{
        InitialDelay: 100 * time.Millisecond,
        MaxDelay:     30 * time.Second,
        Multiplier:   2.0,
        Jitter:       FullJitter,
        MaxAttempts:  8,
    })

    var lastErr error
    for {
        lastErr = fn()
        if lastErr == nil {
            return nil
        }

        delay, hasMore := bo.NextDelay()
        if !hasMore {
            break
        }

        log.Warnf("attempt %d failed: %v — backing off %v", bo.attempt, lastErr, delay)

        select {
        case <-time.After(delay):
        case <-ctx.Done():
            return ctx.Err()
        }
    }

    return fmt.Errorf("all attempts exhausted: %w", lastErr)
}

Dart/Flutter Implementation #

// Backoff calculator in Go — used in an HTTP client or retry middleware
type ExponentialBackoff struct {
    initialDelay time.Duration
    maxDelay     time.Duration
    multiplier   float64
    useJitter    bool
    maxAttempts  int
}

func NewExponentialBackoff() *ExponentialBackoff {
    return &ExponentialBackoff{
        initialDelay: 100 * time.Millisecond,
        maxDelay:     30 * time.Second,
        multiplier:   2.0,
        useJitter:    true,
        maxAttempts:  5,
    }
}

func (b *ExponentialBackoff) Calculate(attempt int) time.Duration {
    // Base exponential delay
    base := float64(b.initialDelay) * math.Pow(b.multiplier, float64(attempt-1))
    capped := math.Min(base, float64(b.maxDelay))

    if !b.useJitter {
        return time.Duration(capped)
    }

    // Full jitter: random between 0 and capped
    jittered := rand.Int63n(int64(capped) + 1)
    return time.Duration(jittered)
}

func (b *ExponentialBackoff) HasMoreAttempts(attempt int) bool {
    return attempt <= b.maxAttempts
}

// Usage in the repository layer
type OrderRepository struct {
    client  *http.Client
    backoff *ExponentialBackoff
}

func (r *OrderRepository) FetchOrder(ctx context.Context, orderID string) (*Order, error) {
    var lastErr error

    for attempt := 1; attempt <= r.backoff.maxAttempts; attempt++ {
        order, err := r.client.Get(ctx, "/orders/"+orderID) // pseudo — your HTTP client call
        if err == nil {
            return order, nil
        }
        if !isRetryable(err) {
            return nil, err // non-retryable error — return immediately
        }
        lastErr = err

        if !r.backoff.HasMoreAttempts(attempt + 1) {
            break
        }

        delay := r.backoff.Calculate(attempt)
        log.Printf("[Backoff] attempt %d failed, retrying in %v", attempt, delay)
        select {
        case <-time.After(delay):
        case <-ctx.Done():
            return nil, ctx.Err()
        }
    }

    return nil, fmt.Errorf("all attempts exhausted for order %s: %w", orderID, lastErr)
}

Backoff and HTTP 429 (Rate Limit) #

HTTP 429 (Too Many Requests) is a special case — the server usually includes a Retry-After header telling you when it’s OK to retry. A good backoff must respect this header.

flowchart TD
    A["Response 429\nToo Many Requests"] --> B{"Has a\nRetry-After header?"}
    B -- "Yes, seconds format" --> C[delay = Retry-After seconds]
    B -- "Yes, HTTP date format" --> D[delay = target time - now]
    B -- "No header" --> E["Normal backoff\nbut more conservative\nbase × 5"]
    C --> F[Wait then retry]
    D --> F
    E --> F

// Backoff that respects the Retry-After header
func getRetryDelay(resp *http.Response, attempt int, cfg BackoffConfig) time.Duration {
    if resp != nil && resp.StatusCode == 429 {
        // Check the Retry-After header — two possible formats
        if retryAfter := resp.Header.Get("Retry-After"); retryAfter != "" {
            // Format 1: seconds (integer)
            if seconds, err := strconv.Atoi(retryAfter); err == nil {
                return time.Duration(seconds) * time.Second
            }
            // Format 2: HTTP date
            if t, err := http.ParseTime(retryAfter); err == nil {
                remaining := time.Until(t)
                if remaining > 0 {
                    return remaining
                }
            }
        }
        // No header — use normal backoff but more conservative
        return calculateExponentialBackoff(attempt, cfg.InitialDelay*5, cfg.MaxDelay)
    }

    // For other errors — use normal backoff with jitter
    return fullJitter(attempt, cfg.InitialDelay, cfg.MaxDelay)
}

Integration with the Circuit Breaker #

Backoff and circuit breakers are two resilience mechanisms that complement each other — not alternatives.

sequenceDiagram
    participant C as Client
    participant CB as Circuit Breaker
    participant D as Downstream

    Note over C,D: Backoff Only (Without CB)
    C->>D: Attempt 1, wait 100ms
    D--xC: Failed
    C->>D: Attempt 2, wait 200ms
    D--xC: Failed
    Note over C,D: ... all attempts still happen\neven though the downstream is clearly unreachable

    Note over C,D: Backoff + Circuit Breaker (Ideal)
    C->>D: Attempts 1-3, normal backoff
    D--xC: Keeps failing
    Note over CB: High failure rate → OPEN
    C->>CB: Attempts 4-10
    CB-->>C: Fail fast — without hitting the downstream
    Note over CB: After cooldown → HALF-OPEN
    C->>CB: Try once
    CB->>D: Probe request
    D-->>CB: Success
    Note over CB: CLOSED again → normal with backoff

// Backoff + circuit breaker combination
type ResilientClient struct {
    cb      *gobreaker.CircuitBreaker
    backoff BackoffConfig
}

func (c *ResilientClient) Call(ctx context.Context, fn func() error) error {
    bo := NewBackoff(c.backoff)

    var lastErr error
    for {
        // The circuit breaker wraps every attempt
        _, cbErr := c.cb.Execute(func() (interface{}, error) {
            return nil, fn()
        })

        if cbErr == nil {
            return nil
        }

        // If the circuit breaker is open, fail fast — no need to wait for backoff
        if errors.Is(cbErr, gobreaker.ErrOpenState) {
            return fmt.Errorf("circuit open, fast fail: %w", cbErr)
        }

        lastErr = cbErr
        delay, hasMore := bo.NextDelay()
        if !hasMore {
            break
        }

        select {
        case <-time.After(delay):
        case <-ctx.Done():
            return ctx.Err()
        }
    }

    return fmt.Errorf("exhausted after %d attempts: %w", bo.attempt, lastErr)
}

Anti-Patterns to Avoid #

// ✗ Backoff without jitter in a system with many instances
delay := baseDelay * time.Duration(1<<attempt) // all instances have the same delay
// ✓ Always add full jitter

// ✗ No max delay — the delay can grow absurdly long
delay := 100 * time.Millisecond * time.Duration(1<<attempt)
// attempt 20: ~100 seconds, attempt 30: ~100 million seconds
// ✓ Always set a maxDelay

// ✗ Ignoring context cancellation — thread leak when the user cancels a request
time.Sleep(delay) // can't be cancelled
// ✓ Use a select with ctx.Done()
select {
case <-time.After(delay):
case <-ctx.Done():
    return ctx.Err()
}

// ✗ Backoff for all errors, including non-retryable ones
// A validation error (400) won't change no matter how long you wait
// ✓ Always classify errors before deciding on backoff

// ✗ Resetting the backoff counter after one success within a batch
// If there's 1 success among many failures, the attempt counter resets
// → backoff returns to the initial delay, the thundering herd can return
// ✓ Backoff state per operation, not per batch

Production-Ready Backoff Checklist #

CONFIGURATION:
  □ Initial delay matches the SLA — not too short (< 50ms for external APIs)
  □ Max delay set — don't let backoff grow without bound
  □ Multiplier of 2.0 (the exponential standard)
  □ Max attempts set (recommendation: 3-8 depending on context)
  □ Jitter enabled — full jitter as the default

ERROR CLASSIFICATION:
  □ Only transient errors are retried (5xx, timeout, 429)
  □ Deterministic errors fail fast (4xx except 429)
  □ HTTP 429 respects the Retry-After header when present

INTEGRATION:
  □ Context cancellation respected between retries
  □ Circuit breaker integrated to prevent futile retries
  □ DLQ available as a safety net after all retries are exhausted

OBSERVABILITY:
  □ Every attempt logged with the attempt number and delay used
  □ Metrics: attempt count, backoff exhausted, success after retry
  □ Alerts if the backoff exhausted rate exceeds the threshold

Summary #

Backoff strategies determine when and how long to wait before retrying — the complement of retry strategies, which determine whether a retry is worth it.
The thundering herd problem happens when many clients retry simultaneously, worsening the condition of an already-overloaded downstream — backoff + jitter is the primary solution.
Four backoff types: fixed (not adaptive), linear (slow growth), exponential (effective), exponential + jitter (best practice for production).
Three jitter variants: full jitter (most even distribution, recommended), equal jitter (predictable floor), decorrelated jitter (natural/organic, AWS-style).
Full jitter is the best default because it produces the most even distribution and most effectively prevents thundering herds in systems with many concurrent clients.
HTTP 429 must be respected — use the Retry-After header when present, don’t use a normal backoff that may be too fast.
Backoff + circuit breaker is the ideal combination: backoff for transient failures, circuit breaker for fail-fast when the downstream is clearly unavailable.
A max delay is mandatory — without an upper bound, backoff can grow to hundreds of seconds, causing resource leaks and terrible user experience.
Context cancellation must be respected between every attempt — use a select with ctx.Done(), not a plain time.Sleep.

← Previous: Retry Strategy Next: DLQ →