Circuit Breaker #
Imagine an e-commerce system depending on an external payment gateway. One day, the payment gateway has an outage and every request to it takes 30 seconds before timing out. Without protection mechanisms, all checkout-handling threads get held hostage waiting for timeouts — 30 seconds × thousands of concurrent requests — and the entire application becomes unresponsive even though only one dependency has a problem.
Circuit breakers are a pattern inspired by the electrical world: a safety device that automatically cuts circuits when detecting overload, preventing worse damage to the broader system. In software, circuit breakers monitor failures toward a dependency — databases, external APIs, or internal services — and when failures exceed thresholds, they “open” the circuit: requests fail quickly without waiting for timeouts, giving the dependency time to recover.
This pattern is one of the foundations of resilient systems — systems able to face partial failures without total failures.
The Three Circuit Breaker States #
Circuit breakers operate in three states that transition automatically based on dependency conditions.
stateDiagram-v2
[*] --> Closed
Closed --> Open : Failure threshold reached\n(e.g. 5 failures in 60 seconds)
Open --> HalfOpen : Recovery timeout passed\n(e.g. after 30 seconds)
HalfOpen --> Closed : Probe request succeeds\n(e.g. 2 consecutive successes)
HalfOpen --> Open : Probe request fails\n(immediately back to Open)
state Closed {
[*]: Requests allowed\nFailures counted
}
state Open {
[*]: Requests rejected immediately\nNo calls to the dependency
}
state HalfOpen {
[*]: Some requests allowed\nas probes
}Details of each state:
CLOSED (Normal):
→ All requests forwarded to the dependency
→ Every failure recorded
→ If the failure rate exceeds the threshold → move to OPEN
→ If failures within the window aren't enough → counters reset
OPEN (Circuit Open — Protecting):
→ All requests fail immediately (fast fail)
→ No requests sent to the dependency
→ After the recovery_timeout passes → move to HALF-OPEN
→ Benefits: low latency (no timeouts), dependencies get recovery time
HALF-OPEN (Probe):
→ Some probe requests sent to the dependency
→ If successful → move to CLOSED (system normal)
→ If failed → back to OPEN (dependency hasn't recovered)
→ Prevents flooding dependencies just starting to recover
From-Scratch Implementations #
Understanding implementations from the ground up helps engineers configure and debug circuit breakers better.
// A circuit breaker with a sliding window for failure tracking.
// Thread-safe for concurrent use.
type CircuitState int
const (
Closed CircuitState = iota
Open
HalfOpen
)
// ErrCircuitOpen is returned when the circuit breaker is in the OPEN state.
var ErrCircuitOpen = errors.New("circuit is OPEN")
type CircuitBreakerConfig struct {
// How many failures in the window before OPEN
FailureThreshold int
// The time window for counting failures
FailureWindow time.Duration
// How long in OPEN state before trying HALF-OPEN
RecoveryTimeout time.Duration
// How many successes in HALF-OPEN before back to CLOSED
SuccessThreshold int
// How many requests allowed in HALF-OPEN per window
HalfOpenMaxCalls int
// Which errors count as failures (nil = all errors)
IsFailure func(err error) bool
}
type CircuitBreaker struct {
name string
config CircuitBreakerConfig
mu sync.Mutex
state CircuitState
// Sliding window: store the timestamp of every failure
failureTimestamps []time.Time
successCount int
halfOpenCalls int
openedAt time.Time
// Callbacks for observability
OnStateChange func(name string, from, to CircuitState)
}
func NewCircuitBreaker(name string, config CircuitBreakerConfig) *CircuitBreaker {
return &CircuitBreaker{name: name, config: config, state: Closed}
}
func (cb *CircuitBreaker) Call(fn func() error) error {
cb.mu.Lock()
cb.evaluateState()
if cb.state == Open {
cb.mu.Unlock()
return ErrCircuitOpen
}
if cb.state == HalfOpen {
if cb.halfOpenCalls >= cb.config.HalfOpenMaxCalls {
cb.mu.Unlock()
return ErrCircuitOpen
}
cb.halfOpenCalls++
}
cb.mu.Unlock()
// Execute outside the lock so other goroutines aren't blocked
err := fn()
if err != nil {
if cb.config.IsFailure == nil || cb.config.IsFailure(err) {
cb.recordFailure()
}
return err
}
cb.recordSuccess()
return nil
}
func (cb *CircuitBreaker) evaluateState() {
// Check whether the state needs changing based on current conditions.
if cb.state == Open && cb.shouldAttemptRecovery() {
cb.transitionTo(HalfOpen)
}
}
func (cb *CircuitBreaker) recordSuccess() {
cb.mu.Lock()
defer cb.mu.Unlock()
if cb.state == HalfOpen {
cb.successCount++
if cb.successCount >= cb.config.SuccessThreshold {
cb.transitionTo(Closed)
}
}
}
func (cb *CircuitBreaker) recordFailure() {
cb.mu.Lock()
defer cb.mu.Unlock()
now := time.Now()
cb.failureTimestamps = append(cb.failureTimestamps, now)
// Remove failures outside the window
cutoff := now.Add(-cb.config.FailureWindow)
for len(cb.failureTimestamps) > 0 && cb.failureTimestamps[0].Before(cutoff) {
cb.failureTimestamps = cb.failureTimestamps[1:]
}
if cb.state == HalfOpen {
// Immediately back to OPEN if probes fail
cb.transitionTo(Open)
} else if cb.state == Closed && len(cb.failureTimestamps) >= cb.config.FailureThreshold {
cb.transitionTo(Open)
}
}
func (cb *CircuitBreaker) transitionTo(newState CircuitState) {
oldState := cb.state
cb.state = newState
switch newState {
case Open:
cb.openedAt = time.Now()
cb.successCount = 0
cb.halfOpenCalls = 0
case Closed:
cb.failureTimestamps = nil
cb.successCount = 0
cb.halfOpenCalls = 0
case HalfOpen:
cb.successCount = 0
cb.halfOpenCalls = 0
}
if cb.OnStateChange != nil && oldState != newState {
cb.OnStateChange(cb.name, oldState, newState)
}
}
func (cb *CircuitBreaker) shouldAttemptRecovery() bool {
if cb.openedAt.IsZero() {
return false
}
return time.Since(cb.openedAt) >= cb.config.RecoveryTimeout
}
// FailureCount returns the number of failures in the current window.
func (cb *CircuitBreaker) FailureCount() int {
cb.mu.Lock()
defer cb.mu.Unlock()
now := time.Now()
cutoff := now.Add(-cb.config.FailureWindow)
count := 0
for _, ts := range cb.failureTimestamps {
if !ts.Before(cutoff) {
count++
}
}
return count
}
#
// A circuit breaker with a sliding window for failure tracking.
// Thread-safe for concurrent use.
type CircuitState int
const (
Closed CircuitState = iota
Open
HalfOpen
)
// ErrCircuitOpen is returned when the circuit breaker is in the OPEN state.
var ErrCircuitOpen = errors.New("circuit is OPEN")
type CircuitBreakerConfig struct {
// How many failures in the window before OPEN
FailureThreshold int
// The time window for counting failures
FailureWindow time.Duration
// How long in OPEN state before trying HALF-OPEN
RecoveryTimeout time.Duration
// How many successes in HALF-OPEN before back to CLOSED
SuccessThreshold int
// How many requests allowed in HALF-OPEN per window
HalfOpenMaxCalls int
// Which errors count as failures (nil = all errors)
IsFailure func(err error) bool
}
type CircuitBreaker struct {
name string
config CircuitBreakerConfig
mu sync.Mutex
state CircuitState
// Sliding window: store the timestamp of every failure
failureTimestamps []time.Time
successCount int
halfOpenCalls int
openedAt time.Time
// Callbacks for observability
OnStateChange func(name string, from, to CircuitState)
}
func NewCircuitBreaker(name string, config CircuitBreakerConfig) *CircuitBreaker {
return &CircuitBreaker{name: name, config: config, state: Closed}
}
func (cb *CircuitBreaker) Call(fn func() error) error {
cb.mu.Lock()
cb.evaluateState()
if cb.state == Open {
cb.mu.Unlock()
return ErrCircuitOpen
}
if cb.state == HalfOpen {
if cb.halfOpenCalls >= cb.config.HalfOpenMaxCalls {
cb.mu.Unlock()
return ErrCircuitOpen
}
cb.halfOpenCalls++
}
cb.mu.Unlock()
// Execute outside the lock so other goroutines aren't blocked
err := fn()
if err != nil {
if cb.config.IsFailure == nil || cb.config.IsFailure(err) {
cb.recordFailure()
}
return err
}
cb.recordSuccess()
return nil
}
func (cb *CircuitBreaker) evaluateState() {
// Check whether the state needs changing based on current conditions.
if cb.state == Open && cb.shouldAttemptRecovery() {
cb.transitionTo(HalfOpen)
}
}
func (cb *CircuitBreaker) recordSuccess() {
cb.mu.Lock()
defer cb.mu.Unlock()
if cb.state == HalfOpen {
cb.successCount++
if cb.successCount >= cb.config.SuccessThreshold {
cb.transitionTo(Closed)
}
}
}
func (cb *CircuitBreaker) recordFailure() {
cb.mu.Lock()
defer cb.mu.Unlock()
now := time.Now()
cb.failureTimestamps = append(cb.failureTimestamps, now)
// Remove failures outside the window
cutoff := now.Add(-cb.config.FailureWindow)
for len(cb.failureTimestamps) > 0 && cb.failureTimestamps[0].Before(cutoff) {
cb.failureTimestamps = cb.failureTimestamps[1:]
}
if cb.state == HalfOpen {
// Immediately back to OPEN if probes fail
cb.transitionTo(Open)
} else if cb.state == Closed && len(cb.failureTimestamps) >= cb.config.FailureThreshold {
cb.transitionTo(Open)
}
}
func (cb *CircuitBreaker) transitionTo(newState CircuitState) {
oldState := cb.state
cb.state = newState
switch newState {
case Open:
cb.openedAt = time.Now()
cb.successCount = 0
cb.halfOpenCalls = 0
case Closed:
cb.failureTimestamps = nil
cb.successCount = 0
cb.halfOpenCalls = 0
case HalfOpen:
cb.successCount = 0
cb.halfOpenCalls = 0
}
if cb.OnStateChange != nil && oldState != newState {
cb.OnStateChange(cb.name, oldState, newState)
}
}
func (cb *CircuitBreaker) shouldAttemptRecovery() bool {
if cb.openedAt.IsZero() {
return false
}
return time.Since(cb.openedAt) >= cb.config.RecoveryTimeout
}
// FailureCount returns the number of failures in the current window.
func (cb *CircuitBreaker) FailureCount() int {
cb.mu.Lock()
defer cb.mu.Unlock()
now := time.Now()
cutoff := now.Add(-cb.config.FailureWindow)
count := 0
for _, ts := range cb.failureTimestamps {
if !ts.Before(cutoff) {
count++
}
}
return count
}
Usage in Real Applications #
// Set up a circuit breaker with monitoring callbacks
var paymentBreaker = NewCircuitBreaker("payment-gateway", CircuitBreakerConfig{
FailureThreshold: 5,
FailureWindow: 60 * time.Second,
RecoveryTimeout: 30 * time.Second,
SuccessThreshold: 2,
})
func init() {
paymentBreaker.OnStateChange = func(name string, from, to CircuitState) {
log.Printf("Circuit breaker state change: %s %v -> %v", name, from, to)
// Send metrics to monitoring
metrics.Increment("circuit_breaker.state_change", map[string]string{
"circuit": name,
"state": fmt.Sprint(to),
})
}
}
// The function protected by the circuit breaker
func chargePayment(orderID string, amount float64) (PaymentResult, error) {
var result PaymentResult
err := paymentBreaker.Call(func() error {
resp, err := http.Post(
"https://payment-gateway.com/charge",
"application/json",
strings.NewReader(fmt.Sprintf(`{"order_id": %q, "amount": %f}`, orderID, amount)),
)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
return fmt.Errorf("payment gateway returned %d", resp.StatusCode)
}
return json.NewDecoder(resp.Body).Decode(&result)
})
if err != nil {
if errors.Is(err, ErrCircuitOpen) {
// The circuit is open — use the fallback
log.Printf("Payment circuit open for order %s", orderID)
return fallbackPaymentHandler(orderID, amount)
}
// The request failed (circuit still closed/half-open, failure recorded)
log.Printf("Payment request failed for order %s: %v", orderID, err)
return PaymentResult{}, fmt.Errorf("payment processing failed: %w", err)
}
return result, nil
}
func fallbackPaymentHandler(orderID string, amount float64) (PaymentResult, error) {
// Fallback when the payment gateway is unavailable.
// Options:
// 1. Queue for later processing (async)
// 2. Try an alternative payment provider
// 3. Return an informative error to users
// Option 1: Queue to a background job
queuePendingPayment(orderID, amount)
return PaymentResult{
Status: "queued",
Message: "The payment will be processed in a few minutes",
OrderID: orderID,
}, nil
}
#
// Set up a circuit breaker with monitoring callbacks
var paymentBreaker = NewCircuitBreaker("payment-gateway", CircuitBreakerConfig{
FailureThreshold: 5,
FailureWindow: 60 * time.Second,
RecoveryTimeout: 30 * time.Second,
SuccessThreshold: 2,
})
func init() {
paymentBreaker.OnStateChange = func(name string, from, to CircuitState) {
log.Printf("Circuit breaker state change: %s %v -> %v", name, from, to)
// Send metrics to monitoring
metrics.Increment("circuit_breaker.state_change", map[string]string{
"circuit": name,
"state": fmt.Sprint(to),
})
}
}
// The function protected by the circuit breaker
func chargePayment(orderID string, amount float64) (PaymentResult, error) {
var result PaymentResult
err := paymentBreaker.Call(func() error {
resp, err := http.Post(
"https://payment-gateway.com/charge",
"application/json",
strings.NewReader(fmt.Sprintf(`{"order_id": %q, "amount": %f}`, orderID, amount)),
)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
return fmt.Errorf("payment gateway returned %d", resp.StatusCode)
}
return json.NewDecoder(resp.Body).Decode(&result)
})
if err != nil {
if errors.Is(err, ErrCircuitOpen) {
// The circuit is open — use the fallback
log.Printf("Payment circuit open for order %s", orderID)
return fallbackPaymentHandler(orderID, amount)
}
// The request failed (circuit still closed/half-open, failure recorded)
log.Printf("Payment request failed for order %s: %v", orderID, err)
return PaymentResult{}, fmt.Errorf("payment processing failed: %w", err)
}
return result, nil
}
func fallbackPaymentHandler(orderID string, amount float64) (PaymentResult, error) {
// Fallback when the payment gateway is unavailable.
// Options:
// 1. Queue for later processing (async)
// 2. Try an alternative payment provider
// 3. Return an informative error to users
// Option 1: Queue to a background job
queuePendingPayment(orderID, amount)
return PaymentResult{
Status: "queued",
Message: "The payment will be processed in a few minutes",
OrderID: orderID,
}, nil
}
Available Circuit Breaker Libraries #
For production, it’s better to use battle-tested libraries than custom implementations.
# Python: pybreaker
from pybreaker import CircuitBreaker, CircuitBreakerError
payment_breaker = CircuitBreaker(
fail_max=5,
reset_timeout=30,
exclude=[ValueError] # these exceptions don't count as failures
)
@payment_breaker
def call_payment_api(order_id: str, amount: float):
response = httpx.post("https://payment-gateway.com/charge", ...)
return response.json()
try:
result = call_payment_api(order_id, amount)
except CircuitBreakerError:
# The circuit is open
handle_payment_unavailable(order_id)
// Go: sony/gobreaker
package main
import (
"github.com/sony/gobreaker"
"time"
)
var paymentBreaker = gobreaker.NewCircuitBreaker(gobreaker.Settings{
Name: "payment-gateway",
MaxRequests: 2, // max requests in HALF-OPEN
Interval: 60 * time.Second, // window for resetting counters
Timeout: 30 * time.Second, // recovery timeout (OPEN → HALF-OPEN)
ReadyToTrip: func(counts gobreaker.Counts) bool {
// Custom logic: open the circuit if the failure rate > 60% with a minimum of 5 requests
if counts.Requests < 5 {
return false
}
failureRatio := float64(counts.TotalFailures) / float64(counts.Requests)
return failureRatio >= 0.6
},
OnStateChange: func(name string, from, to gobreaker.State) {
log.Printf("Circuit %s: %s → %s", name, from, to)
},
})
func chargePayment(orderID string, amount float64) (PaymentResult, error) {
result, err := paymentBreaker.Execute(func() (interface{}, error) {
return callPaymentAPI(orderID, amount)
})
if err != nil {
if err == gobreaker.ErrOpenState {
return PaymentResult{}, ErrPaymentUnavailable
}
return PaymentResult{}, err
}
return result.(PaymentResult), nil
}
// Bulkhead: isolate worker pools per dependency.
// Prevents one slow dependency from exhausting all goroutines.
type BulkheadExecutor struct {
name string
sem chan struct{} // buffered channel = semaphore
workers chan func()
wg sync.WaitGroup
}
func NewBulkheadExecutor(name string, maxWorkers, queueSize int) *BulkheadExecutor {
b := &BulkheadExecutor{
name: name,
sem: make(chan struct{}, maxWorkers+queueSize),
workers: make(chan func(), queueSize),
}
for i := 0; i < maxWorkers; i++ {
b.wg.Add(1)
go func() {
defer b.wg.Done()
for fn := range b.workers {
fn()
}
}()
}
return b
}
// Submit a task to the isolated pool; returns a channel carrying the result.
func (b *BulkheadExecutor) Submit(fn func() interface{}) (chan interface{}, error) {
select {
case b.sem <- struct{}{}: // acquire
default:
return nil, ErrBulkheadFull
}
resultCh := make(chan interface{}, 1)
b.workers <- func() {
defer func() { <-b.sem }() // release
resultCh <- fn()
}
return resultCh, nil
}
// Setup: every dependency has its own worker pool
paymentPool := NewBulkheadExecutor("payment", 10, 5)
inventoryPool := NewBulkheadExecutor("inventory", 20, 10)
emailPool := NewBulkheadExecutor("email", 5, 50)
// If the payment API is slow, only 10 workers are affected
func processCheckout(order Order) (interface{}, error) {
paymentCh, err := paymentPool.Submit(func() interface{} {
return chargePayment(order)
})
if err != nil {
return map[string]string{"error": "Service busy, please try again"}, nil
}
inventoryCh, _ := inventoryPool.Submit(func() interface{} {
return reserveInventory(order)
})
select {
case paymentResult := <-paymentCh:
inventoryResult := <-inventoryCh
return finalizeOrder(order, paymentResult, inventoryResult), nil
case <-time.After(15 * time.Second):
return nil, errors.New("payment timeout")
}
}
#
# Python: pybreaker
from pybreaker import CircuitBreaker, CircuitBreakerError
payment_breaker = CircuitBreaker(
fail_max=5,
reset_timeout=30,
exclude=[ValueError] # these exceptions don't count as failures
)
@payment_breaker
def call_payment_api(order_id: str, amount: float):
response = httpx.post("https://payment-gateway.com/charge", ...)
return response.json()
try:
result = call_payment_api(order_id, amount)
except CircuitBreakerError:
# The circuit is open
handle_payment_unavailable(order_id)
// Go: sony/gobreaker
package main
import (
"github.com/sony/gobreaker"
"time"
)
var paymentBreaker = gobreaker.NewCircuitBreaker(gobreaker.Settings{
Name: "payment-gateway",
MaxRequests: 2, // max requests in HALF-OPEN
Interval: 60 * time.Second, // window for resetting counters
Timeout: 30 * time.Second, // recovery timeout (OPEN → HALF-OPEN)
ReadyToTrip: func(counts gobreaker.Counts) bool {
// Custom logic: open the circuit if the failure rate > 60% with a minimum of 5 requests
if counts.Requests < 5 {
return false
}
failureRatio := float64(counts.TotalFailures) / float64(counts.Requests)
return failureRatio >= 0.6
},
OnStateChange: func(name string, from, to gobreaker.State) {
log.Printf("Circuit %s: %s → %s", name, from, to)
},
})
func chargePayment(orderID string, amount float64) (PaymentResult, error) {
result, err := paymentBreaker.Execute(func() (interface{}, error) {
return callPaymentAPI(orderID, amount)
})
if err != nil {
if err == gobreaker.ErrOpenState {
return PaymentResult{}, ErrPaymentUnavailable
}
return PaymentResult{}, err
}
return result.(PaymentResult), nil
}
// Bulkhead: isolate worker pools per dependency.
// Prevents one slow dependency from exhausting all goroutines.
type BulkheadExecutor struct {
name string
sem chan struct{} // buffered channel = semaphore
workers chan func()
wg sync.WaitGroup
}
func NewBulkheadExecutor(name string, maxWorkers, queueSize int) *BulkheadExecutor {
b := &BulkheadExecutor{
name: name,
sem: make(chan struct{}, maxWorkers+queueSize),
workers: make(chan func(), queueSize),
}
for i := 0; i < maxWorkers; i++ {
b.wg.Add(1)
go func() {
defer b.wg.Done()
for fn := range b.workers {
fn()
}
}()
}
return b
}
// Submit a task to the isolated pool; returns a channel carrying the result.
func (b *BulkheadExecutor) Submit(fn func() interface{}) (chan interface{}, error) {
select {
case b.sem <- struct{}{}: // acquire
default:
return nil, ErrBulkheadFull
}
resultCh := make(chan interface{}, 1)
b.workers <- func() {
defer func() { <-b.sem }() // release
resultCh <- fn()
}
return resultCh, nil
}
// Setup: every dependency has its own worker pool
paymentPool := NewBulkheadExecutor("payment", 10, 5)
inventoryPool := NewBulkheadExecutor("inventory", 20, 10)
emailPool := NewBulkheadExecutor("email", 5, 50)
// If the payment API is slow, only 10 workers are affected
func processCheckout(order Order) (interface{}, error) {
paymentCh, err := paymentPool.Submit(func() interface{} {
return chargePayment(order)
})
if err != nil {
return map[string]string{"error": "Service busy, please try again"}, nil
}
inventoryCh, _ := inventoryPool.Submit(func() interface{} {
return reserveInventory(order)
})
select {
case paymentResult := <-paymentCh:
inventoryResult := <-inventoryCh
return finalizeOrder(order, paymentResult, inventoryResult), nil
case <-time.After(15 * time.Second):
return nil, errors.New("payment timeout")
}
}
Configuring Correct Thresholds #
Wrong configurations can make circuit breakers ineffective or overly aggressive.
Threshold selection guidance:
failure_threshold (how many failures before OPEN):
→ Too low (1-2): circuits open on fluke errors, overly aggressive
→ Too high (50+): too slow to detect real problems
→ Recommendation: 5-10 failures within a 60 second window
→ Or: failure rates (50-60%) with a minimum request volume
failure_window (how long failures are counted):
→ Too short (5 seconds): old failures keep piling up
→ Too long (10 minutes): slow to reset after problems finish
→ Recommendation: 60 seconds for most cases
recovery_timeout (how long OPEN before probing):
→ Adjust to the dependency's expected recovery time
→ Database restarts: ~30 seconds
→ External APIs: ~60 seconds (give them time to notice problems)
→ Recommendation: start at 30-60 seconds, adjust per dependency SLAs
success_threshold (how many successes before CLOSED):
→ Too low (1): could immediately CLOSED before stable
→ Recommendation: 2-3 for adequate confidence
Don't use the same numbers for everything:
→ Critical payment APIs: stricter thresholds, more careful recovery
→ Optional cache layers: looser thresholds
→ Fast internal services: shorter windows
The Bulkhead Pattern: Complementing Circuit Breakers #
Circuit breakers protect against cascade failures from poor quality. Bulkheads protect against cascade failures from excessive quantity — preventing one slow dependency from exhausting all threads/connections.
// Metrics to expose for every circuit breaker
type MonitoredCircuitBreaker struct {
*CircuitBreaker
// Prometheus counters and gauges
requestsTotal *prometheus.CounterVec // labels: circuit, result
stateGauge *prometheus.GaugeVec // labels: circuit
failureCountGauge *prometheus.GaugeVec // labels: circuit
}
func NewMonitoredCircuitBreaker(name string, config CircuitBreakerConfig) *MonitoredCircuitBreaker {
m := &MonitoredCircuitBreaker{
CircuitBreaker: NewCircuitBreaker(name, config),
requestsTotal: prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "circuit_breaker_requests_total",
Help: "Total requests through circuit breaker",
},
[]string{"circuit", "result"}, // result: success, failure, rejected
),
stateGauge: prometheus.NewGaugeVec(
prometheus.GaugeOpts{
Name: "circuit_breaker_state",
Help: "Current state (0=closed, 1=open, 2=half_open)",
},
[]string{"circuit"},
),
failureCountGauge: prometheus.NewGaugeVec(
prometheus.GaugeOpts{
Name: "circuit_breaker_failure_count",
Help: "Current failure count in window",
},
[]string{"circuit"},
),
}
m.OnStateChange = m.updateStateMetric
return m
}
func (m *MonitoredCircuitBreaker) updateStateMetric(name string, from, to CircuitState) {
stateMap := map[CircuitState]float64{
Closed: 0, Open: 1, HalfOpen: 2,
}
m.stateGauge.WithLabelValues(name).Set(stateMap[to])
}
func (m *MonitoredCircuitBreaker) Call(fn func() error) error {
defer func() {
m.failureCountGauge.WithLabelValues(m.name).
Set(float64(m.FailureCount()))
}()
err := m.CircuitBreaker.Call(fn)
if err != nil {
if errors.Is(err, ErrCircuitOpen) {
m.requestsTotal.WithLabelValues(m.name, "rejected").Inc()
} else {
m.requestsTotal.WithLabelValues(m.name, "failure").Inc()
}
return err
}
m.requestsTotal.WithLabelValues(m.name, "success").Inc()
return nil
}
#
// Metrics to expose for every circuit breaker
type MonitoredCircuitBreaker struct {
*CircuitBreaker
// Prometheus counters and gauges
requestsTotal *prometheus.CounterVec // labels: circuit, result
stateGauge *prometheus.GaugeVec // labels: circuit
failureCountGauge *prometheus.GaugeVec // labels: circuit
}
func NewMonitoredCircuitBreaker(name string, config CircuitBreakerConfig) *MonitoredCircuitBreaker {
m := &MonitoredCircuitBreaker{
CircuitBreaker: NewCircuitBreaker(name, config),
requestsTotal: prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "circuit_breaker_requests_total",
Help: "Total requests through circuit breaker",
},
[]string{"circuit", "result"}, // result: success, failure, rejected
),
stateGauge: prometheus.NewGaugeVec(
prometheus.GaugeOpts{
Name: "circuit_breaker_state",
Help: "Current state (0=closed, 1=open, 2=half_open)",
},
[]string{"circuit"},
),
failureCountGauge: prometheus.NewGaugeVec(
prometheus.GaugeOpts{
Name: "circuit_breaker_failure_count",
Help: "Current failure count in window",
},
[]string{"circuit"},
),
}
m.OnStateChange = m.updateStateMetric
return m
}
func (m *MonitoredCircuitBreaker) updateStateMetric(name string, from, to CircuitState) {
stateMap := map[CircuitState]float64{
Closed: 0, Open: 1, HalfOpen: 2,
}
m.stateGauge.WithLabelValues(name).Set(stateMap[to])
}
func (m *MonitoredCircuitBreaker) Call(fn func() error) error {
defer func() {
m.failureCountGauge.WithLabelValues(m.name).
Set(float64(m.FailureCount()))
}()
err := m.CircuitBreaker.Call(fn)
if err != nil {
if errors.Is(err, ErrCircuitOpen) {
m.requestsTotal.WithLabelValues(m.name, "rejected").Inc()
} else {
m.requestsTotal.WithLabelValues(m.name, "failure").Inc()
}
return err
}
m.requestsTotal.WithLabelValues(m.name, "success").Inc()
return nil
}
Monitoring Circuit Breakers #
Circuit breakers without monitoring are useless — you won’t know when or why circuits open.
// Go equivalent: a generic breaker with the same event-based API
var paymentBreaker = NewCircuitBreaker("payment-gateway", CircuitBreakerConfig{
FailureThreshold: 5, // volume threshold: min requests before calculating error rates
FailureWindow: 60 * time.Second, // window for resetting counters
RecoveryTimeout: 30 * time.Second, // resetTimeout: try again after 30 seconds
SuccessThreshold: 2,
})
// Event listeners for monitoring
paymentBreaker.OnStateChange = func(name string, from, to CircuitState) {
switch to {
case Open:
log.Println("Payment circuit OPENED")
metrics.Increment("circuit.opened", map[string]string{"circuit": "payment"})
case HalfOpen:
log.Println("Payment circuit HALF-OPEN — probing")
case Closed:
log.Println("Payment circuit CLOSED — recovered")
}
}
// Fallback: called when the circuit is OPEN
func chargePayment(orderID string, amount float64) (PaymentResult, error) {
var result PaymentResult
err := paymentBreaker.Call(func() error {
// timeout: 10000 (per-request timeout) is handled by the HTTP client
var err error
result, err = callPaymentAPI(orderID, amount)
return err
})
if err != nil {
if errors.Is(err, ErrCircuitOpen) {
return queuePaymentForLater(orderID, amount)
}
return PaymentResult{}, err
}
return result, nil
}
// Usage
func checkoutHandler(w http.ResponseWriter, r *http.Request) {
orderID, amount := parseCheckout(r)
result, err := chargePayment(orderID, amount)
if err != nil {
// The circuit is open or the request failed
w.WriteHeader(http.StatusServiceUnavailable)
json.NewEncoder(w).Encode(map[string]string{
"error": "Payment service temporarily unavailable",
})
return
}
json.NewEncoder(w).Encode(result)
}
Alerts to install:
Critical alerts:
→ Circuits OPEN for critical-path dependencies
circuit_breaker_state{circuit="payment"} == 1
→ PagerDuty/on-call engineers
Warning alerts:
→ Circuits OPEN for non-critical dependencies
circuit_breaker_state{circuit="recommendation"} == 1
→ Slack notifications
→ Failure rates approaching thresholds
circuit_breaker_failure_count > threshold * 0.8
→ Early warnings before circuits open
Useful dashboards:
→ The state of every circuit breaker (closed/open/half-open)
→ Failure rates per circuit over time
→ Numbers of rejected requests (circuits open)
→ Durations of circuits in OPEN states
Anti-Patterns to Avoid #
// ✗ Anti-pattern 1: circuit breakers not configured per dependency
// One global circuit breaker for all calls
globalBreaker := NewCircuitBreaker("global", defaultConfig) // DON'T
// ✓ Solution: separate circuit breakers per dependency
paymentBreaker := NewCircuitBreaker("payment-gateway", defaultConfig)
inventoryBreaker := NewCircuitBreaker("inventory-service", defaultConfig)
emailBreaker := NewCircuitBreaker("email-service", defaultConfig)
// ✗ Anti-pattern 2: no fallbacks when circuits are OPEN
func getProduct(productID string) (Product, error) {
var product Product
err := productBreaker.Call(func() error {
var err error
product, err = fetchFromDB(productID)
return err
})
// If the circuit is OPEN → errors propagate → 500 errors to users
return product, err
}
// ✓ Solution: always have fallbacks
func getProductSafe(productID string) (Product, error) {
var product Product
err := productBreaker.Call(func() error {
var err error
product, err = fetchFromDB(productID)
return err
})
if errors.Is(err, ErrCircuitOpen) {
// Fallback: check caches, or return minimal data
cached, cacheErr := redis.Get(fmt.Sprintf("product:%s", productID))
if cacheErr == nil {
if err := json.Unmarshal([]byte(cached), &product); err == nil {
return product, nil
}
}
return Product{ID: productID, Status: "limited"}, nil
}
return product, err
}
// ✗ Anti-pattern 3: overly sensitive thresholds
sensitiveConfig := CircuitBreakerConfig{FailureThreshold: 1} // 1 error → OPEN
// Regular network flukes immediately open circuits
// ✗ Anti-pattern 4: overly long recovery timeouts
slowConfig := CircuitBreakerConfig{RecoveryTimeout: time.Hour} // 1 hour
// Dependencies recover in 30 seconds but circuits stay OPEN for 1 hour
// Users experience degraded services far longer than necessary
// ✗ Anti-pattern 5: circuit breakers without monitoring
// Nobody knows circuits opened until users complain
// ✗ Anti-pattern 6: including all exceptions as failures
allErrorsConfig := CircuitBreakerConfig{IsFailure: func(err error) bool { return true }}
// Code bugs count as "dependency failures"
// ✓ Solution: only network and HTTP errors count as failures
networkOnlyConfig := CircuitBreakerConfig{IsFailure: func(err error) bool {
var netErr *net.OpError
return errors.As(err, &netErr) || isHTTPError(err)
}}
#
// ✗ Anti-pattern 1: circuit breakers not configured per dependency
// One global circuit breaker for all calls
globalBreaker := NewCircuitBreaker("global", defaultConfig) // DON'T
// ✓ Solution: separate circuit breakers per dependency
paymentBreaker := NewCircuitBreaker("payment-gateway", defaultConfig)
inventoryBreaker := NewCircuitBreaker("inventory-service", defaultConfig)
emailBreaker := NewCircuitBreaker("email-service", defaultConfig)
// ✗ Anti-pattern 2: no fallbacks when circuits are OPEN
func getProduct(productID string) (Product, error) {
var product Product
err := productBreaker.Call(func() error {
var err error
product, err = fetchFromDB(productID)
return err
})
// If the circuit is OPEN → errors propagate → 500 errors to users
return product, err
}
// ✓ Solution: always have fallbacks
func getProductSafe(productID string) (Product, error) {
var product Product
err := productBreaker.Call(func() error {
var err error
product, err = fetchFromDB(productID)
return err
})
if errors.Is(err, ErrCircuitOpen) {
// Fallback: check caches, or return minimal data
cached, cacheErr := redis.Get(fmt.Sprintf("product:%s", productID))
if cacheErr == nil {
if err := json.Unmarshal([]byte(cached), &product); err == nil {
return product, nil
}
}
return Product{ID: productID, Status: "limited"}, nil
}
return product, err
}
// ✗ Anti-pattern 3: overly sensitive thresholds
sensitiveConfig := CircuitBreakerConfig{FailureThreshold: 1} // 1 error → OPEN
// Regular network flukes immediately open circuits
// ✗ Anti-pattern 4: overly long recovery timeouts
slowConfig := CircuitBreakerConfig{RecoveryTimeout: time.Hour} // 1 hour
// Dependencies recover in 30 seconds but circuits stay OPEN for 1 hour
// Users experience degraded services far longer than necessary
// ✗ Anti-pattern 5: circuit breakers without monitoring
// Nobody knows circuits opened until users complain
// ✗ Anti-pattern 6: including all exceptions as failures
allErrorsConfig := CircuitBreakerConfig{IsFailure: func(err error) bool { return true }}
// Code bugs count as "dependency failures"
// ✓ Solution: only network and HTTP errors count as failures
networkOnlyConfig := CircuitBreakerConfig{IsFailure: func(err error) bool {
var netErr *net.OpError
return errors.As(err, &netErr) || isHTTPError(err)
}}
Circuit Breaker Checklist #
DESIGN:
□ Separate circuit breakers for every external dependency
□ Thresholds configured per dependency characteristics (not one value for all)
□ Expected exceptions configured correctly (only network/timeout errors)
□ Fallback strategies defined for every circuit breaker
CONFIGURATION:
□ failure_threshold: 5-10 failures (or 50-60% failure rates)
□ failure_window: 60 seconds (adjust per traffic patterns)
□ recovery_timeout: per dependency recovery time expectations
□ success_threshold: 2-3 for adequate HALF-OPEN confidence
FALLBACKS:
□ When circuits are OPEN, useful responses exist (not just errors)
□ Fallbacks tested regularly (chaos engineering)
□ Users get informative messages about degraded services
□ Stale cache data used as fallbacks when possible
MONITORING:
□ Every circuit breaker's state exposed as metrics
□ Alerts installed for OPEN circuits (especially critical paths)
□ Dashboards displaying the health of all circuit breakers
□ Failure rates tracked per circuit for early warnings
BULKHEADS:
□ Separate thread pools for potentially slow dependencies
□ Per-dependency connection pools configured appropriately
□ Max concurrent calls per dependency limited
Summary #
- Circuit breakers prevent cascade failures — one dependency’s failure doesn’t spread across the entire system. Requests fail fast instead of waiting for timeouts and blocking threads.
- Three states working together — CLOSED (normal), OPEN (protecting the dependency), HALF-OPEN (probing recovery). Transitions are automatic based on configured thresholds.
- Fast fail is the main benefit — OPEN circuits return errors in microseconds instead of waiting for 30-second timeouts. This keeps thread pools available for other requests.
- Every dependency needs its own circuit breaker — payment gateways, databases, email services, search services — each has different characteristics needing different thresholds.
- Fallbacks are non-negotiable parts — circuit breakers without fallbacks just move problems from “slow” to “error”. Good fallbacks provide degraded but still useful services.
- Thresholds must be calibrated — too sensitive and circuits open on small errors, too loose and protection comes too late. Use failure rates with minimum volumes, not just absolute counts.
- Bulkheads complement circuit breakers — circuit breakers protect against poor quality, bulkheads protect against excessive quantity. Both are needed for comprehensive resilience.
- Monitoring isn’t optional — without metrics and alerts, opened circuits go unnoticed until users complain. Every circuit’s state must be exposed and alerted.
- Tested libraries beat custom implementations — pybreaker, gobreaker, and opossum already handle edge cases you haven’t thought of. Use libraries for production; custom implementations only for understanding concepts.
- Expected exceptions must be configured carefully — only network errors, timeouts, and HTTP errors indicate dependency problems. Bugs in your own code (ValueError, TypeError) must not count as “dependency failures”.
#
- Circuit breakers prevent cascade failures — one dependency’s failure doesn’t spread across the entire system. Requests fail fast instead of waiting for timeouts and blocking threads.
- Three states working together — CLOSED (normal), OPEN (protecting the dependency), HALF-OPEN (probing recovery). Transitions are automatic based on configured thresholds.
- Fast fail is the main benefit — OPEN circuits return errors in microseconds instead of waiting for 30-second timeouts. This keeps thread pools available for other requests.
- Every dependency needs its own circuit breaker — payment gateways, databases, email services, search services — each has different characteristics needing different thresholds.
- Fallbacks are non-negotiable parts — circuit breakers without fallbacks just move problems from “slow” to “error”. Good fallbacks provide degraded but still useful services.
- Thresholds must be calibrated — too sensitive and circuits open on small errors, too loose and protection comes too late. Use failure rates with minimum volumes, not just absolute counts.
- Bulkheads complement circuit breakers — circuit breakers protect against poor quality, bulkheads protect against excessive quantity. Both are needed for comprehensive resilience.
- Monitoring isn’t optional — without metrics and alerts, opened circuits go unnoticed until users complain. Every circuit’s state must be exposed and alerted.
- Tested libraries beat custom implementations — pybreaker, gobreaker, and opossum already handle edge cases you haven’t thought of. Use libraries for production; custom implementations only for understanding concepts.
- Expected exceptions must be configured carefully — only network errors, timeouts, and HTTP errors indicate dependency problems. Bugs in your own code (ValueError, TypeError) must not count as “dependency failures”.