SPoF — Single Point of Failure #
Imagine a system that’s been running well for months, and then suddenly one component fails — and the entire system dies with it. Not because many things broke at once, but because there was one point that, when it failed, nothing could replace it. This is what’s called a Single Point of Failure (SPoF): a single component whose failure makes the entire system or an important feature unavailable. SPoF isn’t just a technical reliability issue — it’s a real business risk. Every minute of downtime has a cost: failed transactions, frustrated users, eroded trust. Understanding SPoF means understanding where the system is most vulnerable, and what needs to be done to ensure one component’s failure doesn’t become the failure of the whole system. This article covers how to identify SPoFs, the five most common categories in modern backend systems, concrete techniques to eliminate or mitigate them, and — just as importantly — when accepting SPoF risk is a rational decision.
What Is an SPoF? #
A Single Point of Failure is a condition where a single component, if it fails, makes the entire system or a critical part of it unavailable — with no fallback or redundancy mechanism to take over.
flowchart TD
subgraph S1["System WITHOUT an SPoF"]
direction TB
Client1["Client"] --> LB["LB (Load Balancer)"]
LB --> AppA["App A"]
LB --> AppB["App B"]
LB --> AppC["App C"]
AppA --> DBP["DB Primary"]
AppB --> DBP
AppC --> DBP
DBP <--> DBR["DB Replica"]
end
subgraph S2["System WITH an SPoF"]
direction TB
Client2["Client"] --> App["App"]
App --> DB["Single DB (SPoF)"]
style DB stroke:#D9534F,stroke-width:2px
endSPoFs can exist anywhere in a system’s architecture:
SPoF category Concrete example
───────────────────── ────────────────────────────────────────────
Infrastructure Single database instance without a replica
Single server without a load balancer
Single availability zone for the whole system
Networking Single network path between datacenters
Single DNS provider without fallback
Single CDN for all static assets
Application A service that can't scale horizontally
Shared mutable state that isn't distributed
Single consumer for a critical queue
External dependency Single payment gateway without fallback
Single SMS provider
Third-party API without a circuit breaker
Operational One person who knows how to deploy
One person with production access
Documentation that exists in only one place
The most effective way to identify an SPoF in a system is to ask: “If component X suddenly died right now, what would happen?” If the answer is “the system can’t function” or “users can’t do something important” — that’s an SPoF that needs handling.
flowchart TD
Q1{"If this component\nfails right now,\nwhat happens?"}
Q2{"Is there another component\nthat can take over\nits function?"}
Q3{"Does the takeover\nhappen automatically?"}
SPOF["Critical SPoF ✗\nNeeds redundancy\nand elimination"]
MANUAL["Partial SPoF ⚠\nNeeds automated\nfailover"]
SAFE["Not an SPoF ✓\nRedundancy exists\nand is automatic"]
Q1 -->|"system/important\nfeature can't\nfunction"| Q2
Q1 -->|"minimal impact,\nthere's a degraded mode"| SAFE
Q2 -->|"No"| SPOF
Q2 -->|"Yes, but manual"| MANUAL
Q2 -->|"Yes, automatic"| SAFE
style SPOF fill:#D9534F,color:#fff
style MANUAL fill:#F0AD4E,color:#fff
style SAFE fill:#5CB85C,color:#fffCategory 1 — SPoFs in Databases #
Databases are the most common and most dangerous SPoF in backend systems. A single database instance without a replica means: if that database server dies, every application depending on it dies too.
ANTI-PATTERN: single database instance
flowchart LR
App1["App Server 1"] --> DB["Single PostgreSQL (SPoF)"]
App2["App Server 2"] --> DB
App3["App Server 3"] --> DB
DB --> Disk["single disk"]
style DB stroke:#D9534F,stroke-width:2px
style Disk stroke:#D9534F,stroke-width:2pxCORRECT: primary-replica with automatic failover
flowchart LR
App1["App Server 1"] --> DBP["DB Primary"]
App2["App Server 2"] --> DBP
App3["App Server 3"] --> DBP
DBP -. "replication stream (WAL)" .-> DBR1["Replica 1 (hot standby)"]
DBP -. "replication stream (WAL)" .-> DBR2["Replica 2 (read replica)"]Implementing connection handling that respects the primary-replica topology:
// ANTI-PATTERN: one database connection for all operations
type Repository struct {
db *sql.DB // one pool — writes and reads all go to Primary
}
// Problems:
// - Primary bears all the load including read-heavy analytics queries
// - If Primary is unavailable, there's no fallback for read-only operations
// CORRECT: separate writes (primary) and reads (replica)
type Repository struct {
primary *sql.DB // for INSERT, UPDATE, DELETE, transactions
replica *sql.DB // for SELECTs that don't need the freshest data
}
func NewRepository(primaryDSN, replicaDSN string) (*Repository, error) {
primary, err := sql.Open("postgres", primaryDSN)
if err != nil {
return nil, fmt.Errorf("connect primary: %w", err)
}
primary.SetMaxOpenConns(20)
primary.SetMaxIdleConns(10)
primary.SetConnMaxLifetime(5 * time.Minute)
replica, err := sql.Open("postgres", replicaDSN)
if err != nil {
return nil, fmt.Errorf("connect replica: %w", err)
}
replica.SetMaxOpenConns(30) // the replica can take more reads
replica.SetMaxIdleConns(15)
replica.SetConnMaxLifetime(5 * time.Minute)
return &Repository{primary: primary, replica: replica}, nil
}
// Writes always go to the primary
func (r *Repository) SaveOrder(ctx context.Context, order Order) error {
_, err := r.primary.ExecContext(ctx,
"INSERT INTO orders (id, user_id, total, status) VALUES ($1, $2, $3, $4)",
order.ID, order.UserID, order.Total, order.Status,
)
return err
}
// Reads needing high consistency → primary
func (r *Repository) FindOrderForPayment(ctx context.Context, id string) (*Order, error) {
// Right after a user created an order, read from primary
// to avoid replication lag that could cause "order not found"
return r.queryOrder(ctx, r.primary, id)
}
// Analytics or list reads tolerant of slight lag → replica
func (r *Repository) ListUserOrders(ctx context.Context, userID string) ([]Order, error) {
return r.queryOrders(ctx, r.replica, userID)
}
For systems with high availability requirements, consider a connection pooler like PgBouncer or pgpool-II that can automatically route to a replica when the primary is unavailable for read operations.
Category 2 — SPoFs in the Application Layer #
An application server running as a single instance is an SPoF often missed in early development. When you redeploy, the server dies, or there’s a crash, nothing can serve requests.
// Deployment patterns that eliminate SPoFs in the application layer:
// 1. Design the service to be stateless — don't store state in memory between requests
// ANTI-PATTERN: state stored in application memory
var activeSessions = map[string]Session{} // SPoF: lost if the server restarts
// CORRECT: state stored in a shared external store
type SessionStore struct {
redis *redis.Client
}
func (s *SessionStore) Get(ctx context.Context, token string) (*Session, error) {
data, err := s.redis.Get(ctx, "session:"+token).Bytes()
if err == redis.Nil {
return nil, ErrSessionNotFound
}
if err != nil {
return nil, fmt.Errorf("get session: %w", err)
}
var session Session
if err := json.Unmarshal(data, &session); err != nil {
return nil, fmt.Errorf("unmarshal session: %w", err)
}
return &session, nil
}
func (s *SessionStore) Set(ctx context.Context, token string, session Session, ttl time.Duration) error {
data, _ := json.Marshal(session)
return s.redis.Set(ctx, "session:"+token, data, ttl).Err()
}
// 2. Graceful shutdown — don't drop requests currently being processed
func main() {
srv := &http.Server{
Addr: ":8080",
Handler: router,
}
// Channel to catch shutdown signals
quit := make(chan os.Signal, 1)
signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
go func() {
if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
log.Fatalf("server error: %v", err)
}
}()
// Wait for the shutdown signal
<-quit
slog.Info("shutting down server gracefully")
// Give in-flight requests time to finish (max 30 seconds)
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
if err := srv.Shutdown(ctx); err != nil {
slog.Error("forced shutdown", "error", err)
}
slog.Info("server shutdown complete")
}
At the deployment level, eliminating SPoFs in the application layer requires:
Kubernetes deployment with multiple replicas:
replicas: 3 → at least 3 pods running at once
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1 → at most 1 pod down during deploys
maxSurge: 1 → 1 extra pod allowed during rollout
readinessProbe: → pods only receive traffic when ready
httpGet:
path: /health/ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
livenessProbe: → pods auto-restarted if unresponsive
httpGet:
path: /health/live
port: 8080
initialDelaySeconds: 15
periodSeconds: 20
podAntiAffinity: → pods must not all be on the same node
requiredDuringSchedulingIgnoredDuringExecution:
- topologyKey: kubernetes.io/hostname
Category 3 — SPoFs in External Dependencies #
External dependencies like payment gateways, SMS providers, or third-party APIs are SPoFs often ignored because “that’s their problem, not ours”. But from the user’s perspective, if checkout fails because Midtrans is down, it’s still a bad experience on our platform.
Circuit Breaker is the most effective pattern for preventing an unresponsive external dependency from draining resources and causing cascading failures:
// Circuit Breaker pattern for external dependencies
type CircuitState int
const (
StateClosed CircuitState = iota // Normal: requests forwarded
StateOpen // Open: requests directly rejected (fallback)
StateHalfOpen // Probing: one request forwarded as a test
)
type CircuitBreaker struct {
mu sync.Mutex
state CircuitState
failureCount int
successCount int
lastFailure time.Time
// Configuration
maxFailures int // how many failures before opening
resetTimeout time.Duration // how long before retrying (half-open)
halfOpenMaxCalls int // how many requests may pass while half-open
}
func NewCircuitBreaker(maxFailures int, resetTimeout time.Duration) *CircuitBreaker {
return &CircuitBreaker{
state: StateClosed,
maxFailures: maxFailures,
resetTimeout: resetTimeout,
halfOpenMaxCalls: 1,
}
}
func (cb *CircuitBreaker) Execute(ctx context.Context, fn func() error) error {
cb.mu.Lock()
state := cb.state
switch state {
case StateOpen:
// Check if it's time to try half-open
if time.Since(cb.lastFailure) > cb.resetTimeout {
cb.state = StateHalfOpen
cb.successCount = 0
slog.Info("circuit breaker: transitioning to half-open")
} else {
cb.mu.Unlock()
return ErrCircuitOpen // reject immediately, don't wait for timeouts
}
case StateHalfOpen:
if cb.successCount >= cb.halfOpenMaxCalls {
cb.mu.Unlock()
return ErrCircuitOpen
}
}
cb.mu.Unlock()
// Run the original function
err := fn()
cb.mu.Lock()
defer cb.mu.Unlock()
if err != nil {
cb.failureCount++
cb.lastFailure = time.Now()
if cb.state == StateHalfOpen || cb.failureCount >= cb.maxFailures {
cb.state = StateOpen
slog.Warn("circuit breaker: opened",
"failures", cb.failureCount,
"last_failure", cb.lastFailure,
)
}
return err
}
// Success
if cb.state == StateHalfOpen {
cb.successCount++
if cb.successCount >= cb.halfOpenMaxCalls {
cb.state = StateClosed
cb.failureCount = 0
slog.Info("circuit breaker: closed (recovered)")
}
} else {
cb.failureCount = 0 // reset on success in the Closed state
}
return nil
}
var ErrCircuitOpen = errors.New("circuit breaker is open — dependency unavailable")
// Usage: PaymentService with a circuit breaker and fallback
type PaymentService struct {
primaryGateway PaymentGateway
fallbackGateway PaymentGateway // alternative provider
circuitBreaker *CircuitBreaker
}
func (s *PaymentService) Charge(ctx context.Context, req ChargeRequest) (*ChargeResult, error) {
var result *ChargeResult
// Try the primary gateway through the circuit breaker
err := s.circuitBreaker.Execute(ctx, func() error {
var e error
result, e = s.primaryGateway.Charge(ctx, req)
return e
})
if err == nil {
return result, nil
}
// Circuit open or primary failed — try the fallback
if errors.Is(err, ErrCircuitOpen) || isRetryable(err) {
slog.Warn("primary payment gateway unavailable, trying fallback",
"order_id", req.OrderID,
"error", err,
)
return s.fallbackGateway.Charge(ctx, req)
}
return nil, err
}
stateDiagram-v2
[*] --> Closed: Initial state
Closed --> Closed: Request success\n(failure count reset)
Closed --> Open: failure count >= maxFailures
Open --> HalfOpen: resetTimeout elapsed
HalfOpen --> Closed: Test request succeeds
HalfOpen --> Open: Test request fails
Open --> Open: Requests directly\nrejected (ErrCircuitOpen)
note right of Closed
Normal operation
All requests forwarded
end note
note right of Open
Dependency considered down
Requests go straight to fallback
No requests to the provider
end note
note right of HalfOpen
One probe request
Determines whether the provider recovered
end noteCategory 4 — SPoFs in Message Queues #
A message queue serving as the backbone of async processing is an SPoF often not thought about until an incident happens. A single broker, single consumer, or a queue without a dead letter queue (DLQ) are all forms of SPoF.
// ANTI-PATTERN: single consumer without error handling and DLQ
func startConsumer(queue *kafka.Reader) {
for {
msg, err := queue.ReadMessage(context.Background())
if err != nil {
log.Printf("read error: %v", err) // log and continue — message lost!
continue
}
if err := processMessage(msg.Value); err != nil {
log.Printf("process error: %v", err) // message failed, no retry
// No DLQ → the failed message is lost forever
}
}
}
// CORRECT: consumer with retry, DLQ, and graceful shutdown
type MessageConsumer struct {
reader *kafka.Reader
dlqWriter *kafka.Writer // dead letter queue for unprocessable messages
maxRetries int
}
func (c *MessageConsumer) Start(ctx context.Context) error {
for {
select {
case <-ctx.Done():
return c.reader.Close()
default:
}
msg, err := c.reader.FetchMessage(ctx)
if err != nil {
if errors.Is(err, context.Canceled) {
return nil
}
slog.Error("failed to fetch message", "error", err)
time.Sleep(time.Second) // backoff before retrying the fetch
continue
}
if err := c.processWithRetry(ctx, msg); err != nil {
// After maxRetries are exhausted, send to the DLQ for later investigation
slog.Error("sending message to DLQ after max retries",
"topic", msg.Topic,
"offset", msg.Offset,
"error", err,
)
if dlqErr := c.sendToDLQ(ctx, msg, err); dlqErr != nil {
slog.Error("failed to send to DLQ", "error", dlqErr)
// Don't commit — will be retried on consumer restart
continue
}
}
// Commit only after successful processing or DLQ entry
if err := c.reader.CommitMessages(ctx, msg); err != nil {
slog.Error("failed to commit message", "error", err)
}
}
}
func (c *MessageConsumer) processWithRetry(ctx context.Context, msg kafka.Message) error {
var lastErr error
for attempt := 1; attempt <= c.maxRetries; attempt++ {
if err := processMessage(msg.Value); err != nil {
lastErr = err
backoff := time.Duration(attempt) * 500 * time.Millisecond
slog.Warn("processing failed, retrying",
"attempt", attempt,
"max_retries", c.maxRetries,
"backoff", backoff,
"error", err,
)
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(backoff):
}
continue
}
return nil
}
return fmt.Errorf("all %d retries exhausted: %w", c.maxRetries, lastErr)
}
Category 5 — Hidden SPoFs in Configuration and Operations #
The most often ignored SPoFs are operational ones — not technical components, but processes and knowledge that exist in only one place.
OPERATIONAL SPoFs OFTEN MISSED:
1. "Only one person knows how to deploy"
→ Solution: documented deployment runbooks, CI/CD anyone can run,
knowledge rotation within the team
2. "Secrets/credentials only exist on one person's laptop"
→ Solution: centralized secret management (HashiCorp Vault, AWS Secrets Manager,
GCP Secret Manager) with role-based access
3. "Database backups exist but restore was never tested"
→ A backup never tested for restore is an SPoF — you don't know
whether that backup works when you need it
→ Solution: scheduled restore tests into a separate environment
4. "One wrong environment variable can take down every service"
→ Solution: config validation at startup, fail fast with clear messages
5. "A single DNS provider for all domains"
→ If the DNS provider goes down, all domains are unreachable
→ Solution: secondary DNS provider, or multi-provider with failover
Config validation that eliminates configuration SPoFs:
// Fail fast at startup if the configuration is invalid
// Better to not start with a clear message than to start and crash mysteriously
func (c *Config) Validate() error {
var errs []string
if c.Database.DSN == "" {
errs = append(errs, "DATABASE_URL is required")
}
if c.Database.MaxOpenConns <= 0 {
errs = append(errs, "DB_MAX_OPEN_CONNS must be positive")
}
if c.HTTP.Timeout <= 0 {
errs = append(errs, "HTTP_TIMEOUT must be positive")
}
if c.Auth.JWTSecret == "" {
errs = append(errs, "JWT_SECRET is required")
}
if len(c.Auth.JWTSecret) < 32 {
errs = append(errs, "JWT_SECRET must be at least 32 characters")
}
if c.Queue.Brokers == "" {
errs = append(errs, "KAFKA_BROKERS is required")
}
if len(errs) > 0 {
return fmt.Errorf("invalid configuration:\n - %s", strings.Join(errs, "\n - "))
}
return nil
}
func main() {
cfg, err := config.Load()
if err != nil {
log.Fatalf("failed to load config: %v", err)
}
// Fail fast — better to crash at startup with a clear message
// than to crash midway without context
if err := cfg.Validate(); err != nil {
log.Fatalf("config validation failed:\n%v", err)
}
// Only after validation passes, initialize all components
startServer(cfg)
}
Health Checks as an Early Detection System for SPoFs #
Proper health checks are the first infrastructure needed to mitigate SPoFs — because without them, the load balancer doesn’t know an instance is unhealthy and keeps sending traffic to it.
// Health checks with two endpoints serving different purposes:
// /health/live → is this process still alive? (for liveness probes)
// /health/ready → is this process ready to receive traffic? (for readiness probes)
type HealthChecker struct {
db *sql.DB
redis *redis.Client
kafka *kafka.Writer
}
// Liveness: only checks whether the app is still running
// If this fails, Kubernetes restarts the pod
func (h *HealthChecker) LiveHandler(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
json.NewEncoder(w).Encode(map[string]string{"status": "alive"})
}
// Readiness: checks whether all dependencies are ready
// If this fails, Kubernetes won't send traffic to this pod
// but won't restart the pod either
func (h *HealthChecker) ReadyHandler(w http.ResponseWriter, r *http.Request) {
ctx, cancel := context.WithTimeout(r.Context(), 5*time.Second)
defer cancel()
checks := map[string]error{
"database": h.checkDB(ctx),
"redis": h.checkRedis(ctx),
}
allHealthy := true
results := map[string]interface{}{}
for name, err := range checks {
if err != nil {
allHealthy = false
results[name] = map[string]string{
"status": "unhealthy",
"error": err.Error(),
}
slog.Warn("dependency unhealthy", "component", name, "error", err)
} else {
results[name] = map[string]string{"status": "healthy"}
}
}
w.Header().Set("Content-Type", "application/json")
if !allHealthy {
w.WriteHeader(http.StatusServiceUnavailable) // 503 — not ready for traffic
} else {
w.WriteHeader(http.StatusOK)
}
json.NewEncoder(w).Encode(map[string]interface{}{
"status": map[bool]string{true: "ready", false: "not_ready"}[allHealthy],
"checks": results,
})
}
func (h *HealthChecker) checkDB(ctx context.Context) error {
return h.db.PingContext(ctx)
}
func (h *HealthChecker) checkRedis(ctx context.Context) error {
return h.redis.Ping(ctx).Err()
}
flowchart LR
LB["Load Balancer"]
P1["Pod 1\n✓ healthy"]
P2["Pod 2\n✗ DB timeout"]
P3["Pod 3\n✓ healthy"]
LB -->|"traffic"| P1
LB -->|"readiness probe 503\n→ removed from rotation"| P2
LB -->|"traffic"| P3
P2 -->|"/health/ready → 503"| LB
style P2 fill:#D9534F,color:#fff
style P1 fill:#5CB85C,color:#fff
style P3 fill:#5CB85C,color:#fffThe Trade-off: Reliability vs Cost #
Eliminating all SPoFs isn’t always practical or cost-effective. Every redundancy layer adds cost — both infrastructure cost and operational complexity cost.
A FRAMEWORK FOR PRIORITIZING SPoF ELIMINATION:
Priority = Impact of Failure × Probability of Failure
High impact, high probability → eliminate immediately (database without a replica)
High impact, low probability → mitigate with monitoring + runbooks
Low impact, high probability → accept or lightly mitigate
Low impact, low probability → accept, focus on more critical items
EXAMPLE PRIORITIES IN A TYPICAL BACKEND SYSTEM:
🔴 Highest priority (eliminate):
- Database without a replica (impact: all operations fail)
- Auth service without HA (impact: no user can log in)
- Payment gateway without fallback (impact: revenue directly affected)
🟡 Medium priority (mitigate):
- Single availability zone (big impact, low probability)
- Email/notification service without fallback (degraded, but core still runs)
- Redis without a replica (cache miss → fallback to DB, slower but works)
🟢 Acceptable for now:
- Admin dashboard without HA (non-critical)
- Reporting service without a replica (can be delayed, not a blocker)
- Development/staging environments without redundancy
Redundancy adds operational complexity. A primary-replica database requires monitoring replication lag, periodic failover testing, and clear procedures for handling split-brain scenarios. Before adding redundancy, make sure the team is ready to operate it — misunderstood redundancy can create a new SPoF more dangerous than the one eliminated.
SPoF and Its Relationship with Other Principles #
SPoF is an architectural problem whose solutions often involve the principles discussed earlier:
flowchart TD
SPOF["SPoF\n(Single Point of Failure)"]
SRP2["SRP\nServices with one responsibility\nare easier to scale\nindependently"]
SOC2["SoC\nSeparated components\ncan be replaced or\nfailed over independently"]
DIP["DIP (from SOLID)\nDepending on abstractions\nallows swapping implementations\nwithout changing consumers"]
SSOT2["SSOT\nOne configuration source\nprevents config drift\nbetween instances"]
SPOF -->|"prevented by"| SRP2
SPOF -->|"prevented by"| SOC2
SPOF -->|"facilitated by"| DIP
SPOF -->|"risk reduced by"| SSOT2
style SPOF fill:#D9534F,color:#fff
style SRP2 fill:#5CB85C,color:#fff
style SOC2 fill:#5CB85C,color:#fff
style DIP fill:#4C9BE8,color:#fff
style SSOT2 fill:#4C9BE8,color:#fffThe DIP principle is especially relevant: when a service depends on an interface rather than a concrete implementation, swapping implementations (for example, adding a fallback payment gateway) doesn’t require changes in business logic. The circuit breaker pattern can also only be applied cleanly when dependencies are injected as interfaces.
Anti-Patterns at a Glance #
// ✗ Single database without a replica
db, _ := sql.Open("postgres", "host=db-primary-only port=5432 ...")
// If db-primary-only dies → the app dies
// ✗ State in memory — lost on restart
var cache = map[string]User{} // not shared state, an SPoF per instance
// ✗ External calls without timeouts and circuit breakers
resp, err := http.Get("https://api.payment.com/charge")
// If payment.com is slow → goroutines hang → connection pool exhausted → cascading failure
// ✗ Queue consumers without a DLQ
for msg := range queue.Messages() {
if err := process(msg); err != nil {
log.Println(err) // failed messages are lost forever
}
}
// ✗ No config validation — the app starts with incomplete config
func main() {
cfg := loadConfig() // no validation
startServer(cfg) // mysterious crash later at runtime
}
// ✗ Health checks that aren't meaningful
func healthHandler(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(200) // always 200, even when the database is down
w.Write([]byte("ok"))
}
// The load balancer thinks the pod is healthy, keeps sending traffic → users get errors
SPoF Review Checklist #
DATABASE AND STORAGE:
□ Database has at least one replica with automatic failover
□ Writes and reads separated (writes to primary, reads to replica)
□ Backups scheduled and restores tested periodically
□ Connection pools configured with appropriate timeouts
APPLICATION LAYER:
□ Service can run with more than one instance simultaneously
□ No state stored in process memory (sessions, caches, counters)
□ Graceful shutdown implemented — in-flight requests are completed
□ Health check endpoints (liveness and readiness) return accurate status
EXTERNAL DEPENDENCIES:
□ Every external dependency has an explicitly configured timeout
□ Critical dependencies have circuit breakers or exponential backoff retries
□ Fallbacks exist for dependencies affecting core business flows
□ One dependency's failure doesn't cascade into other dependencies
MESSAGE QUEUES:
□ Consumers have retry mechanisms with backoff
□ A Dead Letter Queue exists for messages failing after max retries
□ Consumers can restart without losing messages (manual commit)
□ Multiple consumer instances run for availability
CONFIGURATION AND OPERATIONS:
□ Config validated at startup — fail fast with clear messages
□ Secrets stored in centralized secret management, not local files
□ Deployment runbooks documented and executable by more than one person
□ Database failover procedures documented and practiced
Summary #
- An SPoF is a single component whose failure makes the entire system or an important feature unavailable. Identify it with one question: “If component X suddenly died right now, what would happen?”
- Five common SPoF categories: databases without replicas, single-instance application servers, external dependencies without fallbacks, message queues without DLQs, and operational SPoFs (knowledge or access that exists in only one person).
- Databases: primary-replica with automatic failover is the first step. Separate writes to primary and reads to replica to reduce load and increase read availability.
- Application layer: design services stateless — no state in memory. Graceful shutdown ensures in-flight requests complete before the pod closes. Multiple replicas with anti-affinity rules prevent all pods from landing on one node.
- External dependencies: a circuit breaker cuts connections to unresponsive providers before they drain resources. Fallback providers for dependencies critical to revenue flows.
- Message queues: retry with backoff and a Dead Letter Queue ensure no message is lost — whether from processing errors or consumer restarts.
- Accurate health checks: readiness probes checking real dependencies ensure the load balancer doesn’t send traffic to unhealthy instances.
- Reliability vs cost trade-offs: prioritize SPoF elimination by impact × probability. A database without a replica is the highest priority; an admin dashboard without HA is acceptable for now.
- Redundancy adds operational complexity — make sure the team understands how to operate a redundant system before adding it. Misunderstood redundancy can create new SPoFs.
- Relationship with other principles: SRP enables independent scaling, SoC enables per-component failover, DIP facilitates implementation swaps for fallbacks, SSOT prevents config drift between instances.