Race Condition #

A race condition is one of the most dangerous bugs in software engineering — not because it’s hard to fix, but because it’s hard to detect. It almost never shows up in development, sails through every unit test, and only appears in production under high traffic. And when it does appear, the effect isn’t a clear error — it’s silently corrupted data: balances going negative, stock selling more than what’s available, or one user getting access that should belong to another. This article covers race conditions from their root principles, which layers they can occur at, concrete examples in HTTP, memory, database, and distributed systems, and prevention patterns you can apply right away.

What Is a Race Condition? #

A race condition is when the final outcome of a process depends on the execution order of several operations running at the same time, and that order isn’t controlled. Two or more processes “race” to access or modify the same resource — and whoever “wins” determines the system’s final state.

What makes race conditions dangerous is their non-deterministic nature: identical code can produce different outputs depending on CPU execution timing, system load, or how many connections arrive at once. This isn’t a random bug — it’s a deterministic bug that depends on timing.

flowchart TD
    A[Two Requests Arrive Together] --> B[Request A: Read balance = 100]
    A --> C[Request B: Read balance = 100]
    B --> D[Request A: Subtract 100]
    C --> E[Request B: Subtract 100]
    D --> F[Request A: Save balance = 0]
    E --> G[Request B: Save balance = 0]
    F --> H["❌ Balance should be -100\nbut 0 is saved"]
    G --> H

A race condition happens when three conditions are present at the same time: there’s a shared resource, there’s more than one executor (thread, goroutine, request, process), and there’s no correct synchronization mechanism.

The Three Principles That Trigger Race Conditions #

Understanding these three principles is the key to anticipating race conditions at the design phase, not after production breaks.

Shared Mutable State #

A race condition always involves shared data that can change. If the data is read-only, there’s no race. If the data is only accessed by one process, there’s no race. Problems arise when there’s a shared resource that can be written to by more than one executor.

The most common shared mutable state:

Concurrency or Parallelism #

A race condition can’t happen in a truly sequential single-threaded system. It appears when there’s more than one execution path that can run “at the same time” — either physically (parallelism on multi-core) or logically (concurrency through context switching).

flowchart LR
    subgraph Sequential["Sequential — Can't Race"]
        S1[Request A] --> S2[Process A] --> S3[Request B] --> S4[Process B]
    end
    subgraph Concurrent["Concurrent — Can Race"]
        C1[Request A] --> C3["Process A\n& Process B\nrunning overlapped"]
        C2[Request B] --> C3
        C3 --> C4["Final state\nunpredictable"]
    end

Execution paths that can trigger a race: multi-thread, multi-goroutine, simultaneous HTTP requests, parallel queue consumers, and multiple service instances in a distributed system.

Non-Atomic Operation #

This is the technical root of most race conditions. An operation that looks like “one step” at the business level actually consists of several technical steps that can be interrupted.

// Looks like a single operation:
counter++

// It's actually three steps that can be interrupted in between:
// 1. LOAD  — read the counter value from memory
// 2. ADD   — add 1
// 3. STORE — write the new value to memory

// Goroutine B can read the old value between A's LOAD and STORE

The same applies to the read-check-write pattern so common in business code:

1. Read balance from the database  ← Goroutine B also reads here
2. Check balance >= amount
3. Subtract the balance
4. Save the new balance            ← Two goroutines save different results

If these steps aren’t atomic — meaning they’re not guaranteed to run without interruption — a race condition is almost certain.

Race Conditions at the HTTP / API Level #

Many engineers assume race conditions only happen at the thread or goroutine level. In fact, they can happen at the HTTP level even if your backend is single-threaded, as long as more than one request is processed at the same time.

Case: Double Submit Payment #

The most classic scenario: a user clicks the “Pay” button twice because the response is slow, or a mobile client retries automatically because of a network timeout. Two HTTP requests arrive almost simultaneously at the same endpoint.

sequenceDiagram
    participant ClientA as Request A\n(first click)
    participant ClientB as Request B\n(second click / retry)
    participant Server
    participant DB

    ClientA->>Server: POST /payments
    ClientB->>Server: POST /payments
    Server->>DB: SELECT status WHERE invoice_id=1
    Server->>DB: SELECT status WHERE invoice_id=1
    DB-->>Server: status = UNPAID
    DB-->>Server: status = UNPAID
    Note over Server: Both requests see UNPAID
    Server->>DB: Process payment A
    Server->>DB: Process payment B ❌
    DB-->>Server: OK
    DB-->>Server: OK
    Note over DB: Invoice paid twice!

// ANTI-PATTERN: check-then-act without concurrency protection
func processPayment(invoiceID string, amount int) error {
    invoice, _ := db.FindInvoice(invoiceID)
    
    if invoice.Status == "PAID" {
        return errors.New("already paid")
    }
    
    // GAP HERE — another request can slip past the check above
    
    db.CreatePayment(invoiceID, amount)
    db.UpdateInvoiceStatus(invoiceID, "PAID")
    return nil
}

// CORRECT: use an atomic update with a WHERE clause as the guard
func processPayment(invoiceID string, amount int) error {
    // UPDATE only succeeds if the status is still UNPAID — atomic at the DB level
    result := db.Exec(`
        UPDATE invoices 
        SET status = 'PAID' 
        WHERE id = ? AND status = 'UNPAID'
    `, invoiceID)
    
    if result.RowsAffected == 0 {
        return errors.New("invoice already paid or not found")
    }
    
    return db.CreatePayment(invoiceID, amount)
}

Common Causes at the HTTP Layer #

Race Conditions at the Application / Memory Level #

At the application level, race conditions happen when goroutines or threads share in-memory state without proper synchronization.

Case: An Inaccurate Counter #

// ANTI-PATTERN: counter variable accessed by many goroutines without a mutex
var requestCount int

func handleRequest(w http.ResponseWriter, r *http.Request) {
    requestCount++ // NOT SAFE — not an atomic operation
    // process request...
}

// With 1000 concurrent goroutines:
// The final requestCount could be 847, 923, or any other unpredictable number
// Depending on CPU scheduling timing

// CORRECT: use sync/atomic for counter operations
var requestCount int64

func handleRequest(w http.ResponseWriter, r *http.Request) {
    atomic.AddInt64(&requestCount, 1) // Truly atomic, safe from races
    // process request...
}

// Or use a mutex for more complex code blocks
var (
    mu           sync.Mutex
    activeUsers  = make(map[string]bool)
)

func addActiveUser(userID string) {
    mu.Lock()
    defer mu.Unlock()
    activeUsers[userID] = true // Safe — only one goroutine in here
}

sequenceDiagram
    participant G1 as Goroutine 1
    participant G2 as Goroutine 2
    participant Mem as Memory\ncounter=5

    G1->>Mem: LOAD counter (reads 5)
    G2->>Mem: LOAD counter (reads 5)
    G1->>G1: ADD 1 → result = 6
    G2->>G2: ADD 1 → result = 6
    G1->>Mem: STORE 6
    G2->>Mem: STORE 6
    Note over Mem: counter = 6, not 7!
    Note over G1,G2: One increment lost

Real Examples in Production #

Race conditions at the memory level often show up in these scenarios:

// ANTI-PATTERN: concurrent write to a map without a mutex — crashes the Go runtime
var cache = make(map[string]string)

func getFromCache(key string) string {
    // Concurrent reads are actually safe, but if there's a concurrent write: FATAL
    return cache[key]
}

func setCache(key, value string) {
    cache[key] = value // Concurrent write to a map = panic
}

// CORRECT: use sync.RWMutex for a read-heavy cache
var (
    cacheMu sync.RWMutex
    cache   = make(map[string]string)
)

func getFromCache(key string) string {
    cacheMu.RLock() // Multiple readers allowed at once
    defer cacheMu.RUnlock()
    return cache[key]
}

func setCache(key, value string) {
    cacheMu.Lock() // Only one writer, blocks all readers
    defer cacheMu.Unlock()
    cache[key] = value
}

In Go, a concurrent write to a map without a mutex doesn’t just corrupt data — it panics at runtime. Go’s race detector (go test -race) is strongly recommended in your CI pipeline to catch race conditions before they reach production.

Race Conditions at the Database Level #

This one is the most often underestimated. Many engineers assume that because they use a database, the data is automatically safe from race conditions. That assumption is wrong.

Case: Stock Going Negative #

-- ANTI-PATTERN: separate read-check-write without locking
-- Transactions A and B can both read stock = 1 before either updates

-- Transaction A
SELECT stock FROM products WHERE id = 1;  -- stock = 1
-- (Transaction B also SELECTs here, getting stock = 1)

-- Application: if stock > 0 → allowed to buy

-- Transaction A
UPDATE products SET stock = stock - 1 WHERE id = 1;  -- stock = 0

-- Transaction B (also passed the check because it read stock = 1 earlier)
UPDATE products SET stock = stock - 1 WHERE id = 1;  -- stock = -1 ❌

-- CORRECT: atomic update with a WHERE guard, no separate SELECT needed
UPDATE products 
SET stock = stock - 1 
WHERE id = 1 AND stock > 0;

-- Check rows_affected: if 0, the stock ran out or a race happened
-- No gap between check and update — one atomic operation

// Go implementation checking rows affected
func purchaseProduct(productID int) error {
    result := db.Exec(
        "UPDATE products SET stock = stock - 1 WHERE id = ? AND stock > 0",
        productID,
    )
    
    if result.Error != nil {
        return result.Error
    }
    
    if result.RowsAffected == 0 {
        // ANTI-PATTERN: returning a generic error
        return errors.New("failed")
        
        // CORRECT: return an informative error
        return errors.New("product out of stock")
    }
    
    return nil
}

Case: Lost Update #

A lost update is a subtler form of database race condition — two transactions read the same value, modify it based on that value, and one of the updates “disappears” because it was overwritten.

sequenceDiagram
    participant TxA as Transaction A
    participant TxB as Transaction B
    participant DB

    TxA->>DB: SELECT login_count WHERE user_id=1 → 10
    TxB->>DB: SELECT login_count WHERE user_id=1 → 10
    TxA->>TxA: Calculate: 10 + 1 = 11
    TxB->>TxB: Calculate: 10 + 1 = 11
    TxA->>DB: UPDATE login_count = 11
    TxB->>DB: UPDATE login_count = 11
    Note over DB: login_count = 11, not 12!
    Note over TxA,TxB: One increment lost — Lost Update

-- ANTI-PATTERN: read in the app, calculate in the app, write to the DB
-- Vulnerable to lost updates

-- CORRECT: do the increment directly in the database — atomic
UPDATE users SET login_count = login_count + 1 WHERE id = 1;

-- Or for cases that need the previous value:
-- Use SELECT FOR UPDATE to lock the row
BEGIN;
SELECT login_count FROM users WHERE id = 1 FOR UPDATE;
-- The row is now locked — other transactions will wait
UPDATE users SET login_count = login_count + 1 WHERE id = 1;
COMMIT;

Isolation Levels and Their Impact #

Databases provide isolation levels that control how protected a transaction is from the effects of other concurrent transactions:

PostgreSQL’s default is READ COMMITTED. MySQL/InnoDB’s default is REPEATABLE READ. Understand the default of the database you’re using and adjust per transaction when needed.

Race Conditions in Distributed Systems #

In distributed systems, race conditions go beyond the boundaries of a single process or machine. This is the most complex level because there’s no shared memory — synchronization must happen over the network, which is inherently unreliable.

flowchart TD
    A[Job Scheduler] --> B["Instance 1\nProcessing Job X"]
    A --> C["Instance 2\nProcessing Job X"]
    B --> D{"Distributed Lock\nAvailable?"}
    C --> D
    D -- Lock acquired → Instance 1 --> E[Instance 1 processes]
    D -- Lock failed → Instance 2 --> F["Instance 2 waits / skips"]
    E --> G[Release lock]
    G --> H[Instance 2 can process the next job]

// ANTI-PATTERN: no coordination between instances
// If 3 instances are running, the same job can be processed 3 times
func processScheduledJob(jobID string) {
    job := db.FindJob(jobID)
    if job.Status == "PENDING" {
        executeJob(job)
        db.UpdateJobStatus(jobID, "COMPLETED")
    }
}

// CORRECT: use a distributed lock before processing
func processScheduledJob(jobID string) error {
    lockKey := fmt.Sprintf("job-lock:%s", jobID)
    
    // Try to acquire the lock with a TTL — atomic in Redis
    acquired, err := redis.SetNX(ctx, lockKey, "locked", 30*time.Second)
    if err != nil || !acquired {
        // Another instance is processing this job
        return nil
    }
    defer redis.Del(ctx, lockKey) // Release the lock when done
    
    job := db.FindJob(jobID)
    if job.Status != "PENDING" {
        return nil // Already processed by another instance
    }
    
    return executeJob(job)
}

The most common distributed race condition scenarios:

Race Condition Prevention Patterns #

There’s no silver bullet for race conditions. Choose the solution based on the layer where the race happens.

1. Atomic Operations #

Turn the race-prone operation into a single atomic step that can’t be interrupted.

// Memory level: use sync/atomic
atomic.AddInt64(&counter, 1)
atomic.CompareAndSwapInt64(&value, old, new)

// Database level: let the DB do the math
db.Exec("UPDATE products SET stock = stock - 1 WHERE id = ? AND stock > 0", id)
db.Exec("UPDATE users SET login_count = login_count + 1 WHERE id = ?", id)

2. Locking #

Use locks to protect critical sections — the code blocks that access shared resources.

// In-process: sync.Mutex or sync.RWMutex
var mu sync.Mutex
mu.Lock()
// critical section — only one goroutine in here
mu.Unlock()

// Database: SELECT FOR UPDATE
// BEGIN; SELECT ... FOR UPDATE; UPDATE ...; COMMIT;

// Distributed: Redis SetNX
redis.SetNX(ctx, lockKey, "locked", ttl)

A lock is a solution, not a free safety guarantee. Locks that aren’t released cause deadlocks. Locks that are too wide kill throughput. Locks that are too narrow don’t protect every path. Always measure the performance impact after adding a lock to a hot path.

3. Optimistic Locking #

Instead of locking upfront, detect conflicts when you’re about to save. Effective for read-heavy cases with rare write conflicts.

// ANTI-PATTERN: no version check — lost updates can happen
func updateUserProfile(userID string, data ProfileData) error {
    return db.Exec("UPDATE users SET name=?, bio=? WHERE id=?",
        data.Name, data.Bio, userID)
}

// CORRECT: optimistic locking with a version column
func updateUserProfile(userID string, data ProfileData, currentVersion int) error {
    result := db.Exec(`
        UPDATE users 
        SET name=?, bio=?, version=version+1 
        WHERE id=? AND version=?
    `, data.Name, data.Bio, userID, currentVersion)
    
    if result.RowsAffected == 0 {
        // Version mismatch — someone else updated first
        return errors.New("conflict: data was modified by another process, please refresh")
    }
    return nil
}

4. The Right Transaction and Isolation Level #

Choose the isolation level based on your needs, not just the default.

-- For operations that need high consistency (balance transfers)
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
BEGIN;
SELECT balance FROM accounts WHERE id = ? FOR UPDATE;
UPDATE accounts SET balance = balance - ? WHERE id = ?;
UPDATE accounts SET balance = balance + ? WHERE id = ?;
COMMIT;

-- For read-heavy operations tolerant of stale data
-- READ COMMITTED is sufficient and more performant

5. Design That Avoids Shared State #

The best solution is not having the problem in the first place. Design systems that minimize shared mutable state.

// ANTI-PATTERN: one global counter accessed by everyone
var globalOrderCounter int

// CORRECT: use a database sequence or UUID — no shared state needed
orderID := uuid.New().String()
// or
orderNumber, _ := db.QueryRow("SELECT nextval('order_seq')").Scan(&orderNumber)

Anti-Patterns to Avoid #

// ✗ Check-then-act without atomicity
stock := db.GetStock(productID)  // read
if stock > 0 {                   // check
    db.DecrementStock(productID) // act — race window here
}
// ✓ Atomic update with a WHERE guard
db.Exec("UPDATE products SET stock = stock - 1 WHERE id = ? AND stock > 0", productID)

// ✗ Assuming a DB transaction automatically prevents races
db.Transaction(func(tx *DB) error {
    user := tx.FindUser(id)           // read
    user.Balance -= amount            // modify in memory
    return tx.Save(user)              // save — can still be a lost update!
})
// ✓ Use explicit locking inside the transaction
db.Transaction(func(tx *DB) error {
    var balance int
    tx.QueryRow("SELECT balance FROM users WHERE id = ? FOR UPDATE", id).Scan(&balance)
    if balance < amount { return errors.New("insufficient balance") }
    return tx.Exec("UPDATE users SET balance = balance - ? WHERE id = ?", amount, id)
})

// ✗ Distributed scheduler without coordination
func runEvery5Minutes() {
    job()  // All instances run at the same time
}
// ✓ Distributed lock before execution
func runEvery5Minutes() {
    if acquireDistributedLock("scheduler-job", 5*time.Minute) {
        defer releaseLock("scheduler-job")
        job()
    }
}

Race Condition Review Checklist #

IDENTIFICATION:
  □ All shared resources (DB, cache, memory) have been identified
  □ All write operations checked for concurrent access
  □ Read-check-write patterns identified across the codebase

APPLICATION LEVEL:
  □ All access to shared maps uses sync.RWMutex
  □ Counters use sync/atomic or a mutex
  □ No global variables modifiable concurrently without a lock
  □ Go: CI pipeline runs go test -race

DATABASE LEVEL:
  □ Increment/decrement operations done directly in SQL (not via the app)
  □ Critical UPDATEs use WHERE guards (AND stock > 0, AND status = ?)
  □ Row locking (SELECT FOR UPDATE) used in transactions that need consistency
  □ Isolation level chosen by need, not just the default

DISTRIBUTED SYSTEM:
  □ Scheduled jobs use distributed locks
  □ Queue consumers are idempotent with event_id checks
  □ No assumption of "only one instance is running"

TESTING:
  □ Concurrent tests (multiple goroutines) exist for critical operations
  □ Race detector enabled in CI
  □ Load tests run to trigger timing-dependent race conditions

Summary #

• A race condition is a timing bug — it happens when two processes access a shared resource at the same time without coordination, and the outcome depends on who “wins”.
• Three triggering conditions — shared mutable state + more than one executor + non-atomic operations. Remove any one of them and a race condition can’t happen.
• It’s not only about threads — race conditions can happen at the HTTP level (double submit), database level (lost updates, phantom reads), and distributed system level (duplicate jobs).
• A database isn’t a silver bullet — transactions don’t automatically prevent race conditions. Isolation levels, SELECT FOR UPDATE, and atomic SQL are still required.
• Atomic SQL is the best DB solution — UPDATE ... WHERE stock > 0 is far safer and simpler than a separate read-check-write.
• Distributed locks for cross-instance — Redis SetNX is the most common way to coordinate in distributed systems.
• Optimistic locking for rare conflicts — a version column in the database works well for read-heavy cases where conflicts are rare.
• Go’s race detector is mandatory in CI — go test -race catches race conditions invisible to regular tests.
• Design without shared state is the best solution — UUIDs, DB sequences, and event sourcing eliminate the need for coordination from the start.

← Previous: Idempotency   Next: Replay Strategy →