Idempotency #

Every internet-connected system will experience network failures, timeouts, and retries. This isn’t an edge scenario — it’s a normal condition. What distinguishes robust systems from fragile ones is how they respond when the same request is sent more than once: does the system process twice and produce two charges on the user’s credit card, or does it recognize this as the same request and return the existing result without additional side effects? Idempotency is the property answering this question. This article covers why idempotency matters, how to implement it with the Idempotency-Key pattern, atomic claims with databases, request hashing for security, and anti-patterns to avoid.

The Problems Idempotency Solves #

To understand why idempotency matters, imagine a checkout scenario in an e-commerce app:

sequenceDiagram
    participant U as User/Client
    participant S as Server
    participant PG as Payment Gateway
    participant DB as Database

    Note over U,DB: Without Idempotency — Retry = Double Charge

    U->>S: POST /checkout { items, payment }
    S->>PG: Charge credit card Rp 500,000
    PG-->>S: Success
    S->>DB: Insert order
    Note over S: Server crash / timeout before sending the response!
    U--xU: Timeout — no response received

    U->>S: POST /checkout { items, payment } (RETRY)
    S->>PG: Charge credit card Rp 500,000 AGAIN
    PG-->>S: Success
    S->>DB: Insert order AGAIN
    S-->>U: 200 OK

    Note over U,DB: The user is charged Rp 1,000,000 for one purchase!

    Note over U,DB: With Idempotency — Retry Is Safe

    U->>S: POST /checkout { Idempotency-Key: key-abc }
    S->>PG: Charge Rp 500,000
    PG-->>S: Success
    S->>DB: Insert order + save key-abc + response
    Note over S: Server crash / timeout!
    U--xU: Timeout

    U->>S: POST /checkout { Idempotency-Key: key-abc } (RETRY)
    S->>DB: Check key-abc → already exists, get the cached response
    S-->>U: 200 OK (same response, no second charge)

This scenario doesn’t only happen because users deliberately submit twice. More often it happens because:

Invisible retry causes:

1. Network timeouts — mobile SDKs auto-retry after 3 seconds without a response
2. Server restarts during deployments — in-flight requests resent by the load balancer
3. Client-side retry logic — JavaScript fetch with retries on network errors
4. Message queue at-least-once delivery — events sent more than once
5. Webhook retries — payment gateways resend webhooks without a 200 response
6. Service mesh retries — Istio/Envoy automatically retry failed requests

Definition and Its Relationship with HTTP Methods #

An operation is idempotent if executing it once or several times produces the same final state.

Idempotent operations:
  DELETE /users/123 — executed 1x or 5x, user 123 still doesn't exist
  PUT /users/123 { name: "Budi" } — executed repeatedly, the name stays "Budi"
  GET /users — doesn't change state, always safe

Operations not idempotent by nature:
  POST /payments { amount: 500000 } — executed 2x = 2 payments
  POST /orders — executed 2x = 2 orders
  PATCH /counter { increment: 1 } — executed 2x = counter increases by 2

Relationship with HTTP methods (specification):

Method      Idempotent    Safe (doesn't change state)
GET         ✓             ✓
HEAD        ✓             ✓
PUT         ✓             ✗
DELETE      ✓             ✗
POST        ✗             ✗
PATCH       ✗ (generally) ✗

IMPORTANT: Idempotency per the HTTP spec is about
           "whether the same request produces the same effect"
           But the BACKEND implementation determines whether
           this actually happens or not.

DELETE that isn't idempotent:
  DELETE /notifications/oldest → deletes the oldest, not a specific ID
  Executed 2x → deletes 2 different notifications
  This isn't idempotent even though it uses DELETE

POST made idempotent:
  POST /payments with an Idempotency-Key → made idempotent through implementation

The Idempotency-Key Pattern #

The most common and recommended pattern for making non-idempotent operations idempotent is the Idempotency-Key header.

flowchart TD
    Start["Request arrives with\nIdempotency-Key: key-xyz"]

    CheckKey{"Check key-xyz\nin storage"}
    KeyFound["Key found"]
    KeyNotFound["Key not found"]

    CheckStatus{"Key status?"}
    Completed["Status: completed"]
    Processing["Status: processing"]
    ReturnCached["Return cached response\n(don't reprocess)"]
    ReturnConflict["Return 409 Conflict\n(still being processed)"]

    HashCheck{"Request hash\nmatches the\nstored one?"}
    HashMismatch["Return 422\nDifferent payload\nfor the same key"]

    ClaimKey["Atomic: Insert key\nwith status 'processing'\n(UNIQUE constraint)"]
    ClaimSuccess{"Insert succeeded?\n(affected rows = 1)"}
    Duplicate["Return 409\nConcurrent duplicate"]

    Execute["Execute business logic"]
    StoreResult["Save response +\nupdate status to 'completed'"]
    ReturnResponse["Return response"]

    Start --> CheckKey
    CheckKey -->|"Exists"| KeyFound
    CheckKey -->|"Not found"| ClaimKey

    KeyFound --> CheckStatus
    CheckStatus -->|"completed"| Completed
    CheckStatus -->|"processing"| Processing
    Completed --> HashCheck
    HashCheck -->|"Matches"| ReturnCached
    HashCheck -->|"Doesn't match"| HashMismatch
    Processing --> ReturnConflict

    ClaimKey --> ClaimSuccess
    ClaimSuccess -->|"Yes"| Execute
    ClaimSuccess -->|"No (duplicate insert)"| Duplicate

    Execute --> StoreResult
    StoreResult --> ReturnResponse

    style ReturnCached fill:#27AE60,color:#fff
    style HashMismatch fill:#E74C3C,color:#fff
    style ReturnConflict fill:#E67E22,color:#fff
    style Duplicate fill:#E74C3C,color:#fff

Implementation in Go #

// Table for storing idempotency keys
// CREATE TABLE idempotency_keys (
//   key         VARCHAR(128) PRIMARY KEY,
//   request_hash VARCHAR(64) NOT NULL,
//   status      VARCHAR(16) NOT NULL DEFAULT 'processing',  -- processing | completed
//   response    JSONB,
//   status_code INT,
//   user_id     VARCHAR(64) NOT NULL,
//   endpoint    VARCHAR(256) NOT NULL,
//   expires_at  TIMESTAMP NOT NULL,
//   created_at  TIMESTAMP NOT NULL DEFAULT NOW()
// );

type IdempotencyKey struct {
    Key         string
    RequestHash string
    Status      string
    Response    []byte
    StatusCode  int
    UserID      string
    Endpoint    string
    ExpiresAt   time.Time
}

func (h *Handler) HandleWithIdempotency(w http.ResponseWriter, r *http.Request) {
    idempotencyKey := r.Header.Get("Idempotency-Key")
    if idempotencyKey == "" {
        http.Error(w, "Idempotency-Key header required", 400)
        return
    }

    // Compute a hash from the request body to validate payload consistency
    body, _ := io.ReadAll(r.Body)
    r.Body = io.NopCloser(bytes.NewReader(body))  // reset the body
    requestHash := sha256Hex(body)

    userID := getAuthenticatedUser(r).ID

    // Step 1: Check whether the key already exists
    existing, err := h.repo.GetIdempotencyKey(r.Context(), idempotencyKey, userID)
    if err == nil && existing != nil {
        // Key exists — validate the payload hash
        if existing.RequestHash != requestHash {
            // Different payload for the same key — this is a client bug
            http.Error(w, "Payload mismatch for existing idempotency key", 422)
            return
        }

        if existing.Status == "completed" {
            // Return the cached response
            w.Header().Set("Idempotent-Replayed", "true")
            w.WriteHeader(existing.StatusCode)
            w.Write(existing.Response)
            return
        }

        // Status still "processing" — concurrent request
        http.Error(w, "Request still being processed", 409)
        return
    }

    // Step 2: Atomic claim — insert the key with a UNIQUE constraint
    err = h.repo.ClaimIdempotencyKey(r.Context(), &IdempotencyKey{
        Key:         idempotencyKey,
        RequestHash: requestHash,
        Status:      "processing",
        UserID:      userID,
        Endpoint:    r.URL.Path,
        ExpiresAt:   time.Now().Add(24 * time.Hour),
    })
    if err != nil {
        // Insert failed — concurrent request with the same key
        http.Error(w, "Concurrent request with same key", 409)
        return
    }

    // Step 3: Execute the business logic
    responseBody, statusCode, bizErr := h.executeBusinessLogic(r.Context(), body)
    if bizErr != nil {
        // Update the status to "failed" so it doesn't get stuck on "processing"
        h.repo.UpdateIdempotencyKey(r.Context(), idempotencyKey, "failed", nil, 0)
        http.Error(w, bizErr.Error(), 500)
        return
    }

    // Step 4: Save the response and update the status to "completed"
    h.repo.UpdateIdempotencyKey(r.Context(), idempotencyKey, "completed", responseBody, statusCode)

    // Step 5: Send the response to the client
    w.WriteHeader(statusCode)
    w.Write(responseBody)
}

Request Hashing for Security #

Request hashing is a mechanism to detect when clients send different payloads with the same idempotency key — which could be a client-side bug or an abuse attempt.

Scenarios to prevent:

Request 1 (original):
  POST /payments
  Idempotency-Key: key-abc-123
  Body: { "amount": 500000, "recipient": "acc_budi" }

Request 2 (suspicious retry):
  POST /payments
  Idempotency-Key: key-abc-123  ← the same key
  Body: { "amount": 50000000, "recipient": "acc_fraud" }  ← different payload!

Without a request hash check: the server might return the cached response from request 1
→ The client thinks a Rp 50 million transfer succeeded (when it was only Rp 500 thousand)
→ This is a security issue and a dangerous bug

With a request hash check:
→ The server detects the different hash → returns 422 Unprocessable Entity
→ The client must handle this error

// How to compute a consistent request hash
func calculateRequestHash(body []byte) string {
    // Normalize the JSON before hashing to avoid false mismatches
    // from key ordering or whitespace differences
    var normalized interface{}
    json.Unmarshal(body, &normalized)
    normalizedJSON, _ := json.Marshal(normalized)

    hash := sha256.Sum256(normalizedJSON)
    return hex.EncodeToString(hash[:])
}

// Fields that must NOT be included in the hash:
// - Timestamps that change on every request
// - Nonces or random values
// - Fields genuinely different across retries (not business payload)

TTL and Lifecycle Management #

Idempotency keys must not be stored forever — this causes unbounded storage growth.

TTL guidance by use case:

Financial operations (payment, transfer):
  TTL: 24-48 hours
  Reason: payment gateways usually retry within this window

Order creation:
  TTL: 4-12 hours
  Reason: users won't retry checkout after several hours

Short idempotent operations (data updates):
  TTL: 1-4 hours

Webhook processing:
  TTL: 72 hours
  Reason: some systems retry webhooks for up to 3 days

Key lifecycle:
  processing  → completed (succeeded)
  processing  → failed (failed, can be retried)
  Expired keys → deleted via scheduled jobs

Cleanup strategies:
  Option 1: A daily cron job deleting keys with expires_at < NOW()
  Option 2: Native TTLs in Redis (if using Redis as storage)
  Option 3: Partitioned tables in PostgreSQL with DROP PARTITION for old data

Don’t forget to implement cleanup for expired idempotency keys. Without cleanup, the idempotency_keys table grows linearly over time and can become a performance problem source. A nightly scheduled job deleting expired keys is the minimum requirement.

Natural Idempotency with Database Constraints #

Besides the explicit Idempotency-Key pattern, many operations can be made naturally idempotent using database constraints.

-- Natural idempotency for payment processing
-- No two payments for the same order with the same payment method

CREATE TABLE payments (
    id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    order_id    UUID NOT NULL,
    method      VARCHAR(50) NOT NULL,
    amount      BIGINT NOT NULL,
    status      VARCHAR(20) NOT NULL,
    external_id VARCHAR(128),
    created_at  TIMESTAMP NOT NULL DEFAULT NOW(),

    UNIQUE (order_id, method)  -- ← natural idempotency constraint
);

-- First insert: succeeds
INSERT INTO payments (order_id, method, amount, status)
VALUES ('ord-123', 'credit_card', 500000, 'pending');

-- Second insert (retry): rejected by the database
INSERT INTO payments (order_id, method, amount, status)
VALUES ('ord-123', 'credit_card', 500000, 'pending');
-- ERROR: duplicate key value violates unique constraint "payments_order_id_method_key"

// Handling natural idempotency in the application
func (r *PaymentRepo) CreatePayment(ctx context.Context, p *Payment) (*Payment, error) {
    _, err := r.db.ExecContext(ctx,
        "INSERT INTO payments (order_id, method, amount, status) VALUES ($1, $2, $3, $4)",
        p.OrderID, p.Method, p.Amount, p.Status,
    )
    if err != nil {
        // Check whether this is a unique constraint violation (duplicate = idempotent retry)
        if isUniqueConstraintError(err) {
            // Return the existing payment — this is an idempotent response
            return r.GetByOrderAndMethod(ctx, p.OrderID, p.Method)
        }
        return nil, err
    }
    return p, nil
}

Natural idempotency advantages:
  ✓ Simpler — no separate tables needed
  ✓ Automatically atomic — the database guarantees it
  ✓ No state machine to manage

Natural idempotency limitations:
  ✗ Only works for insert operations (not for more complex operations)
  ✗ Doesn't store the first execution's response
  ✗ Doesn't handle concurrent requests well
  → Use together with the Idempotency-Key pattern for more complex use cases

Concurrent Requests with the Same Key #

One scenario that needs handling is two requests with the same idempotency key arriving almost simultaneously — this happens when clients retry very fast or there’s a client bug sending two requests at once.

Concurrent request scenario:

T=0ms: Request A arrives with key-xyz
T=5ms: Request B arrives with key-xyz (before A finishes)

Without proper handling:
  Both check the DB: key-xyz doesn't exist
  Both execute business logic
  Double payment again!

With an atomic claim (INSERT ... ON CONFLICT):
  Request A: INSERT key-xyz → succeeds (affected rows = 1)
  Request B: INSERT key-xyz → fails (UNIQUE constraint) → return 409
  Only Request A continues execution

-- PostgreSQL: Atomic claim with INSERT ... ON CONFLICT DO NOTHING
INSERT INTO idempotency_keys (key, request_hash, status, user_id, endpoint, expires_at)
VALUES ($1, $2, 'processing', $3, $4, $5)
ON CONFLICT (key) DO NOTHING;

-- Check whether the insert succeeded (affected rows)
-- If 0: the key already exists → duplicate request
-- If 1: claim succeeded → continue execution

func (r *Repo) ClaimIdempotencyKey(ctx context.Context, key *IdempotencyKey) error {
    result, err := r.db.ExecContext(ctx, `
        INSERT INTO idempotency_keys (key, request_hash, status, user_id, endpoint, expires_at)
        VALUES ($1, $2, 'processing', $3, $4, $5)
        ON CONFLICT (key) DO NOTHING`,
        key.Key, key.RequestHash, key.UserID, key.Endpoint, key.ExpiresAt,
    )
    if err != nil {
        return err
    }

    rowsAffected, _ := result.RowsAffected()
    if rowsAffected == 0 {
        return ErrDuplicateIdempotencyKey  // concurrent duplicate
    }
    return nil
}

Informative Response Headers #

When returning a cached response from idempotency, add headers helping clients and operators distinguish original responses from cached ones.

Useful headers:

Idempotent-Replayed: true
  → Indicates this is a cached response, not a new execution result

X-Idempotency-Key: key-abc-123
  → Echo back the used key

X-Original-Request-Time: 2024-01-27T14:32:00Z
  → When the original request was executed

Example cached response:
  HTTP 200 OK
  Idempotent-Replayed: true
  X-Idempotency-Key: key-abc-123
  X-Original-Request-Time: 2024-01-27T14:32:00Z
  Content-Type: application/json

  { "order_id": "ord_789", "status": "confirmed", "total": 500000 }

This is useful for:
  → Monitoring: track how many replays vs new executions
  → Debugging: distinguish original and cached responses when investigating incidents
  → Clients: know whether to update the UI or whether it's already up to date

Idempotency Anti-Patterns to Avoid #

Keys That Are Too Generic or Predictable #

// ✗ Anti-pattern: Keys that can collide
Idempotency-Key: user-123-payment
→ If the user buys twice within the same TTL window,
  the second purchase will be treated as a duplicate!

// ✓ Solution: Keys unique per operation
Idempotency-Key: <uuid-v4 freshly generated for every operation>
→ Clients must generate a new UUID for each different operation
→ The server must not generate keys — that's the client's job

Not Storing the Original Response #

// ✗ Anti-pattern: Only storing "already processed" without the response
if exists(idempotencyKey) {
    return genericSuccessResponse()  // not the original response!
}

// Problem: responses are inconsistent between the original execution and replays
// Clients can't distinguish which is original and which is a replay

// ✓ Solution: Store the full response (body + status code)
// On replay, return EXACTLY the same response
if existing.Status == "completed" {
    w.WriteHeader(existing.StatusCode)
    w.Write(existing.Response)  // byte-for-byte identical
}

“processing” Status Getting Stuck #

// ✗ Anti-pattern: processing status not handled on failures
func processPayment() {
    claimKey()         // status: processing
    charge()           // if it panics here...
    updateKey("completed")  // ...this is never called
}
// The key is stuck on "processing" forever
// All retries are treated as "still being processed" and get 409

// ✓ Solution: Always update the status to "failed" on errors
defer func() {
    if r := recover(); r != nil {
        repo.UpdateStatus(ctx, key, "failed")
        panic(r)
    }
}()
// Or use a defer with a captured error

Storing Idempotency Keys in Client Cookies or LocalStorage #

// ✗ Anti-pattern: Keys generated and stored client-side
// localStorage.setItem('idempotencyKey', uuid())
// → Keys can be deleted by users, browser refreshes, incognito mode
// → Losing the key = retries without idempotency protection

// ✓ Solution: Keys freshly generated per operation
// BUT stored in the right state (React state, server sessions)
// Or: generate when the Submit button is clicked, not when the page loads

Idempotency Implementation Checklist #

DESIGN:
  □ Idempotency mandatory for financial operations (payment, transfer, refund)
  □ Idempotency applied to operations with irreversible side effects
  □ TTLs chosen to match the expected retry window
  □ Status lifecycle defined: processing → completed / failed

IMPLEMENTATION:
  □ Idempotency-Key header accepted and validated (must not be empty)
  □ Request hashes computed from normalized payloads
  □ Atomic claims using INSERT ... ON CONFLICT DO NOTHING
  □ Original responses fully stored (body + status code)
  □ Status updated to "failed" when business logic fails (not stuck on processing)
  □ Idempotent-Replayed: true header returned for cached responses

SECURITY:
  □ Keys scoped per user (user A can't use keys created by user B)
  □ Request hashes validated to detect payload conflicts
  □ Payload mismatches return 422, not silently ignored

STORAGE:
  □ UNIQUE constraint on the key column
  □ Index on (key, user_id) for efficient lookups
  □ Cleanup jobs run periodically for expired keys
  □ Storage chosen by need (Redis for native TTLs, DB for audit trails)

TESTING:
  □ Test: the same request twice produces only one side effect
  □ Test: concurrent requests with the same key — only one processed
  □ Test: different payloads with the same key return 422
  □ Test: expired keys no longer treated as duplicates
  □ Test: "failed" status doesn't block retries with the same key

Summary #

• Idempotency is a property, not a feature — an operation is idempotent if running it once or several times produces the same state. GET and DELETE are idempotent by spec, POST isn’t — but the implementation decides.
• Retries happen more often than you think — mobile SDKs, load balancers, service meshes, and message queues all retry automatically. Non-idempotent systems produce bugs that are hard to reproduce.
• The Idempotency-Key header is the most common pattern — clients generate a unique UUID per operation and send it as a header; the server stores and uses it for deduplication.
• Atomic claims with INSERT … ON CONFLICT — the simplest way to handle concurrent requests with the same key. The one that inserts successfully proceeds; failed inserts return 409.
• Request hashing detects payload conflicts — the same key with a different payload is a client bug or an abuse attempt. Hash payloads and validate their consistency.
• Store original responses, not generic ones — on replay, return exactly the same response (byte-for-byte). This matters for consistency from the client’s perspective.
• “failed” statuses must be retryable — don’t let keys get stuck on “processing” when business logic fails. Use defer to ensure statuses are always updated.
• TTLs chosen by retry windows — payment gateways usually retry within 24-48 hours; use this as TTL guidance. Not too short (retries become dangerous again) or too long (storage bloats).
• Natural idempotency via database constraints — for simpler operations, a UNIQUE constraint on the right field combination is enough. Simpler than a full idempotency layer.
• Monitor replay rates — track what percentage of requests are idempotent replays vs new executions. High replay rates indicate client-side or infrastructure problems.

#

← Previous: N+1 Query   Next: CSR