DLQ #

In modern distributed systems, failure isn’t something you can avoid. Network glitches, logic bugs, slow or down external dependencies, anomalous data — all of these are guaranteed to happen. What separates a mature system from a fragile one isn’t whether it fails, but how it fails. One of the most important mechanisms for handling failures in message-queue-based systems is the Dead Letter Queue (DLQ) — a “quarantine room” for messages that keep failing to process, so they don’t clog the main queue or get processed in an endless loop. This article covers the DLQ from the concrete problems it solves, why retry alone isn’t enough, DLQ fundamentals, a real implementation case study in Amazon SQS with the complete message flow from start to DLQ, handling strategies, best practices, and common mistakes.

What Is a Dead Letter Queue (DLQ)? #

A Dead Letter Queue (DLQ) is a dedicated queue used to store messages that failed to process after a set number of retry attempts.

Instead of:

• retrying endlessly,
• dropping messages without a trace,
• or grinding the system to a halt,

the message is moved to the DLQ for analysis, debugging, repair, or manual/automatic reprocessing.

A DLQ is the quarantine for problematic messages.

flowchart LR
    P[Producer] --> Q[("(Main Queue\norder-created)")]
    Q --> C["Consumer\nOrder Processor"]
    C -- "✅ Success" --> Done[Done, message deleted]
    C -- "❌ Failed" --> R{"Retry\ncount < max?"}
    R -- Yes --> Q
    R -- "No\nmax reached" --> DLQ[("(Dead Letter Queue\norder-created-dlq)")]
    DLQ --> M["Manual Inspection\n/ Reprocessing"]

The Real Problem Without a DLQ #

Let’s look at the classic problem of not using a DLQ.

Example Case #

A system has an SQS Queue order-created and an order-processor consumer whose job is: validate the order, save it to the database, call the payment service.

The Problem Happens #

Suddenly a message arrives in a broken state — amount is null, the JSON format doesn’t match, or there’s a logic bug in the consumer.

{
  "order_id": "123",
  "amount": null
}

As a result, the consumer always fails to process this message. The message reappears after the visibility timeout, gets processed again, and fails again — repeating endlessly.

The Impact #

sequenceDiagram
    participant Q as Main Queue
    participant C as Consumer

    loop Without a DLQ — repeating forever
        Q->>C: deliver poison message
        C->>C: error! amount = null
        Note over C: message not deleted
        Note over Q: visibility timeout expires
        Q->>C: deliver poison message AGAIN
        C->>C: error! amount = null again
    end
    Note over Q,C: 🔥 Resources wasted\n🔥 Queue blocking\n🔥 Hard debugging

Without a DLQ, this message gets processed over and over without end — wasting resources, disrupting other messages (queue blocking), and making debugging difficult. This is what’s called a poison message.

Why Retry Alone Isn’t Enough #

Retry is important, but not every error can be solved by retrying.

// ANTI-PATTERN: unlimited retry for all errors
func consumeMessage(msg Message) {
    for {
        err := process(msg)
        if err == nil {
            return
        }
        // If the error is permanent (invalid data), this loop NEVER ends
        time.Sleep(time.Second)
    }
}

// CORRECT: bounded retry, then send to the DLQ
func consumeMessage(msg Message, maxReceiveCount int) error {
    if msg.ApproximateReceiveCount >= maxReceiveCount {
        return sendToDLQ(msg, "max retry exceeded")
    }

    if err := process(msg); err != nil {
        if !isRetryable(err) {
            // Permanent error — straight to the DLQ, don't burn through retries
            return sendToDLQ(msg, fmt.Sprintf("permanent error: %v", err))
        }
        return err // let SQS redeliver, ApproximateReceiveCount rises
    }
    return nil
}

Without a DLQ: retry → fail → retry → fail → infinite loop.

With a DLQ: retries are bounded, and once the limit is reached, the message moves to the DLQ.

DLQ Fundamentals #

In general, a DLQ has these characteristics:

flowchart TD
    DLQ[Dead Letter Queue] --> A["A separate queue\nfrom the main queue"]
    DLQ --> B["Not processed\nby the main consumer"]
    DLQ --> C["Stores the payload\n+ failure metadata"]
    DLQ --> D["Used for:\ndebugging, reprocessing, audit"]

Dead Letter Queues in Amazon SQS #

Amazon SQS has native DLQ support through the Redrive Policy feature.

Redrive Policy #

A redrive policy lets the main queue send messages to the DLQ after a message fails to process maxReceiveCount times.

Components #

• Main Queue: order-created
• Dead Letter Queue: order-created-dlq

flowchart TD
    subgraph SQS["Amazon SQS"]
        MQ[("(order-created\nMain Queue)")]
        DLQ[("(order-created-dlq\nDead Letter Queue)")]
    end
    MQ -- "Redrive Policy:\nmaxReceiveCount = 3" --> DLQ

Case Study: DLQ in SQS #

Architecture #

flowchart TD
    P[Producer] --> Q["(SQS: order-created)"]
    Q --> C["Order Processor\nConsumer"]
    C -- success --> Done["✅ Done\nmessage deleted"]
    C -- failure --> Retry{Retry}
    Retry --> Q
    Retry -- "max retry reached" --> DLQ["(SQS: order-created-dlq)"]

The Complete Message Flow #

1. Message Sent #

The producer sends a message:

{
  "order_id": "123",
  "amount": null
}

2. Consumer Fails to Process #

The consumer errors during validation because amount = null. The consumer doesn’t delete the message from the queue.

3. Visibility Timeout #

The message becomes invisible during the visibility timeout period. After the timeout expires, the message reappears in the queue so it can be processed again (by the same consumer or another instance).

4. Repeated Retries #

SQS increments the ApproximateReceiveCount counter every time the message is received:

receive #1 → failed → ApproximateReceiveCount = 1
receive #2 → failed → ApproximateReceiveCount = 2
receive #3 → failed → ApproximateReceiveCount = 3

5. Entering the DLQ #

If maxReceiveCount = 3, then on the 3rd failure, the message is automatically moved to the DLQ.

sequenceDiagram
    participant Q as Main Queue
    participant C as Consumer
    participant DLQ as Dead Letter Queue

    Q->>C: deliver (ApproximateReceiveCount=1)
    C->>C: ❌ failed, amount=null
    Note over Q: visibility timeout expires
    Q->>C: deliver (ApproximateReceiveCount=2)
    C->>C: ❌ failed again
    Note over Q: visibility timeout expires
    Q->>C: deliver (ApproximateReceiveCount=3)
    C->>C: ❌ failed again
    Note over Q: maxReceiveCount reached!
    Q->>DLQ: move message automatically
    Note over DLQ: message + metadata\nawaiting investigation

DLQ Configuration in SQS #

Create the DLQ #

Create a new queue: order-created-dlq. Usually:

• Longer retention (7–14 days) — give enough time for investigation
• No automatic consumer — prevents messages from being processed unknowingly

Attach the Redrive Policy #

On the main queue order-created, set:

• Dead-letter queue: order-created-dlq
• maxReceiveCount: e.g. 3 or 5

{
  "deadLetterTargetArn": "arn:aws:sqs:ap-southeast-1:123456789:order-created-dlq",
  "maxReceiveCount": 3
}

Meaning: if a message is received more than N times without being successfully deleted, send it to the DLQ.

The DLQ must have a longer retention period than the main queue. If the DLQ retention is the same as or shorter than the time your team needs to notice and investigate the problem, messages will be permanently deleted before they can be analyzed.

What’s in the DLQ? #

Messages in the DLQ keep the original payload, plus SQS metadata:

This information is critical for root cause analysis and replaying messages after the bug is fixed.

Strategies for Handling DLQ Messages #

The DLQ isn’t the end — it’s part of the workflow.

flowchart TD
    DLQ["(Message in DLQ)"] --> A{What kind of problem?}
    A -- "Bug in the consumer" --> B["Manual Inspection\nEngineer reads the message,\nfinds the root cause, fixes the bug"]
    A -- "Bug already fixed" --> C["Automated Reprocessing\nMessage sent back\nto the main queue"]
    A -- "Data truly invalid" --> D["Filtering & Discard\nKeep as an audit,\ndon't reprocess"]
    B --> C

Manual Inspection #

An engineer reads the message, finds the root cause, and fixes the bug that caused the failure.

Automated Reprocessing #

After the bug is fixed, messages are moved back to the main queue for reprocessing.

// CORRECT: replay messages from the DLQ to the main queue after a bug fix
func replayFromDLQ(ctx context.Context, dlqURL, mainQueueURL string, maxMessages int) error {
    for i := 0; i < maxMessages; i++ {
        msgs, err := sqsClient.ReceiveMessage(ctx, &sqs.ReceiveMessageInput{
            QueueUrl:            aws.String(dlqURL),
            MaxNumberOfMessages: 1,
        })
        if err != nil || len(msgs.Messages) == 0 {
            break
        }

        msg := msgs.Messages[0]

        // Send back to the main queue
        _, err = sqsClient.SendMessage(ctx, &sqs.SendMessageInput{
            QueueUrl:    aws.String(mainQueueURL),
            MessageBody: msg.Body,
        })
        if err != nil {
            log.Errorf("failed to replay message: %v", err)
            continue
        }

        // Delete from the DLQ after successfully resending
        sqsClient.DeleteMessage(ctx, &sqs.DeleteMessageInput{
            QueueUrl:      aws.String(dlqURL),
            ReceiptHandle: msg.ReceiptHandle,
        })

        log.Infof("replayed message %s from DLQ to main queue", *msg.MessageId)
    }
    return nil
}

Filtering & Discard #

If the data is truly invalid (e.g. old duplicates or stray test data), keep it as an audit log but don’t reprocess it.

DLQ Best Practices in SQS #

Don’t Make the DLQ a Trash Bin #

The DLQ must be monitored, cleaned up, and followed up on — not left to pile up unnoticed.

// ANTI-PATTERN: DLQ left to pile up without monitoring
// After 6 months: 50,000 messages in the DLQ, nobody knows why

// CORRECT: dashboard and a routine process for reviewing the DLQ
func reportDLQStatus(ctx context.Context) {
    attrs, _ := sqsClient.GetQueueAttributes(ctx, &sqs.GetQueueAttributesInput{
        QueueUrl: aws.String(dlqURL),
        AttributeNames: []types.QueueAttributeName{
            types.QueueAttributeNameApproximateNumberOfMessages,
        },
    })

    count := attrs.Attributes["ApproximateNumberOfMessages"]
    metrics.Gauge("dlq.message_count", parseFloat(count))

    if parseFloat(count) > 0 {
        log.Warnf("DLQ order-created-dlq has %s messages — needs investigation", count)
    }
}

Set Up Alarms #

Use a CloudWatch Alarm: if the DLQ has messages > 0, that’s a signal of a serious problem needing immediate attention.

CloudWatch Alarm:
  Metric: ApproximateNumberOfMessagesVisible
  Queue: order-created-dlq
  Threshold: > 0
  Period: 5 minutes
  Action: notify on-call engineer via SNS/Slack

Choose maxReceiveCount Wisely #

Separate Transient vs Permanent Errors #

// CORRECT: classify errors before deciding retry or DLQ
func handleMessage(msg Message) error {
    err := process(msg)
    if err == nil {
        return nil
    }

    if isPermanentError(err) {
        // Permanent error — send to the DLQ immediately, don't burn through retries
        log.Errorf("permanent error, sending to DLQ immediately: %v", err)
        return sendToDLQDirectly(msg, err)
    }

    // Transient error — let SQS redeliver according to the redrive policy
    log.Warnf("transient error, will retry: %v", err)
    return err
}

func isPermanentError(err error) bool {
    var validationErr *ValidationError
    var schemaErr *SchemaError
    return errors.As(err, &validationErr) || errors.As(err, &schemaErr)
}

Store Error Context #

Add correlation IDs, request IDs, and schema/app versions to message metadata so DLQ debugging is easier.

// CORRECT: enrich the message with error context before entering the DLQ
type DLQMessage struct {
    OriginalBody  string            `json:"original_body"`
    ErrorMessage  string            `json:"error_message"`
    CorrelationID string            `json:"correlation_id"`
    ReceiveCount  int               `json:"receive_count"`
    FailedAt      time.Time         `json:"failed_at"`
    ConsumerVersion string          `json:"consumer_version"`
    Attributes    map[string]string `json:"attributes"`
}

Common DLQ Implementation Mistakes #

// ✗ Never looking at the DLQ — messages pile up unnoticed
// ✓ Dashboard + alarm for ApproximateNumberOfMessages > 0

// ✗ maxReceiveCount too large — permanent-error messages clog the main queue longer
// maxReceiveCount: 100  ← too large, longer queue blocking
// ✓ maxReceiveCount: 3-5 for most cases

// ✗ DLQ processed automatically without filtering — invalid messages re-enter the main queue
func autoReplay() {
    for _, msg := range getAllDLQMessages() {
        sendToMainQueue(msg) // ← if still invalid, it returns to the DLQ (loop!)
    }
}
// ✓ Filter and validate before replaying, or fix the root cause first

// ✗ Deleting DLQ messages without analysis — losing important information
deleteAllMessages(dlqURL) // ← the root cause is never found
// ✓ Inspect, record the root cause, then delete or replay

When Is a DLQ Mandatory? #

flowchart TD
    A["System uses a\nmessage queue?"] --> B{"Async /\nevent-driven?"}
    B -- No --> C[DLQ not relevant]
    B -- Yes --> D{"Can you afford\nto lose data on failure?"}
    D -- Yes, non-critical --> E["DLQ optional\nbut recommended"]
    D -- No --> F[DLQ MANDATORY]
    F --> G["SQS / Pub/Sub / Kafka\nall have a DLQ concept"]

A DLQ is mandatory if the system is async/event-driven, can’t afford to lose data, does complex processing, or depends on many dependencies. If you use SQS, Pub/Sub, or Kafka, the DLQ concept (or its equivalent) isn’t optional.

DLQ Implementation Checklist #

CONFIGURATION:
  □ DLQ separate from the main queue, with a clear name (order-created-dlq)
  □ maxReceiveCount set according to error type (generally 3-5)
  □ DLQ retention period longer than the main queue (7-14 days)
  □ No automatic consumer subscribing to the DLQ

ERROR CLASSIFICATION:
  □ Transient errors left to retry via the normal redrive policy
  □ Permanent errors (validation, schema) sent to the DLQ immediately without burning retries
  □ Error context (correlation ID, request ID, version) included in metadata

MONITORING:
  □ CloudWatch Alarm (or equivalent) for ApproximateNumberOfMessages > 0
  □ Dashboard shows the message count per DLQ
  □ Routine process (daily/weekly) for reviewing DLQ messages

HANDLING:
  □ Tools/scripts for replaying messages from the DLQ to the main queue
  □ Documented process: who investigates, when, and how
  □ No DLQ message deletion without root cause analysis

Summary #

• The DLQ is the quarantine for problematic messages — not a trash bin, but an important part of the failure-handling workflow.
• A poison message is a message that always fails to process; without a DLQ, it gets processed endlessly, wasting resources and blocking the queue.
• Retry suits transient errors (network timeout, temporarily down dependency); it doesn’t suit permanent errors (invalid data, logic bug, schema mismatch).
• SQS’s Redrive Policy automatically moves messages to the DLQ after ApproximateReceiveCount exceeds maxReceiveCount.
• DLQ messages carry important metadata — message body, attributes, receive count, and timestamp — all needed for root cause analysis.
• Three handling strategies: manual inspection (find the root cause), automated reprocessing (replay after the fix), filtering & discard (for truly invalid data).
• DLQ retention must be longer than the main queue — give the team enough time to notice and investigate before messages are permanently deleted.
• Alarms for the DLQ are mandatory — messages in the DLQ > 0 is a signal of a serious problem needing attention, not a normal condition to ignore.
• Separate transient and permanent errors in the consumer — permanent errors should go straight to the DLQ without wasting the retry quota meant for transient errors.
• A DLQ is mandatory for async/event-driven systems that can’t afford data loss — SQS, Pub/Sub, and Kafka all have DLQ concepts or equivalents.

← Previous: Backoff Strategy   Next: Unit Test →