```meta
title: Poison messages on the fulfillment queue
subtitle: How one malformed message stops blocking the queue, where it goes, and who replays it.
tag: RUNBOOK
```

A poison message is a message that fails on every delivery. Before this design, the `fulfillment-worker` crashed on such a message, the broker redelivered it to the next worker, and every message behind it waited. The fix has two halves. The worker moves a failing message out of the way after a bounded number of tries. The broker does the same when the worker itself dies.

```callout
tone: note
title: Assumptions
body: "The queue is `orders.fulfillment`, a RabbitMQ quorum queue. The consumer is `fulfillment-worker` (3 replicas). The producer is the Orders API, which publishes `order.placed`. Retry delay uses a second queue, `orders.fulfillment.retry`, with a per-message TTL. Names and numbers below are the current production values; change them in one place, the `spec` block, and carry the change into the other blocks."
```

## Where a poison message goes

The worker never requeues a failed message to the head of `orders.fulfillment`. A transient failure goes to the retry queue, which returns it to the main queue after the backoff expires. A permanent failure, or the fifth failure of any kind, goes to `orders.fulfillment.dlq`. The main queue therefore never holds a message that already failed, and the messages behind it keep flowing.

```block
id: topology
preset: event
groups:
  - { id: workers, col: 3, row: 2, cols: 1, rows: 1, label: Consumer group }
nodes:
  - { id: orders, col: 1, row: 2, kind: producer, name: Orders API, tech: order.placed }
  - { id: retry, col: 2, row: 1, kind: queue, name: orders.fulfillment.retry, tech: "TTL, DLX back to main" }
  - { id: q, col: 2, row: 2, kind: queue, name: orders.fulfillment, tech: "quorum, delivery-limit 5" }
  - { id: worker, col: 3, row: 2, kind: consumer, name: fulfillment-worker, replicas: 3 }
  - { id: dlq, col: 4, row: 2, kind: queue, name: orders.fulfillment.dlq, tech: "14 day retention" }
  - { id: replay, col: 4, row: 1, kind: service, name: dlq replay CLI, tech: fulfillment-cli }
edges:
  - orders -> q: publish
  - q -> worker: deliver
  - worker --> retry: "nack transient, set expiration"
  - retry --> q: "TTL expires, redeliver"
  - worker -x-> dlq: "permanent, or 5th failure"
  - q -x-> dlq: "broker: 5 deliveries without ack"
  - replay --> q: "republish, retry count 0"
```

Two paths lead to the DLQ, and they cover different failures. The worker path handles errors the handler can see and classify. The broker path (`delivery-limit: 5` on the quorum queue) handles a worker that dies before it can ack or nack, for example an out-of-memory kill. The broker counts each redelivery in `x-delivery-count`. It dead-letters the message itself on the sixth attempt. Without the broker path, a crash that happens before any handler code runs would loop forever.

## What the worker decides on each delivery

The worker acks the original message in every branch. In the transient branch it acks only after the copy is safely in the retry queue. An ack is what frees the queue head. The order of the checks matters. Schema goes first because an invalid body cannot be classified further. The retry count goes last because it applies to every error class.

```flow
id: decision
title: One delivery, from receipt to ack
nodes:
  - { id: start, col: 1, row: 1, kind: start, label: Message delivered }
  - { id: schema, col: 2, row: 1, kind: decision, label: Body matches order.placed schema? }
  - { id: run, col: 3, row: 1, kind: process, label: Run handler }
  - { id: ok, col: 4, row: 1, kind: decision, label: Handler succeeded? }
  - { id: ack, col: 5, row: 1, kind: end, label: "Ack, done" }
  - { id: perm, col: 4, row: 2, kind: decision, label: Error is permanent? }
  - { id: dlq, col: 5, row: 2, kind: end, label: "Publish to DLQ with reason, ack original" }
  - { id: count, col: 4, row: 3, kind: decision, label: x-retry-count >= 5? }
  - { id: retry, col: 3, row: 3, kind: process, label: "Publish to retry queue with backoff, ack original" }
edges:
  - start -> schema
  - schema -> run: "yes"
  - schema -x-> dlq: "no: schema_invalid"
  - run -> ok
  - ok -> ack: "yes"
  - ok -> perm: "no"
  - perm -x-> dlq: "yes"
  - perm -> count: "no"
  - count -x-> dlq: "yes: retries_exhausted"
  - count -> retry: "no"
  - retry --> start: "after TTL, x-retry-count + 1"
```

## Failure classes

The handler maps every exception to one class. The class decides the branch in the flow above and the team that gets the alert. An exception with no mapping is `unknown`, which retries and then dead-letters, so an unclassified error can delay a message but never block the queue.

```table
id: classes
title: Failure class, action, and owner
columns: [Class, Detected by, Permanent?, Action, Owner]
rows:
  - [schema_invalid, "JSON schema check before the handler runs", { v: "yes", tone: neg }, "DLQ at once, reason in x-death-reason", Orders API team]
  - [unknown_sku, "Catalog lookup returns 404", { v: "yes", tone: neg }, "DLQ at once", Catalog team]
  - [duplicate_order, "Idempotency key already processed", { v: "yes", tone: muted }, "Ack and drop, log at info", Fulfillment team]
  - [downstream_timeout, "Warehouse API call exceeds 5 s", { v: "no", tone: pos }, "Retry with backoff, then DLQ", Fulfillment team]
  - [db_conflict, "Postgres serialization failure or deadlock", { v: "no", tone: pos }, "Retry with backoff, then DLQ", Fulfillment team]
  - [unknown, "Any exception without a mapping", { v: "no", tone: warn }, "Retry with backoff, then DLQ", Fulfillment on-call]
  - [worker_crash, "Broker: no ack within the delivery limit", { v: "n/a", tone: warn }, "Broker dead-letters on the 6th delivery", Fulfillment on-call]
```

`duplicate_order` is the one class that never reaches the DLQ. A replayed message can arrive after a retry of the same message already succeeded. The idempotency key catches that case, and dropping it is the correct result.

## Retry and dead-letter numbers

These values are the contract between the worker, the broker policy, and the alert. The same limit of 5 appears in the worker header check and in the broker `delivery-limit` policy. Keep them equal. If they differ, one path dead-letters earlier than the other and the reason header lies.

```spec
id: numbers
title: Retry and DLQ policy
accent: amber
rows:
  - { label: Max attempts, value: "5 per message. The worker reads x-retry-count; the broker policy delivery-limit is also 5." }
  - { label: Backoff, value: "1 s, 5 s, 30 s, 2 min, 10 min, each with 20% jitter. Set as the message expiration on the retry queue." }
  - { label: Ack mode, value: "Manual. The worker acks after the handler returns or after the copy is in the retry or DLQ queue." }
  - { label: Prefetch, value: "10 per worker replica, so one slow message delays at most 9 others on that replica." }
  - { label: DLQ retention, value: "14 days. After that the message is deleted and the order is lost unless the producer re-emits it." }
  - { label: DLQ headers, value: "x-death-reason (class), x-retry-count, x-first-failed-at, x-last-error (first 512 bytes of the exception)." }
  - { label: Alert, value: "PagerDuty pages Fulfillment on-call when DLQ depth > 0 for 10 min, or > 50 at any time." }
  - { label: Replay, steps: [Inspect, Fix the cause, "fulfillment-cli dlq replay", Confirm DLQ depth returns to 0] }
```

## Replay a message from the DLQ

Replay is manual on purpose. A message in the DLQ failed five times, or failed in a way that a retry cannot fix. A replay without a fix repeats the failure. Fix the cause first, then replay.

```steps
id: replay
title: Replay after a fix
items:
  - title: Read the failed messages and their reasons
    body: The command peeks without consuming. Group the output by x-death-reason before you decide anything.
    code: fulfillment-cli dlq peek --queue orders.fulfillment.dlq --limit 50
    lang: bash
  - title: Fix the cause for that class
    body: For schema_invalid, the Orders API team ships a producer fix. For unknown_sku, the Catalog team adds the SKU. For downstream_timeout, wait until the Warehouse API is healthy.
    note: Do not replay before the fix is in production. A replay without a fix returns the same message to the DLQ with x-retry-count reset to 0.
  - title: Replay one message and watch it
    body: Replay by message id first. The worker logs the outcome with the same id within one backoff window.
    code: fulfillment-cli dlq replay --queue orders.fulfillment.dlq --id 7f3c1a --wait
    lang: bash
  - title: Replay the rest of that class
    body: Replay by reason so messages from other classes stay in the DLQ.
    code: fulfillment-cli dlq replay --queue orders.fulfillment.dlq --reason unknown_sku
    lang: bash
  - title: Confirm the queue is clear
    body: DLQ depth must return to 0 and the PagerDuty alert must resolve on its own. If a message returns to the DLQ, its x-retry-count is 5 again and the fix was incomplete.
    code: fulfillment-cli dlq depth --queue orders.fulfillment.dlq
    lang: bash
  - title: Purge only what is truly dead
    body: Purge by id only a message that no fix can repair, for example a test order from a deleted tenant. Record the reason in the incident ticket.
    code: fulfillment-cli dlq purge --queue orders.fulfillment.dlq --id 7f3c1a --ticket INC-2291
    lang: bash
    note: Purge is irreversible. Retention deletes the rest after 14 days.
```