```meta
title: Outbox for order events
subtitle: Why a committed order can lose its event today, and the transactional outbox that stops it.
tag: DECIDED
```

The Orders API writes the order row, commits, and then publishes `order.placed`
to Kafka. When the broker is down or the process dies between those two
steps, the order exists and the event does not. Billing and Notifications
never hear about it, and nothing retries. The fix moves the event into the
same Postgres transaction as the order, and a separate relay carries it to
the broker.

```callout
tone: note
title: Assumptions
body: "One Postgres database per service, Kafka as the broker, and one Outbox Relay process per service with SKIP LOCKED for safe scale-out. Consumers already key on order_id. The change is scoped to the Orders API; other services adopt the same table when they next touch their publish path."
```

## One transaction, then a relay

The order row and the outbox row commit together or not at all. The API
returns `201` as soon as the commit lands, and never talks to Kafka. The
Outbox Relay polls `outbox` every 200 ms, publishes each `PENDING` row, and
marks it `PUBLISHED` only after the broker acknowledges the write. A broker
outage leaves rows `PENDING`; it cannot lose them.

```sequence
id: outbox-sequence
actors:
  - { id: Client, name: Client }
  - { id: OrdersAPI, name: Orders API }
  - { id: Postgres, name: Postgres }
  - { id: Relay, name: Outbox Relay }
  - { id: Kafka, name: Kafka }
  - { id: Billing, name: Billing }
messages:
  - Client -> +OrdersAPI: POST /orders
  - OrdersAPI -> +Postgres: BEGIN
  - OrdersAPI -> Postgres: INSERT orders (status PAID)
  - OrdersAPI -> Postgres: INSERT outbox (order.placed, PENDING)
  - OrdersAPI -> Postgres: COMMIT
  - Postgres --> -OrdersAPI: committed
  - OrdersAPI --> -Client: 201 Created
  - loop: every 200 ms
  - Relay -> +Postgres: SELECT outbox WHERE status = PENDING FOR UPDATE SKIP LOCKED
  - Postgres --> -Relay: up to 100 rows
  - Relay --> +Kafka: publish order.placed (key = order_id)
  - alt: broker acknowledges
  - Kafka --> -Relay: ack (partition, offset)
  - Relay -> Postgres: UPDATE outbox SET status = PUBLISHED, published_at = now()
  - Kafka --> Billing: order.placed
  - Billing -> Billing: skip if event_id already in processed_events
  - else: broker down or timeout after 5 s
  - Kafka -x-> -Relay: no ack
  - Relay -> Postgres: UPDATE outbox SET attempts = attempts + 1, next_attempt_at = backoff
  - end
  - end
```

## Outbox row lifecycle

A row is born `PENDING` inside the order transaction and leaves that state
only on a broker acknowledgement. Ten failed attempts move it to `FAILED`,
which pages the on-call engineer; a requeue puts it back to `PENDING` with
`attempts` reset to zero. Published rows stay seven days for audit and
replay, then a nightly job deletes them.

```state
id: outbox-row-states
dir: LR
states:
  - { id: s0, col: 1, row: 1, kind: start }
  - { id: pending, col: 2, row: 1, kind: wait, name: PENDING }
  - { id: published, col: 3, row: 1, kind: active, name: PUBLISHED }
  - { id: purged, col: 4, row: 1, kind: terminal, name: PURGED }
  - { id: failed, col: 2, row: 2, kind: wait, name: FAILED }
transitions:
  - s0 -> pending: INSERT in the order transaction
  - pending -> published: broker ack
  - pending -> failed: 10 attempts without ack
  - failed -> pending: operator requeues
  - published -> purged: 7 days after published_at
```

## Guarantees and numbers

The pattern trades exactly-once for at-least-once plus idempotent consumers.
A relay that crashes after the broker ack but before the `UPDATE` republishes
the same row, so every consumer must treat `event_id` as the dedupe key.
These values are the contract; change them in config and in this table
together.

```spec
id: outbox-invariants
title: Outbox contract
accent: teal
rows:
  - { label: Atomicity, value: "The orders row and the outbox row commit in one Postgres transaction. No code path commits one without the other." }
  - { label: Delivery, value: "At-least-once. A row is PUBLISHED only after the broker ack; a crash between ack and UPDATE republishes it." }
  - { label: Idempotency, value: "Consumers store event_id in processed_events for 7 days and skip a duplicate before any side effect." }
  - { label: Ordering, value: "Partition key is order_id. The relay publishes rows in id order, so events for one order stay in sequence." }
  - { label: Retry, steps: ["Attempt 1 at once", "Backoff 1 s, 2 s, 4 s ... capped at 60 s", "FAILED after attempt 10", "Page on-call"] }
  - { label: Poll, value: "Every 200 ms, batch of 100 rows, SELECT ... FOR UPDATE SKIP LOCKED so two relay instances never take the same row." }
  - { label: Lag target, value: "p99 commit-to-ack under 2 s. Alert when the oldest PENDING row is older than 60 s." }
  - { label: Retention, value: "PUBLISHED rows are deleted 7 days after published_at. FAILED rows are kept until an operator requeues or deletes them." }
```

## What each fault does now

Every fault that lost events before now either delays the event or produces
a duplicate that the consumer drops. No fault deletes a committed event.

```table
id: outbox-faults
columns: [Fault, Before the outbox, With the outbox]
rows:
  - ["Broker down when the order commits", { v: "Event lost; order exists with no downstream work", tone: neg }, { v: "Row stays PENDING; relay retries with backoff until the broker returns", tone: pos }]
  - ["API process dies after COMMIT", { v: "Event lost", tone: neg }, { v: "Row is already committed; relay publishes it on the next poll", tone: pos }]
  - ["Relay dies after ack, before UPDATE", "Not applicable", { v: "Row is republished; consumer drops the duplicate by event_id", tone: warn }]
  - ["Two relay instances poll at once", "Not applicable", { v: "SKIP LOCKED gives each row to one instance", tone: pos }]
  - ["Postgres down during POST /orders", "503 to the client, nothing written", { v: "503 to the client, nothing written; no event to lose", tone: muted }]
  - ["Payload the broker rejects every time", { v: "Silent loss after one attempt", tone: neg }, { v: "FAILED after 10 attempts; on-call is paged with the row id", tone: warn }]
note: "Attempt count, backoff cap, and retention are the production values on the day of this decision."
```