Checkout outage — 14 Aug postmortem document

```meta
title: Checkout outage — 14 Aug
subtitle: "INC-2026-0814 · SEV-1 · Orders API unavailable for 47 minutes"
tag: POSTMORTEM
```

On 14 Aug the Orders API returned errors for 47 minutes and no customer could complete checkout. The trigger was a routine deploy that enabled the `price-check-v2` flag. The flag made Orders call Pricing synchronously on every request. Pricing ran out of database connections, and retries without backoff turned a slowdown into a full outage.

Assumptions in this document: all times are UTC. The deploy, flag, and service names come from the incident channel. The customer counts come from the checkout funnel dashboard, not from support tickets.

## Detection to resolution

```timeline
items:
  - "[done] 09:41 · Deploy 2026.08.14-3 ships · Enables price-check-v2 for 100% of traffic in one step"
  - "[done] 09:46 · Pricing p95 crosses 2 s · Pool wait time alert fires but pages nobody"
  - "[done] 09:52 · Orders 5xx rate reaches 38% · Load balancer marks 6 of 8 Orders pods unhealthy"
  - "[done] 09:55 · Pager fires for checkout errors · On-call (M. Okafor) acknowledges in 2 minutes"
  - "[done] 10:09 · Root cause suspected · Deploy diff shows the flag flip; Pricing pool at 100% for 20 minutes"
  - "[done] 10:14 · Flag set to 0% · Orders pods recover within 3 minutes; Pricing pool drains by 10:20"
  - "[done] 10:28 · Incident resolved · Error rate below 0.1% for 10 consecutive minutes"
  - "[done] 15 Aug · Deploy reverted, flag rollout policy changed · Any flag over 10% needs a canary stage"
```

Fourteen minutes passed between the first symptom and the page. The pool wait alert existed but routed to a Slack channel, not to the pager, so the first human signal was the customer-facing error rate. The diagnosis itself took 14 minutes. The deploy shipped four unrelated changes, and the flag flip was not in the release notes.

## How the failure spread

```sequence
id: failure-mechanism
actors:
  - { id: Client, name: Web checkout }
  - { id: LB, name: Load balancer }
  - { id: Orders, name: Orders API }
  - { id: Pricing, name: Pricing service }
  - { id: DB, name: Pricing DB }
messages:
  - "Client -> LB: POST /orders"
  - "LB -> +Orders: forward"
  - "Orders -> +Pricing: get price (sync, 5 s timeout)"
  - "Pricing -> DB: wait for pool connection"
  - "loop: retry ×3, no backoff"
  - "Orders -x-> Pricing: timeout after 5 s"
  - "Orders -> Pricing: same request again"
  - "end"
  - "Pricing --> -Orders: 503 pool exhausted"
  - "Orders --> -LB: 502 after 20 s"
  - "LB -x-> Client: 502 Bad Gateway"
  - "LB -> Orders: health check"
  - "Orders --> LB: timeout (threads busy retrying)"
  - { from: LB, to: Orders, kind: note, label: "LB removes the pod; the remaining pods take more traffic" }
foot:
  - { label: Amplification, value: "4 Pricing calls per checkout" }
  - { label: Pool size, value: "20 connections per Pricing pod" }
```

Each checkout held a Pricing connection open for up to 20 seconds and retried three times without backoff, so one slow query cost four connections. With the pool at 20 connections per pod, 45 concurrent checkouts were enough to block every Pricing pod. The load balancer then removed Orders pods whose threads were stuck in retries, which pushed the remaining pods over the same cliff.

## Why it broke

```fishbone
id: root-causes
effect: Orders API unavailable for 47 minutes
causes:
  - label: Change
    items:
      - "Flag went 0% to 100% in one step"
      - "Flag flip bundled with 4 unrelated changes"
      - "No canary for the new Pricing call path"
  - label: Code
    items:
      - "Synchronous Pricing call on the checkout hot path"
      - "3 retries with no backoff and no jitter"
      - "5 s timeout per attempt, 20 s worst case per request"
  - label: Capacity
    items:
      - "Pricing pool sized for 1 call per checkout, not 4"
      - "No circuit breaker between Orders and Pricing"
  - label: Detection
    items:
      - "Pool wait alert routed to Slack, not the pager"
      - "Health check shared a thread pool with request handling"
```

The direct cause was the flag flip, but the outage needed all four bones. Without the retry storm, the pool degrades gracefully. Without the shared thread pool, the load balancer keeps healthy pods. With a paged alert, the response starts nine minutes earlier.

## Impact

```stats
stats:
  - { value: "47 min", label: Checkout unavailable, delta: "+47 min", trend: down }
  - { value: "6,130", label: Failed checkouts, delta: "+6,130", trend: down }
  - { value: "$184k", label: Estimated lost revenue, delta: "-$184k", trend: down }
  - { value: "14 min", label: Time to page, delta: "+9 min vs target", trend: down }
```

Failed checkouts are counted once per session. The revenue figure uses the average order value of the prior seven days. It does not subtract orders that customers retried after 10:28. The funnel shows that 41% of affected sessions converted later that day.

## Remediation

```steps
items:
  - title: Make the Pricing call asynchronous
    body: Orders reads a cached price and enqueues a verification job; a stale price never blocks checkout. Owner M. Okafor, due 28 Aug.
  - title: Add backoff, jitter, and a circuit breaker to the Pricing client
    body: Retries use exponential backoff from 200 ms with jitter, and the breaker opens after 50% failures in 30 s. Owner R. Lindqvist, due 21 Aug.
    code: "retry: { attempts: 3, backoff: exponential, base: 200ms, jitter: full }"
    lang: yaml
  - title: Route the pool wait alert to the pager
    body: Pool wait over 500 ms for 2 minutes pages the Pricing on-call. Owner A. Ferreira, done 15 Aug.
  - title: Separate the health check thread pool
    body: The load balancer probe answers from its own pool so busy request threads never mark a pod unhealthy. Owner R. Lindqvist, due 4 Sep.
  - title: Require a canary stage for every flag above 10%
    body: The flag service refuses a jump from under 10% to over 50% without a 30 minute canary. Policy shipped 15 Aug; enforcement owner A. Ferreira, due 11 Sep.
    note: Applies to all services, not only Orders.
```

Items 1 and 2 remove the mechanism; items 3 and 4 shorten detection and stop the cascade; item 5 removes the trigger class. The team accepts that item 1 can show a price up to 60 seconds stale at checkout, which is within the pricing contract.