```meta
title: Pricing breaker — the 60 seconds of the 8 Sep outage
subtitle: "INC-2026-0908 · SEV-3 · Pricing down for 33 s, checkout served cached prices for 44 s"
tag: INCIDENT
```

On 8 Sep at 14:02 UTC the Pricing database failed over and the Pricing service refused connections for 33 seconds. The circuit breaker in the Orders API pricing client shipped on 21 Aug, after the 14 Aug outage. It opened after 3 seconds and checkout kept working on cached prices. No checkout failed. The breaker stayed open for 14 seconds after Pricing was healthy again, which is the one thing to fix.

```callout
tone: note
title: Assumptions
body: "The outage is INC-2026-0908, 8 Sep, 14:02:00–14:03:00 UTC; t=0 is 14:02:00. The downstream is the Pricing service; the breaker is the one in the Orders API pricing client (config as deployed on 21 Aug). Breaker states come from the breaker state metric at 1 s resolution. Pricing states come from its health endpoint and the database failover log. Checkout traffic during the window was about 40 requests per second."
```

## Breaker, Pricing, and checkout over the minute

```timing
id: outage-minute
title: 14:02:00 to 14:03:00 UTC
unit: s
lanes:
  - label: Breaker
    states:
      - { state: closed, from: 0, to: 11, accent: green }
      - { state: open, from: 11, to: 31, accent: red }
      - { state: half-open, from: 31, to: 33, accent: amber }
      - { state: open, from: 33, to: 53, accent: red }
      - { state: half-open, from: 53, to: 55, accent: amber }
      - { state: closed, from: 55, to: 60, accent: green }
  - label: Pricing
    states:
      - { state: healthy, from: 0, to: 8, accent: green }
      - { state: "timing out", from: 8, to: 14, accent: amber }
      - { state: "connection refused", from: 14, to: 37, accent: red }
      - { state: "warming pool", from: 37, to: 41, accent: amber }
      - { state: healthy, from: 41, to: 60, accent: green }
  - label: Checkout
    states:
      - { state: "fresh price", from: 0, to: 8, accent: green }
      - { state: "slow (800 ms waits)", from: 8, to: 11, accent: amber }
      - { state: "cached price", from: 11, to: 55, accent: blue }
      - { state: "fresh price", from: 55, to: 60, accent: green }
events:
  - { at: 8, label: "Pricing DB failover starts", lane: Pricing }
  - { at: 11, label: "10 of last 20 calls failed", lane: Breaker }
  - { at: 31, label: "3 probes: connection refused", lane: Breaker }
  - { at: 41, label: "Pricing health check green", lane: Pricing }
  - { at: 53, label: "3 probes: 200 OK", lane: Breaker }
constraints:
  - { from: 8, to: 11, label: "3 s to trip" }
  - { from: 11, to: 31, label: "open 20 s" }
  - { from: 33, to: 53, label: "open 20 s" }
  - { from: 41, to: 55, label: "14 s cached while Pricing healthy" }
```

The breaker tripped 3 seconds after the first timeout. At 40 requests per second the 20-call window fills in under a second, and half of it failed by t=11. Checkout was slow for those 3 seconds: each request waited for the 800 ms timeout before it fell back to the cache. The half-open probe at t=31 hit Pricing while it still refused connections, so the breaker reopened for a full second 20 s interval. Pricing was healthy from t=41, but the next probe was not due until t=53. Those 14 seconds are the cost of a fixed open interval with no early probe.

## What moves the breaker between states

```state
id: breaker-states
title: Pricing client breaker
dir: LR
states:
  - { id: s0, col: 1, row: 1, kind: start }
  - { id: closed, col: 2, row: 1, kind: active, name: CLOSED }
  - { id: open, col: 3, row: 1, kind: wait, name: OPEN }
  - { id: halfopen, col: 4, row: 1, kind: wait, name: HALF-OPEN }
transitions:
  - { from: s0, to: closed, event: client starts }
  - { from: closed, to: open, event: call fails, guard: "10 or more of last 20 calls failed" }
  - { from: open, to: halfopen, event: open timer expires, guard: "20 s since open" }
  - { from: halfopen, to: open, event: probe fails, guard: "any of 3 probes" }
  - { from: halfopen, to: closed, event: probe succeeds, guard: "all 3 probes returned 200" }
```

One failed probe sends the breaker back to OPEN for the full 20 s, which is what happened at t=31. The window counts timeouts and connection errors as failures, but not 4xx responses. A 20 s open interval was chosen so that a Pricing pod restart, which takes about 15 s, completes before the first probe.

## What one checkout sees in each state

```sequence
id: checkout-by-state
actors:
  - { id: Client, name: Web checkout }
  - { id: Orders, name: Orders API }
  - { id: Breaker, name: Pricing client breaker }
  - { id: Pricing, name: Pricing service }
  - { id: Cache, name: Price cache }
messages:
  - "Client -> +Orders: POST /orders"
  - "Orders -> +Breaker: get price"
  - "alt: breaker CLOSED (t 0–11, 55–60)"
  - "Breaker -> Pricing: GET /prices/{sku} (800 ms timeout)"
  - "Pricing --> Breaker: 200 price"
  - "Breaker -> Cache: store price, TTL 60 s"
  - "Breaker --> -Orders: fresh price"
  - "else: breaker OPEN (t 11–31, 33–53)"
  - "Breaker -> Cache: read price"
  - "Cache --> Breaker: price, age ≤ 60 s"
  - "Breaker --> -Orders: cached price, under 1 ms"
  - "else: breaker HALF-OPEN (t 31–33, 53–55)"
  - "Breaker -> Pricing: probe, 3 calls allowed"
  - "Pricing -x-> Breaker: connection refused (t 31)"
  - "Breaker -> Cache: read price"
  - "Breaker --> -Orders: cached price"
  - "end"
  - "Orders --> -Client: 201 Created"
foot:
  - { label: Cache hits while open, value: "1,760 of 1,760 checkouts" }
  - { label: Oldest price served, value: "58 s" }
```

Every checkout in the window returned 201, because the cache held a price for every SKU that customers requested. The cache TTL is 60 s and the outage was 44 s of cached serving, so no price expired. In a longer outage, cache entries expire and those checkouts fail with 503, as the pricing contract allows.

## The numbers the breaker ran with

```spec
id: breaker-config
title: Pricing client breaker, config deployed 21 Aug
accent: navy
rows:
  - { label: Call timeout, value: "800 ms per call, no retries while the breaker is closed." }
  - { label: Failure window, value: "Sliding window of the last 20 calls; timeouts and connection errors count, 4xx does not." }
  - { label: Trip threshold, value: "50% of the window failed, so 10 of 20 calls." }
  - { label: Open interval, value: "Fixed 20 s; the same on every reopen, no backoff and no early probe." }
  - { label: Half-open probes, value: "3 calls pass through; all 3 must return 200 to close." }
  - { label: Fallback, value: "Price cache, TTL 60 s; a miss returns 503 to checkout." }
  - { label: Fix proposed, steps: [Watch the Pricing health check while open, "Probe within 2 s of health green", Keep the 20 s cap] }
```

The proposed fix keeps the 20 s cap as the upper bound and adds a health-check trigger for an early probe. With that trigger the breaker closes at about t=43 in this outage, not t=55. Owner R. Lindqvist, due 25 Sep.