Generated from: “The median looks fine but people complain checkout is slow — show the latency picture per endpoint.”

Checkout latency — the tail per endpoint

Written by an agent from the skill, validated by chiltepin check, rendered by the renderer — shown as generated, 13 September 2026.

Quality & audits stats percentiles spans callout

DOCUMENTPERF

Checkout latency — the tail per endpoint

Why the median passes and the users still wait.

The checkout median sits under the 300 ms target on every endpoint. The complaints are real because users hit the tail, not the median. One checkout makes four calls in sequence. The chance that at least one call lands in its p99 is close to 4%, one slow checkout in every 25.

Three assumptions hold for this run. The data is the last 7 days of server-side timings from the API gateway. The target is 300 ms at p99 for every endpoint, and POST /payments calls the card provider on the request path.

Median against tail, checkout as a whole

SECTION 01 · Metrics

210ms

End-to-end p50

— -4ms

1.9s

End-to-end p99

▼ +620ms

4.1%

Checkouts over 1s

▼ +1.8pt

p99 to p50 ratio

▼ +3x

Latency per endpoint, last 7 days

SECTION 02 · Latency percentiles

Checkout endpoints — p50 to max

PERCENTILES

Legendp50p90 · p95p99p99 over the SLOmaxSLO

Two endpoints own the tail. POST /payments breaks the target at p95, so one checkout in twenty waits on it. POST /shipping/quote passes at p95 and fails at p99, which points at a slow dependency rather than slow code. GET /cart and POST /cart/validate are within target at every percentile, so cache work there gains nothing that users notice.

Where a p99 checkout spends its time

SECTION 03 · Trace

One checkout at the p99 — 1 880 ms end to end

SPANS

Attributes

carrier rate lookup (UPS)no cache hit; carrier p99
card-provider authorize (attempt 1)timeout at 500 ms

Legendnested spannested twice or morecritical patherrorCLIENTclientDBdb

The trace explains the shape of the percentiles. The card provider times out at 500 ms and the retry doubles the payment span, so the payment p99 is close to two provider calls. The carrier lookup has no cache, so every quote for a new address pays the carrier's own tail.

SECTION 04 · Note

Where the fix is

Warning

Cut the payment retry budget to one attempt with a 300 ms timeout and fail to a retry queue. Cache carrier rates per postcode and weight band for 10 minutes. Do not touch the cart endpoints; they are already within target at p99.

View the Markdown

```meta
title: Checkout latency — the tail per endpoint
subtitle: Why the median passes and the users still wait.
tag: PERF
```

The checkout median sits under the 300 ms target on every endpoint. The complaints are real because users hit the tail, not the median. One checkout makes four calls in sequence. The chance that at least one call lands in its p99 is close to 4%, one slow checkout in every 25.

Three assumptions hold for this run. The data is the last 7 days of server-side timings from the API gateway. The target is 300 ms at p99 for every endpoint, and `POST /payments` calls the card provider on the request path.

## Median against tail, checkout as a whole

```stats
stats:
  - { value: 210ms, label: End-to-end p50, delta: "-4ms", trend: flat }
  - { value: 1.9s, label: End-to-end p99, delta: "+620ms", trend: down }
  - { value: 4.1%, label: Checkouts over 1s, delta: "+1.8pt", trend: down }
  - { value: 9x, label: p99 to p50 ratio, delta: "+3x", trend: down }
```

## Latency per endpoint, last 7 days

```percentiles
id: endpoints
title: Checkout endpoints — p50 to max
unit: ms
slo: 300
rows:
  - { label: GET /cart, p50: 18, p90: 35, p95: 48, p99: 90, max: 410 }
  - { label: POST /cart/validate, p50: 42, p90: 80, p95: 110, p99: 260, max: 1200 }
  - { label: POST /shipping/quote, p50: 65, p90: 140, p95: 210, p99: 540, max: 2800, accent: amber }
  - { label: POST /payments, p50: 240, p90: 380, p95: 470, p99: 900, max: 3100, accent: red }
  - { label: POST /orders, p50: 55, p90: 95, p95: 130, p99: 310, max: 1400, accent: amber }
```

Two endpoints own the tail. `POST /payments` breaks the target at p95, so one checkout in twenty waits on it. `POST /shipping/quote` passes at p95 and fails at p99, which points at a slow dependency rather than slow code. `GET /cart` and `POST /cart/validate` are within target at every percentile, so cache work there gains nothing that users notice.

## Where a p99 checkout spends its time

```spans
id: slow-trace
title: One checkout at the p99 — 1 880 ms end to end
unit: ms
spans:
  - web/checkout: POST /checkout · 0 · 1880
  - api/cart: GET /cart · 6 · 22 · checkout
  - api/validate: POST /cart/validate · 30 · 48 · checkout
  - api/quote: POST /shipping/quote · 80 · 560 · checkout
  - { id: carrier, service: carrier, name: "rate lookup (UPS)", start: 92, duration: 540, parent: quote, kind: client, note: "no cache hit; carrier p99" }
  - api/pay: POST /payments · 650 · 1120 · checkout
  - { id: auth1, service: card-provider, name: "authorize (attempt 1)", start: 660, duration: 500, parent: pay, kind: client, error: true, note: "timeout at 500 ms" }
  - { id: auth2, service: card-provider, name: "authorize (attempt 2)", start: 1165, duration: 580, parent: pay, kind: client }
  - api/order: POST /orders · 1780 · 90 · checkout
  - { id: db, service: postgres, name: "INSERT order + lines", start: 1790, duration: 70, parent: order, kind: db }
```

The trace explains the shape of the percentiles. The card provider times out at 500 ms and the retry doubles the payment span, so the payment p99 is close to two provider calls. The carrier lookup has no cache, so every quote for a new address pays the carrier's own tail.

```callout
tone: warn
title: Where the fix is
body: Cut the payment retry budget to one attempt with a 300 ms timeout and fail to a retry queue. Cache carrier rates per postcode and weight band for 10 minutes. Do not touch the cart endpoints; they are already within target at p99.
```