Checkout test matrix

```meta
title: Checkout test matrix
subtitle: Which browser and payment method pairs we test, how often, and what each run proves.
tag: QA
```

Checkout has seven browser targets and six payment methods, which is 42 pairs. We do not run all 42 on every pull request. A pair runs on every PR when it carries more than 5% of checkouts, nightly when it carries more than 1%, and weekly otherwise. Pairs the wallets do not support are marked as not applicable and never counted as a gap.

```callout
tone: note
title: Assumptions
body: "Browser targets are the seven that carried 98% of checkouts in the 30 days to 12 Sep 2026. Payment methods are the six live in production. Tests are Playwright end-to-end runs against the staging storefront. The PR tier uses a mocked payment provider; the nightly and weekly tiers use the provider sandbox with real 3-D Secure challenges. Apple Pay runs only in Safari and Google Pay runs only in Chrome and Samsung Internet, which is a platform rule, not a choice."
```

## The matrix

Card is the only method that runs on every PR in every browser. It carries 61% of checkouts and its failure modes differ per browser. Klarna and the gift card run weekly on the low-share browsers. Every cell that says Manual is a pair that Playwright cannot drive: the wallet sheet is a native surface outside the page.

```matrix
id: checkout-matrix
title: Browser × payment method
corner: Browser / Method
cols: [Card, Apple Pay, Google Pay, PayPal, Klarna, Gift card]
rows:
  - { label: Chrome desktop, cells: [Every PR, "n/a", Nightly, Every PR, Nightly, Nightly] }
  - { label: Safari desktop, cells: [Every PR, Manual, "n/a", Nightly, Weekly, Weekly] }
  - { label: Firefox desktop, cells: [Every PR, "n/a", "n/a", Nightly, Weekly, Weekly] }
  - { label: Edge desktop, cells: [Every PR, "n/a", Nightly, Weekly, Weekly, Weekly] }
  - { label: Safari iOS, cells: [Every PR, Manual, "n/a", Nightly, Nightly, Weekly] }
  - { label: Chrome Android, cells: [Every PR, "n/a", Manual, Nightly, Nightly, Weekly] }
  - { label: Samsung Internet, cells: [Nightly, "n/a", Manual, Weekly, Weekly, Weekly] }
```

## What each tier proves

The tiers differ in what they mock. A PR run proves the checkout page and our own API; it cannot prove the provider. The nightly run proves the provider integration on the real sandbox, including the 3-D Secure challenge. The weekly run adds the slow paths: a declined card, a Klarna credit refusal, and a gift card with a partial balance.

```spec
id: checkout-tiers
title: Test tiers
accent: navy
rows:
  - { label: Every PR, value: "7 browsers × card, plus 2 PayPal pairs. Mocked provider. 14 runs, about 6 minutes in parallel. Blocks the merge." }
  - { label: Nightly, value: "17 pairs on the provider sandbox at 02:00 UTC. Real 3-D Secure challenge. About 38 minutes. A failure opens a Jira ticket on the checkout board." }
  - { label: Weekly, value: "The remaining 11 automated pairs on Sunday at 03:00 UTC, plus the decline and partial-balance paths on every automated pair. About 2 hours." }
  - { label: Manual, value: "4 wallet pairs on the device lab before every release candidate. A QA engineer follows the checklist in the release runbook. About 40 minutes." }
  - { label: Escalation, steps: [PR run fails, Author fixes or reverts, Nightly fails twice, Pair moves to Every PR until fixed] }
```

## Browser targets and where they run

Each browser target pins a version range and a runner. Desktop browsers run in Playwright containers. Mobile browsers run on real devices in the lab, because the iOS keyboard and the Android autofill sheet are where mobile-only bugs live.

```table
id: checkout-browsers
title: Browser targets
columns: [Browser, Versions tested, Runner, Share of checkouts]
rows:
  - [Chrome desktop, "Current and current − 1", Playwright container, "34%"]
  - [Safari desktop, "Current (macOS 15)", macOS runner, "9%"]
  - [Firefox desktop, "Current and ESR", Playwright container, "4%"]
  - [Edge desktop, "Current", Playwright container, "3%"]
  - [Safari iOS, "iOS 17 and 18", iPhone 13 and iPhone 15 in the lab, "31%"]
  - [Chrome Android, "Current", Pixel 7 and Galaxy A54 in the lab, "15%"]
  - [Samsung Internet, "Current", Galaxy A54 in the lab, "2%"]
note: "Share is the 30 days to 12 Sep 2026. A browser that falls under 1% for two quarters leaves the matrix; one that rises over 1% joins at the weekly tier."
```

## The matrix by the numbers

Thirty-one of the 42 pairs are automated. The four manual pairs are the wallets, and the seven not-applicable cells are platform rules. Nightly runtime is the number to watch. It grew 11 minutes this quarter as Klarna pairs moved up a tier. Past 60 minutes, the run no longer finishes before the 03:00 deploy window.

```stats
id: checkout-matrix-stats
title: Matrix at a glance · 12 Sep 2026
stats:
  - { value: "42", label: Pairs in the matrix, delta: "0", trend: flat }
  - { value: "31", label: Automated pairs, delta: "+3", trend: up }
  - { value: "4", label: Manual pairs, delta: "0", trend: flat }
  - { value: "38 min", label: Nightly runtime, delta: "+11 min", trend: down, accent: amber }
  - { value: "98.4%", label: Nightly pass rate, delta: "+0.6 pp", trend: up }
```