Start with a template

Choose from 18 example documents, grouped by task. Replace the sample system details with your own. Try a template in the Studio demo or create a local file with chiltepin new <name>.

Looking for a worked example? Browse the gallery.

Architecture, APIs and data

Explain a system, propose a change, or document its contracts.

DOCUMENTService · Tier 1

orders-api

Owns the order lifecycle from cart submission to fulfilment handoff.

At a glance

SECTION 01 · Spec

The facts you need at 3am

Owner

Orders team · #team-orders

On-call

PagerDuty schedule orders-primary

Repo

github.com/acme/orders-api

Runtime

Go 1.23 · Kubernetes · 12 pods, HPA to 40

Datastores

Postgres (orders) · Redis (idempotency) · SQS (fulfilment)

Dashboards

Grafana Orders-overview · Honeycomb dataset orders-api

Deploy

Merge to main → canary 10% → full, roughly 18 minutes

Tier

Tier 1 — customer-facing, revenue path

What it does

Accepts a submitted cart, prices it, reserves stock, takes payment through the payments service, and hands a confirmed order to fulfilment. It owns order state; nothing else writes the orders table. Everything downstream reacts to the order.confirmed event rather than calling back in.

SECTION 02 · Architecture

What it talks to

ARCH

1POST /orders2reserve3charge4order.confirmed

LegendCLIENTclientSVCserviceDBdatabaseQUEUEqueuecalls

Service levels

SECTION 03 · Service objectives

What we promise, and where we are

Order submission availability30d

Non-5xx on POST /orders

Target99.95%

Current99.97%

42% of error budget used

Submission latency30d

p95 end to end

Target< 800ms

Current610ms

31% of error budget used

Fulfilment handoff30d

order.confirmed published within 5s

Target99.9%

Current99.4%

87% of error budget used

Dependencies

SECTION 04 · Comparison

Dependency	Kind	If it is down
payments	Sync, blocking	Submission fails — this is a hard dependency
inventory	Sync, blocking	Falls back to optimistic reservation for 10 minutes
Postgres	Sync, blocking	Total outage — fail fast and page
Redis	Sync, degraded	Retries lose idempotency; we fail closed
fulfilment.jobs (SQS)	Async	Orders confirm and queue locally; handoff catches up

Common operations

SECTION 05 · Steps

The three things people ask for

1
Find one order end to end
The order id is in every log line and every span.
bash
```
kubectl logs -n orders -l app=orders-api --since=1h | grep ord_01H…
```
2
Replay a stuck fulfilment handoff
Safe to run twice — the consumer is idempotent on order id.
bash
```
./bin/orders replay-handoff --order ord_01H… --dry-run=false
```
3
Roll back a bad release
bash
```
kubectl -n orders rollout undo deploy/orders-api
```
Schema changes are expand-only, so a rollback never needs a database change.

SECTION 06 · FAQ

Questions we answer weekly

Can I write to the orders table from my service?

No. Order state has one owner. Subscribe to order.confirmed, or ask us for an endpoint.

Why did an order confirm but never ship?

Almost always a stuck handoff — check the fulfilment.jobs DLQ, then replay it with the command above.

How do I test against it locally?

docker compose up in the repo root brings up the API, Postgres and a fake payments service seeded with test cards.

What is the retention on orders?

Seven years, legally required. Deletion requests redact PII in place and keep the financial record.

Service overview

The page you want at 3am — owner, architecture, SLOs, dependencies, and common operations.

specblockslotablesteps+1

7 blocksTry in Studio demo

DOCUMENTRFC · Draft

Media processing off the upload path

A proposal to move thumbnailing and transcoding to an async pipeline.

Problem

Uploads block on media processing today: a 40 MB video holds an API worker for up to 90 seconds. One slow transcode can starve the pool. Describe the pain in two or three sentences — numbers beat adjectives.

Where the time goes

SECTION 01 · Hierarchy

Why uploads are slow

MECE

Legendquestionbranchtestable claim

Alternatives

SECTION 02 · Options

Approaches explored

Option 1Bigger workers

Scale the API pool vertically.

No code change

Cost scales with the worst upload
Still blocks the pool

REJECTED — buys months, not a fix

Option 2Async pipeline

Accept the upload, enqueue processing, notify on completion.

Upload latency independent of media size
Retryable steps

Eventual consistency for previews

CHOSEN — isolates the slow work

Proposed design

SECTION 03 · Architecture

The async pipeline

ARCH

LegendSVCserviceQUEUEqueueDBdatabasecallsasync / optional

Behavior

SECTION 04 · Sequence

One upload, end to end

SEQUENCE

Legendcallresponseasyncthe answer the caller getsactive

Open questions

SECTION 05 · Status

To resolve before build

Question	Where it stands	Status
Poison-pill handling — max retries before dead-letter?	Needs a decision from the platform team	todo
Do previews need a placeholder rendition synchronously?	Prototype scheduled for next sprint	todo

Design doc

An RFC-style proposal — problem, goals, proposed design, alternatives, rollout.

treeoptionsblocksequencestatustable

6 blocksTry in Studio demo

DOCUMENTADR · Accepted · 2026-03-04

ADR-014 — Idempotency keys on the payments API

Clients supply a key; the API replays the first response instead of charging twice.

Status

Accepted — 2026-03-04. Supersedes ADR-009 (retry budgets). Proposed → Accepted → Superseded.

Context

Mobile clients retry POST /payments whenever the network drops, and a retry that lands after the charge succeeded bills the customer a second time. We saw nine duplicate charges last quarter, every one of them refunded by hand. The PSP deduplicates on its own reference, but only within a five-minute window, which is shorter than our longest client retry backoff.

SECTION 01 · Note

Decision

Note

Clients send an Idempotency-Key header on every mutating payments call. The API stores the key with the response for 24 hours and replays that response verbatim for any repeat. A retry can never produce a second charge.

What forced it

SECTION 02 · Drivers

The forces behind this decision

Retries are not optional

Mobile networks drop mid-request, and the client cannot tell a lost response from a failed charge.

Constraint

The PSP window is too short

Provider-side dedup lasts five minutes; our longest client backoff is thirty.

Constraint

Money is not idempotent by nature

Every duplicate is a refund, a support ticket and a trust cost we cannot automate away.

Risk

The ecosystem already agreed

Stripe and Adyen both use a client-supplied key header — our SDK users expect it.

Precedent

How it works

SECTION 03 · Sequence

The second call never reaches the PSP

SEQUENCE·POST/payments

Legendcallresponsethe answer the caller getsEXTexternal actoractive

Charges: exactly oneReplay cost: one Redis GET

Options considered

SECTION 04 · Options

Three ways to make a retry safe

ChosenClient-supplied idempotency key

The client sends `Idempotency-Key` per user action; the API claims it before charging and replays the stored response for repeats.

The client defines what "the same request" means, so a deliberate second charge still works
One header, one lookup — the replay path is trivial to reason about and to test
Matches Stripe and Adyen, so SDK users need no explanation

Clients must generate and persist a key across retries
Adds Redis to the payments write path

Cheapest correct answer, and the one our clients already know.

RejectedServer-side request fingerprinting

Hash method, path, body and caller, and treat a repeat hash inside a window as a duplicate.

Nothing changes for existing clients
No new client-side state

Guesses at intent — two legitimate identical charges silently collapse into one
Needs a body-hash index on every mutating route, not one header

Rejected — a payments API must never decide for the caller that a charge was a mistake.

RejectedLean on the PSP

Pass our order reference through and let the provider deduplicate.

No code, no new dependency

Five-minute window, shorter than our own retry backoff
Our own writes still double up even when the charge does not
Couples correctness to a vendor behaviour we do not control

Rejected — this is the status quo, and it is what produced the nine duplicates.

Architecture

SECTION 05 · Architecture

Where the key is checked

ARCH

LegendCLIENTclientSVCserviceCACHEcacheEXTexternalcalls

Consequences

SECTION 06 · Status

Consequence	What it means for us	Status
Retries are safe by default	Clients drop their bespoke retry guards; the SDK sends a UUID per user action	completed
A new Redis dependency on the write path	Payments now fails closed if the key store is down — capacity and alerting owned by Platform	in progress
Keys expire after 24 hours	A retry the next day charges again. Support needs the runbook line, and we watch the replay-rate metric	todo

Legendcompletedin progresstodo

What could go wrong

SECTION 07 · Risk register

Risks we are accepting, and what we do about them

highKey store outage blocks every paymentPlatformmitigating

L: low · I: high

Mitigation: Fail closed with 503 + Retry-After; Redis runs multi-AZ with a 60s failover drill each release

highA client reuses one key for genuinely different chargesPaymentsmitigating

L: med · I: high

Mitigation: Store a request fingerprint beside the key and return 422 when it disagrees

mediumClients never send the headerPaymentsopen

L: med · I: med

Mitigation: Log the miss rate per client; the header becomes required at v2

Rollout

SECTION 08 · Steps

From merged to required

1
Ship the middleware, keys optional
A request without a key behaves exactly as it does today, so nothing breaks on deploy.
bash
```
PAYMENTS_IDEMPOTENCY=shadow
```
Send keys from the SDKs
One UUID per user action, persisted with the pending payment so a cold start reuses it.
iOS and Android ship on their own cadence — expect six weeks before the miss rate flattens.
3
Watch the replay rate
Replays should track the retry rate. A replay rate near zero means clients regenerate keys per attempt.
promql
```
sum(rate(payments_idempotent_replay_total[5m])) by (client)
```
Make the header required
Once the miss rate is under 1% for two weeks, missing keys get 400 on v2. v1 keeps the old behaviour until we retire it.

ADR

An architecture decision record — status, context, the decision, options, consequences.

calloutdriverssequenceoptionsblock+3

9 blocksTry in Studio demo

DOCUMENTSystem design · Draft

Webhook delivery service

Design for delivering customer webhooks reliably at platform scale.

Requirements

SECTION 01 · Drivers

What the design must satisfy

At-least-once delivery

Every event reaches every subscribed endpoint.

MUST

Ordered per subscription

Events for one subscription arrive in order.

MUST

Slow consumers isolated

One dead endpoint never delays the rest.

MUST

Replay window

Customers can replay the last 7 days.

SHOULD

Envelope math

SECTION 02 · Capacity math

Delivery-path capacity

Events per day

40M

Subscriptions per event

1.5 avg

Deliveries per day40M × 1.560M/day

Steady QPS60M / 86,400 s≈ 700 rps

Peak QPS700 × 4 (bursts)≈ 2,800 rps

Provision for

~5,600 rps (2× peak headroom)

Contract

SECTION 03 · API endpoint

Subscribe to events

POST/webhook-subscriptions

Request body

Field	Type	Description
url required	string	HTTPS endpoint to deliver to
events required	string[]	Event types to receive

Responses

Status	Description
201	Subscription created
422	URL failed the handshake ping

Architecture

SECTION 04 · Architecture

Delivery topology

INFRA

LegendSVCserviceQUEUEqueueDBdatabasecallsasync / optional

The bottleneck — slow consumers

SECTION 05 · Options

Isolating slow endpoints

Option 1Shared worker pool

All deliveries through one pool with timeouts.

Simple

One stuck endpoint consumes the pool

REJECTED — fails isolation

Option 2Per-subscription queues

Scheduler drains each subscription independently with backoff.

Head-of-line blocking scoped to one customer
Natural per-endpoint rate limits

More queue state to manage

CHOSEN

Delivery objectives

SECTION 06 · Service objectives

Delivery latency30d

Deliveries attempted under 60 s of event time

Target99%

Current99.4%

40% of error budget used

Delivery success30d

Deliveries succeeding within the retry policy

Target99.9%

20% of error budget used

Plan

SECTION 07 · Roadmap

current

Ingest + log

Events API writing to the log.

Scheduler + senders

Per-subscription delivery with backoff.

future

Replay API

7-day replay from the log.

Legendcurrentnextfuture

System design

A system-design write-up — requirements, capacity math, architecture, data model, SLOs.

driversenvelopeendpointblockoptions+2

8 blocksTry in Studio demo

DOCUMENTAPI · v1

Sessions API

Create and inspect login sessions for first-party clients.

All routes are under https://api.example.com/v1 and require a client key.

Create a session

SECTION 01 · API endpoint

POST/sessionsClient key (X-Client-Key header)

Exchange user credentials for a session token.

Request body

Field	Type	Description
email required	string	Account email
password required	string	Account password
remember	boolean	Extend the session to 30 days

Responses

Status	Description
201	Session created
401	Bad credentials
429	Too many attempts — retry after the indicated delay

Example request

{ "email": "sam@example.com", "password": "…", "remember": true }

Example response

{ "token": "ses_9f2…", "expiresAt": "2026-08-01T12:00:00Z" }

Request flow

SECTION 02 · Sequence

Credential exchange

SEQUENCE·POST/sessions

Legendcallresponsethe answer the caller getsactive

Data touched

SECTION 03 · Entity model

Legend#primary key→foreign key1 / Ncardinalityaggregate root

Status codes

SECTION 04 · Comparison

Status	Meaning	Client action
201	Session created	Store the token
401	Bad credentials	Prompt the user again
429	Rate limited	Retry after the Retry-After header

API spec

An API reference — endpoint cards with params, bodies, responses, and examples.

endpointsequenceerdtable

5 blocksTry in Studio demo

DOCUMENTData model · v1

Orders data model

The tables behind ordering, and the lifecycle of an order row.

Entities

SECTION 01 · Entity model

Orders and their line items

Legend#primary key→foreign key1 / Ncardinality

Field semantics

SECTION 02 · Comparison

Field	Semantics
orders.status	Enum order_status — see the lifecycle below; only the state machine writes it
orders.total_cents	Integer cents, never floats; recomputed from items on every mutation
order_items.sku	Denormalized from the catalog at purchase time — price changes never rewrite history

Record lifecycle

SECTION 03 · State machine

order_status transitions

STATE

Legend●startstatewaitingtransitionerror exiterror transitionsuccess exit

№	From	Event	Guard	To
1	s0	create	—	PENDING
2	PENDING	capture	—	PAID
3	PENDING	cancel / timeout	—	CANCELLED
4	PAID	fulfil	—	SHIPPED
5	SHIPPED	deliver	—	done

Data model

A data-model reference — the ERD, per-entity tables, and lifecycle states.

erdtablestate

4 blocksTry in Studio demo

Operations and releases

Prepare, ship, and support a change.

DOCUMENTRunbook · On-call

Runbook — elevated 5xx on the API

Triage and mitigate a 5xx spike on the public API.

SECTION 01 · Note

When to use this

Warning

The api-5xx-rate alert fired (>1% of requests over 5 minutes). If the whole site is down, use the total-outage runbook instead.

Triage

SECTION 02 · Flowchart

Decide which case you are in

FLOW

1yes2no3yes4no

Legendstartstepdecision (diamond)exitnexterror path

Act

SECTION 03 · Steps

Roll back the last deploy

1
Confirm the suspect release
The release that started serving right before the spike.
bash
```
kubectl rollout history deploy/api | tail -3
```
2
Roll back
bash
```
kubectl rollout undo deploy/api
```
Rollback takes ~3 minutes to settle; watch the error rate, not the pods.
3
Verify
5xx rate should drop below 0.1% within five minutes.
bash
```
curl -s https://status.internal/api/error-rate
```

Symptom → action

SECTION 04 · Comparison

Symptom	Likely cause	Action
5xx only on POST /orders	Payment provider degraded	Enable the capture circuit breaker
Timeouts across all routes	DB primary saturated	Fail over to the replica
Spike right after a deploy	Bad release	Roll back (steps above)

Runbook

An operational how-to — prerequisites, numbered steps with commands, verification, rollback.

calloutflowstepstable

5 blocksTry in Studio demo

DOCUMENTMigration · In flight

Orders MySQL → Postgres

Dual-write, backfill, verify, cut over — with a rollback that stays open until the end.

SECTION 01 · Note

The one rule

Warning

Rollback stays possible until the final step. Every phase below is reversible on its own, and nothing deletes the MySQL data until two weeks after cutover.

Where we are, where we land

SECTION 02 · Before / after

What actually changes

MySQL 5.7 (today)

Single primary, vertical scaling only — already on the largest instance

No native JSON indexing, so the order metadata is a serialized blob

Schema changes lock the table; releases wait for a maintenance window

Read replicas lag 4-9s under load, so reporting reads the primary

→

Postgres 16 (after)

Primary + two replicas, logical replication for zero-downtime upgrades

jsonb with GIN indexes — meta queries no longer scan the full table

Non-blocking DDL for the changes we make weekly

Replicas under 500ms, so reporting moves off the primary

The application data model does not change. This is an engine migration, not a rewrite.

Phases

SECTION 03 · Roadmap

Week 1

done

Schema parity

Postgres schema generated and diffed against MySQL in CI.

Week 2-3

current

Dual write

Writes go to both engines; MySQL stays authoritative.

Week 4

Backfill + verify

Historic rows copied, then row-count and checksum comparison per table.

Week 5

Read cutover

Reads move to Postgres behind a flag, one endpoint at a time.

Week 6

future

Write cutover

Postgres becomes authoritative; MySQL still takes writes as the escape hatch.

Week 8

future

Decommission

MySQL writes stop and the instance is snapshotted, then retired.

Legenddonecurrentnextfuture

Cutover runbook

SECTION 04 · Steps

The 20-minute window

1
Freeze writes at the edge
The API returns 503 with Retry-After for mutating routes. Reads keep serving from Postgres.
bash
```
kubectl -n orders set env deploy/orders-api WRITE_FREEZE=1
```
Clients retry automatically. Anything longer than 90 seconds shows in the checkout funnel.
2
Drain in-flight writes
Wait for the dual-write queue to reach zero before comparing anything.
bash
```
psql -c "select count(*) from replication_lag where behind_ms > 0"
```
3
Verify the two engines agree
Row counts and per-table checksums must match exactly. A mismatch aborts the cutover — this is the go/no-go.
bash
```
./scripts/verify-parity.sh --strict
```
4
Promote Postgres
Flip the authority flag. MySQL keeps taking writes so a rollback is still one flag away.
bash
```
kubectl -n orders set env deploy/orders-api DB_AUTHORITY=postgres WRITE_FREEZE=0
```
Watch for 30 minutes
Error rate, checkout conversion and p95 all back to baseline before anyone leaves the call.

If it goes wrong

SECTION 05 · Flowchart

Rollback decision

FLOW

1yes2no3yes4no

Legendstartstepdecision (diamond)exitnexterror path

Risks

SECTION 06 · Risk register

highSilent data divergence during dual writeDatamitigating

L: med · I: high

Mitigation: Per-table checksums run hourly and gate the cutover

highQuery plans regress on PostgresOrdersmitigating

L: high · I: med

Mitigation: Top 40 queries replayed against a production-sized copy; two rewritten already

highCutover window overrunsOrdersclosed

L: low · I: high

Mitigation: Rehearsed twice against a restored snapshot — last run took 11 minutes

mediumReporting jobs still point at MySQLAnalyticsopen

L: med · I: low

Mitigation: Connection strings inventoried; four jobs left to move

Tracking

SECTION 07 · Status

Workstream	Latest	Status
Schema parity in CI	Green for 9 days — diff runs on every PR	completed
Dual write in production	Live for all writes; queue depth steady under 200	in progress
Backfill of 240M historic rows	Runs nightly in 6h batches — 61% complete	in progress
Query plan review	Two rewrites merged, 6 of 40 queries left	in progress
Reporting jobs moved	Blocked on the Analytics migration window	blocked

Legendcompletedin progressblocked

Migration plan

A phased cutover — before/after, phases, the runbook, rollback, and the risk register.

calloutcvttimelinestepsflow+2

8 blocksTry in Studio demo

DOCUMENTPostmortem · SEV-1 · Resolved

Postmortem — checkout outage 2026-06-12

41 minutes of failed checkouts caused by an expired PSP certificate.

SECTION 01 · Note

Impact

Danger

41 minutes of checkout failures (14:03-14:44 UTC). ~3,100 failed purchases, ~$46k delayed revenue. No data loss; retried payments settled by 16:00.

Timeline

SECTION 02 · Roadmap

14:03

done

Cert expires

PSP client certificate lapses; captures start failing.

14:07

done

Alert fires

checkout-error-rate pages the on-call.

14:31

done

Root cause found

TLS handshake errors traced to the expired cert.

14:44

done

Mitigated

Renewed cert deployed; error rate back to baseline.

Root cause

SECTION 03 · Hierarchy

Why checkouts failed

MECE

Legendquestionbranchtestable claim

Action items

SECTION 04 · Status

Corrective actions

Action	Owner / due	Status
Alert on certs expiring within 21 days	Platform — due Jun 26	in progress
Automate PSP cert rotation	Payments — due Jul 15	todo
Queue captures when the PSP is unreachable	Payments — needs sizing	todo

Legendin progresstodo

Postmortem

An incident review — impact, timeline, root cause, and follow-up actions.

callouttimelinetreestatustable

5 blocksTry in Studio demo

DOCUMENTSecurity · Reviewed 2026-02-18

Threat model — public share links

Anyone with the URL can read the file. What that lets an attacker do, and what stops them.

Scope

SECTION 01 · Spec

What we model

Feature

Public share links for uploaded files

Assets

File contents · file names · uploader identity · the link itself

Trust boundary

The public internet reaches the share service directly

Attacker

Unauthenticated, has a link or can guess one · a logged-in user of another tenant

Out of scope

Physical access · the CI supply chain · the S3 control plane

Reviewed by

Security + Platform · next review at GA

Data flow

SECTION 02 · Data flow

Where the data crosses the boundary

DFD

1GET /s/{token}2resolve + check expiry3signed URL, 60s4create / revoke link

Legendprocessexternal entitydata storedata flow

Threats

SECTION 03 · Risk register

STRIDE, applied to the flow above

criticalToken guessing — enumerate /s/{token} until something resolvesPlatformmitigating

L: high · I: high

Mitigation: 128-bit random tokens, no sequence; 20 req/min/IP; alert on a 404 burst

highSigned URL leaks via Referer to a third-party pagePlatformmitigating

L: med · I: high

Mitigation: Referrer-Policy: no-referrer on the download route; URLs expire in 60 seconds

highRevoked link keeps working from cacheShareopen

L: med · I: high

Mitigation: Revocation writes a tombstone the CDN honours; TTL capped at 30s

mediumUploader identity inferred from the file nameShareaccepted

L: med · I: low

Mitigation: Served under an opaque path with the original name only in Content-Disposition

highStored XSS from an HTML upload rendered inlineSharemitigating

L: med · I: high

Mitigation: Content-Disposition: attachment plus a sandbox CSP on the download origin

mediumDenial of wallet — a hot link drives egress costPlatformopen

L: low · I: med

Mitigation: Per-link bandwidth cap; the link parks itself after 50GB

Controls in place

SECTION 04 · List

What already stops these

128-bit tokensGenerated with a CSPRNG — guessing is not a practical attack
Short-lived signed URLs60-second expiry — a leaked URL is worthless within the minute
Downloads on a separate originNo session cookie exists there, so a malicious upload cannot reach an authenticated context
Rate limit + burst alert20 requests/min/IP, and a 404 burst pages the on-call
CDN revocation tombstonesRevocation must invalidate the edge, not just the origin
Per-link bandwidth capCost control against a link that goes viral

What we test

SECTION 05 · Comparison

Test	Expected	Where
Request 10k random tokens	All 404, IP blocked after 20	e2e/share-enumeration
Fetch a signed URL after 61s	403 from the object store	e2e/share-expiry
Download an uploaded .html	Content-Disposition: attachment	e2e/share-content-type
Revoke, then fetch through the CDN	404 within 30s	manual — automation pending

Threat model

A security review — scope, the data flow across the trust boundary, threats, and controls.

specdfdrisklisttable

6 blocksTry in Studio demo

DOCUMENTTest plan · Release 2026-05

Test plan — checkout rewrite

What we verify before the new checkout takes real traffic, and what stops the release.

SECTION 01 · Note

Exit criteria in one line

Note

All P0 cases pass and the payment matrix is green in staging against the PSP sandbox. A 2-hour soak at 3× peak holds p95 under 800ms.

Scope

SECTION 02 · Before / after

What this plan covers

In scope

The full submit path — cart to confirmed order

All four payment methods, including 3DS challenge and decline paths

Idempotent retries under packet loss

Rollback to the old checkout behind the flag

→

Out of scope

Fulfilment beyond the handoff event (covered by the fulfilment suite)

Marketing pages and the cart UI, unchanged in this release

Load beyond 3× peak — a separate capacity exercise

Coverage

SECTION 03 · Comparison

Case	Priority	Type	Owner
Card payment, happy path	P0	e2e	QA
3DS challenge accepted	P0	e2e	QA
3DS challenge abandoned	P0	e2e	QA
Card declined, customer retries with another card	P0	e2e	QA
Retry after a dropped response charges once	P0	integration	Payments
Stock runs out between reserve and capture	P1	integration	Orders
Wallet payment (Apple Pay)	P1	e2e	QA
Flag off — old checkout still works	P0	smoke	Orders
Screen reader completes the flow	P1	manual	Design

Environments

SECTION 04 · Spec

Where each layer runs

Unit + integration

CI on every PR · PSP stubbed

End to end

staging · PSP sandbox · seeded catalogue

Soak

perf cluster · 3× peak for 2 hours · production-sized data

Manual + accessibility

staging · Safari/VoiceOver and Chrome/NVDA

Production

canary 5% for 24h, then staged to 100%

How to run it

SECTION 05 · Steps

1
The fast loop
Unit and integration against stubs — this is what CI gates on.
bash
```
pnpm test --filter checkout
```
2
End to end against staging
bash
```
pnpm e2e --project=checkout --env=staging
```
Needs a PSP sandbox key in your environment; ask
3
The soak
Two hours at 3× peak. Watch p95 and the error budget, not the total request count.
bash
```
k6 run perf/checkout-soak.js --vus 900 --duration 2h
```

Status

SECTION 06 · Status

Suite	Latest run	Status
Unit + integration	1,412 cases green on main	completed
Payment matrix (e2e)	18 of 20 green — two 3DS cases flaking on sandbox timeouts	in progress
Idempotent retry under loss	Green — 10k retries, zero duplicate charges	completed
Soak at 3× peak	Scheduled for Thursday once the payment matrix is green	todo
Accessibility pass	Blocked on the final focus-order fix	blocked

Legendcompletedin progresstodoblocked

Exit criteria

SECTION 07 · List

Every P0 case greenNo known P0 failure, no skipped P0
Zero duplicate chargesAcross the full retry-under-loss run
p95 under 800ms at 3× peakHeld for the full two-hour soak
Rollback rehearsedFlag flipped off in staging and the old checkout served traffic
Accessibility sign-offBoth screen-reader paths complete without a sighted assist

Test plan

What gets verified — scope, the case matrix, environments, suite status, and exit criteria.

calloutcvttablespecsteps+2

8 blocksTry in Studio demo

DOCUMENTRelease · 2026-04-09

Orders API v2.4.0

Idempotency keys, partial refunds, and a faster order search.

SECTION 01 · Note

One breaking change

Warning

GET /orders no longer returns line_items by default. Ask for it with ?include=line_items. The old shape is available on v2.3 until 2026-07-01.

Highlights

SECTION 02 · Metrics

Duplicate charges since 2.4-rc1

▲ was 9/quarter

180ms

Order search p95

▲ -1.2s

Endpoints deprecated

Changes

SECTION 03 · Changelog

2.4.02026-04-09minor

addedIdempotency-Key on all mutating routes — retries replay the first response

addedPartial refunds against a captured payment

changedOrder search moves to a materialized view — p95 1.4s → 180ms

changedGET /orders omits line_items unless requested

fixedCancelling an already-shipped order returned 500 instead of 409

securitySigned webhook payloads now include the delivery attempt number

2.3.22026-03-21patch

fixedFulfilment handoff retried forever on a poisoned message

Upgrading

SECTION 04 · Steps

Twenty minutes, in this order

1
Bump the SDK
bash
```
npm install @acme/orders-sdk@2.4.0
```
Ask for line items where you need them
The compiler will not catch this — grep for order list calls that read line_items.
Send an idempotency key
One key per user action, persisted with the pending request so a retry after a cold start reuses it.
4
Verify against staging
Replay yesterday's traffic and diff the responses before you promote.
bash
```
./scripts/replay --from staging --since 24h --diff
```

SECTION 05 · Code

What the change looks like in a caller

diff

- const { orders } = await client.orders.list({ status: "open" })+ const { orders } = await client.orders.list({+   status: "open",+   include: ["line_items"],+ }) - await client.orders.create(payload)+ await client.orders.create(payload, { idempotencyKey: actionId })

Deprecations

SECTION 06 · Inventory

On the way out

GET /orders/searchremoved in 3.0

Use GET /orders with the q parameter.

deprecated

order.updated webhookremoved in 3.0

Replaced by the typed order.status_changed event.

deprecated

X-Request-Id headerremoved in 3.0

Use traceparent — we propagate W3C trace context now.

deprecated

Partial refunds2.4

Stable after the PSP reconciliation job runs a full month.

beta

Release notes

What shipped — highlights, the changelog, breaking changes, upgrade steps, deprecations.

calloutstatschangelogstepscode+1

7 blocksTry in Studio demo

Planning and communication

Share a plan, report progress, or help someone get started.

DOCUMENTRoadmap · H2

Payments platform — H2 roadmap

What ships when, and the targets each phase moves.

Targets

SECTION 01 · Metrics

Where H2 should land

99.95%

Checkout availability

▲ +0.15pp

600ms

Checkout p95

▲ -1.8s

New payment methods

Phases

SECTION 02 · Roadmap

Jul

done

Async capture

Capture moves off the request path.

Aug-Sep

current

Wallets

Apple Pay + Google Pay behind a flag.

Oct

Bank transfers

SEPA + ACH via the PSP.

Nov-Dec

future

Chargeback automation

Evidence packets built from order data.

Legenddonecurrentnextfuture

Now / next / later

SECTION 03 · Board

Now

Wallet tokenization

wallets

Capture worker autoscaling

SEPA mandate flow

transfers

Later

Chargeback evidence builder

Workstreams

SECTION 04 · Status

Workstream	Latest update	Status
Wallets GA behind a flag	Payments — tokenization in canary, due Sep 15	on track
PSP contract for bank transfers	Finance — redlines back from legal, due Aug 30	at risk
Deprecate the legacy capture job	Payments — blocked until wallets GA	todo

Legendon trackat risktodo

Roadmap

A plan over time — phases on a timeline, a board of workstreams, and status tracking.

statstimelinekanbanstatustable

5 blocksTry in Studio demo

DOCUMENTStatus · Steering committee

Payments platform — week 12

Wallets are in canary, the PSP contract slipped, and we need a decision on bank transfers.

SECTION 01 · Summary

Situation

Payments is midway through H2 — wallets, bank transfers and chargeback automation, in that order.

Complication

The PSP contract for bank transfers is two weeks late in legal, and that work is on the critical path for the Q4 revenue target.

Question

Do we hold the October date and cut scope, or move the date and keep the full scope?

Answer

Hold the date and cut scope to SEPA only. ACH moves to Q1.

SEPA is 78% of the forecast transfer volume, so most of the revenue lands on time
ACH needs a separate compliance review we have not started, and that is the real long pole
Cutting now keeps the wallets GA date, which is already committed to two enterprise customers

Where we are

SECTION 02 · Metrics

99.97%

Checkout availability

▲ +0.02pp

610ms

Checkout p95

▲ -190ms

12%

Wallet share in canary

▲ +12pp

2 wks

Bank transfers behind

▼ slipped

Workstreams

SECTION 03 · Status

Workstream	Latest	Owner	Status
Wallets GA	Canary at 10% for nine days — zero incidents. GA on the 22nd	Payments	on track
Bank transfers	PSP contract still in legal redlines; build is ready to start	Payments	slipped
Chargeback automation	Design complete, starts when wallets ship	Payments	on track
Idempotency rollout	SDKs shipping; header miss rate down to 4%	Payments	at risk

Legendon trackat riskslipped

Decisions we need today

SECTION 04 · List

Cut ACH to Q1Keeps the October date for SEPA. Needs a call from Finance on the forecast impact.
Approve two contractors for chargebacksSix-week engagement — otherwise it starts in November and misses the quarter.
Confirm the wallets GA comms dateMarketing needs ten working days of lead time.

Risks

SECTION 05 · Risk register

highPSP contract slips a further two weeksFinancemitigating

L: med · I: high

Mitigation: Second provider approached as a fallback — two weeks to integrate

highWallet canary hides a scale problemPaymentsmitigating

L: low · I: high

Mitigation: Load test at 10× canary volume before GA

highChargeback work starts too late for Q4Steeringopen

L: high · I: med

Mitigation: Contractor approval today makes the date; otherwise it moves

SECTION 06 · Takeaways

Takeaways

Wallets are on track and GA on the 22nd
Nine days in canary, zero incidents.
Bank transfers slipped on the contract, not the build
Engineering is ready to start the day it signs.
Cutting ACH to Q1 keeps the October revenue
Two decisions today, or chargebacks miss Q4

Status update

A steering-committee update — SCQA summary, the numbers, workstreams, decisions needed.

scqastatsstatustablelistrisk+1

7 blocksTry in Studio demo

DOCUMENTDeck · Eng review

Fixing checkout latency

What we changed, what it bought us, and what we do next.

Part 1 — the problem

SECTION 01 · Divider

PART 1

Checkout was bleeding conversions

The synchronous capture call dominated checkout time

Every 100ms of checkout latency costs ~0.6% conversion. Our p95 drifted to 2.4s — and one call was 71% of it.

SECTION 02 · Chart

Where the 2.4s went

CHART

Part 2 — the fix

SECTION 03 · Divider

PART 2

Move capture off the request path

One number tells the story

The async capture change removed the 1.7s synchronous call from the request path. Nothing else shipped in the window.

SECTION 04 · Big number

-75%

Checkout p95 after the change

2.4s → 600ms over four weeks of production traffic

What we do next

The same pattern applies to every synchronous third-party call on the hot path — fraud scoring is the next candidate.

SECTION 05 · Roadmap

shipped

done

Async capture

current

Fraud scoring async

Hot-path audit

Legenddonecurrentnext

What to remember

SECTION 06 · Takeaways

Takeaways

The synchronous capture call was the bottleneck
It accounted for 71% of the 2.4s checkout p95.
Moving it to a queue cut p95 by 75%
Conversion recovered within two weeks
Audit every synchronous third-party call on the hot path

Deck

A slide deck — cover, dividers, big numbers, charts, and a takeaways close.

dividerchartbignumbertimelinetakeaways

7 blocksTry in Studio demo

DOCUMENTOnboarding · Engineering

Onboarding — payments platform

From laptop to merged pull request in your first week.

SECTION 01 · Note

The goal for week one

Tip

Ship something small to production by Friday. Everything below exists to make that possible — if a step blocks you for more than 30 minutes, that is a bug in this document. Tell us and we will fix it.

Day one

SECTION 02 · Steps

Get it running locally

1
Clone and install
bash
```
git clone git@github.com:acme/payments.git && cd payments && make bootstrap
```
Bootstrap installs the toolchain versions from .tool-versions — do not install Go or Node by hand.
2
Bring up the stack
Postgres, Redis, a fake PSP and the API, all in containers.
bash
```
make up
```
3
Prove it works
A seeded test card charges through the fake PSP and confirms an order.
bash
```
make smoke
```
Get your accounts
Open the access request template — it grants AWS read-only, Grafana, Honeycomb and PagerDuty shadow in one go.
Production write access comes after your first on-call shadow, not before.

How the code is laid out

SECTION 03 · Hierarchy

The four directories that matter

▸payments/

—cmd/One directory per binary — api, worker, cli

▸internal/

—domain/Pure business rules. No SQL, no HTTP, no clock — this is where logic belongs

—adapters/Postgres, Redis, PSP clients. Everything that talks to the outside world

—http/Handlers and middleware. Thin — they translate, they do not decide

—migrations/Expand-only. Never edit a migration that has run in production

—docs/This document, the ADRs, and the runbooks

How a payment actually flows

SECTION 04 · Sequence

Read this once and the codebase makes sense

SEQUENCE

Legendcallresponsethe answer the caller getsEXTexternal actoractive

Your first tickets

SECTION 05 · Board

Day 1-2

Fix a typo in an error message

warm-up

Add a test for an uncovered branch

warm-up

Day 3-4

Add a field to an existing endpoint

real

Shadow an on-call handover

ops

Week 2

Own a small feature end to end

real

Who to ask

SECTION 06 · Team

Your onboarding buddy

Engineer

First port of call for anything

Tech lead

Payments

Architecture and review

Payments

Expectations, growth, time off

#team-payments

Channel

Anything the above are asleep for

SECTION 07 · FAQ

Things nobody writes down

How do I get a change to production?

Open a PR, get one approval, merge. It canaries at 10% for 20 minutes and promotes itself. There is no release train and no ticket to file.

Am I allowed to break staging?

Yes. That is what it is for. Production is the one that pages people.

Where do decisions live?

docs/adr/. If you wonder why something is the way it is, the answer is usually there — and if it is not, write one.

What if I break production?

Roll back first, understand later. Nobody is in trouble for an outage; people are in trouble for hiding one. We write a blameless postmortem and move on.

When do I join the on-call rota?

After two shadow shifts, and only when you want to. Usually around week six.

Onboarding

A new joiner's first week — local setup, the code map, how a request flows, who to ask.

calloutstepstreesequencekanban+2

8 blocksTry in Studio demo

Agents and interfaces

Document an agent workflow or a UI design system.

DOCUMENTAgent system · v1

Support triage agent

How the ticket-triage agent works — loop, prompts, budget, and evals.

Requirements

SECTION 01 · Drivers

What the agent must do

Resolve or route

Answer from the KB or escalate to the right queue.

JOB

Grounded answers

Every claim cites a KB article.

GUARDRAIL

Bounded cost

One episode stays under 40k tokens.

BUDGET

The loop

SECTION 02 · Agent loop

Triage loop

AGENT

Legendenvironmentagenttool · memoryprompt · tool callresponse · result

stops when: reply sent or ticket escalated

Prompt anatomy

SECTION 03 · Prompt

Triage prompt

SYSTEM— role + guardrails

You are a support agent for {{product}}. Answer only from KB articles; cite the article id.

USER

Customer ({{plan}} plan) writes: {{ticket}}

{{product}}Product name from config

{{plan}}Plan tier of the customer

{{ticket}}The inbound ticket body

Context budget

SECTION 04 · Context window

Where one episode's window goes

CONTEXT

Legend

1system prompt4,000 tokens · 2%

2tool schemas6,000 tokens · 3%

3KB retrieval18,000 tokens · 9%

4history12,000 tokens · 6%

One turn, end to end

SECTION 05 · Sequence

A tool-using turn

SEQUENCE

Legendcallresponsethe answer the caller getsactive

A real episode

SECTION 06 · Trace

Password reset — one episode

USER

I never get the reset email.

ASSISTANT

Could be a bounce — check delivery before blaming spam.

Let me check our email logs.

TOOL

search_kb

args:{ "query": "reset email not delivered" }

→KB-142: bounced mail troubleshooting

ASSISTANT

Your mailbox rejected the email (KB-142) — free up space and I will resend it.

Eval results

SECTION 07 · Comparison

Eval	Target	Current
Resolution without escalation	40%	46%
Citation accuracy	99%	98.7%
Tokens per episode (p95)	40k	31k

Agent system

An AI-agent design — the agent loop, tools, prompt anatomy, context budget, and a trace.

driversagentlooppromptcontextsequence+2

8 blocksTry in Studio demo

DOCUMENTDesign system · v2

Aurora design system

Tokens, type, usage rules, and component status for product teams.

Color tokens

SECTION 01 · Palette

Core palette

#0E54A1

Primary

Buttons and links

#1F2937

Ink

Body text

#F6F8FB

Surface

Card backgrounds

#1F9747

Positive

Success states

Type scale

SECTION 02 · Type scale

Display

40px / weight 700

The quick brown fox jumps over the lazy dog

Heading

24px / weight 700

The quick brown fox jumps over the lazy dog

Body

15px / weight 400

The quick brown fox jumps over the lazy dog

Caption

12px / weight 500

secondary text

The quick brown fox jumps over the lazy dog

Usage guidelines

SECTION 03 · Guidelines

Button usage

✓Do

Use one primary button per view

Write labels as verbs

Save changes

✕Don't

Stack two primary buttons side by side

Disable a button without explaining why

tooltip: Add a line item first

Component status

SECTION 04 · Inventory

Component inventory

Buttonv2

stable

Data table

API may change before GA

beta

Date picker

experimental

Modal (legacy)

Use Dialog instead

deprecated

Key screen

SECTION 05 · Mockup

The settings pattern

Design system

A design-system reference — palette, type scale, do/don't guidance, component inventory.

palettetypescaledodontinventorywireframe

6 blocksTry in Studio demo