```meta
title: Production readiness — search-indexer
subtitle: PRR v4 applied to search-indexer 2.3.0 before the go-live on 2026-09-22.
tag: REVIEW
```

The search-indexer consumes `catalog.product.v2` from Kafka, builds the product document, and writes it to the `products-v7` OpenSearch index. The review applies PRR v4, the standard every tier-1 service passes before it takes production traffic. Each item below carries the artifact that proves its verdict, so a second reviewer can repeat the check without the author.

Assumptions: the review covers the `search-indexer` deployment in `eu-west-1` only, against PRR v4 (2026-05). Peak load is 1,800 events/s, taken from the 2026-08 catalog import. The reviewer is the platform on-call lead. Change any of these and rerun the failed items.

## PRR v4, item by item

```checklist
title: PRR v4 — search-indexer 2.3.0
standard: PRR v4
groups:
  - label: Observability
    items:
      - "[pass] Dashboard covers the four golden signals — grafana.internal/d/search-indexer, panels 1-8"
      - "[pass] Alerts page the owning on-call — PagerDuty service search-indexer, escalation policy search-p1"
      - "[pass] Structured logs carry trace_id and product_id — loki query {app=\"search-indexer\"} | json, 100% of sampled lines"
      - "[partial] Traces sampled on every error — OTEL sampler is 10% flat; PRR v4 §2.4 needs 100% on status=error"
  - label: Resilience
    items:
      - "[fail] Load test at 2× peak within 30 days — last run 2026-06-14 at 1.2× peak, before the Kafka partition change"
      - "[pass] Consumer restarts from committed offsets without duplicate writes — chaos run 2026-09-08, 0 duplicate docs across 41,200 events"
      - "[fail] Poison message goes to a DLQ after 3 retries — retry loop is unbounded; incident INC-4471 held partition 3 for 52 min"
      - "[na] Multi-region failover — single-region service by design, exception EXC-118 approved"
  - label: Data and security
    items:
      - "[pass] Index writes are idempotent by product_id — OpenSearch _id = product_id, upsert with version_type external"
      - "[pass] No PII in the index — schema audit 2026-09-05, 0 of 38 fields tagged pii"
      - "[pass] Secrets come from Vault, not env files — deploy manifest search-indexer.yaml, vault-agent sidecar"
      - "[pending] Dependency scan has 0 high CVEs — trivy run of 2026-09-10 reports 1 high in libxml2 2.9.14; fix build in CI"
  - label: Operations
    items:
      - "[pass] Runbook covers the top 3 alerts — runbooks/search-indexer.md, sections lag, error-rate, index-red"
      - "[pass] Rollback completes in under 10 min — release 2.2.1 rollback on 2026-08-27 took 6 min 40 s"
      - "[partial] Capacity plan for the next 2 quarters — plan exists for Q4 only; Q1 numbers missing"
      - "[fail] Reindex procedure tested end to end — full reindex of products-v7 never run in staging"
```

Ten of sixteen items pass. The four fails and two partials cluster in resilience and operations, not in the code path that serves reads. The standard does not accept a launch with an open fail in Resilience. The go-live date holds only if the two resilience fails close by 2026-09-19.

## Latency and lag against the PRR budgets

PRR v4 sets fixed budgets for consumer lag and write latency. The measured values come from the 7-day production-mirror run that ended on 2026-09-11, p95 unless the row says otherwise.

```perfbudget
title: PRR v4 budgets — search-indexer
context: p95, production mirror in eu-west-1, 7 days ending 2026-09-11
metrics:
  - { metric: Consumer lag, budget: 30, measured: 12, unit: s, note: "max lag during the 08:00 catalog sync" }
  - { metric: Event-to-index latency, budget: 5000, measured: 3400, unit: ms }
  - { metric: Bulk write latency, budget: 800, measured: 910, unit: ms, note: "1,000-doc batches; over budget on the 3 largest categories" }
  - { metric: Error rate, budget: 0.5, measured: 0.08, unit: "%" }
  - { metric: Throughput at 1× peak, budget: 1800, measured: 2150, unit: "events/s", lowerIsBetter: false }
  - { metric: CPU headroom at peak, budget: 40, measured: 31, unit: "%", lowerIsBetter: false }
```

Bulk write latency is the only budget over the line, and the cause is batch size, not OpenSearch. The 1,000-doc batch was tuned for the old 6-partition topic; the topic now has 12 partitions, so each consumer sends smaller, more frequent batches. Throughput clears the budget at 1× peak, but the 2× run has not been done, so the CPU headroom figure is an estimate above 1×.

## Findings that block or shape the launch

```audit
title: PRR v4 findings — search-indexer 2.3.0
scope: search-indexer deployment, eu-west-1
date: 2026-09-12
auditor: Platform on-call lead
findings:
  - { id: F1, title: Retry loop on a malformed event is unbounded, severity: critical, area: Resilience, evidence: "INC-4471: partition 3 stalled 52 min on one event with a null price; consumer log shows 3,118 retries", fix: "Cap retries at 3, then publish to catalog.product.v2.dlq with the original headers", owner: search-team, status: fixing }
  - { id: F2, title: No load test at 2× peak since the partition change, severity: high, area: Resilience, evidence: "k6 report 2026-06-14 ran 2,160 events/s against 6 partitions; topic moved to 12 partitions on 2026-07-30", fix: "Run the k6 suite at 3,600 events/s against the mirror; attach the report to the PRR ticket", owner: search-team, status: open }
  - { id: F3, title: Full reindex never run in staging, severity: high, area: Operations, evidence: "runbooks/search-indexer.md §reindex has steps but no run log; staging index products-v7 was created by copy, not by reindex", fix: "Run the reindex in staging with the 2026-08 snapshot; record duration and doc count", owner: search-team, status: open }
  - { id: F4, title: Bulk write p95 over budget, severity: medium, area: Performance, evidence: "perfbudget row: 910 ms measured against 800 ms; OpenSearch slow log shows 1,000-doc batches on categories furniture, lighting, garden", fix: "Set batch size to 400 docs or 2 MB, whichever comes first", owner: search-team, status: fixing }
  - { id: F5, title: Error traces sampled at 10%, severity: medium, area: Observability, evidence: "otel-collector config sampler: probabilistic 0.10; no tail sampler on status=error", fix: "Add a tail sampler that keeps 100% of spans with status=error", owner: platform, status: open }
  - { id: F6, title: libxml2 2.9.14 has one high CVE, severity: medium, area: Supply chain, evidence: "trivy 2026-09-10, CVE-2025-31115 high", fix: "Bump the base image to debian-12.7; CI build #4412 in progress", owner: platform, status: fixing }
  - { id: F7, title: Capacity plan stops at Q4, severity: low, area: Operations, evidence: "capacity/search-indexer.md has Q4 2026 rows only", fix: "Add Q1 2027 from the catalog growth forecast", owner: search-team, status: open }
```

F1 is the finding that decides the launch. An unbounded retry turns one bad event into a stalled partition. The 2026-09-08 chaos run did not cover that path, because it injected restarts, not malformed events. F2 and F3 are proof gaps rather than known defects, but the standard treats a missing proof as a fail.

## Verdict

```callout
tone: warn
title: Not ready on 2026-09-12
body: "search-indexer 2.3.0 does not pass PRR v4. Two Resilience fails (F1, F2) and one Operations fail (F3) block the go-live. The launch on 2026-09-22 holds if F1 and F2 close and re-check by 2026-09-19. F3 can move to a launch condition, with a reindex freeze until it passes. The reviewer will rerun the four failed items and the perfbudget on the re-check."
```