Rotate the payments database credentials runbook

```meta
title: Rotate the payments database credentials
subtitle: Swap the password payments-api uses for Postgres while every pod keeps serving.
tag: RUNBOOK
```

The payments database has two application roles, `payments_app_a` and `payments_app_b`. Only one is active at a time; the other holds the previous password and stays valid until the next rotation. Rotation writes a new password to the standby role, points Vault at it, and reloads the pods one by one. No pod ever holds a password that Postgres has already rejected.

Assumptions in this runbook: Postgres 15 on RDS, eight `payments-api` pods, and Vault at `secret/payments/db`. A pod reads the secret at boot and again on SIGHUP. The rotation takes about 20 minutes and needs no change window. Run it from the ops bastion with the `payments-ops` role.

## Procedure

```steps
id: rotate-steps
items:
  - title: Find the active role
    body: The active role is the one Vault serves today. The other role is the target of this rotation.
    code: vault kv get -field=username secret/payments/db
    lang: bash
    note: If the output is payments_app_a, the target is payments_app_b, and the reverse.
  - title: Generate the new password and set it on the target role
    body: The password is 32 bytes from /dev/urandom. It never appears in shell history because it lives in a variable.
    code: |
      NEWPW=$(openssl rand -base64 32)
      psql "$PAYMENTS_ADMIN_DSN" -c "ALTER ROLE payments_app_b WITH PASSWORD '$NEWPW' VALID UNTIL 'infinity';"
    lang: bash
  - title: Prove the target role connects
    body: A failed login here costs nothing. A failed login after step 4 costs a pod.
    code: PGPASSWORD="$NEWPW" psql -h payments-db.internal -U payments_app_b -d payments -c "select 1;"
    lang: bash
    note: Stop if this fails. The pods still use the old credential.
  - title: Write the new credential to Vault
    body: Vault keeps the previous version, so the old username and password stay readable for the rollback.
    code: vault kv put secret/payments/db username=payments_app_b password="$NEWPW"
    lang: bash
  - title: Reload the pods one at a time
    body: On SIGHUP a pod opens a new pool with the new credential. It drains the old pool over 30 seconds. Wait for the pod to report healthy before the next one.
    code: |
      for p in $(kubectl -n payments get pods -l app=payments-api -o name); do
        kubectl -n payments exec "$p" -- kill -HUP 1
        kubectl -n payments wait --for=condition=Ready "$p" --timeout=90s
      done
    lang: bash
    note: Watch the payments-api error rate during the loop. Stop the loop on any rise above 0.1%.
  - title: Confirm no session uses the old role
    body: The old pools drain 30 seconds after the last reload. A session that stays means one pod did not reload.
    code: psql "$PAYMENTS_ADMIN_DSN" -c "select count(*) from pg_stat_activity where usename = 'payments_app_a';"
    lang: bash
  - title: Expire the old role's password
    body: The old role keeps its grants, so the next rotation reuses it. Only its password ends.
    code: psql "$PAYMENTS_ADMIN_DSN" -c "ALTER ROLE payments_app_a VALID UNTIL '$(date -u -d '+1 hour' +%FT%TZ)';"
    lang: bash
    note: One hour, not now. A pod that restarts during the drain must still be able to reconnect with the old credential.
  - title: Record the rotation
    body: Post the date, the new active role, and your name in #payments-ops. The next rotation starts from this record.
```

## Checks and where each one sends you

```flow
id: rotate-checks
dir: TB
nodes:
  - { id: start, col: 1, row: 1, kind: start, label: "Step 3: target role login" }
  - { id: q1, col: 1, row: 2, kind: decision, label: "select 1 returns?" }
  - { id: fix, col: 2, row: 2, kind: end, label: "Fix the ALTER ROLE; nothing to roll back" }
  - { id: q2, col: 1, row: 3, kind: decision, label: "Every pod Ready after SIGHUP?" }
  - { id: roll, col: 2, row: 3, kind: process, label: "Roll back: vault kv rollback -version=<prev>" }
  - { id: rehup, col: 3, row: 3, kind: end, label: "SIGHUP the failed pods; they reload the old credential" }
  - { id: q3, col: 1, row: 4, kind: decision, label: "Old-role sessions at 0?" }
  - { id: find, col: 2, row: 4, kind: end, label: "Find the pod that did not reload; SIGHUP it again" }
  - { id: ok, col: 1, row: 5, kind: end, label: "Expire the old password" }
edges:
  - start -> q1
  - q1 -> q2: "yes"
  - q1 -x-> fix: "no"
  - q2 -> q3: "yes"
  - q2 -x-> roll: "no"
  - roll -> rehup
  - q3 -> ok: "yes"
  - q3 -x-> find: "no"
```

Rollback is always a Vault version rollback plus a SIGHUP. Postgres still accepts the old credential until step 7, so a rollback at any earlier step needs no database change. After step 7 the old password expires in one hour. A rollback inside that hour still works. After the hour, the fix is a fresh rotation onto the old role.

## Who holds what

```table
id: rotate-inventory
columns: [Credential, Holder, Where it lives, Changes when]
rows:
  - [payments_app_a password, "Postgres, Vault version N-1", "Vault secret/payments/db, previous version", "Every second rotation"]
  - [payments_app_b password, "Postgres, Vault version N", "Vault secret/payments/db, current version", "Every second rotation"]
  - [Pod connection pool, payments-api pods, "Process memory; refreshed on SIGHUP", "Each rotation, one pod at a time"]
  - [PAYMENTS_ADMIN_DSN, Ops bastion, "Vault secret/payments/admin, 1-hour lease", "Never rotated by this runbook"]
note: "The admin credential is a Vault lease. Run vault login before the runbook if the lease is older than an hour."
```

```callout
tone: danger
title: Never drop a role
body: "Both roles own no objects, but both hold grants on every payments table. Dropping one breaks the next rotation and needs a schema migration to recreate the grants. Expire the password; keep the role."
```