```meta
title: Why deploys fail on Fridays
subtitle: The candidate causes, ranked by how likely they are and how cheap they are to confirm.
tag: INVESTIGATION
```

Deploys of `checkout-api` fail about three times more often on Fridays than on any other weekday. This document does not name the cause. It lists every cause that fits the evidence and gives a plan to confirm or reject each one.

```callout
tone: note
title: Assumptions
body: "The pipeline is GitHub Actions for build and test, Argo CD for rollout to the `prod` cluster. A deploy \"fails\" when the Argo rollout aborts or the post-deploy smoke test fails. Failure counts come from the last 12 weeks of pipeline runs. The pipeline did not change in that period."
```

## Failure rate by weekday

The pattern is a weekday effect, not a volume effect. Fridays carry the same number of deploys as Wednesdays but four times the failures. Thursdays are already above the Monday to Wednesday baseline, which points at something that builds through the week.

```chart
title: Deploy failure rate, last 12 weeks
kind: bar
unit: "%"
labels: [Mon, Tue, Wed, Thu, Fri]
series:
  - { label: Failed deploys, accent: red, values: [4, 5, 6, 9, 24] }
```

## Candidate causes

Each bone groups causes that share a mechanism. A cause belongs here only if it can explain why Friday differs from Thursday.

```fishbone
id: causes
effect: Friday deploys fail 3x more often
causes:
  - label: Change size
    items:
      - Week's work merged in one batch before the weekend
      - Feature-flag cleanups bundled with product changes
      - Release train cuts on Friday morning, so Friday deploys carry the most commits
  - label: Scheduled jobs
    items:
      - Weekly TLS certificate rotation runs Friday 13:00 UTC
      - Nightly-on-Thursday data refresh leaves staging with a different schema than prod
      - Friday index rebuild on the orders database holds long locks
  - label: Pipeline capacity
    items:
      - Runner quota resets Monday, so Friday runs hit the cap and time out
      - Dependency cache TTL of 7 days expires on Friday
      - Canary gate compares against a Thursday baseline with higher traffic
  - label: People
    items:
      - Half the reviewers are out on Friday afternoons, so approvals are shallower
      - On-call handover at Friday 16:00 leaves rollouts without an owner
      - Engineers rush to ship before the weekend and skip the staging soak
  - label: Traffic
    items:
      - Friday evening is the weekly peak for checkout
      - Marketing sends the weekly promo email Friday 15:00
      - Autoscaler cooldown is too long for the Friday ramp
```

## What would confirm each cause

A cause is confirmed by a signal that is present on failed Friday deploys and absent on successful ones. Most signals already exist in logs or metrics. Two of them need a change to the pipeline before they can be measured.

```table
title: Evidence per cause
columns: [Cause, Confirming signal, Where to look, Status]
rows:
  - [Batch size, "Failed deploys carry more than 15 commits; successful ones carry fewer", GitHub deploy annotations, { v: Can check now, tone: pos }]
  - [Certificate rotation, "Failures cluster at 13:00 to 13:30 UTC with TLS handshake errors", Argo rollout logs and cert-manager events, { v: Can check now, tone: pos }]
  - [Staging schema drift, "Migration step fails in prod but passed in staging on the same commit", Migration job logs, { v: Can check now, tone: pos }]
  - [Index rebuild locks, "Smoke test times out on order reads while pg_locks shows the rebuild", Postgres slow-query log, { v: Can check now, tone: pos }]
  - [Runner quota, "Job queue time above 20 minutes on Friday only", GitHub Actions usage report, { v: Can check now, tone: pos }]
  - [Cache expiry, "Cold build on Friday takes 2x the Thursday build", Build step durations, { v: Can check now, tone: pos }]
  - [Canary baseline, "Canary aborts on latency while the new pods are healthy", Argo analysis run results, { v: Needs baseline logging, tone: warn }]
  - [Shallow review, "Failed deploys have one approver; successful ones have two", Pull request metadata, { v: Can check now, tone: pos }]
  - [On-call handover, "Failures start after 16:00 and no one acknowledges the rollout page", PagerDuty timeline, { v: Can check now, tone: pos }]
  - [Skipped soak, "Staging soak shorter than 30 minutes on failed deploys", Pipeline step timestamps, { v: Needs soak duration metric, tone: warn }]
  - [Traffic peak, "Failures track the 15:00 to 19:00 traffic ramp, not deploy time", Checkout RPS dashboard, { v: Can check now, tone: pos }]
```

## Where to look first

Likelihood comes from how well each cause matches the Thursday rise and the Friday spike. Cost is the engineer hours needed to confirm or reject it. Start in the top left.

```quadrant
title: Likelihood against cost to confirm
xAxis: { label: Cost to confirm, low: Hours, high: Days }
yAxis: { label: Likelihood, low: Low, high: High }
items:
  - { x: 0.15, y: 0.85, label: Batch size }
  - { x: 0.2, y: 0.8, label: Certificate rotation }
  - { x: 0.25, y: 0.7, label: Traffic peak }
  - { x: 0.3, y: 0.55, label: Skipped soak }
  - { x: 0.35, y: 0.5, label: Staging schema drift }
  - { x: 0.4, y: 0.45, label: On-call handover }
  - { x: 0.2, y: 0.35, label: Runner quota }
  - { x: 0.25, y: 0.3, label: Cache expiry }
  - { x: 0.7, y: 0.6, label: Canary baseline }
  - { x: 0.65, y: 0.3, label: Index rebuild locks }
  - { x: 0.75, y: 0.25, label: Shallow review }
```

## Plan for the next two Fridays

The plan tests the cheap, likely causes first and adds the two missing metrics so the expensive causes become cheap the week after. Do not change the deploy process until a cause is confirmed; a process change now would hide the signal.

```steps
title: Confirm or reject
items:
  - title: Pull the failed-deploy set
    body: Export every deploy from the last 12 weeks, failed and successful. Include commit count, start time, approver count, and soak duration.
    code: gh api repos/acme/checkout-api/deployments --paginate > deploys.json
    lang: bash
  - title: Split by batch size and time of day
    body: Compare commit counts and start hours between the failed and successful sets. This rejects or confirms batch size, certificate rotation, on-call handover, and traffic peak in one pass.
    note: If failures cluster at 13:00 UTC, stop here and check cert-manager events before anything else.
  - title: Add the two missing metrics
    body: Emit staging soak duration and the canary baseline window as pipeline annotations.
    code: argo rollouts get rollout checkout-api -n prod --watch
    lang: bash
  - title: Hold one Friday deploy to Thursday
    body: Move one normal-size deploy from Friday to Thursday 14:00 and keep everything else the same. If it succeeds and the Friday deploys keep failing, the cause is the Friday environment. The change itself is not the cause.
    note: This is a probe, not a policy. Do not adopt a Friday freeze from one data point.
  - title: Write down the confirmed cause
    body: Record the confirming signal and the rejected causes. The next investigation then starts from this list.
```