```meta
title: What breaks deploys
subtitle: The 140 failed deploys of 2026 by root cause, ranked so the few causes behind most failures stand out.
tag: ANALYSIS
```

The 140 failed deploys this year come from eight root causes, but three of them account for 99 failures. Fixing those three removes 71% of the failures. The other five causes together fail fewer deploys than the second cause alone.

```callout
tone: note
title: Assumptions
body: "A deploy is one pipeline run of `checkout-api` to the `prod` cluster between 1 January and 12 September 2026. A deploy failed when the Argo CD rollout aborted or the post-deploy smoke test failed. Each failure carries one root cause, taken from the incident label that the on-call engineer set when the pipeline was rerun. Failures without a label were re-labeled from the pipeline log; 5 of them did not fit any category and are counted as Other."
```

## The headline

```bignumber
value: "71%"
label: Three causes explain 99 of the 140 failed deploys
context: "Config drift, failed migrations, and flaky integration tests. The remaining five causes explain 41."
accent: navy
```

## Failures by root cause

The bars are sorted by count and the line is the running share. The line crosses 80% at the fourth cause, so anything past expired secrets is a long tail and not worth a project of its own.

```chart
id: pareto
title: Failed deploys by root cause, 2026 to date
kind: pareto
unit: deploys
items:
  - { label: Config drift between staging and prod, value: 46, accent: navy, desc: "An environment variable or Helm value differs between staging and prod, so the pod starts with a wrong or missing setting" }
  - { label: Failed database migration, value: 31, accent: navy, desc: "The migration job errors on prod data that staging does not have, or holds a lock past the rollout timeout" }
  - { label: Flaky integration test, value: 22, accent: navy, desc: "A test that passes on rerun fails the gate and aborts the rollout" }
  - { label: Expired secret or certificate, value: 14, accent: gray, desc: "A TLS certificate or API key rotated and the new value was not in the deploy manifest" }
  - { label: Image pull timeout, value: 9, accent: gray, desc: "The registry did not serve the image within the kubelet timeout" }
  - { label: Runner capacity timeout, value: 7, accent: gray, desc: "The job waited in the GitHub Actions queue past the 60 minute limit" }
  - { label: Canary latency abort, value: 6, accent: gray, desc: "The canary analysis saw p95 above the threshold and rolled back" }
  - { label: Other, value: 5, accent: gray, desc: "Single occurrences that fit no category" }
```

## What each cause costs and what fixes it

Count is not the only measure. A failed migration takes about three times longer to recover than a flaky test, because the rollback needs a human to check the schema first. The fix column names the change that removes the cause, not a mitigation.

```table
title: Root causes with recovery time and fix
columns: [Root cause, { label: Failures, align: r }, { label: Share, align: r }, { label: Cumulative, align: r }, { label: Median recovery, align: r }, Fix that removes the cause]
rows:
  - [{ v: Config drift between staging and prod, lead: true }, 46, "33%", "33%", 25 min, { v: "Generate both environments from one values file with a diff gate in CI", tone: pos }]
  - [{ v: Failed database migration, lead: true }, 31, "22%", "55%", 70 min, { v: "Run every migration against a nightly prod snapshot before merge", tone: pos }]
  - [{ v: Flaky integration test, lead: true }, 22, "16%", "71%", 20 min, { v: "Quarantine tests that fail and pass on rerun; retry once inside the gate", tone: pos }]
  - [Expired secret or certificate, 14, "10%", "81%", 40 min, "Alert 14 days before expiry; read secrets from the vault at start-up"]
  - [Image pull timeout, 9, "6%", "87%", 15 min, "Pre-pull the image to every node before the rollout starts"]
  - [Runner capacity timeout, 7, "5%", "92%", 65 min, "Add a dedicated runner pool for deploy jobs"]
  - [Canary latency abort, 6, "4%", "96%", 30 min, { v: "Keep. Four of the six aborts were correct", tone: muted }]
  - [Other, 5, "4%", "100%", 35 min, { v: No shared fix, tone: muted }]
note: Shares are rounded to whole percent, so the cumulative column can differ from the sum of the shares by one point. Recovery is the median time from the failed run to the next successful rollout of the same commit.
```

## What to do with this

```takeaways
title: Takeaways
items:
  - text: Fix config drift first
    detail: It causes one failure in three and the fix is a CI gate, not a platform change.
  - text: Failed migrations are the second cause and the slowest to recover
    detail: 31 failures at a 70 minute median is more lost time than config drift.
  - text: Three fixes remove 71% of failures
    detail: The pareto chart at #pareto shows the line crossing 80% at the fourth cause.
  - text: Leave the canary gate alone
    detail: Its aborts are mostly correct. Removing them would trade six failed deploys for four bad releases.
```