```meta
title: Feed fan-out
subtitle: How a new post reaches every follower, and why accounts with millions of followers take a different path.
tag: DESIGN
```

A post is written once and read by every follower. We move that work off the
read path. A fan-out worker pushes the post id into each follower's feed cache
right after the post commits, so a feed read is one cache lookup. Above a
follower-count threshold that push is too expensive. Those accounts switch to
pull: followers fetch their latest posts at read time and merge them in.

```callout
tone: note
title: Assumptions
body: "The request did not fix the stack or the numbers. This document assumes Postgres for posts and the follow graph, Kafka for the post.created topic, and Redis sorted sets for the feed cache. It assumes a 5 s p95 delivery target for pushed posts and a 10,000-follower threshold between push and pull. Correct any of these and the envelope and spec below change with them."
```

## Who is wired to whom

The Post API never talks to a follower's feed. It commits the post and an
outbox row, and everything after that flows through the `post.created` topic.
A fan-out worker outage delays feeds; it cannot lose a post, because the
outbox row stays pending until the relay publishes it.

```block
preset: event
dir: LR
groups:
  - { id: workers, col: 3, row: 1, cols: 1, rows: 2, label: Fan-out consumer group }
nodes:
  - { id: postapi, col: 1, row: 1, kind: producer, name: Post API, tech: post.created }
  - { id: postsdb, col: 1, row: 2, kind: store, name: Posts DB, tech: Postgres + outbox }
  - { id: topic, col: 2, row: 1, kind: topic, name: post.created, tech: "Kafka, key = author_id" }
  - { id: w1, col: 3, row: 1, kind: consumer, name: Fan-out worker, replicas: 12 }
  - { id: graph, col: 3, row: 2, kind: store, name: Follow graph, tech: Postgres }
  - { id: feed, col: 4, row: 1, kind: store, name: Feed cache, tech: Redis sorted sets }
  - { id: celeb, col: 4, row: 2, kind: store, name: Pull-account posts, tech: Redis list per author }
  - { id: feedapi, col: 5, row: 1, kind: service, name: Feed API }
edges:
  - postapi -> postsdb: insert post + outbox row
  - postsdb --> topic: outbox relay publishes
  - topic -> w1
  - w1 -> graph: page follower ids
  - w1 -> feed: push post id per follower
  - w1 -> celeb: append when author is pull
  - feed -> feedapi: read cached feed
  - celeb --> feedapi: merge at read time
```

## One post, hop by hop

The client waits only for the database commit. Every hop after `201 Created`
is asynchronous, so a slow fan-out never slows the author. The feed cache
write is a `ZADD` keyed by post id, so a redelivered event writes the same
member again and changes nothing.

```sequence
id: post-fanout
actors:
  - { id: Client, name: Client }
  - { id: PostAPI, name: Post API }
  - { id: PostsDB, name: Posts DB }
  - { id: Relay, name: Outbox relay }
  - { id: Topic, name: post.created }
  - { id: Worker, name: Fan-out worker }
  - { id: Graph, name: Follow graph }
  - { id: Feed, name: Feed cache }
messages:
  - Client -> +PostAPI: POST /posts
  - PostAPI -> +PostsDB: insert post and outbox row in one transaction
  - PostsDB --> -PostAPI: committed
  - PostAPI --> -Client: 201 Created with post_id
  - Relay -> PostsDB: poll pending outbox rows
  - { from: Relay, to: Topic, label: "publish post.created (key = author_id)", kind: async }
  - { from: Topic, to: Worker, label: "deliver post.created", kind: async }
  - Worker -> Graph: followers of author_id, active in last 30 days
  - Graph --> Worker: follower ids in pages of 1,000
  - loop: per page of 1,000 followers
  - { from: Worker, to: Feed, label: "ZADD feed:{follower_id} post_id, score = created_at" }
  - Feed --> Worker: OK
  - end
  - alt: feed cache write fails
  - Worker -x-> Feed: timeout
  - Worker -> Feed: retry with backoff, 5 attempts max
  - else: 5 attempts exhausted
  - { from: Worker, to: Topic, label: "move event to post.created.dlq", kind: async }
  - end
  - Worker -> Topic: commit offset
```

## Why push stops at 10,000 followers

Push costs one cache write per follower per post. For a median author that is
a few hundred writes and the fleet absorbs it. For one account with five
million followers, a single post occupies the whole fleet for minutes and every
other author's post waits behind it. The threshold falls out of the delivery
budget, not out of taste.

```envelope
title: Push cost per post
assumptions:
  - { label: Posts per day, value: 2M }
  - { label: Median followers per author, value: "300" }
  - { label: Followers of one large account, value: 5M }
  - { label: "Feed cache write throughput, whole fleet", value: "30,000 writes/s" }
  - { label: Concurrent posts in fan-out at peak, value: "~15" }
  - { label: Delivery target for a pushed post, value: "5 s (p95)" }
steps:
  - { label: "Feed writes per day, push for everyone", calc: "2M × 300", result: 600M/day }
  - { label: Average write rate, calc: "600M / 86,400 s", result: "≈ 7,000/s" }
  - { label: Peak write rate, calc: "7,000 × 3 (peak factor)", result: "≈ 21,000/s, inside the 30,000/s fleet" }
  - { label: "One large-account post, pushed", calc: "5M / 30,000 per s", result: "≈ 170 s with the whole fleet, everyone else waits" }
  - { label: Write share per post at peak, calc: "30,000 / 15 concurrent posts", result: "≈ 2,000/s per post" }
  - { label: Followers that fit the budget, calc: "2,000/s × 5 s", result: "10,000 followers" }
result: { label: "Push below 10,000 followers", value: "Pull at or above; 5M followers would take 34× the budget" }
```

## What the worker does with each event

The worker decides once per event, from the author's follower count at that
moment. A pull author's post is one list append; the followers pay a small
read-time cost instead. A push author's post is paged through the follow graph
in batches so one slow batch retries alone.

```flow
dir: LR
nodes:
  - { id: start, col: 1, row: 1, kind: start, label: post.created consumed }
  - { id: count, col: 2, row: 1, kind: process, label: Count followers of author }
  - { id: dec, col: 3, row: 1, kind: decision, label: "10,000 followers or more?" }
  - { id: celeb, col: 4, row: 1, kind: process, label: Append post id to author's pull list }
  - { id: page, col: 4, row: 2, kind: process, label: "Load next 1,000 active followers" }
  - { id: write, col: 5, row: 2, kind: process, label: Write post id to each follower's feed cache }
  - { id: more, col: 6, row: 2, kind: decision, label: More followers? }
  - { id: done, col: 7, row: 1, kind: end, label: Commit offset }
  - { id: retry, col: 5, row: 3, kind: decision, label: Fewer than 5 attempts? }
  - { id: dlq, col: 6, row: 3, kind: end, label: post.created.dlq }
edges:
  - start -> count
  - count -> dec
  - dec -> celeb: "yes, pull author"
  - dec -> page: "no, push author"
  - celeb -> done
  - page -> write
  - write -> more
  - more -> page: "yes"
  - more -> done: "no"
  - write -x-> retry: cache write fails
  - retry --> write: "yes, back off and retry"
  - retry -x-> dlq: "no"
```

An account that crosses the threshold switches to pull for its next post; its
older pushed posts stay in follower caches until they age out. It switches back
to push only under 8,000 followers, so an account that hovers at the line does
not flip on every post.

## The numbers we hold to

```spec
title: Fan-out contract
accent: teal
rows:
  - { label: Delivery target, value: "A pushed post is in every active follower's feed cache within 5 s (p95)." }
  - { label: Push or pull, value: "Push below 10,000 followers. Pull at or above. Switch back to push only under 8,000." }
  - { label: Batch size, value: "1,000 follower ids per feed cache write batch." }
  - { label: Delivery guarantee, value: "At-least-once from post.created. The write is ZADD keyed by post_id, so a redelivery changes nothing." }
  - { label: Ordering, value: "Partition key = author_id. One author's posts arrive in order; two authors' posts do not." }
  - { label: Feed cache entry, value: "One Redis sorted set per follower, score = created_at, trimmed to 800 post ids, TTL 30 days." }
  - { label: Inactive followers, value: "No push to a follower with no session in 30 days. The feed rebuilds from the follow graph on the next login." }
  - { label: Read path, steps: [Read one page of the feed cache, "Fetch the 20 latest post ids from each followed pull author", Merge by created_at, Hydrate post bodies] }
  - { label: Retry and dead letter, value: "5 attempts, backoff 200 ms to 3 s, then post.created.dlq. Page on-call at 10 dead letters in 5 min." }
```

The read-time merge is bounded. A reader follows at most a few dozen pull
authors, so the Feed API adds a few dozen small list reads to one cache read.
That bound is what makes the split safe. If pull authors ever number in the
thousands per reader, the threshold or the merge has to change.