Designing a News Feed / Timeline

The news feed is one of the most impactful system design problems because it sits at the intersection of data modeling, caching, fan-out strategies, ranking algorithms, and real-time delivery. You are designing the system that powers Twitter's home timeline, Instagram's feed, or Facebook's news feed -- showing each user a personalized, ranked list of posts from the people they follow.

Requirements

Functional Requirements

• A user can create a post (text, image, video).
• A user can follow other users.
• A user's home feed displays posts from people they follow, ordered by relevance or recency.
• The feed updates in near real-time when someone they follow posts.

Non-Functional Requirements

• Low latency: feed should load in under 200ms.
• High availability: the feed is the core product experience.
• Scale: 500 million daily active users, each following hundreds of accounts. Some accounts (celebrities) have tens of millions of followers.

Capacity Estimation

Assumptions:
  - 500M DAU
  - Average user checks feed 10 times/day → 5 billion feed requests/day
  - QPS: 5B / 86,400 ≈ 58,000 feed reads/sec
  - 10M new posts/day
  - Average user follows 200 accounts

Feed generation:
  - Naive approach: for each feed request, query posts from 200 followees
  - 58,000 × 200 = 11.6M database queries/sec (unsustainable)

This back-of-the-envelope calculation immediately shows why you cannot compute feeds on the fly. You need to pre-compute them.

Fan-Out Strategies

This is the core design decision, and the one interviewers care about most.

Fan-Out on Write (Push Model)

When a user creates a post, the system immediately pushes it to the feed cache of every follower. Each user's feed is pre-computed and ready to serve.

Alice creates a post:
1. Store the post in the posts table.
2. Look up Alice's follower list: [Bob, Carol, Dave, ... (500 followers)]
3. For each follower, insert Alice's post ID into their feed cache.

When Bob opens his feed:
1. Read Bob's pre-computed feed cache.
2. Return the top N posts. Done.

def fan_out_on_write(post, author):
    # Store the post
    db.save(post)

    # Push to every follower's feed cache
    followers = get_followers(author.id)
    for follower_id in followers:
        feed_cache.push(follower_id, post.id, post.timestamp)
        # Trim cache to keep only top 800 posts
        feed_cache.trim(follower_id, max_size=800)

Pros:

• Feed reads are extremely fast -- just a cache lookup.
• Feed is always up to date (eventually consistent within seconds).

Cons:

• Celebrity problem. If a celebrity with 50 million followers posts, that is 50 million cache writes. This takes minutes and wastes resources for followers who may never log in.
• Wasted work. Many followers are inactive. Pre-computing their feeds is pointless.
• Write amplification. Every post generates O(followers) writes.

Fan-Out on Read (Pull Model)

The feed is computed at read time. When a user opens their feed, the system queries the posts table for recent posts from all accounts they follow.

When Bob opens his feed:
1. Look up Bob's following list: [Alice, Eve, Frank, ... (200 people)]
2. Query recent posts from each of these 200 accounts.
3. Merge and rank the posts.
4. Return the top N.

Pros:

• No wasted computation for inactive users.
• No celebrity problem on the write path.
• Simple write path: just store the post.

Cons:

• Feed reads are slow. Merging posts from 200+ sources in real time is expensive.
• Harder to rank and personalize in real-time.

Hybrid Approach (The Real-World Solution)

This is what Twitter, Instagram, and Facebook actually use. Combine both strategies based on the follower count of the author:

When a user creates a post:
  - If they have < 10,000 followers → Fan-out on write (push to followers' caches)
  - If they have ≥ 10,000 followers → Skip fan-out (pull at read time)

When a user reads their feed:
  1. Fetch pre-computed feed from cache (contains posts from non-celebrity followees).
  2. Fetch recent posts from celebrity followees (pull model).
  3. Merge and rank both sets.
  4. Return the top N.

This eliminates the celebrity problem while keeping reads fast for the 99% of posts that come from non-celebrity accounts.

Ranking and Scoring

Modern feeds are not purely chronological. They use a ranking model to surface the most relevant content.

Ranking Signals

Post-level signals:
  - Recency (time since posted)
  - Engagement (likes, comments, shares, click-through rate)
  - Content type (video tends to get higher engagement)
  - Post length, has image, has link

User-level signals:
  - Relationship strength (how often you interact with the author)
  - Past engagement (do you typically like this author's posts?)
  - Your interests (inferred from your activity)

Negative signals:
  - Posts you've already seen
  - Posts from accounts you've muted
  - Content you've reported or hidden

Scoring Function

A simplified scoring model:

def score_post(post, viewer):
    recency = time_decay(post.created_at)  # Exponential decay
    affinity = get_affinity(viewer.id, post.author_id)  # 0.0 to 1.0
    engagement = normalize(post.likes + 2 * post.comments + 3 * post.shares)

    return (0.3 * recency) + (0.4 * affinity) + (0.3 * engagement)

In practice, companies use machine learning models (gradient-boosted trees, neural networks) trained on engagement data to predict the probability that a user will interact with a post.

Caching Strategy

Feed Cache

Each user's feed is stored in a sorted set in Redis, ordered by ranking score or timestamp.

Redis sorted set for user Bob:
  Key: "feed:bob"
  Members: post_ids sorted by score

  ZADD feed:bob 0.95 post_123
  ZADD feed:bob 0.87 post_456
  ZADD feed:bob 0.82 post_789

  ZREVRANGE feed:bob 0 19  → Top 20 posts

Keep only the most recent 500-800 posts per user. Older feed items can be fetched from the database on scroll.

Hot Timeline Cache

For very active users (those checking the feed frequently), keep their feed in a hot cache layer with aggressive TTLs. For dormant users, evict the cache and rebuild on the next visit.

Post Content Cache

Separate the feed list (post IDs) from post content. The feed cache stores only post IDs and scores. A second cache layer stores the actual post content (text, images, author info). This avoids duplicating post content across millions of user feeds.

Feed request flow:
1. Get post IDs from feed cache: [post_123, post_456, post_789]
2. Multiget post content from content cache
3. Cache misses → fetch from database, populate cache
4. Assemble and return the feed

Data Model

Table: users
  id, username, name, avatar_url, follower_count, ...

Table: posts
  id, author_id, content, media_url, created_at, like_count, comment_count

Table: follows
  follower_id, followee_id, created_at
  Index on follower_id (who do I follow?)
  Index on followee_id (who follows me?)

Table: feed (materialized, for fan-out-on-write)
  user_id, post_id, score, created_at
  Partition key: user_id
  Clustering key: score DESC

Real-Time Updates

When a user is viewing their feed and a new post arrives, you want to show it without requiring a page refresh.

Approach 1: Polling

The client polls the server every 30 seconds: "Any new posts since my last fetch?" Simple but wasteful.

Approach 2: Long Polling / SSE

The server holds the connection open until new data is available. More efficient than polling.

Approach 3: WebSocket

The client maintains a persistent connection. When a new post is fanned out to the user's feed, the server pushes a notification through the WebSocket. The client can then fetch the new post or display a "New posts available" banner (the Twitter approach).

The Celebrity Problem in Detail

Consider a celebrity with 100 million followers. Fan-out on write for this user would mean:

100 million cache writes × 1ms per write = 100,000 seconds = 27.7 hours

By the time the fan-out completes, the post is already old. Solutions:

1. Hybrid approach (described above): celebrities use pull model.
2. Tiered fan-out: First push to active followers (who logged in within the last 24 hours), then lazily compute for others on demand.
3. Precompute celebrity feeds separately. Maintain a "celebrity feed" per user that is recomputed periodically (every few minutes).

High-Level Architecture

                         ┌───────────────┐
     Clients ──────────> │ Load Balancer  │
                         └───────┬───────┘
                                 │
                    ┌────────────┼────────────┐
                    ▼            ▼            ▼
             ┌───────────┐ ┌──────────┐ ┌──────────────┐
             │ Post Svc  │ │ Feed Svc │ │ Follow Svc   │
             └─────┬─────┘ └────┬─────┘ └──────────────┘
                   │             │
                   ▼             ▼
            ┌────────────┐ ┌────────────┐
            │  Posts DB  │ │ Feed Cache │
            │            │ │  (Redis)   │
            └────────────┘ └────────────┘
                   │
                   ▼
            ┌────────────┐
            │ Fan-out    │
            │ Workers    │
            │ (async via │
            │  Kafka)    │
            └────────────┘

Interview Tips

• Always start with the fan-out trade-off. This is the central design decision. Clearly explain push vs pull, then propose the hybrid approach.
• Name the celebrity problem before the interviewer asks. This shows you have thought deeply about edge cases.
• Separate feed list from content. Storing post IDs in the feed cache (not full post objects) is a critical optimization that interviewers look for.
• Discuss ranking. A purely chronological feed is a starting point, but mention that production systems rank by relevance. Even a simple scoring formula demonstrates awareness.
• Quantify the fan-out cost. Show with numbers why pure fan-out-on-write breaks down at scale (celebrity with millions of followers).
• Mention feed pagination. Users scroll. Discuss cursor-based pagination (using the last post's score/ID as the cursor) rather than offset-based pagination, which has consistency issues when new posts arrive.