Designing Recommendation Systems at Scale: A Complete System Design Deep Dive

A user opens an app for 15 seconds while waiting for coffee. In those seconds, the system must decide which content to show first, which products to rank, which creators to highlight, and which exploration opportunities are worth spending impressions on.

One wrong recommendation costs almost nothing.

One million wrong recommendations per minute reshape retention, revenue, creator distribution, and trust.

Recommendation systems are not a single model problem. They are end-to-end systems problems:

• data freshness,
• online serving latency,
• feedback loops,
• business constraints,
• experimentation discipline,
• resilience when models fail.

In interviews, many candidates over-focus on model types and under-focus on architecture. Senior candidates treat recommendation as a distributed system with machine learning components, strict SLOs, and operational guardrails.

This guide follows that senior path.

Table of Contents

• Requirements Analysis
• Back-of-Envelope Calculations
• High-Level Architecture
• API Design
• Data Modeling
• Candidate Generation Pipeline (Core 1)
• Retrieval System and ANN Infrastructure (Core 2)
• Ranking and Re-Ranking Layers (Core 3)
• Feature Store Architecture (Core 4)
• Real-Time Signal Ingestion (Core 5)
• Exploration, Diversity, and Constraints (Core 6)
• Feedback Loops and Model Governance (Core 7)
• Serving Reliability and Fallbacks (Core 8)
• Training, Validation, and Deployment Pipelines
• Experimentation and Causal Measurement
• Caching and Sharding Strategy
• Multi-Region and Disaster Recovery
• Security, Privacy, and Responsible AI Controls
• Observability and SLO Design
• Interview Tips and Strategy
• Anti-Patterns to Avoid
• Conclusion
• References
• Quick Reference

Requirements Analysis

A recommendation system can be built for feed ranking, ecommerce discovery, ad ranking, video suggestions, or social graph suggestions. This guide uses a generalized architecture that applies to all of them.

Functional Requirements

1. Generate personalized candidate items for a user.
2. Rank candidates by predicted utility under business constraints.
3. Return recommendations under tight latency budget.
4. Update recommendations with fresh user interactions.
5. Support cold-start users and items.
6. Provide fallback recommendations when models are unavailable.
7. Expose explainability metadata where needed.
8. Support A/B experimentation and canary rollouts.

Non-Functional Requirements

Success Metrics

System should optimize a portfolio of metrics, not one scalar:

1. CTR (click-through rate)
2. watch time or dwell time
3. conversion rate
4. session depth
5. retention impact
6. creator/seller fairness metrics
7. latency and error budgets

Clarifying Questions in Interviews

1. Is this for feed, ecommerce, or video recommendations?
2. What is the max latency budget?
3. How fresh must recommendations be?
4. Is real-time personalization required?
5. Are fairness/diversity constraints in scope?

For this design we assume:

• feed-style and catalog-style recommendations,
• p99 <= 120ms end-to-end serving,
• near-real-time behavior updates,
• experimentation and fallback are mandatory.

Back-of-Envelope Calculations

These estimates drive architecture decisions.

Assumed Scale

DAU: 400 million
Recommendation requests/day: 120 billion
Avg recommendations returned per request: 20
Candidate pool per request before ranking: 2,000
Item catalog size: 1.5 billion active items
User events/day: 2.8 trillion interactions
Peak multiplier: 5x

QPS Estimates

Req/sec avg = 120,000,000,000 / 86,400 ~= 1,388,889
Req/sec peak (5x) ~= 6,944,445

Ranking Compute Estimate

If ranking 2,000 candidates/request naively:

6.94M req/sec * 2,000 candidates ~= 13.9 billion item scores/sec

This is expensive. Hence multi-stage architecture:

• lightweight retrieval reduces to hundreds,
• heavy ranker runs on smaller set,
• final re-ranker handles top slice.

Event Ingestion Load

2.8T events/day ~= 32.4M events/sec average
Peak (3x) ~= 97M events/sec

This demands partitioned streaming architecture and careful schema governance.

Storage Directional Estimate

Assume 200 bytes/event compressed average for logged interactions.

2.8T * 200B ~= 560TB/day raw logical
with compression and tiering still massive

Hence strong sampling policies, retention tiers, and feature materialization are required.

Core Insight

Recommendation systems are dominated by:

1. retrieval and ranking efficiency,
2. event freshness,
3. online/offline feature consistency,
4. operational controls against regressions.

High-Level Architecture

Architecture Principles

1. Multi-stage retrieval and ranking to control compute.
2. Clear separation between serving plane and training plane.
3. Online/offline feature parity contracts.
4. Fallback path as first-class design element.
5. Continuous experimentation and guardrail monitoring.

API Design

Design API around ranking intent and context quality.

Core Endpoint

POST /v1/recommendations

Request:

{
  "request_id": "req_2381",
  "user_id": "u_991",
  "surface": "HOME_FEED",
  "context": {
    "device": "ios",
    "locale": "pt-BR",
    "timezone": "America/Sao_Paulo",
    "network_type": "wifi"
  },
  "session": {
    "session_id": "s_882",
    "entry_point": "app_open"
  },
  "constraints": {
    "max_items": 20,
    "allow_sensitive": false,
    "diversity_level": "medium"
  },
  "debug": false
}

Response:

{
  "request_id": "req_2381",
  "model_version": "ranker_v278",
  "items": [
    {
      "item_id": "i_101",
      "score": 0.9281,
      "reason_codes": ["similar_interest", "fresh_content"],
      "rank": 1
    }
  ],
  "served_at": "2026-02-22T23:40:10Z",
  "fallback_used": false,
  "trace_id": "tr_5502"
}

Interaction Event API

POST /v1/recommendations/events

{
  "event_id": "evt_901",
  "request_id": "req_2381",
  "user_id": "u_991",
  "item_id": "i_101",
  "event_type": "CLICK",
  "position": 1,
  "surface": "HOME_FEED",
  "event_ts": "2026-02-22T23:40:21Z"
}

Idempotency for Events

Use dedupe key:

dedupe_key = (event_id) or (request_id, user_id, item_id, event_type, event_ts_bucket)

Data Modeling

Recommendation architecture relies on multiple data shapes.

Core Data Objects

1. User profile vectors and sparse features.
2. Item metadata, embeddings, and policy tags.
3. Interaction events stream.
4. Model metadata and rollout configs.
5. Experiment assignment logs.

User Feature Schema (Example)

{
  "user_id": "u_991",
  "embedding": [0.12, -0.44, 0.89],
  "recent_topics": ["ml", "system-design", "cloud"],
  "activity_1d": {
    "views": 52,
    "clicks": 13,
    "likes": 2
  },
  "quality_signals": {
    "session_depth_avg_7d": 8.2,
    "negative_feedback_ratio": 0.07
  },
  "updated_at": "2026-02-22T23:31:01Z"
}

Item Feature Schema

{
  "item_id": "i_101",
  "creator_id": "c_55",
  "embedding": [0.02, 0.88, -0.31],
  "tags": ["distributed-systems", "architecture"],
  "freshness_hours": 3,
  "quality_score": 0.82,
  "policy": {
    "eligible_surfaces": ["HOME_FEED", "DISCOVER"],
    "sensitivity_level": "LOW"
  },
  "updated_at": "2026-02-22T23:29:44Z"
}

Experiment Assignment Table

CREATE TABLE experiment_assignments (
  experiment_id VARCHAR(64) NOT NULL,
  user_id BIGINT NOT NULL,
  variant VARCHAR(32) NOT NULL,
  assigned_at TIMESTAMP NOT NULL,
  PRIMARY KEY (experiment_id, user_id)
);

ER Diagram (Simplified)

Candidate Generation Pipeline (Core 1)

Ranking every item is impossible. Candidate generation narrows search space.

Candidate Sources

1. collaborative filtering candidates.
2. embedding nearest-neighbor candidates.
3. popularity-trending candidates.
4. follow-graph or social proximity candidates.
5. business-priority candidates.

Candidate Blending

Combine multiple candidate sets and deduplicate.

final_candidates = unique(
  topK(collab, 500)
  U topK(embedding, 600)
  U topK(trending, 200)
  U topK(business, 100)
)

Candidate Generation Sequence

Cold-Start Strategy

For new users:

• use region/language popularity,
• infer interests from onboarding signals,
• use exploration-heavy mix initially.

For new items:

• use metadata-based similarity,
• controlled exploration budget,
• creator priors and quality filters.

Retrieval System and ANN Infrastructure (Core 2)

Embedding retrieval is usually the largest latency-sensitive component.

ANN Index Requirements

1. high recall at low latency.
2. online updates or frequent rebuild support.
3. partitioning by language/region/surface.
4. memory efficiency for large catalogs.

Retrieval Path

1. fetch user embedding.
2. query ANN index for nearest item vectors.
3. apply coarse filters (policy, language, availability).
4. return top N item ids to ranker.

ANN Retrieval Diagram

Partition Strategy

• shard by item id hash for balance,
• optionally route by locale or vertical first,
• keep replicas for fast failover.

Index Refresh Modes

Retrieval Latency Budget Example

Total budget: 120ms
- API + orchestration: 15ms
- retrieval calls: 30ms
- feature fetch: 25ms
- rank + rerank: 35ms
- serialization/network: 15ms

Ranking and Re-Ranking Layers (Core 3)

Ranking predicts user-item utility under constraints.

Typical Multi-Stage Ranking

1. lightweight pre-ranker on ~2k candidates.
2. heavy ranker on top ~300.
3. re-ranker on top ~50 for constraints and diversity.

Ranking Features

• user features: affinity, recency, session intent,
• item features: quality, freshness, creator trust,
• context features: device, time, network,
• cross features: user-item interaction history.

Ranking Service Interface

{
  "user_id": "u_991",
  "surface": "HOME_FEED",
  "candidate_ids": ["i_101", "i_102"],
  "feature_refs": {
    "online_snapshot_id": "ofs_788"
  },
  "model_version": "ranker_v278"
}

Re-Ranking Constraints

Common constraints:

1. source diversity,
2. creator fairness caps,
3. policy and safety exclusion,
4. recency quotas,
5. sponsored placement rules.

Re-Ranking Flow

Ranking Pitfall

Optimizing a single short-term metric can hurt long-term retention and ecosystem health. Always include guardrail metrics.

Feature Store Architecture (Core 4)

Feature consistency is one of the hardest practical problems.

Feature Store Goals

1. online low-latency feature serving.
2. offline batch feature generation.
3. consistent feature definitions across training and serving.
4. point-in-time correctness for model training.

Online vs Offline Stores

• Offline store: large historical datasets for training.
• Online store: latest feature values for inference.

Feature Pipeline

Point-in-Time Correctness

During training, never join future features accidentally.

Bad:

• using post-click aggregated features to predict pre-click outcomes.

Good:

• enforce training snapshot timestamp boundaries.

Feature Definition Example

feature_name: user_click_rate_7d
entity: user_id
logic: clicks_7d / impressions_7d
source: interaction_events
update_mode: streaming
online_ttl: 10m
offline_backfill: daily

Real-Time Signal Ingestion (Core 5)

Fresh events are essential for personalization quality.

Event Types

1. impression,
2. click,
3. like/favorite,
4. add-to-cart,
5. purchase,
6. hide/not-interested,
7. session end.

Ingestion Pipeline

1. SDK emits event.
2. edge collector validates schema and auth.
3. stream bus appends event.
4. stream processors update online features.
5. durable storage for analytics/training.

Event Pipeline Diagram

Data Quality Controls

1. schema registry with compatibility checks,
2. event dedupe,
3. clock skew handling,
4. bot/spam filtering,
5. delayed event watermark handling.

Freshness Target

High-impact signals should reach online features in seconds, not hours.

Exploration, Diversity, and Constraints (Core 6)

Pure exploitation creates filter bubbles and long-term stagnation.

Exploration Strategies

1. epsilon-greedy buckets,
2. Thompson sampling variants,
3. contextual bandit policies,
4. controlled randomization by user segment.

Diversity Constraints

1. source diversity (avoid same creator repetition),
2. topic diversity,
3. freshness quotas,
4. novelty quotas.

Constraint-Aware Objective

maximize expected_utility(items)
subject to:
- policy_eligible(item) == true
- creator_repeat_cap <= threshold
- min_diversity_score >= threshold
- sponsored_slots <= configured_limit

Practical Deployment

Start with deterministic ranker + lightweight controlled exploration. Increase sophistication only after observability and safeguards mature.

Trade-off Table

Feedback Loops and Model Governance (Core 7)

Recommendation systems influence the data they train on.

Feedback Loop Risks

1. rich-get-richer item exposure bias,
2. creator starvation,
3. popularity collapse into narrow topics,
4. self-reinforcing bias on historical demographics.

Governance Controls

1. monitor distribution shift in exposure,
2. maintain minimum exposure quotas for eligible inventory,
3. add long-term utility objectives,
4. run fairness diagnostics by cohort.

Offline Evaluation Should Include

• precision@k / recall@k,
• calibration metrics,
• coverage and novelty,
• fairness and creator distribution,
• counterfactual estimators where applicable.

Model Registry Metadata Example

{
  "model_name": "ranker_v278",
  "training_data_window": "2026-01-20..2026-02-18",
  "feature_set_version": "fs_v91",
  "offline_metrics": {
    "auc": 0.812,
    "ndcg_20": 0.442,
    "coverage_1k": 0.71
  },
  "guardrails": {
    "creator_gini_max": 0.82,
    "latency_p99_ms_max": 40
  }
}

Promotion Rules

Model promotion should require both metric gains and guardrail compliance.

Serving Reliability and Fallbacks (Core 8)

Recommendation failures should degrade quality, not availability.

Fallback Hierarchy

1. primary personalized ranker output,
2. cached recent recommendations,
3. popularity-by-context lists,
4. safe global defaults.

Serving Flow with Fallbacks

Circuit Breakers

• breaker on ANN latency spikes,
• breaker on rank model inference failures,
• timeout budgets with partial results.

Cache Strategy for Reliability

Maintain short-lived per-user cached results and longer-lived context fallback caches.

Contract Rule

Never return empty list unless policy requires it. Empty surfaces usually destroy UX.

Training, Validation, and Deployment Pipelines

Model quality depends on data and deployment discipline.

Training Pipeline Stages

1. data extraction and cleansing,
2. feature materialization,
3. train/validation split with time boundaries,
4. model training and hyperparameter tuning,
5. offline evaluation and guardrail checks,
6. artifact registration and reproducibility metadata.

Deployment Patterns

1. shadow mode inference,
2. canary rollout by traffic slice,
3. progressive ramp with guardrail checks,
4. automatic rollback if guardrails breached.

Deployment Flow

Reproducibility Requirements

Every model artifact should link to:

• data snapshot id,
• feature definitions version,
• training code commit hash,
• hyperparameter config,
• evaluation report.

Experimentation and Causal Measurement

Recommendation changes require controlled measurement.

Experiment Framework Essentials

1. deterministic user bucketing,
2. holdout groups,
3. guardrail metrics,
4. segment-level analysis,
5. sequential testing controls.

Metrics for Decision

Primary metric depends on surface:

• feed: session depth + retention,
• ecommerce: conversion + revenue per session,
• video: watch time + return rate.

Guardrails:

• latency,
• crash/error rate,
• diversity/fairness measures,
• negative feedback rate.

Experiment Assignment API (Concept)

{
  "experiment_id": "exp_ranker_278",
  "user_id": "u_991",
  "variant": "B",
  "assigned_at": "2026-02-22T23:35:00Z"
}

Common Measurement Pitfalls

1. running too many overlapping experiments on same surface,
2. stopping tests too early,
3. optimizing short-term CTR while harming retention,
4. ignoring cohort heterogeneity.

Caching and Sharding Strategy

Serving performance depends on smart partitioning and caching.

Cache Layers

1. per-user recommendation cache (very short TTL),
2. context/popularity cache by locale/surface,
3. feature cache for hot entities,
4. model artifact/config cache.

Cache Keys

rec:user:{user_id}:surface:{surface}
rec:context:{locale}:{surface}:{hour_bucket}
features:user:{user_id}
features:item:{item_id}
model:config:{surface}

Sharding Strategy

Hot Key Mitigation

• replicate extremely hot context keys,
• randomize small suffix for ultra-hot fallback keys,
• precompute and distribute popular lists.

Multi-Region and Disaster Recovery

Recommendation serving is global and latency-sensitive.

Regional Design

• region-local serving clusters,
• regional feature store replicas,
• global model registry with regional rollout controls,
• asynchronous replication for events and offline stores.

Multi-Region Diagram

Failover Rules

1. if primary region degraded, shift to nearest healthy region.
2. use stale-but-valid feature snapshots instead of hard failure.
3. downgrade to context fallback if personalized path unavailable.
4. protect global control plane from cascading failures.

RPO/RTO Guidance

• serving path RTO minutes,
• online feature freshness may degrade briefly,
• offline retraining can tolerate larger RTO windows.

Security, Privacy, and Responsible AI Controls

Recommendation systems process sensitive behavioral data.

Security Controls

1. strict auth and authorization on feature stores.
2. encryption in transit and at rest.
3. audit trails for model and config changes.
4. secrets management for model endpoints and pipelines.

Privacy Controls

1. data minimization by feature necessity.
2. retention policies for interaction events.
3. user deletion/export workflows.
4. anonymization/pseudonymization where possible.

Responsible AI Controls

1. policy filters for unsafe content categories.
2. fairness diagnostics across user and creator cohorts.
3. explainability hooks for regulated contexts.
4. human override controls in high-risk domains.

Abuse and Manipulation Risks

• bot-generated fake engagement,
• coordinated boosting attacks,
• model poisoning attempts.

Mitigation requires anomaly detection and trust-weighted signals.

Observability and SLO Design

SLOs must include both system and model health.

SLO Table

Golden Signals

System signals:

• request latency,
• request error rate,
• dependency timeouts,
• queue lag.

Model signals:

• score distribution drift,
• feature null rate drift,
• calibration drift,
• diversity collapse indicators.

Trace IDs

x-request-id
x-user-id
x-surface
x-model-version
x-experiment-variant

Alerting Priorities

1. page for serving outages and extreme latency.
2. page for feature freshness failures on critical features.
3. ticket for quality drift without immediate user harm.

Interview Tips and Strategy

Strong Interview Structure

1. Clarify use case and optimization objective.
2. Show scale and compute constraints.
3. Present multi-stage retrieval and ranking architecture.
4. Explain online/offline feature stores.
5. Cover experimentation and fallback reliability.
6. Add fairness, governance, and observability.

Common Candidate Mistakes

1. designing only a model, not a system.
2. no freshness or feature consistency strategy.
3. no fallback when ranker/index fails.
4. no experimentation framework.
5. ignoring feedback loop risks.

Trade-off Table

Whiteboard Script

If asked "design a recommendation system":

1. Start with candidate generation, retrieval, ranking, reranking.
2. Add online/offline feature stores and event ingestion.
3. Add experimentation and model deployment lifecycle.
4. Add fallback path and reliability controls.
5. Finish with fairness, privacy, and key SLOs.

Anti-Patterns to Avoid

1) Running expensive ranker on full catalog

Problem: impossible compute and latency profile.

Fix: use candidate generation + retrieval to shrink search space first.

2) Training-serving feature mismatch

Problem: offline gains do not reproduce online.

Fix: versioned feature definitions and point-in-time training joins.

3) No fallback path

Problem: model/index incident causes blank recommendation surfaces.

Fix: layered fallback hierarchy with tested degradation modes.

4) Optimizing CTR only

Problem: short-term click gains but long-term retention and ecosystem damage.

Fix: multi-objective optimization with guardrail metrics.

5) Ignoring exploration

Problem: recommendation entropy collapses; discovery quality declines.

Fix: controlled exploration budget and diversity constraints.

6) No drift monitoring

Problem: silent model quality decay in production.

Fix: monitor feature, score, and outcome drift continuously.

7) Weak experiment discipline

Problem: false wins and unstable rollouts.

Fix: robust experimentation framework with causal safeguards.

Conclusion

Recommendation systems at scale are distributed systems with ML intelligence, not ML scripts wrapped in APIs.

The stable architecture pattern is:

• efficient multi-stage retrieval and ranking,
• consistent online/offline feature platform,
• realtime signal ingestion,
• robust fallback and reliability controls,
• disciplined experimentation and governance.

If you can explain this architecture with clear trade-offs, you can deliver strong system design interviews and build recommendation platforms that are fast, resilient, and responsible.

References

• Google - Deep Neural Networks for YouTube Recommendations
• Google - Wide & Deep Learning for Recommender Systems
• TensorFlow Recommenders
• NVIDIA Merlin Recommender Systems
• Feast Feature Store
• Apache Kafka Documentation
• Google SRE Book
• Designing Data-Intensive Applications

Quick Reference

Core Layers

Critical Rules

1. Never depend on a single model path without fallback.
2. Never ignore training-serving feature parity.
3. Never optimize short-term metrics without guardrails.
4. Always monitor drift in both data and outcomes.
5. Always treat experimentation as part of architecture.

Serving Lifecycle Snapshot

request -> retrieval -> ranking -> reranking -> response
       \-> fallback when degraded

Suggested Practice Prompts

1. Design a home feed recommendation system for 1B users.
2. Add near-real-time personalization with feature freshness under 60s.
3. Handle ANN index outage without blank feed.
4. Improve long-term retention while preserving CTR.
5. Add fairness constraints for creator ecosystem health.

10-Minute Interview Compression

1) Scope and objective: define surface + success metrics.
2) Scale: estimate QPS and score compute constraints.
3) Architecture: candidate gen -> retrieval -> ranking -> reranking.
4) Data plane: online/offline feature stores + realtime events.
5) Reliability: fallback hierarchy, circuit breakers, multi-region serving.
6) Governance: experimentation, drift monitoring, fairness/privacy controls.

You now have a production-grade blueprint for recommendation system design with practical architecture decisions and interview-ready trade-off framing.