System design interview guide
Recommendation System Design Interview: Two-Stage Candidate Generation and Ranking at Billion-Item Scale
A modern recommender has to pick a few dozen items to show you out of a catalog that can run to billions of pieces of content, and it has to do it in tens of milliseconds while you wait for a feed to load. YouTube, Instagram Explore, Pinterest, TikTok, Amazon, and Netflix all run some version of the same shape. Instagram Explore alone serves recommendations to hundreds of millions of daily users (Meta engineering, 2023). The hard part is not the math of any single model. It is building a funnel that can go from billions to tens without blowing the latency budget, and keeping the models fresh as user behavior changes minute by minute.
The interview is really about a funnel. You cannot score billions of candidate items with an expensive model on every request, so you split the work into stages. A retrieval or candidate generation stage cuts the catalog from billions down to a few hundred or a few thousand cheaply, usually with a two-tower model whose item embeddings are precomputed and indexed for approximate nearest neighbor search. A ranking stage then scores that short list with a heavier model that can afford richer user-item cross features. Behind both sits a feature store that serves the same feature values online at request time and offline during training, so the model sees consistent inputs. A streaming pipeline turns clicks, watches, likes, and skips into fresh features and training labels. The recurring themes an interviewer probes are the latency budget across stages, how you keep offline and online features in sync, how you handle cold start for new users and new items, and how you measure whether a change actually helped through A/B testing rather than offline metrics alone.
Asked at: Asked at Meta, Google, YouTube, Netflix, Amazon, Pinterest, LinkedIn, TikTok, Spotify, and most consumer companies with a feed or a catalog. It shows up for ML platform, backend, and generalist system design loops, sometimes framed as design the news feed, design the Explore page, or design product recommendations.
Why this question is asked
It sits at the intersection of distributed systems and machine learning, so interviewers can probe both without either dominating. A candidate who only knows model architectures will miss the serving path, the feature store, and the latency budget. A candidate who only knows backend infrastructure will hand-wave the retrieval and ranking split and get stuck on how you actually pick items from billions. The problem also rewards clear thinking about tradeoffs that have no textbook answer: recall versus precision across stages, freshness versus cost, exploration versus exploitation, and how you avoid a feedback loop where the model only ever learns from what it already showed. It maps to real production systems, so strong candidates can reason from how YouTube or Instagram actually built it.
Requirements
Always clarify these in the first 5 minutes of the interview. Do not start drawing boxes until both lists are agreed.
Functional requirements
- Given a user and a context, return a ranked list of items the user is likely to engage with
- Support multiple retrieval sources: collaborative signals, content similarity, trending, and the user's own recent interactions
- Rank retrieved candidates by predicted engagement, optimizing for a real objective like watch time or long-term retention rather than raw clicks
- Personalize using the user's history, demographics, and real-time session behavior
- Handle cold start for brand new users with little history and for brand new items with no interaction data
- Apply business rules and filtering: remove already-seen items, integrity and policy violations, and blocked content
- Inject diversity and exploration so the feed does not collapse into near-duplicates or a narrow filter bubble
- Support A/B testing so new models and features can be measured against a control before full rollout
- Log impressions and outcomes so every recommendation becomes training data for the next model
Non-functional requirements
- End-to-end serving latency in the tens of milliseconds for the ranking path, single-digit to low tens for retrieval
- Scale to billions of candidate items and hundreds of millions of active users
- High availability, since the feed is a primary surface and an outage is very visible
- Feature freshness: recent interactions should influence recommendations within seconds to minutes
- Consistency between the features used at training time and at serving time to avoid training-serving skew
- Elastic capacity to absorb traffic spikes without breaching the latency budget
- Cost efficiency, since scoring is compute-heavy and runs on every feed load
- Graceful degradation: if ranking is slow or down, fall back to a cheaper cached or heuristic feed
Back-of-envelope scale estimates
Show your math. Pulling numbers from thin air signals you have not thought about the load.
Catalog size
1B+ candidate items
Large platforms index on the order of billions of pieces of content. Pinterest reports its ANN system indexes billions of pins (Pinterest engineering). This is the number retrieval must reduce, so it drives the whole funnel.
Daily active users
~500M (order of magnitude)
Instagram Explore serves hundreds of millions of daily users per Meta. Estimate. If each opens the app 10 times a day and each open triggers a rank request, that is roughly 5B rank requests per day.
Rank requests per second
~60,000 QPS average, higher at peak
Derived. 5B requests per day / 86,400 seconds is about 58,000 per second on average. Peak traffic is commonly 3 to 5 times average, so provision for 200,000+ QPS. Each request fans out to score hundreds of candidates.
Retrieval funnel width
1B+ -> ~1,000 -> ~100 -> ~10
Instagram's public funnel: retrieval selects hundreds of items, first-stage ranking narrows thousands to about 100, second-stage ranking scores those, and reranking picks the final set (Meta engineering, 2023). The exact widths vary by product.
Serving latency budget
Tens of milliseconds total
Estimate anchored to feed-load expectations. Retrieval via ANN runs in single-digit to low tens of milliseconds; ranking a few hundred candidates with a neural net must also fit in tens of milliseconds. The heavier the model, the fewer candidates you can afford to score.
Item embedding refresh
Recomputed daily, cached
Two-tower item embeddings are stable enough to precompute offline during off-peak hours and cache, per Meta. User embeddings are computed online with fresh features. This split is what makes ANN retrieval affordable at scale.
High-level architecture
A request arrives with a user id and context such as device, time, and current session. The serving layer fetches the user's features and recent-interaction signals from the feature store and, for a two-tower retriever, computes a fresh user embedding on the fly. That embedding is sent to an approximate nearest neighbor index of precomputed item embeddings, which returns the top few hundred to few thousand nearest items. In parallel the system pulls candidates from other retrieval sources: trending or heuristic lists, items similar to what the user recently liked, and pregenerated lists that capture long-term interests. These candidate sets are merged and deduplicated. A lightweight first-stage ranker, often a two-tower model itself, scores the merged pool and keeps roughly the top hundred. A heavier second-stage ranker, typically a multi-task neural network, scores those survivors using rich user-item cross features that were too expensive to compute earlier, predicting several outcomes at once such as click, like, long watch, and see-less. A value model combines those predictions with tunable weights into a single score. A final reranking pass applies integrity filtering, removes already-seen items, and shuffles for diversity before the list goes back to the client. Every item shown is logged with its features and the eventual outcome, and a streaming pipeline turns those logs into fresh features and labeled training examples that feed the next round of offline model training.
In a real interview, sketch this on the whiteboard before diving into any single box.
Core components
Walk through each service. The interviewer wants to hear what each one owns, not just the names.
Candidate generation (retrieval) service
Reduces the catalog from billions to a few hundred or thousand cheaply. Most systems blend several sources: a two-tower model that retrieves by embedding similarity, collaborative signals, content-based similarity, trending lists, and the user's own recent interactions. Recall matters more than precision here because the ranker fixes ordering later.
ANN index
An approximate nearest neighbor index, commonly FAISS or an HNSW graph, holds precomputed item embeddings and answers the query 'which items are closest to this user embedding' in milliseconds. It trades a small amount of recall for a huge speedup over scanning billions of vectors. The index is rebuilt or updated as item embeddings are refreshed.
Ranking service
Scores the shortlist with progressively heavier models. A first-stage ranker cheaply cuts thousands to about a hundred, then a second-stage multi-task model predicts several engagement outcomes using user-item cross features. The stages exist because you cannot afford the heavy model on every candidate but can afford it on the survivors.
Feature store
Serves feature values with two faces. An online store, typically a low-latency key-value store, returns fresh features at request time. An offline store, usually a data warehouse or lake, holds the same features historically for training. Keeping the two consistent is what prevents training-serving skew, where the model sees different inputs in production than it trained on.
Real-time event pipeline
Streams clicks, watches, likes, skips, and dwell time off the clients into a log like Kafka, then into stream processors that update session features within seconds and emit labeled training rows. This is what makes recommendations react to what you did a minute ago rather than yesterday.
Offline training pipeline
Batch jobs join logged features with observed outcomes to build training sets, retrain the retrieval and ranking models on a schedule, and republish item embeddings and model artifacts. Training and serving are deliberately decoupled: heavy models train offline and are distilled or exported for cheap online inference.
Experimentation and reranking layer
Routes traffic between model variants for A/B tests, applies business rules, integrity filtering, and diversity shuffling, and enforces exploration so the system keeps gathering signal on items it has not shown much. It also holds the fallback path that serves a cached or heuristic feed when ranking is degraded.
Data model
Pick the right store per table. Justify each choice with the access pattern, not by reflex.
item_embeddingsitem_idembedding_vectormodel_versioncontent_featuresupdated_atThe item tower's output, precomputed offline and loaded into the ANN index. Versioned so the index and the online user tower stay on the same embedding space. Refreshed on a batch cadence, often daily.
user_profileuser_idlong_term_interestsdemographicslanguage_regionembedding_vectorupdated_atSlow-moving user features. The stored embedding can seed retrieval, but for freshness the user tower usually recomputes an embedding online from current features plus recent activity.
interaction_eventsuser_iditem_idevent_typedwell_mscontextevent_timeThe raw log of impressions and outcomes: click, like, save, share, skip, see-less, watch duration. Append-only, high volume, partitioned by time. Source of both real-time features and training labels.
online_featuresentity_idfeature_namefeature_valuettlupdated_atThe online feature store, a low-latency key-value store keyed by user, item, or user-item pair. Session counters and recent-interaction features are written here by the stream processors within seconds.
training_examplesrequest_iduser_iditem_idfeature_snapshotlabelcreated_atPoint-in-time joins of the features that were served with the outcome that followed. Snapshotting features as-served is critical; recomputing them later reintroduces skew and leaks future information.
experiment_assignmentsuser_idexperiment_idvariantmodel_versionassigned_atWhich model or config each user is bucketed into. Joined with interaction_events to compute per-variant engagement so an A/B test can be evaluated for statistical significance.
Deep dives
These are the conversations the interviewer is steering you toward. Practice each one until you can talk through it without notes.
Why two stages: retrieval then ranking
You cannot run an expensive ranking model over a billion items per request; the compute alone would cost more than the product earns and would never fit the latency budget. So you split the work by how much you can afford to spend per item. Retrieval is cheap per item and runs over everything, so it optimizes for recall: get the few hundred good items into the pool even if the ordering is rough. Ranking is expensive per item but runs over only the survivors, so it optimizes for precision: order that short list as accurately as possible. The classic reference is YouTube's 2016 paper, which framed the funnel as candidate generation followed by ranking and became the template most large platforms follow. Instagram Explore extends it to four stages, inserting a lightweight first-stage ranker between retrieval and the heavy ranker so the expensive model only ever sees about a hundred candidates.
Collaborative filtering vs content-based vs hybrid
Collaborative filtering learns from behavior: users who engaged with the same items are similar, and items engaged with by the same users are similar, so it recommends what similar users liked. It is powerful but blind to brand new items with no interactions, which is the item cold-start problem. Content-based filtering uses features of the item itself, such as text, category, or a visual embedding, and can recommend a fresh item on day one because it needs no interaction history, but it tends to stay inside what the user already likes and struggles to surprise. Production systems are hybrid. The two-tower model is a natural hybrid: the item tower can consume content features so a new item gets a reasonable embedding immediately, while the user tower learns collaborative patterns from behavior. You then blend collaborative retrieval sources with content-similarity and trending sources so no single weakness dominates.
Two-tower embeddings and ANN retrieval
The two-tower model trains a user network and an item network so that a user embedding and the embeddings of items they engaged with land close together in the same vector space, usually measured by dot product or cosine similarity. The key architectural property is that after training the towers are independent. That lets you precompute every item embedding offline, load them into an ANN index, and at request time only compute the user embedding online and query the index. Instagram computes item embeddings daily during off-peak hours and caches them, and generates the user embedding on the fly from the freshest features. The ANN index, FAISS or an HNSW graph, finds approximate nearest neighbors without scanning all billion vectors, which is the whole reason retrieval fits in milliseconds. The approximation costs a little recall, tuned by graph parameters, in exchange for orders of magnitude less compute.
The feature store and training-serving skew
A recommender's biggest correctness trap is not the model, it is feeding the model different features in production than it saw in training. The feature store exists to prevent that. It has an online path, a low-latency key-value store that answers feature lookups at request time, and an offline path, a warehouse that stores the same feature values historically for building training sets. The discipline that matters is logging features as they were served, then joining them to outcomes with a point-in-time correct join so a training row reflects exactly what the model would have seen at that instant. If instead you recompute features later, you leak information from the future and get a model that looks great offline and disappoints in production. Shared feature definitions across the two paths, so the same code computes a feature online and offline, are what keep the two consistent.
Offline training vs online serving and the batch plus online split
Training and serving have opposite constraints. Training can be slow, huge, and run on GPUs over months of history; serving must be fast, cheap, and predictable. So they are decoupled. Heavy models train offline on batch data, and a cheaper form is exported for online inference. Meta describes training high-capacity teacher models offline, then distilling them into compact quantized student models for serving, which keeps the modeling power without the serving cost. Item embeddings follow the same pattern: computed in batch, served from cache. On top of the batch backbone, a streaming layer keeps a thin slice of state fresh, mostly session and recent-interaction features. This is the lambda-style split in practice: a batch layer for the expensive, slow-changing computation and a speed layer for the recent events, merged at serving time. The interview signal is knowing which computations belong in which layer and why.
Cold start for users and items
Cold start comes in two flavors and they need different answers. New users have no history, so collaborative signals are empty; you fall back to popularity and trending, use whatever context you have such as region, language, device, and the signup source, and lean on fast exploration to learn preferences within the first session. Instagram treats a user's real-time interactions in the current session as a retrieval source precisely so a new session becomes useful immediately. New items have no interactions, so collaborative retrieval will never surface them; content-based embeddings from the item's text or image give it a starting position in the vector space, and a deliberate exploration budget forces the system to show it to some users to gather the first signals. Without that exploration, popular items get shown, get more engagement, and get shown even more, and fresh content never escapes the cold start, which also starves the training data.
Evaluation, A/B testing, and the feedback loop trap
Offline metrics like AUC or recall at K tell you a model ranks held-out data well, but they are computed on impressions the old model chose, so they are biased toward what was already shown and cannot see the counterfactual of items you never surfaced. The decision to ship rests on online A/B tests measuring real objectives: retention, session length, long-term engagement, not just click-through rate, because optimizing clicks alone rewards clickbait, which is exactly why YouTube switched its ranking objective to expected watch time. The deeper trap is the feedback loop. The model learns only from items it chose to show, so it reinforces its own past choices and can collapse users into a narrow filter bubble. Countermeasures are explicit exploration to gather signal on unshown items, diversity injection in reranking, and monitoring for popularity bias so a small set of items does not dominate every feed.
Trade-offs to discuss
Every senior interviewer expects you to surface at least 3 of these. Pick the decisions, state the alternatives, and justify your choice.
More retrieval sources versus serving complexity
Blending collaborative, content, trending, and session sources improves coverage and helps cold start, but each source is a system to build, monitor, and keep fresh. Start with a strong two-tower retriever plus a trending fallback, then add sources where you can measure a lift.
Recall in retrieval versus precision in ranking
Push retrieval toward high recall so good items reach the ranker even if the pool is noisy, and let the expensive ranker handle precision. Over-tuning retrieval for precision wastes the ranker's strength and quietly drops items the ranker would have loved.
Feature freshness versus serving cost
Recomputing user embeddings and session features online every request keeps recommendations reactive but costs compute and latency. Precomputing and caching is cheaper but stale. The usual answer is precompute the slow item side, compute the fast user side online.
Model capacity versus latency budget
A bigger ranking model predicts better but scores fewer candidates in the same budget. Distilling a heavy offline teacher into a light online student, plus the first-stage ranker cutting the pool to about a hundred, is how you keep accuracy without breaking the millisecond budget.
Exploration versus exploitation
Always showing the highest-scoring items maximizes short-term engagement but starves new items and users of signal and hardens filter bubbles. Spending a slice of impressions on exploration costs immediate engagement but is what keeps the model learning and the catalog alive.
Optimizing clicks versus long-term value
Click-through rate is easy to measure and easy to game with clickbait. Optimizing a downstream objective like watch time or multi-week retention is harder to attribute and slower to measure, but it is what the business actually wants and what a strong candidate argues for.
How Recommendation System actually does it
The reference architecture comes straight from published engineering work. YouTube's 2016 RecSys paper, Deep Neural Networks for YouTube Recommendations by Covington, Adams, and Sargin, defined the candidate generation plus ranking funnel and made the case for optimizing expected watch time over click-through rate. Meta's 2023 engineering post on scaling Instagram Explore lays out a concrete four-stage funnel, retrieval with two-tower models and ANN over FAISS or HNSW, a first-stage ranker trained by distillation to imitate the second stage, a multi-task multi-label second-stage ranker, and a value model that combines predicted click, like, and see-less probabilities with tunable weights, all held together with caching and precomputation. Pinterest's PinnerSage and PinSage work shows the same embedding-plus-ANN pattern at billion-pin scale, using HNSW indices and clustering users into multiple embeddings rather than one. These are the systems to name in an interview, and they agree on the shape: precompute the item side, compute the user side online, retrieve with ANN, rank in stages, and decouple offline training from online serving.
Sources
- Deep Neural Networks for YouTube Recommendations (Covington, Adams, Sargin, RecSys 2016)
- Scaling the Instagram Explore recommendations system (Meta Engineering, 2023)
- PinnerSage: Multi-Modal User Embedding Framework for Recommendations at Pinterest (arXiv)
- Graph Convolutional Neural Networks for Web-Scale Recommender Systems (PinSage, arXiv)
- The Two-Tower Model for Recommendation Systems: A Deep Dive (Shaped)
Lessons to study before this interview
If any of these topics are fuzzy, the interviewer will catch it. Each lesson is 15 to 60 minutes with diagrams, code, and a quiz.
Related system design interview questions
Practice these next. They lean on the same core building blocks as Recommendation System.