Design a Search Ranking System
Design a search ranking system: learning to rank, feature engineering, and relevance optimization.
What is a search ranking system?
A search ranking system chooses the most useful ordered results from a very large set of documents. Retrieval finds a broad set of plausible matches quickly; ranking then spends more compute deciding which few deserve to appear first.
This practice problem designs the online ranking path, the feedback loop that improves it, and the guardrails that must still hold when a feature service, cache, or model is unhealthy. The central invariant is simple: never rank or show a document that the requester is not allowed to see.
Candidate generation
Recall before precision
Find hundreds or thousands of eligible candidates cheaply before using an expensive model on a smaller set.
Training-serving parity
One feature meaning
The online feature value must mean the same thing, use the same cutoff time, and carry the same version as its training counterpart.
Hard constraints
Correctness beats a score
Access, legal, moderation, inventory, and deduplication checks are eligibility rules, not optional model features.
Define the outcome before choosing a model
For this interview scenario, design web and product search for 200,000 peak queries per second, a corpus of 10 billion documents, and a 150 ms p95 result deadline. A result page returns ten items, but the system may inspect many more while deciding their order.
The user-visible outcome is useful, trustworthy results. A practical prediction target is the probability that an eligible query-document pair produces a satisfied interaction, such as a long dwell time, successful task completion, or purchase. A click alone is not enough: position, snippet wording, and novelty all influence clicks.
Ask questions that change the design
- Which search surfaces are in scope? Web, product, local, and enterprise search use different eligibility rules and satisfaction signals.
- How fresh must a result be? Breaking-news documents may need to appear within 30 seconds; a catalog price or availability change may require an authoritative lookup before display.
- What does personalization mean? Use a user-controlled cohort or short-lived session context when possible. Do not require an identity profile merely to return useful results.
- Which errors are unacceptable? A private document leak, blocked item, or unsafe result is worse than a lower relevance score and must be prevented before final release.
- What is the evaluation unit? Log the query, candidates shown, positions, model and policy versions, outcomes, and timestamps so that offline data can correct for exposure bias.
Do not train on clicks as if they were labels. A document at position one is seen more often than the same document at position ten. Preserve impressions and positions, then use randomized buckets, propensity weighting, and human judgments to reduce this bias.
Ranking architecture for sub-200ms personalized search
You're ranking results from a huge document index. Each query must return personalized, relevant results. How do you architect the ranking pipeline?
Constraint: Sub-200ms end-to-end latency while delivering personalization and relevance over a huge index.