Advanced Search Systems Architecture

Separate evidence from decisions

Search has several stages because each one answers a different question. Treating a vector score as the final product decision hides these boundaries.

If embeddings, tokenization, or online model serving are unfamiliar, review NLP Systems Architecture and Real-Time ML Inference before tuning an index.

Define the search contract before the architecture

A query path is correct only relative to a product contract. Write that contract in user-visible terms before choosing engines or models.

1. 1

   Input

   Clarify intent

   Define query types, languages, filters, permissions, and the behavior for empty, ambiguous, misspelled, and adversarial input.

2. 2

   Constraints

   Set budgets

   Allocate end-to-end latency across understanding, retrieval, feature access, reranking, assembly, and network time, including a deadline for each child call.

3. 3

   Evidence

   Name quality

   Choose labeled-query metrics, traffic slices, and product outcomes. Define the candidate depth at which retrieval recall will be judged.

4. 4

   Fallback

   Choose failure behavior

   Specify what happens when semantic retrieval, the feature store, or the reranker is late, and which constraints may never be bypassed.

Do not optimize these numbers as one undifferentiated score. A relevance gain does not cancel a permission leak, and a fast response does not make stale inventory correct.

Budget retrieval before buying capacity

Start with quantities that have an owner and can be measured:

• Corpus: searchable documents, vector dimensions, bytes per dimension, graph bytes, metadata, replicas, and growth rate.
• Traffic: peak queries per second, burst factor, fan-out, concurrency per serving copy, and target utilization.
• Per-query work: retrieval depth, filters, shard count, feature reads, reranker depth, and model cost.
• Service objective: the end-to-end percentile target and the smaller deadline assigned to each stage.

The planning relationship is required serving copies = ceil(peak QPS x burst headroom / measured QPS per copy). Index bytes and copies determine memory; per-query work and queueing determine latency. Replicas can absorb more concurrent queries, but they do not make one expensive query cheaper.

Change the retrieval budget

Choose a workload and vector representation, then change candidate depth and the latency target. Watch memory, shard copies, and modeled latency move for different reasons.

Give every stage a deadline and a fallback

Hybrid retrieval should run independent branches concurrently, preserve their source ranks, and stop waiting when a child deadline expires. Fusion and reranking then operate on whatever valid evidence arrived before their budgets closed.

Inject a production failure

Switch among a healthy request, a degraded vector index, a failed reranker, and stale indexes. Select any node to inspect its responsibility and compare the correct response.

The fallback ladder must be deterministic and tested:

1. Return the learned reranker order when all required evidence arrives on time.
2. Return the fused lexical and semantic order when the reranker misses its deadline.
3. Return a healthy single-retriever order when the other retriever is unavailable.
4. Return an explicit empty or unavailable state when authorization, deletion state, or minimum result integrity cannot be proven.

Failing open on permissions is not graceful degradation. Safety and authorization remain hard constraints even when relevance stages are shed.

Fuse ranks before applying expensive models

Lexical and vector scores usually have different scales and distributions. Adding their raw scores makes the result sensitive to engine-specific calibration. Reciprocal rank fusion (RRF) instead combines positions: each result receives 1 / (rank constant + source rank) from every retriever that found it.

RRF is a strong baseline because it is deterministic, handles missing candidates, and does not require the score distributions to match. Its limits are equally important: it does not learn query-specific weights, and it cannot recover a relevant document that both retrievers missed.

After fusion, rerank only a bounded set. Rich features may include exact-term evidence, semantic similarity, freshness, popularity, personalization, document quality, and query-document interactions. Version every feature and model used in a response so an offline replay can reconstruct the decision.

Keep indexing reproducible and freshness visible

Search is a derived view of authoritative data. A safe indexing pipeline can replay changes, produce matching lexical and vector versions, and prove when a deletion or permission update became searchable.

Track at least three freshness clocks:

• Source lag: time from an authoritative commit to ingestion.
• representation lag: time from ingestion to lexical fields and embeddings being ready.
• serving lag: time from a complete index generation to every serving replica using it.

Use idempotent writes keyed by document version. Keep tombstones long enough for lagging workers and replay jobs to observe them, and make permission or deletion checks fail closed when index generations disagree.

Gate quality, latency, freshness, and safety together

Offline evaluation answers whether a candidate is worth exposing. Online evaluation answers whether it helps real users under production traffic. Neither layer is sufficient alone.

Clicks are biased labels: users can click only what the old ranker exposed, position changes click probability, and attractive snippets can hide poor task completion. Use randomized exploration or debiasing where appropriate, and keep human judgments for critical and low-traffic slices.

Diagnose symptoms at the owning stage

Production incidents become easier to reason about when every symptom maps to a stage, a measurement, and a safe first action.

Log one traceable record per query with the normalized and original query, filters, tenant, experiment, index generation, embedding and reranker versions, source candidate ranks, stage timings, fallback reason, returned document IDs, and exposure ID. Sample payloads carefully, but keep aggregate counters for every request.

Make architecture trade-offs explicit

• One engine or specialized services: one engine simplifies operations and filtering; specialized lexical, vector, and ranking services allow independent tuning but add network, consistency, and ownership boundaries.
• Pre-filter or post-filter: pre-filtering protects eligibility and saves work but can reduce ANN recall when the eligible set is sparse; post-filtering may need deeper retrieval and must never be the only authorization check.
• Larger candidates or a stronger retriever: larger sets can improve recall but increase fan-out and reranker cost; better embeddings or routing may improve recall without linear candidate growth.
• Fresh updates or efficient segments: aggressive refresh improves visibility but fragments indexes and consumes merge capacity; generation-based publishing improves reproducibility at the cost of controlled lag.
• Personalization or shared caching: personalized features can improve ordering while lowering cache reuse and increasing privacy obligations; preserve a non-personalized fallback.

Production readiness checklist

• Define candidate depth, relevance datasets, traffic slices, and release thresholds before tuning.
• Allocate child deadlines and load-test every fallback, including partial retriever responses.
• Version indexes, embeddings, features, rerankers, experiments, and judgment sets.
• Make permissions, tombstones, and tenant isolation fail closed across all serving paths.
• Monitor recall proxies, NDCG on judged traffic, task outcomes, p95/p99 latency, timeout rate, zero-result rate, fallback rate, and all three freshness clocks.
• Rehearse rollback for index generations and model versions without requiring a full reindex.