Design ML System: Duplicate Detection
Design an ML system for duplicate detection: embedding strategies, vector search, continuous learning, and production deployment.
What is near-duplicate detection?
Near-duplicate detection finds items that communicate the same information even when their bytes differ. A text pair may be paraphrased, an image may be cropped, or a prompt may contain a changed name while keeping the same intent. Exact hashes catch identical files; an ML system is needed when meaning or visual content must be compared.
For this interview, design a service that checks text and images as they enter customer datasets. Assume 50 million indexed items, 1 million new items per day, and a 500 ms p95 online target. The central invariant is: a similarity score may nominate a pair, but only a versioned decision may merge their duplicate clusters.
Critical path
Retrieve, then verify
Use approximate search to find a small candidate set, then apply a calibrated pairwise decision.
Trust boundary
Isolate each dataset
Never compare or merge customer items across dataset and tenant boundaries without explicit permission.
Recovery
Make merges reversible
Record model, index, threshold, and edge versions so one bad link can be removed without rebuilding everything.
Define what the product decides
The service accepts one new item and returns zero or more candidate duplicates with scores and evidence. It does not delete data automatically. A customer policy decides whether a high-confidence pair is blocked, queued for review, or linked into a duplicate cluster.
Functional requirements
- Detect semantic duplicates among text prompts and responses.
- Detect visually similar images despite resizing, crops, compression, and small edits.
- Return candidate IDs, calibrated confidence, match evidence, and model lineage.
- Accept human labels for duplicate, distinct, and uncertain pairs.
- Support cluster repair when a prior decision is reversed.
Non-functional requirements
- Keep online p95 below 500 ms for an item already inside the supported size limit.
- Isolate vector indices and decision policies by tenant and dataset.
- Continue ingesting when ML dependencies fail by marking the item
pending_detection. - Make every cluster edge attributable to a model, threshold, index, and policy version.
Ask questions that change the architecture
- What counts as a duplicate? A translation, template reuse, and two photographs of the same object may need different labels by dataset purpose.
- Which error costs more? A false merge can hide unique training data; a missed duplicate can contaminate evaluation and inflate dataset size.
- Is blocking required? A synchronous upload gate needs a strict deadline, while an asynchronous quality scan can use deeper models.
- Can content cross boundaries? Default to searching only the current dataset; cross-dataset discovery needs explicit access and privacy policy.
State the initial scope
- Include text and still images; defer video and audio.
- Support one item against an existing dataset, not all-pairs recomputation inline.
- Use human review for uncertain pairs and irreversible customer actions.
- Treat multi-modal relationships as a future extension rather than forcing text and image scores onto one scale.
Near-duplicate detection across billions of items
You must find near-duplicate items in a corpus of billions. Which approach should anchor the matching strategy?
Constraint: Billions of items — the method must scale sub-linearly, and a slight loss of recall is acceptable.