Design ML System: Dataset Diversity Dashboard
Design an ML system for dataset diversity dashboard: analytics pipeline, data quality metrics, and monitoring infrastructure.
What is a dataset diversity dashboard?
A dataset diversity dashboard measures whether training data covers the people, topics, language patterns, task types, and difficulty levels a model is expected to handle. It turns thousands of examples into an evidence-backed release decision: accept the dataset, collect targeted data, or investigate an uncertain slice.
The core invariant is simple: a high average score must never hide a weak critical slice or missing evidence. Every score needs a named population, dimension, target, uncertainty estimate, and metric version.
- Design challenge: Build an ML system that analyzes 10,000 prompt-response pairs per dataset and gives customers trustworthy, actionable diversity evidence.
- Duration: 45 minutes
- Difficulty: Intermediate
- Focus: ML analytics, product decisions, and dataset governance
Question 1
Measure coverage
Does the dataset represent the domains, structures, difficulty levels, and populations required by its intended use?
Question 2
Expose uncertainty
Are missing metadata, unstable clusters, and small sample sizes visible instead of silently becoming reassuring scores?
Question 3
Recommend action
Can a customer move from a failed slice to a specific collection, cleanup, annotation, or taxonomy task?
Define what the product decides
The dashboard helps a customer decide whether a conversational training dataset is ready for delivery. It does not claim that diversity alone proves model quality or fairness. A release decision combines automated coverage metrics with reviewable examples and explicit customer requirements.
Functional requirements
- Upload a dataset and validate its schema, consent metadata, and project boundary.
- Profile topics, prompt structures, response lengths, difficulty, language, and regional representation.
- Compare each dimension and slice with a versioned customer target.
- Drill from a score into representative, missing, and suspicious examples.
- Recommend targeted remediation and preserve the evidence used for approval.
Non-functional requirements
- Analyze a 10,000-item dataset within 30 minutes.
- Keep cached dashboard filtering and drill-down interactions below 500 ms p95.
- Isolate raw examples, embeddings, targets, and reports by tenant and project.
- Mark incomplete or uncertain analysis as
insufficient evidence, never as a passing score. - Reproduce any report from its dataset, taxonomy, metric, and model versions.
Ask questions that change the architecture
- What is the intended model use? A tutoring assistant and a legal support assistant need different domains, language styles, and risk slices.
- Which dimensions are contractual? Some customers require hard floors for language or regional coverage; others treat those dimensions as advisory.
- What metadata is trustworthy? Demographic or regional attributes may be absent, inferred, sensitive, or legally restricted.
- What costs more? A false pass can ship a brittle dataset, while a false failure creates avoidable collection and review work.
- How should examples be exposed? Customer reviewers need evidence, but access must obey consent, retention, and least-privilege rules.
State the initial scope
- Include English prompt-response datasets with supplied project metadata.
- Analyze text only; defer image, audio, and cross-modal coverage.
- Produce decision support, not an automatic claim of fairness or legal compliance.
- Recompute metrics when the dataset, target contract, taxonomy, or analysis model changes.
Computing diversity metrics for the dataset dashboard
A dashboard reports diversity metrics (class balance, demographic coverage, duplication rate) over very large training datasets. How should the metrics be computed to back the dashboard?
Constraint: Metrics span huge datasets but are viewed only occasionally; freshness-to-the-minute is not required, and cost matters.