Design a Content Moderation System
Design an AI content moderation system: multi-modal AI, safety classification, and human-in-the-loop.
What is a content moderation system?
A content moderation system decides whether user-generated text, images, audio, and video may stay visible, needs limited distribution, requires human review, or must be removed. The model supplies evidence; a versioned policy makes the enforcement decision.
For this interview, design a global service for 10 million posts per day, bursts up to 100 million posts per day, 50+ languages, and a sub-second automated path. The core invariant is stronger than model accuracy: every enforcement action must be explainable, auditable, and reversible through an appeal.
Decision
Separate score from policy
Models estimate risk. A policy engine combines those scores with content type, region, user context, and enforcement history.
Operations
Spend humans on ambiguity
Clear cases take the automated path; uncertain, high-impact, and appealed decisions enter bounded specialist queues.
Reliability
Degrade visibility, not safety
When a classifier or review queue is unhealthy, quarantine risky content and preserve evidence instead of silently allowing it.
Define what the system decides
The first version supports posts, comments, images, and short videos. It returns one of four product actions:
- Allow: publish normally because risk is below the allow threshold.
- Limit: keep the content available but reduce recommendations or age-gate it.
- Review: quarantine or restrict the content until a trained reviewer decides.
- Remove: block visibility and trigger any required user, safety, or legal workflow.
The system also accepts user reports and appeals. It does not train a foundation model from scratch, moderate encrypted private messages, or make law-enforcement decisions.
State goals that can conflict
<1s
Automated p95
Ingest through policy decision for ordinary posts.
<0.1%
False-positive target
Legitimate content incorrectly removed or restricted.
<4h
Review target
Average age for ordinary escalations; urgent threats are faster.
100%
Decision trace
Policy, model, evidence, action, and appeal state are versioned.
These goals cannot be compressed into a single 99.5% accuracy number. A rare but severe policy category can have high aggregate accuracy while still missing most violations. Measure precision, recall, latency, and impact separately for each category, language, region, and content modality.
Ask the questions that change the design
- Surface: Is the content public, recommended, monetized, live, or private?
- Policy: Which harms require immediate removal, limited distribution, or specialist review?
- Context: Which decisions depend on conversation history, quoted speech, satire, age, or region?
- Impact: Can an automated decision suspend an account or only restrict one item?
- Recovery: What evidence must an appeal reviewer see, and how quickly must a mistaken action be reversed?
- Privacy and welfare: How long may sensitive evidence be retained, who may view it, and how is repeated reviewer exposure limited?
Design principle:
Use stricter evidence and human oversight as the consequence becomes more severe. An automated ranking reduction and a permanent account ban should not share the same decision threshold.
Tuning the upload moderation classifier
A content classifier scores every user upload before it reaches other users. How should you configure the moderation pipeline?
Constraint: Minimize harmful content reaching users while not over-blocking legitimate uploads — both failure modes are costly.