Design Image Captioning System
Design an image captioning system: vision-language models, multimodal architectures, and caption generation.
What is an image captioning system?
An image captioning system turns visual evidence into a short natural-language description. It combines a vision encoder, which recognizes objects and relationships, with a language model, which expresses that evidence as a caption. The result supports screen readers, image search, cataloging, and content workflows.
The central invariant is simple: describe what the image supports; do not turn a plausible guess into a fact. A fluent caption can still be wrong, biased, or harmful. For this interview, design a service for 100 million images per day, a 5,000 request/s peak, an interactive target under 500 ms at p95, and batch processing for work that does not need an immediate answer.
Correctness
Ground every claim
Objects, text, actions, and relationships need visible or verified evidence. Omit uncertain details instead of inventing them.
Product
Route by consequence
Accessibility, search metadata, social copy, and regulated imagery require different latency, model, and review policies.
Degraded mode
A partial caption is valid
When OCR, grounding, or capacity is weak, return a conservative description, queue review, or return no caption.
Separate the user journeys
The word "caption" hides several products. Ask which one is being built before choosing a model or metric.
- Accessibility alt text
- Describe the information a screen-reader user needs to understand the image in context.
- Prefer factual completeness and predictable wording over creativity.
- Search and catalog metadata
- Extract objects, attributes, and text that improve retrieval.
- Structured tags may matter more than a polished sentence.
- Social assistance
- Suggest editable copy, but never publish it without an explicit user action.
- Brand voice is a soft preference; safety and grounding remain hard gates.
- Regulated imagery
- Route medical, legal, biometric, or safety-critical images to specialized policy and human review.
- Do not present a general captioner as a diagnostic system.
State the service promises
Functional requirements
- Accept common image formats and an optional language, use-case, and style policy.
- Return a caption, grounded entities, confidence metadata, and a review status.
- Support at least 25 languages without translating unsupported claims into new certainty.
- Let authorized users correct captions and send those corrections into a governed feedback pipeline.
Non-functional requirements
- Meet 500 ms p95 for the interactive tier and process the remaining workload asynchronously.
- Keep raw images private, encrypted, region-aware, and short-lived by default.
- Measure quality by use case, language, image type, and user-impact slice.
- Degrade safely during model, OCR, moderation, or regional failure.
Define the first-version boundary
- No diagnosis, identity inference, emotion labeling, or protected-attribute guessing.
- No promise that every image receives a useful caption.
- No permanent image retention unless a separate product policy requires it.
- No open-ended creative generation on the accessibility path.
Captioning strategy for a huge, repetitive image corpus
You must generate alt-text captions for billions of images, where a large fraction are exact or near-duplicate copies of images you have already seen. Which captioning strategy do you build around?
Constraint: High throughput over a huge corpus with heavy image repetition; per-image cost dominates, and marginal caption-quality gains matter little.