Design GenAI System: Image Synthesis
Design an image synthesis system: advanced diffusion models, style control, and high-resolution generation pipeline.
What is an image synthesis system?
An image synthesis system creates or edits an image from a prompt, a reference image, a mask, or a structured control such as a pose. A diffusion model makes pixels, but the product is the system around it: it must validate the request, schedule scarce accelerator time, preserve intended edit boundaries, inspect the result, and record why a result was allowed to leave the service.
For this design exercise, build a commercial image studio that serves 100 million generations per month, offers 512 px previews and 1024 px final renders, and targets 25 seconds p95 for the standard interactive route. These numbers are planning assumptions for the interview, not claims about a specific vendor.
The release invariant is: every delivered image has passed input and output policy, retains the requested edit boundary, and has traceable model and policy provenance. A faster output that fails any part of that invariant is not a valid result.
Critical path
Treat rendering as a job
Durable jobs, quotas, and explicit deadlines prevent accelerator saturation from becoming a lost or duplicated generation.
Correctness
Preserve the user’s control
Text, reference, mask, seed, and model version must be attached to one immutable generation record so edits do not silently drift.
Failure mode
Degrade quality before safety
Under pressure, offer a preview, queue the final, or reduce optional refinement. Never skip safety, rights, or provenance gates.
Start with the user-visible contract
Before discussing model weights, ask which creation modes matter. They share infrastructure but have different correctness rules.
- Text-to-image: a user supplies a prompt, aspect ratio, style category, seed, and output count.
- Reference-guided generation: a user supplies an image plus a permitted transformation; the system must state whether it preserves composition, subject, or only broad style.
- Inpainting and outpainting: a mask defines what can change. Pixels outside a protected region are an edit-boundary constraint, not a preference.
- Controlled generation: pose, depth, edge, or layout guidance makes the desired structure explicit and changes the model-routing decision.
Define functional requirements
- Accept prompts, references, masks, and controls; return a durable job ID, status, result URLs, and generation metadata.
- Offer a fast preview and a higher-quality final render. Users can cancel before work starts and can create a refinement from a prior immutable version.
- Keep a short history of inputs, selected model version, seed, parameters, policy outcome, and output hashes so support and enterprise audit workflows can explain a result.
- Let a user report an output or request deletion. Deletion must remove the result and its accessible derivatives according to retention policy.
Make the boundaries explicit
Hard guardrails
- Block prohibited prompts before allocating expensive GPU work; inspect every output because models can produce unsafe material from an apparently acceptable prompt.
- Apply real-person, trademark, copyrighted-reference, and age-sensitive policies separately from broad visual-safety classification.
- Do not claim ownership clearance from a similarity score. Rights policy should distinguish allowed uploads, licensed asset libraries, and requests that require review.
- Attach provenance to every release: model and adapter version, generation time, policy version, input and output hashes, and any applicable watermark status.
Out of scope for the first version
- Real-time collaborative canvas editing, unbounded video generation, medical or legal advice imagery, and automatic public publishing.
- A guarantee that every request produces an image. A visible review, rejection, or retryable queue state is safer than an unexplained output.
If readers need more foundation on prompts and conditioning before this exercise, link them to the prompt engineering lesson and the text-to-image practice. This page focuses on the system-design decisions around a multi-mode image studio.
Sampling strategy for the interactive image generator
Your text-to-image diffusion product lets users iterate on prompts in real time, then export a final hero image. How should you run the denoising/sampling pipeline?
Constraint: GPU cost and interactive latency are binding during iteration; users only need final-grade quality on the one image they keep, not on every draft.