Text-to-Image System Design

A generation job crosses five boundaries

Long-running image generation should leave the synchronous request path. The API validates intent, creates an idempotent job, and returns a job identifier. Workers then advance that durable record through explicit states.

Keep the state machine explicit

• accepted: the contract and idempotency key are durable.
• queued: a scheduler owns the request, priority, and deadline.
• running: one worker lease owns the current attempt.
• moderating: pixels exist but are not yet releasable.
• succeeded: an accepted asset and its evidence are durable.
• rejected, failed, or cancelled: the terminal reason is recorded and safe to query.

A client retry with the same idempotency key must return the existing job. It must not purchase another generation accidentally.

Write a generation contract before choosing a GPU

The contract separates user intent from mutable infrastructure choices. It also makes a result reproducible enough to investigate, even when hardware kernels or model implementations are not perfectly deterministic.

Reject invalid combinations at admission

• Dimensions must match the selected model and worker pool constraints.
• A reference image must have an authorized purpose, bounded size, and known retention rule.
• Requested adapters and model versions must be compatible and licensed for the product context.
• The maximum work estimate must fit the caller's quota and the product deadline.
• Release policy must be selected before work starts, not inferred after a surprising image appears.

The runnable contract below normalizes a request, validates bounded fields, and derives a stable idempotency fingerprint without storing the raw prompt in the key.

Estimate denoising work before admission

For one calibrated model and worker class, denoising work grows roughly with spatial area × denoising passes × image count. Architecture, precision, attention implementation, guidance method, and batching change the constant, so production planning must use measurements from the exact deployment recipe.

Keep three numbers separate

• Resolution changes the number of spatial positions processed at each denoising pass.
• Steps change how many sequential denoiser evaluations occur.
• Batch size can improve throughput through shared execution, but one batch may take longer and consume more memory.
• GPU count raises pool capacity; it does not make one image faster unless the model supports a measured parallel execution strategy.
• Arrival shape matters: a stable daily average can hide a burst that violates the interactive queue deadline.

The capacity model should be intentionally humble. Calibrate it from load tests, reserve headroom for retries and heterogeneous jobs, then compare its forecast with observed queue age.

Separate model mechanics from product guarantees

Latent diffusion reduces spatial compute by denoising a compressed representation and then decoding it to pixels. Cross-attention or related conditioning mechanisms connect text representations to the image model. The denoiser backbone may be convolutional, transformer-based, or hybrid; the system contract should not depend on one architecture name.

Do not promise a knob you have not evaluated

• More steps do not guarantee a better product result after the model's useful range.
• Stronger guidance can improve prompt adherence while reducing variation or introducing artifacts.
• A higher pixel count can increase latency and memory without improving composition.
• A fixed seed can support comparison, but changed kernels, precision, or model versions can change the output.
• A faster sampler is valuable only if task-slice quality and policy behavior stay inside their release bounds.

Latent Diffusion Models demonstrate text conditioning in a compressed latent space. Diffusion Transformers show that the denoiser backbone can instead operate as a transformer over latent patches.

Schedule around compatibility, fairness, and deadlines

A scheduler should build batches from jobs that can actually share a loaded worker. Mixing incompatible models, adapters, dimensions, or precision modes creates reload churn and unpredictable memory pressure.

1. 1

   Envelope

   Classify the job

   Derive model family, memory class, dimensions, quality tier, deadline, and policy route from the immutable contract.

2. 2

   Isolation

   Choose a compatible queue

   Separate interactive, batch, and review-sensitive work. Preserve tenant fairness and cap one caller's in-flight GPU seconds.

3. 3

   Execution

   Lease a warm worker

   Prefer a worker with the required recipe loaded. Give the attempt a deadline, heartbeat, cancellation token, and stable job ID.

4. 4

   Recovery

   Reconcile the attempt

   Commit output identity atomically with attempt state. Expired leases may be retried only after checking whether an accepted output already exists.

Autoscale from pressure, not CPU

• Oldest runnable job age by service tier.
• Estimated queued GPU seconds by compatibility class.
• Warm-worker availability and model-load duration.
• Memory rejection, out-of-memory, cancellation, and retry rates.
• Sustained utilization with enough spare capacity for one worker failure.

Scale-down policy matters too. Draining a worker must stop new leases, allow bounded in-flight work to finish, and preserve loaded-model economics without trapping stale capacity forever.

Evaluate the product on named task slices

One aggregate image-quality score cannot prove that the system follows prompts, renders text, respects a brand style, avoids unsafe content, and serves every language equally. Build an evaluation set from real product intents and expected failure modes.

Measure generation behavior

• Prompt-object and attribute alignment.
• Counting, spatial relationship, and compositional accuracy.
• Text rendering and typography where the product supports it.
• Anatomy, geometry, repetition, and other artifact classes.
• Diversity across seeds without losing requested constraints.

Measure product behavior

• Task completion or user acceptance by intent slice.
• Regeneration, edit, and abandonment rates.
• End-to-end latency from accepted job to releasable image.
• Policy blocks, review outcomes, appeals, and confirmed escapes.
• Cost per accepted image, not merely cost per GPU attempt.

Compare changes at a fixed contract

Version the prompt set, model recipe, policy, evaluator, and human-review rubric. A cheaper candidate should be compared at a matched quality and safety bar; a better-looking candidate should be compared at a matched latency and cost envelope.

Safety needs an input gate and an output gate

Prompt inspection can reject clearly prohibited requests before expensive work. It cannot determine everything the stochastic model will render. Output inspection sees the candidate but occurs after compute has been spent. Production policy needs both boundaries and an explicit review path.

The policy evaluator below keeps input, output, review, and provenance decisions separate so an audit can explain where a release stopped.

Provenance complements policy; it does not replace it

Content provenance can carry signed claims about an asset's origin and history. It does not prove that an image is true, harmless, unbiased, or legally authorized. Treat provenance, moderation, and product disclosure as separate controls.

The C2PA specifications define interoperable Content Credentials and related implementation guidance. Validate the manifest after storage, transformation, and CDN delivery; attaching one before a pipeline step that strips it creates false confidence.

Design failure behavior before launch

Test the recovery paths

• Duplicate client submission before and after job creation.
• Queue redelivery while the first worker lease is still alive.
• Worker termination before output, after output, and after durable storage.
• Cancellation while queued, while denoising, and during moderation.
• Detector timeout, stale policy configuration, and unavailable review capacity.
• Storage success with lost response, callback replay, and expired delivery URL.

Observe cost, quality, and policy together

Optimization can move harm between metrics. Larger batches may improve throughput while hurting interactive latency. Aggressive output thresholds may reduce escapes while overwhelming reviewers. A faster model may lower GPU cost while increasing regeneration.

Operational signals

• Accepted jobs, terminal outcomes, and state age by tenant and service tier.
• Queue age, estimated queued GPU seconds, warm-worker ratio, and worker utilization.
• Generation time by model recipe, resolution, steps, batch, and hardware class.
• Out-of-memory, timeout, cancellation, retry, and duplicate-output rates.
• Moderation block, review, appeal, and confirmed-escape rates by policy version.
• Storage, egress, and review cost per accepted image.

Evaluation signals

• Prompt alignment and artifact rate by task slice.
• User acceptance, regeneration, and edit rate by quality tier.
• Safety recall and false-positive burden on reviewed, policy-labeled samples.
• Performance and policy gaps across language, identity, geography, and accessibility slices relevant to the product.

The NIST Generative AI Profile emphasizes measuring and managing generative-AI risks across the lifecycle. Use that governance frame to assign owners and evidence, not as a substitute for product-specific tests.