GenAI System Design Process

Write the design contract before drawing boxes

A GenAI design contract is a concise, versioned statement of what the system promises and what it refuses to do. It prevents the team from choosing a model based on a demo and inventing requirements afterward.

A useful contract records

1. The user population, intent distribution, languages, modalities, and accessibility needs.
2. The happy-path output plus refusal, clarification, timeout, insufficient-evidence, and denied-action outcomes.
3. The consequence of a wrong answer or action, including who reviews or owns the decision.
4. Data classification, retention, residency, permissions, deletion, and audit requirements.
5. Representative evaluation cases with expected evidence or behavior, including critical slices and adversarial cases.
6. Token, latency, throughput, money, and review-capacity budgets with units and denominators.

Use the same four-step loop for every candidate

1. 1

   Contract

   Frame the decision

   Freeze the outcome, evaluation slice, constraints, and failure cost. Do not compare models against different prompts, evidence, or success criteria.

2. 2

   Control

   Build a simple baseline

   Start with the shortest viable prompt, no unnecessary tools, and a deterministic application shell. The baseline exposes which capability is actually missing.

3. 3

   Candidate

   Add one mechanism

   Try one change at a time: a different tier, retrieval, structured output, a bounded tool, prompt caching, or a human review route.

4. 4

   Gate

   Compare on the same evidence

   Measure task success, critical slices, harmful failures, tokens, latency, and cost. Keep the mechanism only when the gain justifies its new failure surface.

Select by workload, not provider labels

• Compact tier: bounded extraction, classification, rewriting, or routing after it passes the task-specific evaluation.
• Balanced tier: broader instruction following, grounded generation, multimodal input, or moderate tool selection.
• Deliberate tier: difficult, high-value cases where additional model work produces a measured quality gain worth its latency and cost.
• Specialized route: realtime speech, image generation, embeddings, reranking, or another modality-specific model with its own contract.

Model tiering is an application policy. Route a request by observable task shape and risk; record the selected route and reason; and preserve a known-good fallback. Never route solely from a model's self-reported confidence.

Design the governed request path around the model

The model is one dependency inside a larger control system. Identity, evidence, tool authorization, validation, and release policy remain deterministic application responsibilities.

Keep these responsibilities outside the prompt

• Authentication, authorization, tenant isolation, secrets, rate limits, and tool permissions.
• Schema validation, numerical checks, idempotency keys, transaction boundaries, and side-effect approval.
• Source access, retention, deletion, audit logs, and evidence lineage.
• Request deadlines, retry budgets, circuit breakers, concurrency limits, and cost caps.
• Release gates, rollback, incident response, and ownership of appeals.

Budget one request before estimating the fleet

Token and latency budgets force architectural trade-offs into the open. Use measured distributions from your own prompts and providers; do not multiply a context-window maximum by expected traffic.

For a text request with separate input and output rates:

• cost/request = (input_tokens × input_rate + output_tokens × output_rate) / 1,000,000
• requests/month = requests/minute × 60 × 24 × active_days
• monthly_cost = cost/request × requests/month
• planning_latency = fixed_stages + first_token + output_tokens / output_tokens_per_second
• concurrency_floor ≈ requests/second × request_seconds

The latency equation is an estimate, not an SLO proof. Queueing, provider variance, retries, tool calls, network paths, safety checks, and long-tail outputs must come from traces and load tests.

Evaluate the complete system, not only the model response

A useful evaluation set mirrors the production decision. Each case should preserve the input, authorized evidence, expected behavior, critical defects, and slice labels. Run it against the complete path so retrieval, tools, policy, and fallbacks are evaluated with the model.

Measure four evidence families

1. Task evidence: successful completion, correctness, required-fact coverage, grounded citations, structured-output validity, and human preference where appropriate.
2. Safety evidence: policy violations, prompt injection, sensitive-data disclosure, unauthorized tool attempts, harmful refusals, and unsafe compliance.
3. Operational evidence: p50/p95/p99 stage latency, tokens, cost per successful task, timeout and retry rate, queue depth, fallback rate, and provider errors.
4. Slice evidence: intent, language, modality, customer tier, document type, risk level, model route, and any population for which one aggregate can hide failure.

Do not convert every criterion into one weighted score. A healthy average must not cancel a failed authorization test, critical safety slice, or latency ceiling.

Release through evidence, rollback, and ownership

1. 1

   Evaluate

   Replay offline

   Compare the candidate with the current baseline on frozen representative cases, critical slices, adversarial inputs, and dependency failures.

2. 2

   Observe

   Shadow production

   Run the candidate without user-visible effects. Compare route decisions, outputs, latency, tokens, and policy verdicts on real traffic.

3. 3

   Release

   Canary with limits

   Expose a small eligible cohort, cap spend and side effects, keep a kill switch, and predeclare who may expand or stop the release.

4. 4

   Operate

   Re-evaluate changes

   Treat model, prompt, retrieval, tool, policy, parser, and traffic-distribution changes as versioned releases that can invalidate prior evidence.

Production traces must answer

• Which model, prompt, route, retrieved sources, tools, policies, and configuration versions handled the request?
• How many input, cached, reasoning where reported, and output tokens did each stage consume?
• Where did time accrue, which deadline ended the request, and whether a retry or fallback ran?
• Which evidence, safety, authorization, and approval gates passed, failed, or produced no verdict?
• What did the user see, what side effect occurred, and how can an operator replay or reverse it?

Monitor quality and safety with sampled review, adjudicated feedback, regression suites, and incident analysis. User thumbs-up data alone is neither a correctness label nor a safety evaluation.

Use current primary guidance as evidence, not architecture by copy

• OpenAI model guidance recommends comparing task success, required evidence, tokens, latency, and cost on representative workloads instead of assuming more model work is always better.
• Google Cloud's generative-AI lifecycle places evaluation around model selection, prompting, tuning, optimization, deployment, and monitoring rather than treating it as a final test.
• NIST AI 600-1 describes GenAI risk management across design, development, use, and evaluation.

Provider model names, prices, limits, caching rules, safety behavior, and availability change. Pin the versions you test, retain the evidence, and re-check official documentation before changing production routing or budgets.