OpenAI | System Designer

What is the OpenAI platform?

The OpenAI platform is a hosted API surface for sending inputs to AI models and receiving typed outputs, streamed events, embeddings, or requests to use tools. An application chooses a model, supplies instructions and context, validates the result, and decides whether any requested action is allowed.

In plain language: the model is one dependency inside your system. The platform can generate or classify content, reason over supplied evidence, and propose tool calls, but your application still owns identity, permissions, data boundaries, deadlines, retries, safety policy, and the final user-visible action.

The core production invariant is no model output crosses a trust boundary without an application-owned check. Text needs policy and quality checks, structured data needs schema and semantic validation, and tool calls need authorization plus idempotent execution.

Input

Bound the request

Instructions, user content, retrieved evidence, and tool definitions consume context

Output

Define the contract

Text, JSON, tool calls, and stream events need different validation

Action

Keep authority in code

A tool call is a proposal; trusted application code decides whether it runs

Evidence

Prove every release

Evals, traces, canaries, and rollback criteria turn model changes into controlled deployments

Start with the API boundary

An API boundary is the point where your application turns trusted state and untrusted user input into a model request, then turns a probabilistic response back into application behavior. OpenAI recommends the Responses API for new projects; it represents messages, tool calls, tool results, and other model actions as typed items.

Keep API keys on a server. A browser or mobile client should call your backend, where you can authenticate the user, apply tenant policy, redact sensitive fields, cap work, and record an audit event before contacting OpenAI.

A production Responses API request

The model is inside a controlled path. Validation, authorization, and fallback logic remain application responsibilities.

Intent

Client

Sends the user request and receives progress or a final result. It never receives the OpenAI API key.

Trust boundary

Application gateway

Authenticates the user, checks policy, assigns a request ID and deadline, and enforces request and token budgets.

Evidence

Context builder

Selects instructions, conversation state, retrieved sources, and only the tools allowed for this request.

Inference

Responses API

Runs the configured model and returns typed output items or streaming events.

Control

Validator and executor

Validates output, authorizes tool arguments against trusted state, executes idempotently, and chooses retry, fallback, or refusal.

Select a model from task evidence

Model selection is an evaluation problem, not a naming problem. Start with a capable model that meets the task's quality target, build a representative eval set, and then test whether a smaller or faster model keeps that quality while improving latency and cost. The current model selection guide describes this quality-first sequence.

Choose by workload shape:

Use a smaller general-purpose model for bounded classification, extraction, routing, and rewriting after it passes the task eval.
Use a stronger general-purpose model when instructions, ambiguity, or multimodal evidence require more capability.
Use a reasoning model when the task needs multi-step analysis and the extra latency and output budget fit the service objective.
Use embedding models for similarity and retrieval. Embeddings are not generated answers and should be evaluated with retrieval metrics.
Use specialized image, audio, or realtime models only when the input and interaction mode require them.

Keep the model ID in server configuration. Pin a model snapshot when behavioral stability matters, and require evals before changing it because model behavior can change between snapshots even when the API remains compatible.

Loading request envelope...

Separate output shape from tool authority

Structured Outputs constrain the shape of model-generated data; function tools let the model request application behavior. These are related contracts, but they answer different questions.

Return typed data

Structured response

Use a strict JSON Schema when the model should return data your code will render or process. Schema adherence removes parsing ambiguity, but your code must still validate business rules such as valid account ownership or acceptable totals.

Request an action

Function tool

Define a narrow function with a strict schema when the model may ask your application to read data or perform work. Treat the call as untrusted input until authorization succeeds.

Own the side effect

Executor

Derive tenant and resource identity from the authenticated session, enforce policy, use an idempotency key, record the result, and return a bounded tool output to the model.

The Structured Outputs guide distinguishes schema-constrained responses from function calling. The function calling guide documents the multi-step loop: send tools, receive a call, execute in application code, return the call output, and continue until a final result or a configured limit.

Tool rules that prevent authority leaks

Expose only tools needed for the current route and authenticated user.
Keep trusted identifiers out of model-generated arguments when code already knows them.
Set additionalProperties: false and use strict schemas where supported.
Validate ranges, state transitions, and resource ownership after schema validation.
Require confirmation for destructive or high-impact actions.
Make retried side effects idempotent using a stable operation key.
Cap tool-call count, wall-clock time, and total output returned to the model.

Build context as a bounded evidence set

Context is all input the model can use for one response: instructions, user content, conversation state, retrieved evidence, and tool definitions. A larger context is not automatically better. Irrelevant or unauthorized text adds cost, latency, and opportunities for conflicting instructions.

For retrieval-augmented requests, keep two evaluations separate:

Retrieval quality: Did the system find the eligible source chunks needed to answer the question?
Answer quality: Did the model use those chunks faithfully, cite them, and abstain when evidence was insufficient?

Apply structured eligibility filters before semantic ranking when region, tenant, product version, or effective date determines which documents are allowed. Record source identity, version, score, and the chunks actually sent to the model. The file search guide notes that retrieving fewer results can reduce token use and latency but can also reduce answer quality.

A practical context budget

Reserve space for system instructions and tool schemas.
Bound user input and conversation history before retrieval.
Retrieve enough candidates to protect recall, then rerank or filter before synthesis.
Set an explicit maximum output budget instead of spending every remaining token.
Prefer stable, repeated instructions at the beginning of requests so eligible workloads can benefit from prompt caching.

Stream for perception, not correctness

Streaming delivers response events as work completes so the user can see progress before the final response exists. It can improve perceived latency, but it does not reduce the time required to finish the same generation.

Use the typed event stream documented in the streaming guide. Accumulate only the event types your client understands, handle refusal and error events explicitly, and finalize persistence only after terminal completion.

Streaming changes the safety boundary:

Partial text can reach a user before whole-output checks run.
Client disconnects should cancel upstream work when possible.
A stalled stream needs an inactivity timeout as well as an overall deadline.
Tool calls and structured objects should be executed only after the complete item validates.
High-risk experiences may need buffered output or a post-generation review instead of immediate token display.

The official guide warns that partial completions are harder to moderate. Choose streaming only after the product has defined what may be shown before final validation.

Control latency, cost, and rate pressure together

A request budget is a set of limits on tokens, time, attempts, and downstream work. Optimizing one number in isolation can move pressure elsewhere: more retrieved chunks increase input cost, long outputs occupy capacity, and aggressive retries consume rate-limit headroom.

1
Route
Reject unnecessary work
Use deterministic code, a cache, or a search index when the task does not need generation. Route simple tasks to the least expensive model that passes the eval floor.
2
Budget
Bound each request
Limit input, retrieval count, output tokens, tool calls, per-tool time, and total wall-clock time.
3
Admit
Protect shared capacity
Apply per-tenant quotas and concurrency limits. Shed optional work before queues violate the user-facing deadline.
4
Recover
Retry selectively
Retry transient failures with exponential backoff and jitter, honor server guidance, cap attempts, and never retry a side effect without idempotency.
5
Contain
Degrade deliberately
Fall back to cached evidence, a smaller bounded response, a non-generative path, or a clear retry-later state instead of cascading failure.

OpenAI exposes request- and token-based limits that vary by model and usage tier. Read the live response headers and the current rate-limit guide; do not encode a documentation example as permanent capacity. Unsuccessful requests also consume limit budget, which is why unbounded immediate retries make overload worse.

Make safety a layered control system

Safety is the set of controls that keeps inputs, outputs, retrieved data, and actions within product policy and user authorization. No single prompt or classifier covers every failure mode.

Before inference: authenticate, classify the use case, enforce age and tenant policy, scan uploads, and remove secrets or unnecessary personal data.
During context assembly: isolate untrusted retrieved text from system instructions, filter by access policy, and expose only permitted tools.
After generation: moderate where appropriate, validate factual or policy-critical claims, scan generated code or URLs, and require human review in high-stakes workflows.
At action time: authorize against trusted application state, apply amount and frequency limits, request confirmation, and retain an audit record.
After release: monitor policy slices, refusals, overrides, appeals, and incident reports; feed reviewed failures back into evals.

OpenAI's current safety best practices recommend moderation and human oversight, especially in high-stakes domains. Review the data controls documentation for the exact endpoint, storage, and retention behavior your organization uses rather than assuming every API mode has the same data lifecycle.

Observe the whole request, not prompt text alone

Observability links one user operation to its model requests, retrieval decisions, tool executions, retries, and final outcome. Logs should make a failure diagnosable without collecting more sensitive content than the incident process requires.

Record at least:

your trace ID, user-safe tenant identifier, route, release version, and model configuration;
OpenAI's server request ID and your unique X-Client-Request-Id when supplied;
input and output token usage, cache usage, latency stages, finish status, and retry count;
retrieval source IDs and versions, not only the final prompt;
tool name, authorization result, idempotency key, duration, and bounded result status;
schema validation, safety checks, fallback path, and user-visible outcome;
sampled content only under a documented access, redaction, and retention policy.

The API debugging reference recommends logging request IDs in production. Pair those identifiers with metrics for p50, p95, and p99 latency; error and rate-limit rates; token cost per successful task; eval scores by slice; and tool-side-effect failures.

Loading release controls...

Implement contracts that run without credentials

These examples default to offline behavior so their request shape, validation, and release logic can run in CI. Only the first example has an optional --live mode; it reads the API key and model ID from environment variables instead of embedding either in source.

Build a bounded Responses API payload with a strict structured-output schema. The default command validates and prints the request; --live sends it only when credentials and an explicit model are configured.

Build and optionally send a Responses API request

Process a simulated function call through argument validation, trusted-session authorization, and an idempotent executor. Running the same call twice demonstrates that a retry does not duplicate the side effect.

Authorize and deduplicate a tool call

Evaluate an external fixture across quality, safety, schema, latency, and cost gates. Add --inject-regression to prove that one high-risk failure blocks promotion.

Run an offline OpenAI integration release gate

Fixture: evaluation-cases.json

Migrate through measured compatibility stages

A migration changes an API, model, prompt, tool schema, retrieval policy, or safety control while preserving a tested user contract. Treat each change as a release candidate, not a transparent dependency update.

1
Baseline
Inventory the contract
Capture endpoints, model snapshots, prompts, schemas, tools, storage behavior, event types, retries, and production eval cases.
2
Isolate
Add an adapter
Translate provider items into application-owned request, result, usage, refusal, and error types so business logic does not depend on raw SDK objects.
3
Offline
Replay and compare
Run both paths over fixed and reviewed production-derived cases. Compare task, safety, tool, schema, latency, and cost results by slice.
4
Online
Shadow or canary
Start with no side effects, then a small isolated traffic slice. Keep tool execution disabled or idempotent until equivalence is proven.
5
Decision
Promote with rollback
Advance only while every gate passes. Keep the prior configuration warm and define who can stop the rollout.

For new work, use the Responses API rather than starting on the Assistants API. OpenAI's Assistants migration guide documents the current deprecation and sunset schedule. For any model change, consult the model catalog and deprecations page at release time instead of relying on a lesson's model list.

Define release gates before rollout

A release gate is a measurable condition that must pass before more users depend on a candidate. The threshold belongs to the product risk, not to a generic AI benchmark.

Required gates for a production OpenAI integration include:

Representative task evals meet the baseline overall and on critical slices.
Structured outputs validate, and semantic invariants reject impossible values.
Tool calls stay inside user authorization, confirmation, idempotency, and call-count limits.
Safety tests cover normal use, abuse, prompt injection, retrieval poisoning, and high-risk escalation paths.
Load tests meet end-to-end latency, rate-limit, error, token, and cost budgets under expected concurrency.
Traces connect client operations to provider request IDs, retrieval, tools, retries, and fallbacks.
A bounded canary uses an independent kill switch and a warm rollback configuration.
Migration tests cover refused, incomplete, timed-out, rate-limited, malformed, and newly added event types.

Use OpenAI's evals guide for the current platform workflow, but keep release authority in your deployment system. A dashboard is evidence; it is not the decision policy.

Primary OpenAI documentation

These interfaces and policies change. Check the primary documentation against the SDK and model snapshot you plan to release:

Responses API migration and Responses API reference
Model selection, model catalog, and deprecations
Structured Outputs and function calling
File search, retrieval, and prompt caching
Streaming, latency optimization, and rate limits
Safety best practices, moderation, and data controls
Evals, production best practices, and API request debugging

Model names, prices, limits, supported schema features, event types, storage behavior, and deprecation dates are intentionally linked rather than copied into permanent architecture assumptions.

No quiz questions available

Could not load questions file