OpenAI ChatGPT Architecture
How OpenAI built and scaled ChatGPT: model serving, infrastructure, and handling millions of AI conversations.
What is the ChatGPT architecture?
ChatGPT is a conversational AI product: a user sends a message, the service assembles the allowed conversation context, selects a model and tools, generates a response, applies safety policy, and streams the result back. The language model is one expensive dependency inside a larger distributed system that also owns admission, routing, memory, tool authorization, feedback, storage, and operations.
In plain language, the architecture turns token generation into a responsive and controlled product. The core invariant is: an admitted request follows the required safety and authorization path, and every degradation is explicit to the user and to operators. A faster answer is not correct if it bypasses policy, repeats a tool side effect, leaks another conversation, or silently uses stale state.
This case study presents a measured reference design based on observable product requirements and general LLM-serving principles. It does not claim to describe OpenAI's private deployment. Review Model Serving first if batching, accelerators, and online inference are unfamiliar.
Define what one chat turn must promise
A chat turn begins when the service accepts a user message and ends with a completed, cancelled, refused, or failed response. That lifecycle needs explicit states because a stream can disconnect after some tokens arrive and a tool can finish after its caller times out.
Functional scope
- Accept a text message for an authenticated conversation and assign a unique request ID.
- Build a bounded model input from instructions, recent turns, a conversation summary, and authorized tool results.
- Select an eligible model tier and healthy serving region.
- Stream response tokens, support cancellation, and record a clear finish reason.
- Invoke allowlisted tools through typed arguments and return either a typed result or typed failure to the model.
- Preserve conversation history according to product retention and deletion rules.
Design targets for this exercise
- Responsiveness: keep p95 time to first token below two seconds for admitted, non-tool requests under the planned load.
- Availability: continue safe text generation through a single worker or regional serving failure when reserve capacity exists.
- Isolation: never mix one user's context, cache entries, tool credentials, or streamed output with another request.
- Side effects: execute a tool action at most once for one idempotency key, even if the model or client retries.
- Safety: never turn a required policy dependency failure into an unfiltered model request.
Deliberate exclusions
- Model training and dataset construction are outside the online serving path.
- Voice, image, and video pipelines are omitted so token serving remains the focus.
- Exact model internals, fleet sizes, vendors, prices, and private deployment details are not assumed.
- Perfect factual accuracy is not guaranteed by generation; tools and retrieval need their own freshness and provenance contracts.
Time to first token and tokens per second measure different parts of the experience. Admission, policy, routing, queueing, and prompt processing happen before the first token. Decode throughput determines how quickly the rest of the response arrives.