Production Transformer Architecture

Trace one request through the serving path

A request does not move directly from an API handler to a GPU. Each boundary protects a different resource and produces different evidence.

1. 1

   Control plane

   Validate and admit

   Resolve the model bundle, cap input and output tokens, authenticate the caller, and reserve queue and cache capacity.

2. 2

   Compute-heavy phase

   Prefill the prompt

   Run prompt tokens through every block, create the first-token logits, and write reusable key/value state.

3. 3

   Memory-sensitive phase

   Schedule decode

   Continuously combine active sequences, read their cache pages, run one token step, and retire completed work.

4. 4

   User boundary

   Stream and account

   Emit bounded output, record phase-level latency and resource use, release cache pages, and commit terminal status once.

Measure the phases separately

• Queue delay reveals admission pressure before model execution starts.
• Time to first token combines queueing, tokenization, scheduling, and prefill.
• Inter-token latency exposes decode cadence after streaming begins.
• End-to-end latency includes the generated length and therefore cannot replace phase metrics.
• Goodput counts requests or tokens that meet the declared service objective, not merely raw work completed.

Understand the block before optimizing the server

A decoder-only production model repeats the same logical block many times. A common pre-normalized form moves the representation through normalization, causal self-attention, a residual update, another normalization, a feed-forward network, and a second residual update.

Fused kernels may combine normalization, projections, activation functions, or residual additions to reduce memory traffic. Treat a fused implementation as an optimization of the same tested contract, not as permission to change masks, position handling, precision, or residual order silently.

Choose the attention shape with the checkpoint

The number of query heads and KV heads is part of the learned model architecture. A server cannot safely turn multi-head attention into grouped-query or multi-query attention with a configuration flag unless the checkpoint and runtime support that shape.

FlashAttention changes how exact attention is executed, using IO-aware tiling to avoid materializing the full score matrix in high-bandwidth memory. MHA, GQA, and MQA change how many learned KV projections exist. They solve different problems and can be used together when the kernel supports the checkpoint shape.

Separate prefill pressure from decode pressure

Prefill processes many prompt positions together. It benefits from parallel matrix operations but can create a large one-time attention and cache-write burst. Decode usually processes one new position per active sequence. It repeatedly reads model weights and prior KV state, making memory movement and batch composition central constraints.

Protect both phases

• cap prompt, output, and total-token budgets before admission;
• chunk large prefills so one prompt cannot monopolize the scheduler;
• isolate or prioritize latency-sensitive traffic when long prefills would delay decode;
• measure cache allocation, prefix reuse, and eviction by request class;
• compare throughput only at a fixed latency and quality objective.

Partition the bottleneck, not the diagram

Parallelism adds communication and coordination. Start with the smallest topology that fits the model and its live state, then add a partition only when measured memory, compute, or latency pressure justifies it.

For mixture-of-experts checkpoints, expert parallelism adds another partition over experts and an all-to-all routing path. Treat it as part of that checkpoint's architecture, not a general substitute for tensor or pipeline parallelism.

Schedule tokens instead of waiting for perfect batches

Static request batching waits for a group, executes requests together, and may leave completed slots idle until the longest generation finishes. Iteration-level or continuous batching can admit new sequences as others finish, improving reuse of a fixed decode loop.

1. 1

   Admission

   Bound the request

   Estimate prompt and output tokens, deadline, cache pages, priority, and compatible model pool.

2. 2

   Memory

   Allocate paged state

   Reserve noncontiguous cache blocks so sequence growth does not require one large contiguous allocation.

3. 3

   Scheduler

   Compose an iteration

   Select decode tokens and bounded prefill chunks without exceeding token, sequence, deadline, or memory limits.

4. 4

   Continuous batch

   Retire and refill

   Release completed sequence pages and admit compatible queued work at the next safe iteration boundary.

Prefix caching can avoid recomputing identical, authorized prefixes. It does not make arbitrary responses safe to reuse: cache identity must include the exact model bundle, tokenizer, prompt bytes, adapter, position policy, and any other input that changes execution.

Admit work from measured capacity

Admission control should reject, queue, truncate only under an explicit product rule, or route to another tier before the allocator fails. It needs current evidence, not only a configured maximum.

Preserve these invariants

• the request's declared maximum work fits the selected pool before execution starts;
• reserved KV pages cannot be double-allocated across active sequences;
• queue age is bounded by a deadline or a controlled rejection path;
• retries carry a stable request identity and cannot duplicate the committed response;
• degraded capacity reduces admission immediately instead of waiting for an out-of-memory failure.

Design reliability around serving groups

A distributed model replica is healthy only when every required shard and pipeline stage is compatible and reachable. Losing one device can invalidate the whole group even when other devices are idle.

Contain failures

• keep multiple independent serving groups when availability requires failover;
• pin one versioned bundle of checkpoint, tokenizer, adapters, kernels, and runtime settings to a group;
• drain admission before replacing a shard or changing the parallel topology;
• bound queue length and age per tenant or request class;
• reconcile timed-out generations by request identity before retrying expensive work;
• test allocator exhaustion, device loss, collective timeout, cache corruption, and slow consumers.

Degrade deliberately

• route eligible requests to a smaller or quantized model tier;
• reduce maximum context or output only through a visible product contract;
• shed low-priority work before interactive traffic misses its deadline;
• preserve safety, authorization, and output validation during fallback;
• fail closed when a required shard, policy, or version cannot be proven compatible.

Evaluate the complete release bundle

Kernel, quantization, sharding, batching, and cache changes can alter latency, capacity, or numerical behavior. Release evidence must cover both model quality and systems behavior on the exact production bundle.

Compare variants at fixed workloads, quality criteria, sampling settings, and service objectives. A higher token rate is not an improvement if it shifts failures into tail latency, invalid output, or rejected requests.

Operate with phase-level evidence

Request and scheduler signals

• arrival rate, queue depth, oldest request age, admission decisions, and rejection reasons;
• prompt and requested output token distributions by request class;
• active sequences, batched tokens, prefill chunks, and decode iteration size;
• cancellation delay, client backpressure, and abandoned streamed responses.

Model and memory signals

• time to first token, inter-token latency, completion latency, and goodput percentiles;
• model execution time separated into prefill and decode;
• KV-cache allocation, occupancy, fragmentation, prefix-cache hit rate, and eviction;
• device memory, memory bandwidth, compute utilization, collective latency, and stalled ranks.

Release and quality signals

• checkpoint, tokenizer, adapter, precision, kernel, runtime, and policy version on every trace;
• output-quality slices, safety outcomes, schema validity, and fallback rate;
• canary comparison against the current bundle under the same traffic shape;
• rollback readiness and the last proven compatible bundle for each serving pool.

Review the common failure modes

• Weights fit but traffic does not: model loading succeeds, then concurrent KV growth exhausts device memory.
• Prefill starves decode: a few long prompts consume the token budget and interactive streams stall.
• Sharding adds latency: extra devices reduce per-device memory while collectives lengthen every token step.
• Static batches strand slots: short generations finish, but their capacity cannot serve queued work until the batch ends.
• A partial replica looks healthy: process checks pass on surviving ranks even though the serving group cannot complete a collective.
• A retry duplicates work: the gateway times out after generation commits, then another worker repeats the request without reconciliation.
• A fast kernel changes behavior: a fused or quantized path passes throughput tests but violates precision, mask, or output-quality criteria.

Primary sources

• Attention Is All You Need defines the original Transformer block, scaled dot-product attention, masking, residual paths, and encoder-decoder architecture.
• FlashAttention and FlashAttention-2 describe exact IO-aware attention algorithms and improved work partitioning.
• Fast Transformer Decoding: One Write-Head is All You Need introduces multi-query attention for incremental decoding.
• GQA evaluates grouped-query attention between MHA and MQA.
• Megatron-LM presents intra-layer model parallelism for large Transformer models; the current Megatron Core parallelism guide documents tensor, pipeline, context, data, and expert parallel strategies.
• Orca describes iteration-level scheduling for generative model serving.
• PagedAttention and vLLM describe paged KV-cache management and its role in high-throughput serving.