Gmail Smart Compose GenAI System
Gmail Smart Compose deep dive: transformer architecture, real-time inference, privacy-preserving ML, and GenAI system design patterns.
What is Gmail Smart Compose?
Gmail Smart Compose is an inline writing assistant that predicts a short continuation while someone types an email. At an eligible typing boundary, it turns a small amount of permitted context into candidate tokens, rejects late or unsafe candidates, and displays a suggestion that the writer can accept or ignore.
In plain language: the system tries to finish the next few words, not write or send the email by itself. A useful suggestion must arrive before it interrupts typing, fit the current message, respect private context, and remain completely optional.
The core invariant is simple: a suggestion may disappear, but latency, privacy, safety, and user control may not be bypassed to make one appear.
Read the case through four design targets
The figures below are teaching assumptions for one peak operating scenario, not claims about Gmail's current production fleet. They turn a vague "real-time AI" requirement into capacity and release decisions that can be checked.
100 ms
Modeled p95 deadline
Late output is suppressed
80K/s
Eligible peak requests
After a client-side trigger gate
3-8
Output tokens
A short continuation, not an essay
0
Required suggestions
No suggestion is a valid result
One decision, two systems
The online serving system owns the current keystroke, deadline, candidate, and display decision. The offline learning system owns datasets, evaluation, training, rollout, aggregate feedback, and rollback. Keeping those paths separate prevents training or analytics work from consuming the typing-time budget.
Define completion before choosing a model
Smart Compose is successful when it offers a relevant continuation without taking control away from the writer. The system should optimize helpful opportunities, not maximize how often text appears.
Functional behavior
- Detect a stable, eligible typing boundary before requesting inference.
- Generate a short continuation from the permitted draft context and locale.
- Let the writer accept, keep typing, or ignore the suggestion without blocking input.
- Discard a response when the draft has changed since its request was issued.
- Emit a versioned outcome event only after the display decision is known.
Non-functional guardrails
- Responsiveness: meet a 100 ms modeled p95 end-to-end deadline, including network, context preparation, inference, filtering, and rendering.
- Safety: allow the output gate to abstain when confidence, policy, or sensitive-context checks fail.
- Privacy: send only the context fields approved for this purpose and retain them only as long as the documented policy allows.
- Reliability: degrade to a faster model, a deterministic phrase, or no suggestion without affecting normal email composition.
- User control: never auto-send, silently replace typed text, or treat an ignored suggestion as consent.
Scope boundaries
This design covers inline body-text suggestions during active composition. It does not cover autonomous message generation, inbox ranking, spam classification, or automatic sending. Those products have different risk, context, and evaluation contracts.
Success is a balanced scorecard
Track eligible-request rate, display coverage, acceptance when shown, edit-after-accept behavior, p50/p95 latency, abstention reasons, unsafe-output rate, and slice-level regressions. A higher acceptance rate cannot compensate for a safety or privacy failure.