Design Gmail Smart Compose
Design Gmail Smart Compose: real-time text completion, context understanding, and low-latency ML serving.
What is a Smart Compose system?
A Smart Compose system predicts a short continuation while someone writes. The product succeeds only when a suggestion is useful and arrives before the writer has moved on. Unlike a chatbot response, a completion is optional: returning nothing is better than showing text that is late, unsafe, private, or irrelevant.
For this interview, design a web-first service for an assumed 150 million concurrent compose sessions. Treat that number as a capacity scenario, not a claim about Gmail production traffic. The target is an inline suggestion within 100 ms at p95 while preserving the rule that only the writer can insert or send text.
Product invariant
Predict, never act
The model may propose text. It must never send, commit, or silently mutate a draft.
Critical path
Spend one typing budget
Network, context preparation, inference, safety, and rendering all consume the same 100 ms.
Degraded mode
No suggestion is healthy
Suppress a completion when confidence, policy, capacity, or latency misses its gate.
Frame the user action
The client sends a debounced request after a useful typing boundary, such as a pause or completed token. The service returns zero or one short continuation. The UI renders it as uncommitted ghost text; acceptance requires an explicit key press.
Functional requirements
- Generate a context-aware continuation of roughly one to five words.
- Use the current draft, subject, and a bounded amount of recent thread context.
- Adapt to language and writing style without exposing one user's content to another.
- Let the writer accept, ignore, or dismiss the suggestion without interrupting typing.
Non-functional requirements
- Keep end-to-end p95 latency under 100 ms for the default path.
- Remain available across regions, with a no-suggestion fallback during overload.
- Minimize content in transit and retention; encrypt it and isolate enterprise tenants.
- Measure quality and safety by language, account type, context type, and model version.
State exclusions before architecture
- The system does not compose a whole email, search private mailboxes, or take actions.
- It does not need RAG for ordinary phrase completion; the bounded draft and thread are the primary context.
- It does not train directly on raw request payloads by default.
- It does not promise a suggestion after every keystroke.
Clarifying questions that change the design
- Which clients and regions are in scope? Web first permits a consistent debounce policy; mobile later changes network and device constraints.
- Which context is legally permitted? Thread text, recipient metadata, and personalization signals need separate retention and access decisions.
- What is the product objective? Prefer accepted characters and reduced composition effort over raw suggestion impressions.
- Which mistakes are intolerable? PII leakage, cross-tenant context, harmful text, and text inserted without consent are hard failures.
Where to run Smart Compose inference
Smart Compose predicts the rest of a sentence as the user types, suggesting completions inline. Where should the prediction model run?
Constraint: Suggestions must keep up with keystrokes (single-digit-to-low-tens-of-ms latency) AND the email content being typed must stay private — sending raw draft text to servers is a privacy concern.