Gmail Smart Compose Architecture

Treat the feature as a speculative side path

The compose client owns the draft. The suggestion service may propose text, but it cannot block a keystroke, mutate the message, or assume that a response is still current.

Three identifiers prevent response races

• Compose session ID: binds work to one open editor rather than only an account.
• Draft version: changes when the prefix, cursor, recipient set, or relevant context changes.
• Request ID: correlates one attempt across the client, gateway, inference service, and telemetry.

The client accepts a response only when all three still match. Cancellation saves capacity when possible; version checking preserves correctness even when cancellation arrives too late.

Start from the observed product constraint

The 2019 Gmail Smart Compose paper reports a p90 end-to-end latency requirement below 60 ms. It describes a streaming RPC design in which new keystrokes produce new requests, prefix encoding and beam-search work generate candidates, and host logic handles application work around accelerator inference.

Those details document the published system at that time; they are not a claim that Gmail's current private implementation is unchanged. The durable lesson is to budget the complete interactive path and make abstention cheap.

Budget more than model inference

End-to-end latency includes:

1. client debounce, serialization, and request correlation;
2. network round trip to the selected serving region;
3. authentication, consent, language, and context preparation;
4. queue or dynamic-batch wait;
5. prefix encoding and candidate decoding;
6. ranking, policy, and response serialization;
7. the final client freshness check and render.

Optimizing only step 5 can leave the product slow because network distance, long context, or batching already consumed the deadline.

Estimate demand from eligible typing events

Do not multiply total users by every keystroke. Define when the client is eligible to request a suggestion, then measure the resulting event rate.

Worked design envelope

Assume a hypothetical busy period with 12 million active compose sessions. If client-side eligibility and debouncing produce an average of two requests per minute per active session:

12,000,000 sessions x 2 eligible requests / 60 seconds = 400,000 requests/second

This is an interview design assumption, not a published Gmail traffic figure. Test the architecture at normal load, regional skew, and burst load rather than hiding all three inside one global average.

Capacity questions to answer explicitly

• How many regions can serve a user's data and language under policy?
• What percentage of traffic can one region absorb after another region is removed?
• How much queue time is allowed before the batch is dispatched partially filled?
• Which model tier and context size remain viable during degraded capacity?
• Can the client reduce request frequency before the service starts rejecting work?

Separate the synchronous path from the learning loop

Keep only the work required to produce and release the current suggestion on the keystroke path. Training, aggregate analytics, evaluation, and model rollout use separate asynchronous systems.

1. 1

   Client

   Qualify locally

   Check feature consent, minimum prefix, composition state, recent request timing, and whether the user is still actively editing.

2. 2

   Gateway

   Build bounded context

   Authenticate the account, choose the region, detect the cursor language, and produce a versioned context envelope.

3. 3

   Serving

   Generate candidates

   Use deadline-aware batching, a resident model, bounded decoding, and overload admission control.

4. 4

   Release

   Gate and correlate

   Rank candidates, apply deterministic policy, return request metadata, and let the client reject stale work.

Asynchronous systems own slower feedback

• Training pipeline: prepares governed examples, trains candidate models, and records dataset and model lineage.
• Offline evaluation: measures exact continuation quality, safety, fairness, language slices, and regression sets.
• Online experimentation: assigns stable cohorts, limits blast radius, and compares product outcomes against a control.
• Aggregate telemetry: joins request, render, accept, dismiss, and suppression events without placing analytics on the typing path.
• Rollout control: promotes, pauses, or rolls back model and policy versions independently.

Context is a product contract, not a data grab

The model needs enough context to continue the current thought, but more context increases compute, privacy exposure, and the chance of learning the wrong signal. Define a typed context envelope and justify every field.

The published paper compares multiple ways to combine the current body, subject, and previous message context, and documents a personalized model blended with a global model. Current Gmail Smart Compose help also distinguishes generic suggestions from account-level personalized suggestions that users can turn off.

A context envelope needs explicit fields

• value and token limit;
• purpose in generation or gating;
• account and tenant scope;
• residency and transport rules;
• retention duration and log treatment;
• consent or administrator control;
• fallback behavior when the field is unavailable.

Confidence is not permission to render

Candidate confidence estimates model preference under its own scoring method. It does not prove that the candidate is current, policy-safe, in the right language, or appropriate for this account.

Apply independent release gates

• Confidence: does the candidate clear the product threshold for this language and length?
• Freshness: did it arrive within the client-visible deadline?
• Prefix identity: does the response match the current draft version and cursor state?
• Safety: does deterministic policy allow the content and context class?
• Language: does the continuation follow the language and script at the cursor?
• Repetition and quality: does it avoid duplicated text, malformed spacing, or a trivial echo of the prefix?

A candidate renders only when every required gate passes. The system should record the first suppression reason and enough secondary reasons for aggregate diagnosis without logging unnecessary message content.

Design overload as graceful abstention

The suggestion service is not the email editor. When dependencies fail, the user must still be able to write and send a message normally.

Protect the dependency graph

• Enforce a request deadline at the first server hop and propagate the remaining budget.
• Bound per-account, per-region, and global concurrency before accelerator queues grow without limit.
• Cancel superseded requests, but still make every response safe to discard by version.
• Warm model replicas before adding them to load balancing.
• Keep deployment, model, tokenizer, feature, and policy versions in every trace.
• Make the client back off during overload instead of retrying each abandoned prefix.

Measure usefulness, interruption, and safety together

No single metric proves that the feature is healthy. Latency and availability can improve while suggestions become less useful or more intrusive.

Service and capacity signals

• request rate after client eligibility, split by region and language;
• queue age, batch fullness, inference time, and deadline-abandon rate;
• model load failures, fallback rate, saturation, and regional headroom;
• stale-response discard rate at the client.

Product and quality signals

• render, accept, partial-accept, dismiss, and overwrite behavior;
• accepted characters or keystrokes saved, not only suggestion click-through;
• repetition, formatting breakage, language mismatch, and factual-risk evaluation;
• offline continuation quality correlated with controlled online outcomes.

Privacy, safety, and fairness signals

• policy suppression by reason without raw sensitive content in labels;
• consent state, personalization mode, region, language, and accessibility slices;
• cross-account isolation tests and canary records that must never appear elsewhere;
• deletion, retention, access-audit, and model-lineage evidence.

Primary references

• Gmail Smart Compose: Real-Time Assisted Writing documents the published 2019 model, latency target, production serving design, personalization, multilingual work, evaluation, fairness, and privacy constraints.
• Use Smart Compose in Gmail documents current user-facing behavior, acceptance, personalization controls, and the warning that suggestions are not guaranteed to be factually correct.
• Smart features and controls for Google Workspace documents current consent and personalization controls for Workspace content and smart features.

Treat product-help details as current documentation that may change. The architecture patterns in this lesson are design recommendations unless explicitly attributed to the published Smart Compose paper.