Smart Text Completion

Define the product contract before choosing a model

Start with the interaction promise. It determines which requests are eligible, how long the service has, which context is justified, and what happens when the system is uncertain.

Write measurable non-goals

• Do not answer factual questions unless the product is explicitly designed and evaluated for that job.
• Do not auto-send, auto-save, or execute actions from a completion.
• Do not make document editing depend on model, ranking, policy, or telemetry availability.
• Do not use context merely because it exists; every field needs purpose, scope, retention, and a fallback.
• Do not optimize suggestion coverage as if more visible text were always better.

Make request eligibility cheap and local

The client should eliminate requests that cannot become useful suggestions. Local qualification protects privacy, saves capacity, and prevents a busy model service from becoming part of every keystroke.

Common eligibility checks

1. Confirm the feature and any personalization mode are enabled for this account and surface.
2. Require enough non-whitespace prefix to infer a continuation without guessing blindly.
3. Exclude password, payment, secret, and other configured sensitive fields before network work.
4. Pause during input-method composition, multi-cursor edits, selection replacement, or rapid cursor movement.
5. Debounce recent typing and cancel work associated with superseded draft versions.
6. Apply client backoff when the service reports overload or repeated deadline misses.

Eligibility is a product policy, not a hard-coded model rule. Measure the percentage of editor states rejected by each reason so a client regression cannot silently multiply serving demand.

Build a bounded context and model path

More context can improve relevance, but it also increases encoding time, privacy exposure, and the chance that old or unrelated text dominates the current prefix. Treat context construction as a versioned API contract.

Choose a serving path by product need

• Cached or rules path: useful for stable, non-sensitive phrases with strict scope and invalidation rules.
• Compact model: maximizes responsiveness and capacity for common short continuations.
• Balanced model: spends more time when richer context materially improves ranking.
• Fallback path: returns no suggestion, or an already-approved simpler candidate, when the primary deadline cannot be met.

Do not route every request to the largest model. Measure quality lift against complete-path latency, accelerator memory, queue growth, and the cost of a visible mistake.

Budget the complete interaction path

The 2019 Gmail Smart Compose paper reports a published p90 end-to-end requirement below 60 ms for that system. That number is historical product evidence, not a universal target or a claim about Gmail's current private implementation.

Set a budget from your editor, device, network, language, and user research. Then allocate it across every stage.

Protect tail latency and freshness

• Propagate an absolute deadline from the first server hop.
• Reject work that cannot finish before its result becomes useful.
• Bound dynamic-batch wait instead of hiding queue time inside model latency.
• Cancel superseded requests, but still make every response safe for the client to discard.
• Record queue age, context time, inference time, gate time, and client-visible render time separately.
• Degrade by reducing request frequency, context, candidates, or model tier before queues become unbounded.

Ranking must earn the right to interrupt

Generation answers, "What continuations are plausible?" Ranking answers, "Which candidate, if any, is worth showing here?" Keep those responsibilities separate so policy and product thresholds can change without retraining the generator.

Combine evidence, then apply hard gates

• Relevance: does the continuation follow the active prefix and nearby intent?
• Style fit: does it match language, register, formatting, and account-scoped preferences?
• Utility: does it save meaningful work instead of echoing one or two obvious characters?
• Repetition: does it duplicate nearby text or loop a phrase?
• Safety: is it allowed for this surface, audience, and context class?
• Freshness: does it still belong to the current draft and cursor version?

A weighted ranking score can order candidates, but it must not average away a failed hard gate. A high-relevance candidate that violates policy or belongs to an old draft is still suppressed.

The reject option is a first-class model behavior. SelectiveNet formalizes selective prediction with an integrated reject option; completion products apply the same broad idea by optimizing useful coverage under an acceptable error and interruption budget.

Evaluate the user loop, not just the language model

Offline continuation scores are useful for iteration, but they cannot prove that ghost text helps people write. Build an evaluation portfolio that connects model behavior to the product contract.

Offline evidence

• exact or prefix-aware continuation quality on time-separated data;
• ranking quality and calibrated coverage at each release threshold;
• repetition, language switch, formatting, sensitive-content, and unsafe-completion tests;
• slices by language, locale, device class, document type, and context mode;
• latency and memory benchmarks on the exact model, tokenizer, runtime, and hardware bundle.

Online evidence

• suggestion coverage among eligible editor states;
• accepted characters and retained text after later edits;
• dismiss, overwrite, undo, disable, and negative-feedback behavior;
• p50, p90, and p99 client-visible latency plus stale-response discard rate;
• policy suppression, privacy-control, incident, and complaint rates by governed slice.

Do not optimize acceptance rate alone. A system can inflate acceptance by showing only trivial punctuation, making suggestions too aggressive, or steering users toward text they later delete.

Roll out with reversible evidence gates

A model, tokenizer, context schema, ranker, and release policy can fail independently. Version and deploy them as an observable bundle, but preserve the ability to pause or roll back each layer.

1. 1

   Replay

   Qualify offline

   Pass frozen regression sets, safety suites, slice metrics, latency envelopes, and data-lineage review before exposing users.

2. 2

   No render

   Observe in shadow

   Run the candidate path without showing suggestions. Compare latency, suppression, and model decisions against the current production bundle.

3. 3

   Limited

   Canary by cohort

   Assign stable users, languages, and regions. Keep exposure small enough to stop quickly and large enough to reveal the target slice.

4. 4

   Decision

   Expand or reverse

   Advance only when usefulness clears its evidence floor and every latency, safety, privacy, and reliability guardrail remains healthy.

Every rollout needs explicit stop conditions

• a deadline or stale-response regression;
• a safety, privacy, or cross-account isolation failure;
• a harmful language, locale, accessibility, or device slice;
• a complaint or disable-rate increase beyond the approved budget;
• queue growth or regional headroom below the failover requirement;
• missing telemetry that prevents an accountable decision.

Operate completion as an optional dependency

The editor is the primary product. Completion infrastructure must fail without trapping the writer, corrupting a draft, or creating a retry storm.

Preserve decision evidence without retaining unnecessary text

• Correlate eligibility, request, candidate, render, accept, and discard events with opaque identifiers.
• Log model, tokenizer, context schema, ranker, policy, client, and experiment versions.
• Store numeric scores and reason codes where possible instead of raw document content.
• Set retention and access rules for any sampled text used in debugging or evaluation.
• Monitor per-account and per-region limits, queue age, saturation, and failed-capacity headroom.
• Exercise kill switches, rollback, region isolation, and telemetry-loss procedures before production incidents.

Primary references

• Gmail Smart Compose: Real-Time Assisted Writing documents the published 2019 model-selection, evaluation, personalization, latency, and serving decisions behind an interactive writing system.
• Use Smart Compose in Gmail documents current user-facing controls, explicit acceptance, generic and personalized modes, and the warning that suggestions may not be factually correct.
• SelectiveNet: A Deep Neural Network with an Integrated Reject Option provides a primary research treatment of prediction with abstention.
• NIST AI 600-1 provides current lifecycle guidance for governing, measuring, monitoring, and managing generative-AI risk.

Published system details are attributed to their source and date. Product-help and risk-management guidance can change; the architecture patterns elsewhere in this lesson are design recommendations, not claims about any vendor's current private implementation.