Design an AI Code Assistant
Design an AI code assistant like GitHub Copilot: code completion, context extraction, and latency optimization.
What is an AI code assistant?
An AI code assistant proposes code, explanations, tests, and edits inside a developer's existing workflow. It combines a code-aware language model with repository context, but the model is only one part of the system: the product must select relevant context, answer quickly, isolate tenants, filter secrets, and let the developer decide whether any suggestion becomes real code.
For this interview, design an assistant for 10 million daily developers and a peak of 50,000 requests per second. Optimize inline completion for 100 ms p95. Longer explanations and multi-file edits may use an asynchronous path, but generated code must never execute or merge automatically.
Product invariant
Suggest, never act
The assistant may propose a patch. A developer or an existing trusted workflow must approve every durable change.
Privacy invariant
Retrieve within one tenant
Repository context, indexes, caches, logs, and feedback remain scoped to the authenticated tenant and repository.
Degraded mode
Silence is a valid result
When context, policy, or latency is uncertain, return less context, a smaller model, or no suggestion.
Separate the product modes
The phrase "code assistant" hides workloads with different deadlines and risks:
- Inline completion returns a short continuation while the developer is typing.
- Explanation and chat answer questions about selected code and repository behavior.
- Refactoring and test generation produce a reviewable patch, possibly across several files.
- Debugging assistance interprets diagnostics but does not run commands without explicit permission.
Ask questions that change the architecture
- Which repositories are allowed? Public code, private enterprise code, and air-gapped code require different deployment and retention boundaries.
- Which tools may the assistant call? Read-only search is safer than a shell, package installer, or deployment API.
- What is the user-visible deadline? Inline completion needs a hard latency budget; a repository-wide edit can become a cancellable job.
- What is retained? Raw code should not enter logs or training by default. Feedback needs consent, minimization, and deletion behavior.
State the exclusions
- No automatic commits, merges, command execution, or production changes.
- No cross-tenant retrieval, even when two repositories contain similar code.
- No unbounded repository dump into the prompt.
- No claim that syntactically valid code is secure or correct.
Model choice for inline code completion
Your IDE plugin shows inline "ghost text" completions as the developer types. Each keystroke can trigger a new suggestion. Which serving strategy should power the completions?
Constraint: Completions must feel instant — under ~300ms end-to-end — or developers turn the feature off entirely. Perceived latency dominates raw suggestion quality.