Design a Conversational AI Platform
Design a conversational AI platform: LLM orchestration, multi-turn context, and safety guardrails.
What is a conversational AI platform?
A conversational AI platform receives a user's message, keeps the right context, and either answers or carries out a bounded task. It matters because a fluent response is not enough: the platform must preserve tenant boundaries, choose an affordable model, and refuse unsafe or unsupported actions.
In this practice problem, design a multi-tenant support and operations assistant for 1,000 enterprise customers. It supports text and voice, serves 100,000 concurrent conversations, and may look up account data or begin approved workflows.
Conversation
Keep an explicit state
Store only the user, tenant, task state, and approved memory that the next turn genuinely needs.
Safety
Treat authority as data
Bind every retrieval and tool call to the authenticated tenant, user scope, and action policy.
Reliability
Degrade deliberately
Reduce optional context or tools before serving a response that is slow, unsafe, or ungrounded.
Define the user-visible contract
The assistant handles support questions, account lookups, and low-risk workflow requests. It returns a sourced answer, asks for missing information, invokes a permitted tool, or escalates. It does not silently make high-impact decisions, expose another tenant's information, or treat a model's guess as an account fact.
Functional requirements
- Continue a chat across web, mobile, and voice while preserving only approved session context.
- Answer policy and product questions from tenant-authorized knowledge.
- Read account status through typed, permissioned APIs.
- Start reversible workflows only after the user confirms an action.
- Transfer a conversation to a human with its relevant, redacted summary.
Non-functional requirements
- Serve 100,000 concurrent conversations with a 700 ms text p95 target for routine support.
- Keep tenant isolation, data residency, retention, and audit requirements enforceable end to end.
- Meet a 90% grounded-answer rate, 85% task-completion rate, and 99.9% availability target.
- Support text first; voice and rich media use separate latency and safety budgets.
State the invariant first. Conversation state, retrieved evidence, and tool authority must remain bound to the authenticated user and tenant. Filtering after the model sees data is too late.
Ask questions that change the design
- Which workflows may the assistant execute, and which require a human approval?
- What is the maximum harm from an incorrect answer or duplicated action?
- Which system is authoritative for account state, product policy, and user permissions?
- How long may memory survive, and can a user inspect or delete it?
- What behavior is acceptable when retrieval, a tool, or the model tier is unavailable?
Grounding a support assistant in fast-changing knowledge
Your support assistant must answer questions using company-specific knowledge (policies, product details, pricing) that changes weekly. Replies must reflect the latest facts and be grounded in real sources. Which approach fits best?
Constraint: Knowledge changes frequently; answers must stay current and grounded in authoritative sources.