Conversational AI
Build conversational AI systems: dialogue management, context handling, and creating engaging chatbot experiences.
What is conversational AI?
Conversational AI is a system that accepts natural-language turns, assembles bounded context, generates or executes an answer, and preserves only the state needed for useful continuity. The product is not just a chat box around a model. It is a stateful request protocol with identity, memory, retrieval, tools, streaming, safety, and terminal outcomes.
In plain language, the assistant should understand a follow-up without pretending it remembers information that was never stored, has expired, belongs to another user, or no longer fits in the context window.
The core invariant is every turn must be attributable to one authorized session, and every response or side effect must reflect the context and outcome that actually occurred. A missing memory, failed tool, cancelled stream, or blocked answer cannot be rewritten as success.
Start with LLM fundamentals if tokens, context windows, or model inference are unfamiliar. Add retrieval-augmented generation only when the assistant needs external evidence.
Follow one turn through the runtime
1 Gateway
Accept an authorized turn
Bind user, tenant, session, locale, quotas, request ID, and a cancellation signal before reading state.
2 Orchestrator
Assemble bounded context
Select recent turns, approved memories, permission-filtered evidence, tool schemas, instructions, and output budget.
3 Model
Generate or request a tool
Produce a candidate response or structured action proposal inside the remaining token and time envelope.
4 Policy
Validate the outcome
Check grounding, privacy, tool arguments, output policy, and whether claimed external actions actually completed.
5 Client
Stream an explicit terminal state
Deliver deltas followed by complete, cancelled, timed-out, blocked, or failed so partial text is never mistaken for a final answer.
The topology lab makes degraded paths visible. Change the scenario, then inspect the component that owns the fallback.
Budget context before calling the model
Instructions, conversation history, retrieved evidence, tool results, and generated output all compete for one finite context window. A production system decides what to admit before inference rather than trimming arbitrary text after the provider rejects the request.
Loading context lab
Preparing turn budgets...
Preserve the assembly trace
- Record source IDs, versions, authorization decisions, scores, and token counts for admitted context.
- Give system and safety instructions stable priority over retrieved or user-supplied text.
- Reserve output capacity and tool-result capacity instead of filling the window with history.
- Expose truncation and degraded memory to the user when continuity changes the answer.
Choose a memory contract, not a memory feature
Verbatim and bounded
Recent-turn window
Keep the newest turns exactly as written. It is predictable and correct for local follow-ups, but older dialogue disappears at a declared boundary.
Compressed continuity
Summary plus recency
Summarize older turns and keep recent messages verbatim. Version the summary input because compression can omit or distort important facts.
Selective recall
Retrieved memory
Store approved facts with provenance, confidence, retention, and tenant scope; retrieve only memories relevant to the current task.
Govern memory throughout its lifecycle
- Write: persist only facts with a user-visible purpose and an explicit owner.
- Read: filter by identity and tenant before similarity or ranking.
- Use: label recalled facts separately from current user input and external evidence.
- Correct: support update, deletion, expiry, and user review without rebuilding the whole conversation.
- Audit: retain enough provenance to explain why a memory entered a particular turn.
Do not store secrets, raw tool credentials, unsafe model guesses, or every generated token as long-term memory.
Treat streaming as a state machine
Streaming reduces time to first visible output, not total inference time. The client still needs ordering, backpressure, cancellation, reconnection rules, and one terminal outcome.
A streamed turn has explicit states
Partial text remains provisional until one terminal event closes the request.
Request
Accepted
Request ID, session version, and budgets are fixed.
Events
Generating
Token and tool events carry monotonically ordered sequence IDs.
Policy
Validating
Buffered claims or actions wait for policy and execution evidence.
Outcome
Terminal
Complete, cancelled, timed out, blocked, or failed closes the request.
Define client behavior for every outcome
- Keep partial text visibly provisional until a terminal event arrives.
- Stop provider work when the user cancels or disconnects where the transport supports it.
- On reconnect, resume from an event cursor or start a new request; never append a second stream blindly.
- Preserve typed tool and citation events instead of flattening everything into text.
- Measure time to first meaningful token, completion latency, cancellation, and abandoned partial responses.
Put tools behind an execution boundary
A model proposes an action; it does not authorize or prove the action. Parse structured arguments, validate them against current identity and policy, execute with scoped credentials, and return a typed result the model cannot overwrite.
Validate
Reject unknown tools, excess fields, invalid types, unsafe targets, and arguments derived from untrusted instructions.
Authorize
Evaluate user, tenant, resource, and action at execution time rather than trusting permissions copied into the prompt.
Execute
Use deadlines, idempotency keys, bounded retries, least-privilege credentials, and isolated network access.
Reconcile
Record whether the effect committed, failed, or is unknown; never let generated prose convert ambiguity into success.
Require user confirmation for destructive, expensive, external, or hard-to-reverse actions. Show the exact target and effect before execution.
Evaluate conversations as trajectories
Single-turn answer quality misses context drift, memory contamination, correction handling, tool recovery, and whether a later turn contradicts an earlier commitment.
Build a release suite
- Task completion: multi-turn goals with clarifications, corrections, topic changes, and early termination
- Context fidelity: references resolved from the right turn, explicit behavior when history is absent, and no cross-session leakage
- Grounding: citation support, permission-filtered retrieval, abstention, and contradictory evidence
- Tool behavior: valid proposals, confirmation, idempotency, timeout, partial failure, and reconciliation
- Safety: prompt injection, sensitive data, disallowed requests, unsafe memory writes, and policy-consistent refusal
- Experience: first-token latency, final latency, cancellation, recovery, response length, and user effort
Run deterministic checks for schemas and state transitions, model-based grading for bounded rubrics, and human review for high-impact slices. Compare releases on the same frozen cases before exposing live traffic.
Operate the conversation, not only the model
Monitor each turn
- Request, session, user, tenant, model, prompt version, and experiment identifiers
- Tokens by context source, truncation reason, output reservation, cache use, latency, and cost
- Memory reads/writes, retrieval candidates, authorization filters, citations, and stale-state conflicts
- Tool proposal, confirmation, execution, retries, idempotency, side-effect result, and reconciliation state
- Safety decisions, stream lifecycle, disconnects, cancellations, terminal outcomes, and user feedback
Prepare degraded modes
- Answer from the current turn when memory is unavailable and say that continuity is limited.
- Fall back to no retrieval or a read-only evidence cache when freshness permits it.
- Disable tools independently while preserving safe informational responses.
- Route to a smaller model, concise output, queue, or human handoff when budgets are exhausted.
- Reject the turn when identity, tenant isolation, policy, or action outcome cannot be established.
A trustworthy assistant can explain what it remembered, what evidence it used, what action occurred, and why a partial or degraded response is not the same as success.