GenAI Monitoring
Monitor GenAI systems: observability, performance metrics, quality tracking, and production monitoring strategies.
What is GenAI monitoring?
GenAI monitoring is the continuous collection and interpretation of evidence about how a generative AI application behaves in production. It connects service health, model execution, cost, response quality, safety, and user outcomes so an operator can detect a problem, explain it, and choose a bounded response.
In plain language: an LLM request can return 200 OK while the answer is ungrounded, the wrong tenant's context was retrieved, latency doubled, or the cost per successful task became unacceptable. The core invariant is therefore:
An alert is useful only when it identifies the affected population, links to reproducible evidence, and names an owned action.
Monitor the complete user-visible path
No single signal describes GenAI health. Operators need a layered view because infrastructure, orchestration, model behavior, and user outcomes can fail independently.
Service health
Measure availability, typed errors, queue delay, rate limits, time to first output, end-to-end latency, and dependency saturation.
Execution path
Trace model calls, retrieval, reranking, policy checks, tool use, retries, fallbacks, and the versions that shaped the result.
Economics
Attribute input, output, cached, and reasoning tokens plus provider charges to features, routes, tenants, and successful outcomes.
Response behavior
Sample groundedness, task success, policy violations, refusals, citation support, evaluator disagreement, and human feedback by slice.
Product outcome
Track whether the user completed the task, retried, abandoned, escalated, corrected the answer, or accepted an unsafe recommendation.
Keep signals separate until the decision is clear
- A latency increase can come from a provider, retrieval, a tool, a retry loop, or a longer output.
- A cost increase can reflect more traffic, larger prompts, a route change, lower cache reuse, or repeated failures.
- A quality decline can affect one language, intent, tenant, model route, prompt version, or retrieval corpus while the fleet average stays healthy.
- A user complaint is an important outcome signal, but it still needs trace and version evidence before it identifies a cause.
Build an evidence path, not only a dashboard
Every production interaction needs stable identifiers that survive across the application, model provider, retrieval system, tools, evaluation jobs, and incident records.
Join one user outcome to its production evidence
A shared request and trace identity turns an aggregate signal into a reproducible diagnosis.
Slice
Request context
Record feature, tenant class, language, intent, risk tier, experiment, and privacy policy without using high-cardinality metrics labels.
Path
Execution trace
Join retrieval, model, guardrail, tool, retry, and fallback spans with prompt, model, policy, and corpus versions.
Behavior
Evaluation evidence
Attach deterministic checks, sampled evaluators, reviewer labels, uncertainty, and the sampling reason to the same interaction.
Response
Decision record
Preserve the threshold, affected slice, owner, mitigation, rollback target, exposure limit, and follow-up evaluation.
Separate low-cardinality metrics from detailed traces
- Put bounded dimensions such as service, deployment, model route, region, and status on metrics used for fleet trends and paging.
- Put request IDs, prompt versions, document IDs, tool arguments, and detailed evaluator evidence on traces or controlled event records.
- Keep raw prompts, outputs, tool arguments, and retrieved text opt-in because they can contain personal, confidential, or regulated data.
- Sample traces deliberately: preserve errors and high-risk outcomes, sample healthy traffic, and record the sampling policy so estimates remain interpretable.
OpenTelemetry's current GenAI semantic conventions define common operation, model, token-usage, duration, and streaming-latency fields, but the GenAI conventions remain under active development. Pin a convention version and test migrations instead of assuming every instrumentation library emits identical names.
Diagnose from signal to slice to trace
Start broad enough to notice change, then narrow the population before inspecting individual executions. A fleet average tells you that something moved; the slice and trace explain where and how.
Use the evidence ladder in order
- Validate the signal. Check the event definition, denominator, missing data, duplicates, sampling, and deployment window.
- Compare bounded slices. Break the movement down by feature, route, language, tenant class, intent, risk tier, and version.
- Inspect representative traces. Compare a failing trace with a healthy peer and locate the first divergent span or version.
- Reproduce the behavior. Replay with preserved inputs and controlled dependencies when policy and privacy allow it.
- Record the attribution. Link the alert, trace, changed component, decision, and regression case.
Require evidence before an alert interrupts a human
A useful alert is a decision packet, not a red number. It should prove that a meaningful boundary was crossed and provide enough context for the responder to act without starting from an empty dashboard.
1 Guardrail
Name the boundary
Define the quality floor, cost budget, latency objective, safety condition, or outcome rate before observing the result.
2 Window
Measure persistence
Require an appropriate sample count, time window, confidence bound, or repeated occurrence so one noisy point does not page the team.
3 Slice
Attach the population
Include the affected feature, route, tenant class, language, risk tier, and exposure instead of reporting only a fleet average.
4 Trace
Link the evidence
Attach representative trace IDs, version changes, evaluator results, dependency status, and the last known-good comparison.
5 Owner
Declare the response
Name the owner, runbook, rollback target, exposure cap, escalation path, and condition that resolves the alert.
Match the alert to the failure
- Page immediately for confirmed severe safety events, security incidents, unavailable critical paths, or fast budget exhaustion with a tested containment action.
- Open an investigation for statistically credible quality, cost, or latency regressions that need attribution but do not require immediate shutdown.
- Run an evaluation escalation when impact could be material but production evidence is sparse, delayed, biased, or disagrees across evaluators.
- Keep a dashboard signal when the change is informative but has no predeclared response or owner.
Turn guardrail breaches into bounded responses
Quality, cost, and latency guardrails should not be collapsed into one score. A strong quality gain cannot silently cancel a safety failure, and lower cost cannot excuse an unusable latency regression.
Choose the least disruptive response that controls exposure
- Continue a bounded canary when every hard limit passes and evidence coverage is adequate.
- Route, rate-limit, or disable an expensive optional feature when the breach is isolated and rollback would remove healthy behavior.
- Roll back when a release-attributable quality or latency breach is reproduced and a known-good target exists.
- Escalate evaluation and cap exposure when the signal matters but slice coverage, labels, or attribution are insufficient.
- Reconcile user-visible effects and preserve the trace before closing the incident.
Avoid monitoring systems that create new risk
Telemetry can become a sensitive data store
Prompts, responses, retrieved documents, tool arguments, user identifiers, and evaluator notes can expose personal data, credentials, confidential content, or security decisions.
- Collect content only for a declared observability or evaluation purpose.
- Prefer bounded attributes, derived checks, hashes with understood limitations, and short retention over unrestricted raw content.
- Apply access control, encryption, regional policy, deletion, and audit logging to telemetry pipelines.
- Test collector redaction and filtering before production data reaches an exporter.
Cardinality can make metrics unusable
Request IDs, user IDs, document IDs, and raw error strings create unbounded metric series. Keep them in traces or logs and join by stable IDs during investigation.
Automated evaluators can fail silently
Version evaluator models, prompts, rubrics, calibration sets, and thresholds. Monitor disagreement, missing scores, label delay, and slice coverage so an evaluation pipeline cannot keep the dashboard green by dropping difficult cases.
Operate the monitoring system itself
On every behavior-changing release
- Record model, prompt, retrieval, tool, policy, route, and evaluator versions.
- Run predeclared quality, safety, cost, and latency gates against a known-good baseline.
- Verify alert routing, rollback or feature-disable controls, and trace-to-evaluation joins.
- Start with bounded exposure and preserve the release assignment on every trace.
On a regular cadence
- Review signal coverage, missing spans, dropped telemetry, exporter health, evaluator errors, and label delay.
- Compare aggregate and critical-slice behavior with denominators and uncertainty.
- Revisit thresholds after traffic, pricing, architecture, or product objectives change; preserve the prior decision record.
- Turn confirmed incidents and near misses into regression cases and runbook tests.
After an incident
- Preserve representative traces and the exact version identities that produced the behavior.
- Document detection delay, exposure, containment, recovery, user impact, and evidence gaps.
- Verify the mitigation against a bounded canary before restoring traffic.
- Run an after-action review and update monitoring, evaluation, and response controls.