Agentic AI Evaluation

Evaluate the trajectory, not only the final sentence

Each run should begin with an observable contract and end with an independent state check. Intermediate evidence explains why the run succeeded, failed, recovered, or became unsafe.

1. 1

   Boundary

   Declare the task contract

   Record the starting state, observable success condition, allowed tools, prohibited actions, budgets, and required approvals.

2. 2

   Run

   Execute in a controlled world

   Provide stable identities for the agent, prompt, model, tools, environment, and scenario while preserving every observation and action.

3. 3

   Trajectory

   Score decisions and recovery

   Inspect planning, tool selection, argument validity, evidence use, retries, memory, adaptation, and policy compliance at each step.

4. 4

   Outcome

   Verify authoritative state

   Check the external system for complete, partial, duplicate, or missing effects instead of trusting the agent's narrative.

5. 5

   Robustness

   Repeat under variation

   Change requirements, tools, latency, permissions, and adversarial inputs to expose brittle plans and unsafe recovery behavior.

Explore where decisive evidence comes from

Switch among a healthy run, tool failure, blocked action, and false completion. The active path shows which component supplies the evidence needed to score the run honestly.

Keep outcome, safety, efficiency, and recovery independent

A single blended score hides why the agent failed. Release decisions need separate gates because high task completion cannot compensate for unauthorized actions, and low cost cannot compensate for false success.

Record rates with their denominators

• Verified completion: runs whose authoritative final state satisfies every required success condition divided by eligible runs.
• False-success gap: self-reported completion minus externally verified completion.
• Boundary violation: runs with an attempted or executed prohibited action divided by adversarial and representative runs.
• Tool error and recovery: failed calls by error type, followed by the fraction recovered without duplicate or unsafe side effects.
• Trajectory budget: p50 and p95 steps, tool calls, latency, tokens, and cost for completed and failed runs separately.

Turn trajectory evidence into a release decision

Use the scorecard to compare four agent behaviors. Increase the number of evaluation runs to narrow uncertainty, then change the permitted step budget. Notice that more samples cannot average away a systematic false-completion, retry, or permission failure.

Read the gates in this order

1. Verify final state. Use a conservative completion bound rather than the agent's point estimate or self-report.
2. Measure claim reliability. A large false-success gap means completion language cannot be trusted as evidence.
3. Enforce authorization separately. Treat boundary violations as hard blockers regardless of average task success.
4. Inspect tool and recovery behavior. Separate provider failures from malformed calls and unsafe retry choices.
5. Bound the trajectory. Gate p95 steps, latency, cost, and repeated side-effect risk rather than rewarding eventual success at any price.
6. Grant only bounded exposure. Passing offline evidence earns a canary with abort signals, not unrestricted autonomy.

Build scenarios that expose real decisions

Static question sets underrepresent agent behavior because later observations depend on earlier actions. Use environment-backed scenarios with controlled variation and an evaluator that can inspect both the path and the final state.

Cover the failure surface deliberately

• Ambiguous tool outcomes: timeouts after a provider may have committed the action, delayed webhooks, and stale reads.
• Changing environments: inventory disappears, permissions change, a page layout shifts, or another actor updates shared state.
• Adversarial observations: retrieved pages contain prompt injection, misleading confirmation text, or poisoned tool output.
• Long-horizon drift: the agent forgets constraints, repeats work, changes goals, or exhausts budgets late in the run.
• Human boundaries: approvals arrive late, operators reject a step, or escalation is required before a high-impact action.

Instrument the evaluator for diagnosis

A useful score links every aggregate back to the exact trajectory steps that produced it. Preserve identities and raw evidence before adding dashboards.

Store these fields for every run

• Scenario, environment snapshot, agent, model, prompt, policy, tool, and evaluator versions
• Ordered observations, candidate actions, selected actions, arguments, approvals, results, errors, and timestamps
• External-state changes, idempotency identities, retry relationships, and compensating actions
• Goal, policy, recovery, efficiency, latency, token, cost, and human-review results
• Release assignment, user-impact cohort, incident link, and retention classification

Diagnose efficiency without rewarding shortcuts

Tool-call count is useful only when conditioned on task difficulty and verified outcome. An agent making one unauthorized call is not more efficient than an agent making three safe calls.

Separate healthy efficiency from harmful compression

• Reward correct first-choice tools, valid arguments, parallel independent work, result reuse, and evidence-based stopping.
• Penalize repeated identical calls, ignored results, malformed arguments, avoidable sequential work, and retries without reconciliation.
• Block the score entirely when the run violates permissions, creates an unreconciled side effect, or falsely claims completion.
• Compare within scenario families so a difficult recovery task is not judged against a trivial straight-line task.

Monitor production without grading on user patience

Online evaluation detects behavior that controlled scenarios miss, but production signals are confounded by task mix, changing tools, user intervention, and delayed outcomes. Join them to release and trajectory identity before acting.

Watch three layers together

• Outcome: verified completion, abandonment, corrections, duplicate effects, reversals, and delayed task results.
• Trajectory: steps, backtracking, plan changes, tool errors, retries, approvals, escalations, and false-success gaps.
• Operations: p50/p95 latency, queue time, tokens, tool spend, evaluator failures, trace loss, and policy-gateway health.

Use shadow traffic or a small canary, predeclare abort signals, retain a last known-good route, and sample full traces by risk. A stable fleet average must not hide a failing tenant, tool, language, or high-impact action class.

Operate an evaluation program, not a leaderboard

Public benchmarks can reveal broad capabilities, but production agents need tasks that match their tools, permissions, environment, and failure cost. Treat benchmark results as one input to a versioned evidence portfolio.

Before deployment

1. Define observable success and prohibited outcomes for every representative task family.
2. Build deterministic state checks and policy assertions before adding model-based judges.
3. Test tool errors, changing requirements, adversarial observations, ambiguous side effects, and human approval paths.
4. Calibrate any automated rubric against qualified human review and preserve disagreements.
5. Set independent release thresholds for outcome, safety, recovery, efficiency, latency, and cost.

In production

1. Monitor release- and cohort-specific trajectories with bounded retention and access controls.
2. Pause or route around failing tools before widening autonomy or retry budgets.
3. Review novel, high-impact, and evaluator-disagreement cases by risk rather than only random sampling.
4. Convert validated incidents into a new scenario-set version while retaining an untouched release holdout.
5. Recalibrate thresholds and evaluators when the agent, tools, users, or environment changes.