Agentic AI Systems Design

Choose agentic control only when the path is genuinely uncertain

Agentic control means that a model selects at least part of the execution path at run time. It matters when the required steps cannot be enumerated cleanly in advance, but it adds latency, cost, policy surface, and recovery complexity.

A useful selection test is:

1. Can ordinary code express the path and edge cases clearly?
2. Does model-selected sequencing improve a measurable task outcome?
3. Can the environment expose authoritative feedback after each action?
4. Can every consequential effect be bounded, observed, and recovered?

If the first answer is yes and the others are weak, build a workflow rather than an open loop. Anthropic's official guidance similarly distinguishes predefined workflows from systems where a model dynamically directs tool use, and recommends adding complexity only when simpler patterns fall short.

Separate the model, runtime, tools, and state

A system boundary identifies which component owns a decision and which inputs it may trust. Production failures often begin when a model response is allowed to blur boundaries that normal software would enforce explicitly.

Keep four forms of state distinct:

• Conversation context is model-visible history. It helps continuity but is not an authoritative transaction log.
• Run state records the current step, budget use, pending approval, tool call IDs, and terminal status.
• Domain state lives in the source of truth, such as an order ledger, repository, ticket system, or deployment controller.
• Memory carries selected information across runs. Treat writes to memory as governed mutations because poisoned memory can redirect later work.

The runtime, not the model, maps an authenticated principal to tool credentials. A tool description tells the model how to propose a call; it does not grant authority.

Make the execution loop explicit and bounded

An execution loop repeatedly proposes, checks, acts, observes, and decides whether to continue. A production loop needs named transitions and terminal states so that operators can tell the difference between success, waiting, failure, and abandonment.

1. 1

   Contract

   Bind the goal

   Record the target object, allowed scope, success evidence, and prohibited effects before planning.

2. 2

   Model

   Propose a step

   Request one structured action or a terminal response from the planner.

3. 3

   Runtime

   Authorize and execute

   Validate the tool, arguments, identity, budget, and approval rule before credentials are used.

4. 4

   Evidence

   Validate observation

   Treat tool output as untrusted data, validate its schema and provenance, and reconcile domain state.

5. 5

   State

   Persist or stop

   Checkpoint progress, then complete, wait, retry within policy, or stop with a machine-readable reason.

Set independent limits because each protects a different resource or failure mode:

• Maximum model turns limits recursive planning and token spend.
• Maximum tool calls limits external load and action exposure.
• Per-tool attempts limit repeated pressure on one dependency.
• Wall-clock deadline releases workers and honors caller expectations.
• Financial or compute budget limits paid work.
• Repeated-state detection stops loops that make no measurable progress.
• Cancellation and emergency-stop checks let operators end a run between effects.

Persist durable state before long waits and ambiguous effects

Durable state is run information that survives process restarts, worker loss, and long approval waits. Without it, a resumed worker cannot prove what was proposed, approved, attempted, or committed.

A practical state machine includes:

• created: the goal contract exists, but execution has not started.
• running: one worker owns the current lease and may propose work.
• waiting_for_approval: the exact pending tool call is durable and no effect has executed.
• recovering: the runtime is reconciling an interrupted or ambiguous operation.
• completed: authoritative outcome evidence satisfies the success contract.
• failed, cancelled, or budget_exhausted: a terminal stop reason is recorded.

Checkpoint before releasing a worker for approval, after every external result that can affect later decisions, and before reporting completion. Use optimistic versioning or a lease so two workers cannot advance the same run concurrently.

This sketch deliberately omits model and workflow-framework implementation. It demonstrates the ownership boundary: the planner proposes, the gateway enforces authority, tool output is validated as data, and the run checkpoints before continuing.

Grant the least authority and bind approval to one effect

Least authority means that a run receives only the tools, credential scope, arguments, duration, and effect size needed for its current goal. A broad tool catalog with administrator credentials turns a planning error into a large security incident.

For each tool, define:

• A narrow purpose and an input schema with unknown fields rejected.
• Read, proposal, reversible write, irreversible write, or privileged-administration effect.
• Credential scope derived from the user and current task, never copied from model text.
• Idempotency and reconciliation behavior for mutations.
• Timeouts, rate limits, output schema, and maximum response size.
• Approval policy based on the actual effect, not the model's confidence.

An approval gate pauses a durable run before a sensitive tool call. The reviewer should see canonical server-generated details: tool name, target, arguments, credential scope, expected consequence, and whether the action is reversible. Approval binds to that exact call ID. If arguments change, approval is invalid.

OpenAI's current Agents SDK documentation exposes this pattern as an interruption that can serialize and resume run state. MCP's tool specification likewise recommends confirmation for sensitive operations and requires clients to treat tool annotations as untrusted unless they come from trusted servers.

Treat retries, compensation, and injection as runtime problems

Recovery is the ability to determine what happened after failure and move the system to a known state. Retrying is only one recovery technique, and it can make an incident worse when an external mutation is ambiguous.

For a mutating call, use one stable operation or idempotency key for one user intent. If the network fails after the remote service may have committed, query authoritative state by that identity before replaying. A new key represents a new effect, not a retry.

Use the recovery pattern that matches the effect:

• Read failure: bounded retry with backoff may be safe if the caller deadline and dependency policy allow it.
• Ambiguous write: reconcile by operation ID before deciding whether another delivery is needed.
• Reversible write: checkpoint the effect and provide an audited compensation operation.
• Irreversible effect: stage or preview it, require approval, and reduce authority because rollback may be impossible.
• Partial multi-step workflow: resume from a versioned checkpoint; do not restart from the prompt and hope to reproduce the same path.

Prompt injection occurs when untrusted content changes model behavior in an unintended way. In an agentic system, the injection may arrive through a webpage, document, tool result, memory record, or another agent. A schema-valid result can still contain a hostile instruction.

Contain injection with independent controls:

1. Label external content as data and keep it separate from trusted policy.
2. Allow tools by task and identity, not by instructions found in content.
3. Validate structured output and reject undeclared fields or authority claims.
4. Gate sensitive effects on exact trusted arguments.
5. Minimize data egress paths and block arbitrary destinations.
6. Log the source, tool, arguments, policy decision, and resulting external state.
7. Red-team direct, indirect, multimodal, memory, and cross-agent injection paths.

OWASP's 2026 Agentic Applications list covers goal hijacking, tool misuse, identity and privilege abuse, memory poisoning, cascading failures, and related agent-specific risks. NIST AI 600-1 provides the broader governance, measurement, and risk-management profile; it does not replace application threat modeling.

Evaluate outcomes, trajectories, boundaries, and recovery

Agent evaluation tests the model and its runtime together in an environment where tools can change state. A fluent final message is not proof that the requested outcome exists.

Use separate evidence layers:

Build release suites from representative task distributions and risk-focused cases. Run multiple trials when model variance matters, but do not average away a hard safety or correctness gate. Measure outcomes by slice, including tool, permission scope, language, tenant, task complexity, and failure mode.

Anthropic's agent evaluation guidance distinguishes the transcript or trajectory from the final environment outcome. That distinction is essential: the system may claim success while the database, repository, or external service shows otherwise.

Observe enough to explain and recover every run

Observability is the evidence needed to reconstruct a run, detect abnormal behavior, and support recovery without storing unnecessary secrets or personal data.

Record at minimum:

• Run, task, trace, parent, model, prompt, policy, tool-schema, and deployment versions.
• Authenticated principal, delegated scope, and the policy rule that allowed or denied a call.
• Structured proposals, redacted arguments, call IDs, operation IDs, latency, retries, and error class.
• Approval request, reviewer identity, exact approved call hash, decision, and expiry.
• State transitions, checkpoint version, lease owner, cancellation, and terminal stop reason.
• Verified outcome from the source of truth and any compensation or manual repair.

Trace inputs and outputs may contain secrets, personal data, or hostile content. Apply field-level redaction, access control, retention limits, and regional policy before export. OpenAI's Agents SDK tracing documentation explicitly notes that model and function spans may capture sensitive inputs and outputs.

Operational signals should answer concrete questions:

• Are runs completing according to authoritative state?
• Which tools, policies, or model versions create repeated failure?
• How often do runs exhaust steps, tool calls, time, or money?
• Which mutations are retried, duplicated, rejected, compensated, or manually repaired?
• Are approval queues meeting service objectives without becoming a rubber stamp?
• Did a release change permission violations, false-success gaps, or recovery rates?

Operate a controlled release, not a demo loop

A release gate is a set of independent conditions that must pass before an agent configuration receives real authority. Framework features and benchmark scores do not replace these conditions.

Before production exposure:

1. Version the goal contract, prompts, model, tools, policy, and state schema.
2. Test deterministic validators and tool adapters without a model.
3. Run outcome, trajectory, adversarial, permission, and recovery evaluations in a sandbox.
4. Start with read-only or proposal-only tools and narrow identities.
5. Add exact-call approval before sensitive mutations.
6. Canary on a small task and tenant scope with emergency stop and rollback paths.
7. Compare production traces with evaluation assumptions and feed new failures back into tests.
8. Expand authority only when evidence supports the specific new effect.

Keep multi-agent designs especially narrow. Each handoff adds identity, context, state, timeout, and attribution questions. A manager pattern can centralize policy, while peer handoffs can distribute work, but neither is safer by default. The same runtime controls must apply across nested agents and delegated tools.

Libraries and protocols can supply runners, typed tools, sessions, approvals, tracing, or transport. Verify their current contracts in official documentation, then keep business authorization and recovery semantics in your application control plane.

Primary references

These sources define the current practices and contracts used in this lesson:

• OpenAI, A practical guide to building AI agents for use-case selection, orchestration, tool safeguards, stop conditions, and human intervention.
• OpenAI Agents SDK, Running agents for loop behavior, maximum turns, and durable execution integrations.
• OpenAI Agents SDK, Human-in-the-loop for exact tool-call interruption, serialized run state, approval, rejection, and resume.
• OpenAI Agents SDK, Tracing for trace and span coverage plus sensitive-data handling.
• Anthropic, Building effective agents for workflow-versus-agent selection, environmental feedback, stopping conditions, and tool design.
• Anthropic, Demystifying evals for AI agents for trials, trajectories, outcomes, graders, and environment-based evaluation.
• NIST AI 600-1, Generative AI Profile for cross-sector generative-AI risk management.
• OWASP Top 10 for Agentic Applications 2026 and LLM01:2025 Prompt Injection for agent goal hijack, tool misuse, privilege abuse, memory/context poisoning, and direct or indirect prompt injection.
• Model Context Protocol 2025-11-25 tool specification and MCP security best practices for tool schemas, confirmation, output validation, least privilege, and transport-specific security.