Adversarial Testing

Start with a threat model, not a list of clever prompts

A threat model connects an actor to an asset through a plausible product path. Without it, a large attack corpus can create impressive counts while missing the system's consequential boundaries.

A defensible test case records:

• the actor and realistic capability;
• the entry surface and target asset;
• the exact model, prompt, retrieval, tool, and policy versions;
• the expected control and prohibited outcome;
• the attempt budget, stopping rule, and evidence owner;
• the observed result, severity, containment, and disclosure status.

Do not call all surprising model behavior a security vulnerability. A vulnerability requires a violated boundary or security property in the deployed context. A harmful but non-adversarial model error belongs in the broader safety and quality program, even when the same control helps both.

Follow authority through the complete system

The model is one probabilistic component inside a larger authorization and data path. Retrieved text can be relevant while remaining untrusted; a tool result can be syntactically valid while carrying attacker-controlled content.

Trace a credible attack path

Use the lab to compare direct, retrieval, tool-result, multimodal, and repeated-query paths. Change the control posture and watch which boundary remains exposed. The latency values are illustrative product trade-offs, not measured universal overhead.

A prompt guard can be useful evidence, but it cannot replace least privilege, deterministic authorization, postcondition checks, query governance, or data-handling controls. Defense in depth matters because generative models do not provide a hard instruction-data boundary.

Distinguish the main attack families

Prompt and context manipulation

• Direct prompt injection enters through a user-controlled conversation and tries to redirect behavior or bypass policy.
• Indirect prompt injection arrives through data the application chooses to retrieve or process, such as a document, page, email, memory record, or tool result.
• Multi-turn manipulation distributes the attempt across conversation state so a per-message detector sees only fragments.
• Multimodal instruction injection uses image, audio, document layout, or extracted text paths that a text-only corpus may not exercise.

Prompt injection and jailbreaking overlap but are not identical. A jailbreak generally targets a model's behavioral restriction; prompt injection exploits how an application combines trusted intent with untrusted content. The product impact depends on what data and authority surround the model.

Retrieval, data, and tool paths

• Retrieval poisoning changes content eligible for indexing or ranking so adversarial material reaches the model.
• Tool-result injection uses data returned by a legitimate connector to influence a later proposal.
• Excessive agency turns a model mistake into a consequential effect because tools have broader permissions, destinations, or parameters than the task requires.
• Improper output handling lets model-generated content cross into SQL, HTML, shell, workflow, or messaging contexts without the validation expected at that boundary.

Test whether provenance survives chunking, reranking, summarization, memory, and tool handoffs. Test tool authorization independently from the model response: the same candidate action should be denied when identity, destination, purpose, or scope is not permitted.

Model, privacy, and supply-chain paths

• Behavior extraction uses repeated observations to approximate confidential model behavior or a decision surface.
• Membership inference and memorization probes test whether outputs reveal information about training membership or sensitive records.
• Data or model poisoning changes training, adaptation, retrieval, weights, or dependencies to produce targeted behavior.
• Denial and unbounded consumption exploit token, tool, recursion, retrieval, or compute paths to exhaust a budget or degrade availability.

Run privacy and extraction testing under approved query budgets and data-handling rules. Report rates and uncertainty; do not claim that a failed probe proves no information leakage.

Build a reproducible adversarial case

A useful case is more than attack text. It binds a minimized fixture to an expected security boundary and an exact system identity.

Keep live exploit details and sensitive incident content in an access-controlled store. The release suite needs a stable case identifier, content hash, expected outcome, severity, owner, and fixture location; it does not need to publish the operational payload.

Preserve enough evidence to reproduce

• Freeze model, system prompt, policy, tool schema, retrieval index, parser, and evaluator versions.
• Record attempt identity, random seed where meaningful, conversation state, tool proposals, policy decisions, and observed effects.
• Separate discovery cases from a protected holdout and incident-derived regression corpus.
• Keep the original test result immutable; attach remediation and retest records instead of overwriting history.
• Minimize personal data, credentials, private chain-of-thought, and unnecessary raw content in logs.

Run the program as a learning loop

Adversarial work is not a one-time launch ceremony. Each confirmed finding should change a control, a release condition, or the threat model.

1. 1

   Threat model

   Map the attack surface

   Inventory assets, actors, entry paths, trust boundaries, permissions, and existing controls before generating cases.

2. 2

   Test

   Exercise and adapt

   Run safe, versioned attempts under realistic budgets; change strategy after observing defenses and product responses.

3. 3

   Respond

   Triage and contain

   Confirm reproducibility, severity, affected versions, reachable assets, immediate containment, and disclosure ownership.

4. 4

   Learn

   Gate and monitor

   Promote minimized incidents into protected regressions, enforce release policy, and watch the same boundary in production.

Measure attack success without hiding severity

For one attack family, the basic attack success rate is:

ASR = successful attempts / valid attempts

The denominator must be visible. Exclude invalid harness runs separately; do not silently count tool timeouts or evaluator failures as defended attacks.

Report at least:

• successes and attempts for each attack family, severity, product path, language, and modality;
• attacker access, information, budget, adaptation rule, and stopping condition;
• uncertainty or replication evidence where random generation and sampling affect results;
• evaluator identity, disagreement, and human-confirmed high-severity findings;
• benign false-positive rate, refusal behavior, latency, cost, and product-utility change;
• exact system and corpus versions plus whether cases were discovery, holdout, or incident regressions.

A severity-weighted summary can help prioritize work, but it must not replace the underlying slices. A critical authorization bypass must remain independently blocking even when many moderate attempts fail.

Treat adaptive attackers as a different test condition

A static attack suite replays known cases. An adaptive evaluation lets the tester observe refusals, detectors, rate limits, tool denials, or output transformations and then choose a new strategy within the declared capability and budget.

Adaptive testing changes the evidence contract:

• record what feedback the tester can observe;
• cap attempts, cost, time, accounts, and external effects;
• separate tuning traffic from final holdout evidence;
• compare the same system identity before and after mitigation;
• preserve failed attempts as well as successes so the search is auditable;
• stop immediately when a test risks real data, users, systems, or irreversible effects.

Automation expands coverage and mutation, but humans remain important for novel attack paths, ambiguous consequences, and contextual harm. An automated generator must not be allowed to execute privileged actions directly.

Make release policy explicit

A release gate maps versioned adversarial and benign evidence to an operational decision. Thresholds are governance choices tied to severity and exposure, not universal constants.

The example keeps each attack family independently blocking, adds a benign false-positive limit, protects a utility floor, and requires incident regressions. In a real program, calibrate limits with product owners, security, safety, privacy, affected stakeholders, and incident-response teams before inspecting the candidate.

Compare defense and release decisions

Use the lab to see why one aggregate rate can pass while critical indirect-injection or tool-authorization slices fail. Then compare a bounded runtime with an overblocking filter and add adaptive evidence.

The lab uses illustrative counts to teach the gate mechanics. It does not claim that these rates apply to a provider, model, or production system.

Design defenses around boundaries

Reduce reachable authority

• Give each tool the minimum operation, destination, data scope, and lifetime required for one task.
• Validate action parameters against deterministic policy using authenticated identity and direct user intent.
• Require explicit approval for high-impact, irreversible, external, or novel effects.
• Verify postconditions and make retries idempotent so ambiguous outcomes do not multiply effects.

Preserve data origin and handling rules

• Carry source identity, trust, purpose, tenant, and sensitivity metadata through retrieval and tool adapters.
• Treat retrieved and tool-returned content as data, never as authority.
• Normalize text, images, audio, markup, and document structure before policy checks while retaining source regions for review.
• Validate released output and destination independently from input screening.

Limit probing and leakage

• Apply principal-aware budgets, rate limits, anomaly detection, and response-detail policy to coordinated query campaigns.
• Avoid returning unnecessary confidence, debug, prompt, retrieval, or tool-internal data.
• Test retention, logging, cache, trace, and support workflows for the same privacy boundary as model output.
• Reassess when the model, fine-tune, retrieval corpus, system prompt, tool set, or response format changes.

No single filter closes every path. A model-side mitigation can reduce attack success while system controls bound the consequence when the model still fails.

Monitor the boundary in production

Production monitoring should detect effects and control decisions, not attempt to store every suspicious prompt forever.

Watch for:

• denied and approved tool actions by identity, source type, destination, and policy reason;
• unusual query similarity, account rotation, token or tool budgets, and coordinated campaigns;
• retrieval of newly indexed or low-trust sources before consequential proposals;
• spikes in refusals, overrides, false-positive reports, appeals, and fallback use;
• disclosure-policy failures, anomalous output destinations, and postcondition mismatches;
• recurrence of incident signatures across changed models, prompts, parsers, tools, or corpora.

Join alerts to exact versions and preserve a route to containment: disable one tool, revoke one credential, quarantine one source, narrow one tenant, roll back one policy bundle, or stop one candidate. Monitoring without an executable response path is only observation.

Use frameworks as maps, not certificates

• OWASP LLM01:2025 Prompt Injection distinguishes direct and indirect prompt injection and emphasizes that RAG and fine-tuning do not fully remove the vulnerability.
• OWASP LLM02:2025 Sensitive Information Disclosure and LLM06:2025 Excessive Agency connect model behavior to data exposure and overpowered application actions.
• NIST AI 100-2e2025 organizes adversarial machine learning by lifecycle stage, attacker goal, capability, knowledge, and attack family, including evasion, poisoning, privacy, and misuse for generative AI.
• MITRE ATLAS provides a maintained knowledge base of tactics and techniques for AI systems, including prompt injection, RAG poisoning, agent tool invocation, and model access paths.
• Greshake et al., Indirect Prompt Injection demonstrates why data consumed by an LLM-integrated application can become an instruction channel.
• Microsoft's lessons from red teaming 100 generative AI products emphasizes product context, human expertise, automation as coverage rather than completion, and the difference between red teaming and safety benchmarking.
• Tramèr et al., Stealing Machine Learning Models via Prediction APIs and Shokri et al., Membership Inference Attacks provide foundational evidence for model-extraction and training-membership threat models.

These sources help enumerate risks and terminology. They do not prove that a particular application, model, defense, or release is secure.