Adversarial Robustness: AdvGLUE & TextFooler

Keep model quality, safety, and security separate

Evaluation scope is the set of claims a test can support. Adversarial results become misleading when ordinary quality, harmful behavior, and security boundary failures are averaged into one score.

Use separate dashboards and release gates because the owners and remedies differ:

• A quality failure may require better data, calibration, or model selection.
• A safety failure may require policy data, refusal behavior, and an independent output decision.
• A security failure may require authorization, trust isolation, least privilege, rate limits, or incident response.
• A single attack can cross categories, but each observed consequence still needs its own severity and owner.

Start with a threat model, not an attack library

A threat model states what is protected, what the attacker can do, and what outcome counts as success. It prevents an evaluator from choosing convenient attacks that do not resemble the deployed risk.

Declare five assumptions

1. Protected asset: Name the decision, policy, private data, tool permission, or service capacity that must remain protected.
2. Attacker capability: State whether the attacker sees scores, labels, generated text, gradients, retrieved documents, tool results, or only a final response.
3. Attack budget: Bound queries, time, token edits, restarts, accounts, and any access to the training or evaluation data.
4. Validity rule: Define which properties must remain unchanged, who verifies them, and how disagreements are resolved.
5. Success condition: Name the exact model or system consequence, including severity and affected slice.

Distinguish attack knowledge

• White-box attacker: Can inspect model parameters or gradients and optimize directly against the target.
• Gray-box attacker: Knows architecture, policy, or training patterns but cannot inspect every parameter.
• Black-box attacker: Observes outputs or scores and searches through repeated queries.
• Adaptive attacker: Changes strategy after observing defenses or prior results.
  • The evaluation must keep a held-out adaptive suite.
  • Attack generation and defense tuning must not share every case.
  • Production retry and identity limits belong in the threat model.

Turn a clean case into valid adversarial evidence

An adversarial evaluation case pairs a clean seed with a bounded attack, a validity decision, and an observed consequence. The following sequence keeps the denominator honest.

1. 1

   Baseline

   Qualify the clean seed

   Keep cases with a trusted expected result and record whether the target system is initially correct.

2. 2

   Attack

   Generate under a budget

   Apply a named attack family with fixed query, edit, time, and restart limits.

3. 3

   Evidence

   Validate the change

   Check intent, expected label, fluency, and policy equivalence with independent review where needed.

4. 4

   Measure

   Score the consequence

   Record robust accuracy, attack success, severity, queries used, and results by slice.

5. 5

   Operate

   Preserve the regression

   Version valid failures, defenses, judge policies, and model artifacts for repeatable release testing.

Keep denominators visible

Suppose an evaluation starts with 1,000 labeled cases:

• The model is correct on 920 clean cases, so clean accuracy is 92%.
• Of those 920 seeds, 800 generated attacks pass the semantic and label checks.
• Of the 800 valid adversarial cases, 230 cause the target failure.
• Valid attack success rate is 230 / 800 = 28.75%.
• Robust accuracy on the valid adversarial set is 570 / 800 = 71.25%.
• The 120 invalid attacks are reported as attack-generation or validity failures, not model failures.

Also report query exhaustion. Treating an attack that hit its cap as identical to an attack that never found a promising candidate hides how much effort the search required.

Explore the attack-budget robustness curve

An attack-budget curve plots model performance as the attacker receives more search effort. It answers a different question from clean accuracy: how quickly can a bounded attacker discover valid failures?

The model below is illustrative. Change the attack family, query cap, and maximum edited-token percentage. Watch for two independent reasons to reject a claim: robust accuracy can fall below the release floor, or semantic validity can fall too low for the attacks to remain trustworthy.

Read the curve correctly

• Compare models at the same attack family, budget, validity policy, and seed set.
• Publish the curve or several declared points; a low-budget result alone can overstate robustness.
• Report clean quality beside robust quality so a defense cannot hide behind a damaged baseline.
• Show confidence intervals and slices when the test set is sampled from a larger population.
• Treat the modeled curve as a decision aid. Real release evidence must come from measured attacks against the actual artifact.

Build an attack suite that matches the claim

An attack suite is a versioned collection of attack families, seed slices, budgets, and validity rules. No single benchmark represents every attacker or every system boundary.

AdvGLUE and TextFooler play different roles

• AdvGLUE provides adversarial versions of language-understanding tasks. It is useful for comparable evaluation across bounded perturbations, but it does not represent every generative or agentic attack path.
• TextFooler is a black-box word-substitution method. It ranks influential words, proposes semantically similar replacements, filters candidates, and queries the target until it finds a valid label-changing example or exhausts its budget.
• AdvBench-style harmful prompts and jailbreak suites test harmful-response behavior. Their success criteria and judges differ from label-preserving classification attacks.

Cover different transformations

• Character and encoding attacks: Homoglyphs, spacing, zero-width characters, and misspellings probe normalization and tokenization assumptions.
• Lexical attacks: Synonyms, insertions, and deletions probe sensitivity to surface form.
• Syntactic and paraphrase attacks: Reorder or rewrite sentences while preserving intent.
• Context attacks: Add distractors, conflicting evidence, or untrusted instructions.
• Adaptive attacks: Use prior outputs to choose the next candidate or combine several transformations.

For each family, store the seed ID, attack configuration, model version, judge version, generated candidate, validity result, target consequence, query count, and reviewer decision.

Design the offline evaluation architecture

An adversarial evaluation pipeline separates attack generation from validity review and release decisions. That separation makes failures reproducible and prevents the same model from silently generating, judging, and approving its own evidence.

Estimate the campaign before running it

For a worst-case campaign with 2,000 seeds, four attack families, three restarts, and 400 queries per run:

The cap is not the forecast. Measure early stopping, median and tail queries, invalid-candidate rates, judge calls, retries, and throttling. Capacity planning must also reserve target-model quota so evaluation cannot degrade production.

Trace a threat through the defense stack

A defense stack uses independently owned controls at ingress, trust boundaries, the model, egress, and operations. A control that works for one threat can be irrelevant to another.

Choose a threat scenario, then disable or restore defenses. The visible path distinguishes a robustness weakness in the primary model from a missing system security boundary.

Interpret the defense failure

• Normalization helps encoded-input attacks but does not isolate malicious retrieved instructions.
• Adversarial training can improve behavior on reviewed perturbations but cannot enforce authorization.
• An output gate limits some residual failures but must be independently versioned and tested against transfer attacks.
• Probe monitoring constrains an adaptive attacker's cumulative budget; a per-request benchmark does not measure that boundary.
• A covered scenario still has residual risk. Passing the design review means the required controls exist, not that they are perfectly effective.

Expect defenses to fail in recognizable ways

A defense failure mode is a repeatable reason measured robustness does not transfer to deployment. Diagnosing the pattern is more useful than adding another average score.

• Benchmark overfitting: The defense learns public attack templates but fails on held-out adaptive attacks.
  • Keep hidden attacks and rotate transformations.
  • Compare transfer from attacks generated against another model or control version.
• Invalid perturbations: A semantic judge accepts candidates that changed intent or the correct label.
  • Audit disagreement by language, domain, length, and severity.
  • Send borderline and high-impact cases to human adjudication.
• Gradient masking or detector artifacts: A weak attack appears unsuccessful because optimization or input handling is broken.
  • Increase the budget and use attack families with different search methods.
  • Check whether attacks transfer to related targets.
• Clean-quality regression: Adversarial training improves one suite while reducing ordinary task quality or minority-slice performance.
  • Report clean and robust metrics together.
  • Gate critical slices separately instead of averaging them away.
• System-boundary substitution: A model-level defense is credited for a problem that requires authorization, instruction isolation, or tool permissions.
  • Trace the attack to the protected asset and accountable control owner.
  • Test the deployed system, not only the base model endpoint.

Choose defenses by trade-off and ownership

A defense strategy changes one or more parts of the attack path, but every strategy introduces costs or assumptions that must be measured.

• Adversarial training can improve coverage of known valid attacks.
  • Trade-offs: training cost, clean-quality regressions, catastrophic forgetting, and overfitting to generators.
• Input normalization and preprocessing can collapse harmless encoding variation.
  • Trade-offs: language damage, false positives, canonicalization mismatches, and no protection against semantic attacks.
• Detection and uncertainty routing can abstain or escalate suspicious cases.
  • Trade-offs: added latency, blocked benign users, detector transfer failures, and review load.
• Independent output policy checks can catch failures after generation.
  • Trade-offs: another probabilistic component, correlated blind spots, and exposure of sensitive context before the check.
• Trust and authorization controls prevent untrusted text or users from gaining authority.
  • Trade-offs: policy complexity and integration work, but these controls cannot be replaced by model robustness.

Prefer layered controls whose failures are not perfectly correlated. Measure the entire path under the same scenario rather than testing each component only in isolation.

Implement a reproducible evaluation runner

An evaluation runner records the target, attack, validity, budget, and outcome for every case. The co-located Python example demonstrates those mechanics and aggregates attack results.

The example is a scaffold, not a production attack implementation. Its synonym lookup, lexical-overlap similarity, and masked-language-model candidates are placeholders. A production runner must also:

• Pin target, tokenizer, attack, dataset, judge, and policy versions.
• Make randomness reproducible and store exhausted-budget outcomes.
• Replace lexical overlap with task-appropriate validity checks and sampled human review.
• Isolate secrets and production data from attack workers.
• Write append-only case records before computing aggregates.
• Calculate uncertainty and slice results outside the attack loop.

Operate robustness as a release gate

A release gate converts evaluation evidence into an allow, hold, or rollback decision. Criteria must be declared before seeing candidate results and must reflect consequence severity.

The following example policy uses the same teaching thresholds as the curve lab; a real system should replace them with risk-specific values:

• Evidence validity: At least 85% of generated candidates pass semantic and label review, with human audits for critical slices.
• Robustness: Robust accuracy stays at or above 75% at every declared budget point, and no critical slice falls below its own floor.
• Clean quality: The candidate remains within the allowed clean-task regression budget.
• Safety: No unresolved critical harmful-output failure remains in the release suite.
• Security: Every scenario names its protected asset, required boundary controls, and accountable owner; authorization failures are hard stops.
• Operations: The canary has alert thresholds for attack-shaped traffic, policy escapes, latency, and rollback.

Preserve evidence after release

1. Run the fixed suite in continuous integration for model, prompt, tokenizer, policy, retrieval, and tool changes.
2. Run a held-out adaptive suite on a slower security cadence.
3. Add production incidents only after removing sensitive data and confirming the expected outcome.
4. Track attack-family yield, robust accuracy curves, semantic-validity disagreement, query tails, and critical-slice failures.
5. Expire exceptions, rotate hidden cases, and rerun the release decision when a control or threat assumption changes.

Robustness is not a permanent property of a model name. It is evidence about a versioned system under a versioned threat model and budget.