AI Safety & Guardrails

Safety changes the entire request path

A safety boundary is a control outside or around the model that prevents, detects, contains, or helps recover from a specific failure. Defense in depth matters because controls fail in different ways.

Start with harms, not a generic guardrail list

A harm model connects a plausible failure to affected people, an exposure path, a consequence, and a control owner. It prevents teams from treating every refusal or classifier score as equally important.

For each harm, record:

• Trigger: benign error, misuse, prompt injection, poisoned context, drift, or operator mistake.
• Path: the model, retrieval, policy, tool, human, or monitoring boundary the failure crosses.
• Impact: severity, reach, duration, reversibility, and concentration in an affected cohort.
• Response: prevention, containment, owner, stop condition, evidence contract, and recovery test.

Choose controls that match the product boundary

A control strategy is defensible only when its mechanisms interrupt the actual failure paths of the product. Change the context and strategy below to see why text filters alone do not constrain identity, authority, oversight, or recovery.

The percentage in this lab represents modeled boundary coverage, not a probability that the system is safe. Evidence still has to show that every selected control works against representative failures.

Enforce high-consequence decisions outside the model

A policy gate is deterministic application logic that combines authenticated context with independent safety signals. It can allow a low-risk output, send an uncertain or consequential proposal to review, or block a violated boundary.

The example is intentionally small. In production, replace its boolean inputs with versioned decisions from identity, policy, privacy, authorization, evaluation, and approval services. The important property is that a model cannot grant itself more data or authority by generating persuasive text.

Evaluate failure paths before release

A safety evaluation estimates whether controls work for the intended users, threats, and system versions. It is stronger than a single aggregate score because it exposes where the system fails and what release decision follows.

Build a release-grade evidence contract

• Freeze the model, prompt, policy, retrieval corpus, tools, evaluator, and dataset versions under test.
• Separate evaluation construction, threshold calibration, release evidence, and hidden regression cases.
• Report denominators, uncertainty, severity, and slice results beside every rate.
• Include benign cases so stricter filters do not silently destroy product utility.
• Predetermine which failures block release, cap exposure, require human review, or trigger rollback.
• Convert production incidents and near misses into protected regression tests.

Distinguish safety failures by where they originate

A failure taxonomy speeds diagnosis by separating behavior, security, data, human, and operational causes. The categories overlap, but their first containment steps differ.

Contain incidents before certainty is perfect

An AI incident is an observed or credible failure that causes harm, violates a boundary, or creates unacceptable exposure. Response severity should follow impact, reach, reversibility, and active risk rather than waiting for the model to explain itself.

Minimized evidence and immediate containment solve different problems. Preserve enough identity to reproduce the failing path, but do not create an uncontrolled archive of sensitive prompts and outputs.

Turn an incident into a response contract

A response contract names the owner, first containment action, evidence to preserve, and success condition. This makes escalation executable under pressure.

1. 1

   Detect

   Confirm impact

   Validate the signal without keeping active harm exposed merely to improve statistical confidence.

2. 2

   Contain

   Bound the path

   Restrict the affected cohort, capability, route, or version; shut down when no safe boundary remains.

3. 3

   Evidence

   Preserve identity

   Capture timestamps, cohort, exact component versions, minimized decision traces, and response actions.

4. 4

   Recover

   Prevent recurrence

   Repair the control, validate the known-good path, add regression cases, and review whether similar paths exist.

Operate safety as a continuous control loop

A safety operating loop keeps controls aligned as models, prompts, data, tools, users, and threats change.

Before exposure

• Name system owners, safety owners, incident commanders, and accountable human approvers.
• Map high-severity harms and affected cohorts before selecting controls.
• Version the full system and attach evaluation evidence to the candidate release.
• Test safe fallback, route restriction, rollback, shutdown, and evidence capture.
• Cap initial exposure and define stop conditions before the canary starts.

During operation

• Monitor policy failures, privacy events, tool denials, overrides, appeals, complaints, drift, and critical-slice outcomes.
• Measure false negatives and false positives because both harm users in different ways.
• Reassess after changes to the model, prompt, policy, retrieval data, tools, purpose, or population.
• Limit sensitive trace access and retention while preserving enough version identity for diagnosis.
• Review incidents and near misses for control gaps, organizational delays, and related failure paths.

Use maintained frameworks as inputs, not proof

Frameworks help teams organize risks and controls, but the application must still produce evidence for its own context.

• NIST AI Risk Management Framework organizes risk work across Govern, Map, Measure, and Manage. NIST notes that AI RMF 1.0 is being revised, so preserve the framework version used by each decision.
• NIST Generative AI Profile describes risks that generative AI creates or amplifies and suggests cross-sector risk-management actions.
• OWASP Top 10 for LLM and Generative AI Applications catalogs application security risks including prompt injection, sensitive-information disclosure, improper output handling, and excessive agency.

Treat these sources as maintained catalogs. Your risk model, evaluation slices, thresholds, permissions, and incident actions must still match the actual product boundary.