AI Safety and Guardrails Systems

Start with a risk contract

A guardrail cannot be designed from a generic list of harmful words. It needs a risk contract that connects a use case, an affected party, an unsafe behavior, and an enforceable response.

For each risk, document:

• Affected party: the user, a person described by the output, an operator, a customer tenant, or the public.
• Unsafe behavior: disclosure, discriminatory treatment, harmful assistance, manipulation, unsupported claims, or an unauthorized tool action.
• System path: direct input, retrieved content, memory, model output, tool argument, or downstream side effect.
• Severity and reversibility: the impact of one event and whether the system can undo it.
• Response: allow, transform, refuse, request confirmation, route to review, disable a capability, or stop the rollout.
• Evidence owner: the team or person accountable for the policy, evaluation set, exception, and incident.

Put independent controls around the complete request path

No single classifier can protect every boundary. A classifier may miss an indirect prompt injection; a safe-looking answer may still carry a tool call with excessive authority; and an accurate model can still reveal data from the wrong tenant.

Separate model judgment from application authority

• Treat prompts, retrieved documents, model text, and tool arguments as untrusted data.
• Enforce authorization in deterministic code and downstream services, not in the prompt.
• Prefer narrow tools such as create_refund_draft over open-ended shell, SQL, browser, or HTTP access.
• Require a fresh confirmation for irreversible or externally visible actions.
• Define a non-actioning fallback when a classifier, policy service, model provider, or review queue is unavailable.

Trace the consequence, not just the model response

Use the lab to compare an ordinary answer, an indirect injection, a cross-tenant disclosure, and a high-value write. Change the authority envelope, approval state, and policy-service availability. Watch which independent boundary stops the consequence.

The important result is not whether the model recognized every attack. The important result is whether an unexpected model output can cross a deterministic trust boundary.

Implement a fail-closed runtime policy

The example below consumes the same scenario data as the runtime lab. It gives explicit precedence to policy availability, deny rules, sensitive output, capability scope, and human approval.

Production policy engines also need:

• Atomic policy and configuration versions attached to each decision.
• Short timeouts, bounded retries, and a documented degraded mode.
• Idempotency keys and compensation for retried side effects.
• Tenant-aware authorization at every data and tool boundary.
• Redacted, access-controlled evidence rather than unrestricted prompt logging.

Evaluate the system as a set of independent gates

A release evaluation must exercise the deployed path: model, prompt, retrieval, tools, filters, policy, and fallback behavior. A base-model benchmark does not prove that the application enforces tenant boundaries or handles a policy outage.

Freeze a versioned evidence package with:

• Representative normal, boundary, adversarial, multilingual, and accessibility cases.
• Critical slices for affected groups, system modes, tools, and high-severity failures.
• Expected policy behavior, severity, reviewer rationale, and adjudication status.
• Separate build, calibration, release, and hidden sets with leakage checks.
• Predeclared gates for misses, safe-user interruptions, queue capacity, and hard-stop failures.

Keep the error types visible

For a binary safety triage gate:

• True positive: an unsafe case is flagged.
• False negative: an unsafe case passes through.
• False positive: a safe case is interrupted.
• True negative: a safe case proceeds.

Precision and recall are useful, but neither describes the complete operating burden. Prevalence changes how many alerts are false positives, and a threshold that looks strong offline can still overwhelm the review queue.

Calibrate quality and review capacity together

The fixture below is synthetic and transparent; it does not claim universal detector performance. Choose a use case and one measured operating point, then vary available review capacity. The lab computes the exact confusion matrix and tests three independent release gates.

Do not collapse the result into one “safety score.” A critical recall failure, excessive interruption rate, or unserviceable review queue should remain visible as its own decision.

Measure fairness and human impact by slice

Safety interventions can create harm as well as prevent it. A filter may block one dialect more often, route one group to review more frequently, or produce longer appeal times for a small intersectional slice.

Track at least:

• Outcome, refusal, escalation, override, and appeal rates by relevant protected and intersectional slices.
• False-positive and false-negative rates with sample counts and uncertainty.
• Time to review, time to appeal, reviewer disagreement, and reversal rate.
• Coverage gaps where labels, consent, or sample size are insufficient for a claim.
• Changes after model, prompt, policy, data, or reviewer-guidance updates.

No fairness metric is universally correct. Choose the metric from the decision and harm model, involve domain experts and affected communities, and document conflicts between legitimate objectives.

Operate guardrails as a versioned control plane

1. 1

   Detect

   Observe

   Monitor policy outcomes, slice metrics, tool denials, overrides, user reports, drift, and queue health.

2. 2

   Limit

   Contain

   Disable a capability, lower exposure, switch to answer-only mode, or roll back the model and policy bundle.

3. 3

   Explain

   Investigate

   Replay the versioned path, identify the failed boundary, estimate affected users, and preserve access-controlled evidence.

4. 4

   Learn

   Improve

   Add regression cases, repair the control, review related paths, rehearse the response, and earn exposure again.

Record every consequential decision

Keep the request and tenant scope, model and prompt versions, retrieved-source identifiers, policy and tool versions, decision, reason code, approval, override, and final side-effect identifier. Apply minimization, retention, encryption, and access controls to this evidence.

Rehearse failure before production

• Policy service unavailable or returning stale configuration.
• Safety classifier latency spike or silent version mismatch.
• Review queue saturation during an abuse event.
• Indirect injection through retrieved content or tool output.
• Rollback where the model changes but the policy bundle does not.
• Incident response that must revoke a tool or credential immediately.

Use current guidance as a floor

• NIST AI Risk Management Framework Core organizes continuous risk work across Govern, Map, Measure, and Manage.
• NIST AI 600-1: Generative AI Profile adds cross-sector guidance for generative-AI risks, testing, monitoring, and incident response.
• OWASP LLM01:2025 Prompt Injection recommends least privilege, separation of untrusted content, deterministic validation, human approval, and adversarial testing.
• OWASP LLM06:2025 Excessive Agency distinguishes excessive functionality, permissions, and autonomy.
• NIST SP 1270 frames harmful bias as a socio-technical risk that must be identified, measured, and managed.

These resources are guidance, not a substitute for the legal, safety, security, and domain requirements of the system's actual context.