AI Gateway Patterns

Put the gateway and provider SDK on opposite sides of a clear boundary

A provider SDK is a client for one provider's API. It understands that provider's request fields, authentication mechanism, error types, stream events, usage response, and optional features. An AI gateway coordinates product policy across those clients.

Ownership by layer

Provider APIs are not interchangeable merely because several accept messages and return text. A common envelope can standardize the fields your product needs, but unsupported fields must fail explicitly. Never silently drop a safety option, tool constraint, response format, or cancellation signal to make an adapter appear compatible.

Build the decision from trusted context

The request body is untrusted input. A client may suggest a task or deadline, but it must not authoritatively set tenant_id, residency permission, plan tier, or remaining budget.

1. 1

   Authenticate

   Bind identity

   Verify the caller, derive tenant and plan from trusted identity data, and attach allowed data classes, regions, and policy version.

2. 2

   Classify

   Normalize intent

   Create a product-level envelope with task type, input blocks, required capabilities, output limit, deadline, and cancellation signal.

3. 3

   Authorize

   Filter routes

   Reject candidates that violate capability, context, tenant, safety, residency, health, quota, budget, or deadline constraints.

4. 4

   Execute

   Rank and invoke

   Score only eligible candidates, obtain the chosen adapter's credential just in time, invoke it, and preserve native response semantics.

Keep secret custody inside trusted infrastructure

• Store provider credentials in a secret manager or use workload identity when the platform supports it.
• Give each adapter only the provider, project, model, and region permissions it needs.
• Resolve credentials after policy chooses a route; never send them to the browser, model context, cache key, event payload, or ordinary log.
• Rotate and revoke credentials independently from application releases.
• Attribute usage to the authenticated tenant even when several tenants share one provider project.

The example keeps normalization, policy, route selection, adapter mapping, and usage reconciliation as separate responsibilities.

Route only among eligible candidates

Routing has two phases:

1. Eligibility: deterministic constraints decide which routes may serve the request.
2. Ranking: a versioned policy scores the remaining routes for quality, latency, and cost.

This order prevents an optimization score from overriding a hard rule. Fallback uses the same eligibility filter as the primary route. It must also meet a measured quality floor for the current task; "returns text" is not an acceptable fallback contract.

The model below uses fictional routes and illustrative prices. It teaches the decision mechanics, not provider performance claims.

Reserve budget before the call and reconcile after it

Gateway budgets should control both admission and accounting.

Estimate before admission

For an illustrative request with 1,000 input tokens, 300 maximum output tokens, a $1.00 per million input-token rate, and a $4.00 per million output-token rate:

• Estimated input cost: 1,000 / 1,000,000 x $1.00 = $0.0010
• Estimated output cost: 300 / 1,000,000 x $4.00 = $0.0012
• Estimated request cost: $0.0022, or $2.20 per 1,000 similar requests

Reserve against the maximum permitted output, not an optimistic average. Enforce tenant and route limits across requests per minute, input tokens per minute, output tokens per minute, concurrent streams, and currency budget because providers expose different quota dimensions.

Reconcile after every attempt

Record usage separately for the client request and each provider attempt:

• input, cached-input, output, reasoning, audio, or image units when the provider reports them;
• native model and version actually served;
• primary, retry, hedge, and fallback attempt cost;
• canceled or incomplete stream usage;
• provider request ID and the gateway's request and attempt IDs.

Do not reconstruct billable usage from text length when the provider returns authoritative usage. Do not assume cached tokens are counted or priced the same way across providers.

Distinguish two cache mechanisms

Bound retries, timeouts, streams, and circuit breakers

A timeout means the gateway did not observe completion before its deadline. It does not prove the provider did no work, and a retry can create another billable generation.

Retry by classified failure, not by exception count

• Retry only failures the adapter classifies as transient and only while the original deadline and retry budget have room.
• Respect native retry guidance such as Retry-After when present.
• Add exponential backoff with jitter and cap the total attempts.
• Reuse a gateway request ID to collapse duplicate client submissions. Do not claim that this makes separate provider attempts produce one deterministic completion.
• Charge every attempt to the same tenant budget and expose attempt amplification as a metric.

Make streaming cancellation an end-to-end signal

When a browser disconnects or the product cancels:

1. stop forwarding events to the client;
2. abort the gateway task;
3. ask the active adapter to cancel or close the native stream;
4. record whether the provider confirmed cancellation and any final usage;
5. prevent a canceled result from entering a response cache.

If an API cannot confirm cancellation, mark the outcome unknown and keep the budget reservation until usage is reconciled.

Scope circuit breakers to the failing route

A useful circuit key includes provider, model or deployment, region, operation, and sometimes tenant class. Opening one circuit should eject the unhealthy route without disabling unrelated models. Half-open probes must be few, observable, and excluded from normal fallback storms.

Apply safety and residency policy before and after inference

Safety and data residency are route constraints, not ranking hints.

• Before routing: classify the data, resolve the allowed region and network path, select the required safety profile, and deny requests with no compliant route.
• At the adapter: map the approved policy to native safety controls without weakening unsupported settings. Preserve native blocked, refused, filtered, and incomplete outcomes.
• After inference: apply product-level output checks, data-loss prevention, citation or grounding requirements, and task-specific release rules.
• During fallback: keep the same or stricter tenant, region, retention, safety, and quality contract. A fallback that leaves the allowed geography or loses required filtering is not eligible.
• In storage and telemetry: minimize prompts and outputs, redact sensitive fields, and apply retention by data class. Sampling must not bypass policy.

Regional availability and security controls differ by model and feature. Treat current provider documentation and your own contract as the source of truth; do not infer residency from a provider's global brand or from the gateway's deployment region.

Observe the request, every attempt, and the release policy

One client request may create zero attempts after a policy denial, one normal attempt, or several retry and fallback attempts. Give each level its own identifier and measurements.

Request-level signals

• authenticated tenant or a privacy-preserving tenant key;
• task and policy version, data class, requested capabilities, and deadline;
• eligibility count, selected route class, denial reason, and client-visible outcome;
• time to first token, total latency, cancellation, cache outcome, and final quality cohort.

Attempt-level signals

• provider, native model version, region, adapter version, and provider request ID;
• queue time, connection time, first-token time, stream duration, error class, and retry guidance;
• native usage fields, normalized cost, retry ordinal, fallback reason, and circuit state;
• safety or refusal category without logging sensitive content by default.

Track distributions by bounded dimensions. Keep request IDs and provider request IDs in traces, not metric labels. OpenTelemetry's GenAI conventions are useful vocabulary, but the GenAI attributes are still evolving and some content attributes can be sensitive.

Roll out routing policy with evaluation evidence

A gateway policy change can alter model quality, safety, cost, latency, and failure behavior even when no application code changes.

1. 1

   Evidence

   Evaluate offline

   Run task-specific and safety suites by tenant class, language, input length, tool use, and expected fallback route. Compare against a pinned control.

2. 2

   Observe

   Shadow safely

   Where policy permits, compute route decisions without serving the candidate response. Avoid duplicating restricted data or billable inference merely to create telemetry.

3. 3

   Contain

   Canary by cohort

   Expose a small, reversible share. Keep the stable policy live and cap retry amplification, fallback share, cost, safety failures, and quality regression.

4. 4

   Decide

   Expand or roll back

   Expand only after minimum evidence passes. Automatically stop or roll back on hard policy, safety, quality, or circuit-breaker gates.

Contain failure with narrow policy versions, tenant cohorts, regional routes, circuit keys, retry budgets, and canary percentages. A global switch that changes every tenant and provider at once has a larger blast radius and produces weaker diagnosis evidence.

Current primary and official references

• HTTP Semantics, RFC 9110: method semantics, status codes, retry guidance, and Retry-After.
• IETF Idempotency-Key HTTP Header Internet-Draft: current draft semantics for identifying retries of one request; it is an Internet-Draft, not a published RFC.
• OpenAI streaming responses and usage API reference: examples of provider-specific stream events and token accounting.
• Claude streaming messages, rate limits, and prompt caching: examples of native events, quota dimensions, retry headers, and cache accounting.
• Amazon Bedrock Converse API: a common message API that still exposes model-specific request fields where semantics differ.
• Google Cloud generative AI security controls: feature-specific regional and security-control availability.
• OpenTelemetry GenAI attributes: evolving telemetry vocabulary and warnings for sensitive content fields.

Provider documentation, model availability, prices, limits, and safety features change. Pin the documentation and contract version used by each adapter, and re-run qualification tests before changing routes.

Check your gateway decisions