Token Management
Optimize token usage: cost management, rate limiting, usage tracking, and efficient token utilization strategies.
What is token management?
Token management is the control system that decides how a finite model context is allocated, how much output may be generated, and how token demand is admitted, measured, and stopped in production.
Tokens are model-specific units, not fixed words. A request may spend them on instructions, tool definitions, conversation history, retrieved evidence, the current user input, hidden or visible reasoning, and the response. The core invariant is simple: every token must fit inside an explicit request envelope and remain attributable to a useful outcome.
This matters for more than cost. A poor token policy can remove the evidence needed for a correct answer, leave too little room for the output, overflow a context window, exhaust token-per-minute capacity, increase queue latency, or amplify an outage through retries.
Control one request from evidence to outcome
The model call is only one stage. A deterministic controller owns admission, context assembly, output limits, and final usage accounting.
Intent
Classify the task
Identify required evidence, answer shape, risk, latency objective, and whether the request may be degraded.
Budget
Assemble the envelope
Reserve fixed instructions and output space before selecting history, tools, documents, or images.
Capacity
Admit bounded work
Check context size, rate limits, concurrency, deadline, and retry allowance before calling the provider.
Outcome
Reconcile actual usage
Record provider usage, stop reason, cache details, latency, cost, and whether the request achieved its task.
Treat the context window as shared working memory
A context window includes the information available while the model produces the next response. Its exact accounting is provider and model specific, but the engineering budget can be written as:
fixed instructions + tools + history + retrieved evidence + current input + output reserve <= context window
Give every token class an owner
Protect
Fixed policy
System instructions, safety rules, output schemas, and required tool contracts should be versioned and measured. Do not silently truncate them.
Protect
Current intent
The latest user request and any required attachments define the task. Reject or clarify if they cannot fit safely.
Reserve first
Output reserve
Allocate enough space for the requested answer, structured object, tool call, and any provider-side reasoning that shares the limit.
Compressible
Conversation history
Recent turns preserve continuity, while older turns can be summarized only when the summary is traceable and evaluated.
Selectable
Retrieved evidence
Rank chunks by authorization, relevance, freshness, diversity, and source authority instead of filling the window by count alone.
Expiring
Tool results
Keep the fields needed for the next decision. Move bulky, reproducible results to durable state and retain a reference.
Local tokenizers are useful admission estimates, not timeless billing guarantees. Message wrappers, tools, images, model revisions, and provider-specific accounting can change the final count. Reconcile each completed request against the provider's returned usage.
Allocate the answer before optional context
Use the context lab to vary history, retrieval, and output reserve. Compare policies by what they preserve, not only by whether the arithmetic fits.
Choose truncation by information value
Blind truncation is an availability mechanism, not a relevance policy. Dropping the oldest tokens may work for a casual conversation, but it can delete an early constraint, a security boundary, or the evidence that explains a later decision.
Apply a typed overflow policy
1 Contract
Protect non-negotiable content
Keep required instructions, the current request, authorization context, and the minimum output reserve intact.
2 State
Remove reproducible bulk
Replace old tool payloads and documents with stable references when they can be fetched again under the same permissions.
3 Retrieval
Select evidence by utility
Deduplicate overlapping chunks, preserve source diversity, and require the essential evidence set for grounded tasks.
4 History
Compact with a loss contract
Summarize older turns only when the system tracks provenance, unresolved commitments, and the summary version.
Reject or ask for clarification when protected content cannot fit. Silent clipping is especially dangerous for structured output, legal or policy constraints, tool arguments, and multi-step workflows whose earlier state changes later authority.
Caching changes price and latency, not context capacity
Prompt caching can avoid recomputing an identical prefix, but cached tokens still occupy the model's context. Structure stable instructions and examples before variable user content when the provider matches exact prefixes, then measure cache reads and writes from returned usage.
- Version cached prefixes with the prompt, tool schema, policy, tenant boundary, and model.
- Never treat a cache hit as authorization to reuse private content for another principal.
- Measure task quality alongside cache hit rate; a cheap stale prefix can still produce a wrong answer.
- Keep pricing and cache multipliers in configuration because provider terms change.
Convert request budgets into service capacity
Context fit is a per-request constraint. Production capacity couples many requests through token-per-minute limits, concurrency, generation speed, queues, and retries.
For an initial planning model:
token demand per minute = requests per minute x expected tokens per attempt x attempts per requestrequired concurrency ~= requests per second x service time in secondsservice time ~= time to first token + generated tokens / generation ratecost per successful task = all attempt costs / successful task count
These equations are estimates. Calibrate them with p50, p95, and p99 measurements by route and output profile. Output length often dominates generation time, while streaming improves perceived latency without reducing the underlying work already generated.
Put a failure boundary around token pressure
The capacity lab connects output length, traffic, provider limits, queues, and retry behavior. Push the workload past its envelope and compare bounded admission with retry amplification.
Build one admission ledger for every attempt
The application should make one deterministic decision before a provider call and another after it completes.
Before the call, record
- request, tenant, feature, model, prompt, tool, and policy versions;
- estimated input tokens, output reserve, deadline, and maximum attempts;
- the selected truncation, compaction, retrieval, cache, and degradation policies;
- reserved token-rate and concurrency capacity;
- a cancellation token shared by the queue, provider call, and any downstream tools.
After the call, reconcile
- actual input, cached input, reasoning, output, and total tokens when exposed;
- queue time, time to first token, generation time, and end-to-end latency;
- provider request ID, finish or stop reason, retry count, and ambiguous completion state;
- estimated price from versioned rate configuration;
- task outcome, quality signals, and whether the user abandoned the response.
Do not charge only the winning attempt. Retries, rejected candidates, timed-out calls, and work completed after client cancellation all consume capacity.
Operate on outcomes, not token totals alone
Monitor four views together
- Context health: preflight estimate error, overflow rejection, truncation volume, summary age, retrieved evidence retained, and output-limit stop reasons.
- Capacity health: input and output tokens per minute, active generation concurrency, queue age, admission rate, shedding, cancellations, and retries.
- Unit economics: cost per request, successful task, tenant, feature, route, cache state, and model-policy version.
- Quality: task success, grounded evidence coverage, schema validity, human escalation, rework, and abandonment by token-budget band.
Alert on coupled failures
- token demand rises while request rate stays flat, indicating prompt, retrieval, tool, or output growth;
- p95 output length approaches the configured maximum, suggesting frequent clipping or uncontrolled verbosity;
- queue age and retries rise together, which can turn a slow dependency into a retry storm;
- cache hit rate rises while quality falls, suggesting stale or incorrectly scoped prefixes;
- local estimates drift from billed usage after a model or tool-schema change;
- cost per request falls while cost per successful task rises, indicating harmful degradation.
Recognize the common failure modes
- No output reserve: input fits, but generation ends early or cannot emit the required schema. Reserve the answer before optional context.
- Newest-is-best truncation: an early constraint disappears. Rank content by decision value and protect typed invariants.
- Retrieval stuffing: similar chunks crowd out diverse evidence. Deduplicate and cap per-source contribution.
- Summary drift: repeated compaction turns a mistaken summary into durable state. Keep provenance and periodically rebuild from authoritative events.
- Caching as capacity: cached prefixes are assumed not to count toward the window or rate limit. Treat caching as a provider-specific compute and billing optimization.
- Unbounded retries: timeouts create more token demand against an already saturated dependency. Use retry budgets, jitter, cancellation, and load shedding.
- Static price constants: a model launch changes economics without a code deployment. Load effective rates from versioned configuration.
- Token-only optimization: shorter context reduces cost but removes the fact needed to succeed. Measure cost per successful, policy-compliant outcome.
Primary sources behind the design
- OpenAI's token-counting cookbook explains model-specific encoding and warns that local chat counts are estimates that can change with message and tool formatting.
- OpenAI prompt caching guidance documents exact-prefix matching and usage fields for cached tokens; it also makes clear that caching behavior and billing must be measured rather than assumed.
- OpenAI latency optimization guidance identifies output generation as a major latency driver and distinguishes real streaming improvements from cosmetic loading states.
- OpenAI rate-limit guidance describes request and token limits as separate operating constraints that applications must handle.
- Anthropic's context-window documentation defines the context as working memory, shows that inputs and outputs share it, and states that cached prefixes still occupy context capacity.
- Anthropic's token-counting documentation demonstrates provider-side preflight counting and warns that tokenization can change across model families.
The numerical scenarios in the two labs are illustrative system envelopes, not claims about a current commercial model or price. Replace them with measured route data before capacity planning.