Tokenization Fundamentals

Follow text through the tokenizer pipeline

A tokenizer pipeline is an ordered set of transformations. Production libraries may combine or rename stages, but the responsibilities remain useful for reasoning and debugging.

1. 1

   Unicode policy

   Normalize text

   Preserve the input or apply a declared transformation such as NFC, NFKC, case folding, or accent handling. This step can change code points and offsets.

2. 2

   Pre-tokenize

   Expose boundaries

   Mark candidate regions such as whitespace, punctuation, digits, or raw bytes. These are candidates, not necessarily final tokens.

3. 3

   Segment

   Choose vocabulary pieces

   Apply learned BPE merge ranks, WordPiece rules, or unigram scores to select pieces that exist in the fixed vocabulary.

4. 4

   Post-process

   Insert model structure

   Add trusted beginning, ending, separator, role, or padding IDs and produce masks, type IDs, and offsets expected by the model.

The final IDs index an embedding matrix. If the vocabulary has V entries and each embedding has width d, the lookup table has shape V x d. Token ID i selects row E[i]; the model never receives the token spelling as a separate hint.

See normalization, bytes, pieces, and offsets together

Normalization chooses which Unicode distinctions survive before vocabulary lookup. The same visible text can have different code-point sequences: for example, é can be one precomposed code point or e followed by a combining accent. UTF-8 then represents those code points as bytes.

The lab uses fixed outputs from two small teaching vocabularies. They illustrate BPE-like merge ordering and unigram-style piece selection, but they do not reproduce Hugging Face, SentencePiece, tiktoken, or any vendor tokenizer.

Distinguish vocabulary learning from runtime segmentation

A vocabulary is the finite map from token spellings or byte sequences to IDs. The training algorithm decides what enters that map; the runtime algorithm decides how one input is covered by those entries.

Vocabulary size trades embedding rows against sequence length and coverage. It does not have one universally optimal value: adding pieces can help one language or code pattern while consuming capacity that could serve another.

For the full BPE training loop, continue to BPE Algorithm Deep Dive.

Treat special tokens and boundaries as typed data

A special token is an ID with model-level meaning, such as beginning of sequence, end of sequence, padding, masking, separators, or chat roles. It is not merely decorative text.

• Insert role and control IDs through a trusted template or tokenizer API, not by concatenating user-provided strings into a serialized prompt.
• Encode untrusted text as ordinary content. A literal string such as <|system|> must not gain system authority merely because it resembles a control marker.
• Configure whether special-looking strings are allowed, rejected, or treated as ordinary text. Library defaults differ.
• Apply an attention mask so padding IDs do not behave like content, and verify the padding side expected by the model and serving path.
• Reserve structural tokens when checking context limits; wrappers, tool schemas, and separators consume space too.

Prompt injection can still be expressed in ordinary language, so safe token handling is only one boundary. The application must also separate trusted instructions from untrusted content, restrict tools, validate arguments, and enforce authorization outside the model.

Offsets need a named unit

An offset identifies the source span associated with a token. APIs may count UTF-8 bytes, Unicode code points, UTF-16 code units, characters in normalized text, or positions in original text. These units diverge for multibyte characters, combining marks, and emoji.

Record the offset unit and whether spans refer to original or normalized text. This is essential for named-entity labels, redaction, citations, syntax highlighting, and reconstructing failures. Never compare offsets from two libraries until their units and normalization stages are aligned.

Measure multilingual text and code separately

A tokenization slice is a meaningful subset of traffic measured on its own. An average dominated by English prose can hide severe expansion for another script, source code, identifiers, or unusual whitespace.

• Multilingual text: vocabulary allocation follows training data. Low-resource scripts may fall back to smaller pieces or bytes, increasing sequence length even when the human-visible text is short.
• Languages without spaces: whitespace pre-tokenization is not a universal word boundary. Raw-sentence tokenizers can learn pieces without assuming spaces separate every semantic unit.
• Combining marks and compatibility characters: normalization can improve consistency, but NFKC or case folding can erase distinctions that matter to identifiers, passwords, legal text, or code.
• Source code: indentation, newlines, casing, underscores, operators, and long identifiers carry structure. A tokenizer trained mostly on prose may split them inefficiently or obscure useful boundaries.
• Numbers and structured text: dates, decimals, UUIDs, JSON escapes, and log syntax need dedicated tests because tiny formatting changes can sharply change token count.

Do not use a universal "four characters per token" rule for admission control. Count with the exact production tokenizer and report distributions by language and content type.

Turn token count into a context and cost decision

A context window is the maximum sequence the model can process for one request under a particular serving contract. The budget includes every input token plus the generated output allowance:

system + tools + history + retrieved evidence + user input + output reserve <= context window

The next lab uses fixture token counts and an illustrative price card. It is a planning model, not a provider quote. Change the vocabulary profile, context size, output reserve, and overflow strategy to see different operational consequences.

Choose truncation or chunking deliberately

Truncation drops tokens to satisfy a hard limit. Chunking creates multiple bounded sequences. Both are information-selection policies, not neutral formatting operations.

When truncating

• Protect trusted instructions, the current task, and the output reserve before optional history.
• Decide whether removal occurs on the left, right, or by semantic priority. Right truncation can lose a conclusion; left truncation can lose initial constraints.
• Return metadata that states what was removed. Silent truncation makes missing evidence look like model failure.
• Test the complete post-template sequence because special tokens and tool schemas can push an apparently valid input over the limit.

When chunking

• Split on document structure first, then enforce token limits with the production tokenizer.
• Use overlap only when cross-boundary evidence needs it; overlap is duplicated input and therefore duplicated latency and cost.
• Keep stable source offsets and document IDs so results can be traced and deduplicated.
• Reserve space for the query, instructions, metadata, and output in every chunk request.
• Prefer retrieval or hierarchical summarization when processing every chunk would be wasteful or would destroy global structure.

Evaluate and version the complete contract

A tokenizer evaluation compares a pinned candidate against a pinned baseline on representative and adversarial fixtures. Quality includes semantic coverage, alignment, sequence length, compatibility, and safe structure.

Review medians and tails, not only one average. A practical gate can fail a release when any protected slice exceeds its token-expansion budget, offsets stop aligning, special-token fingerprints change unexpectedly, truncation rises, or downstream task quality regresses.

Keep these mental models

• Models consume token IDs because IDs index learned embedding rows; token spellings are not the model interface.
• Normalization, byte encoding, segmentation, and special-token insertion are separate stages with separate failure modes.
• BPE merges and unigram scores are learned from a corpus; neither produces universal linguistic boundaries.
• Token count depends on the exact tokenizer, language, code pattern, template, and library version.
• Truncation chooses what information disappears, while chunking duplicates boundaries and work.
• The checkpoint, tokenizer artifacts, special-token map, and template must be tested and released as one bundle.

Primary sources and official libraries

• Unicode Standard Annex #15: Unicode Normalization Forms defines NFC, NFD, NFKC, and NFKD and their stability requirements.
• Sennrich, Haddow, and Birch: Neural Machine Translation of Rare Words with Subword Units introduces the influential subword BPE approach for neural machine translation.
• Kudo: Subword Regularization presents unigram language-model segmentation and sampling from multiple segmentations.
• Kudo and Richardson: SentencePiece describes language-independent training from raw sentences.
• Hugging Face Tokenizers documentation documents normalization, models, post-processing, IDs, and offset alignment in the official library.
• OpenAI tiktoken repository documents its byte-pair encoding implementation and model-specific encoding lookup.