Inside the Tokenizer

"@/components/blog/references";

Inside the Tokenizer 🔢

Tokenization is the foundational step in any Natural Language Processing (NLP) pipeline. It involves converting discrete text into numerical tokens that a model can process.

Adapted from A deep understanding of AI language model mechanisms — https://udemy.com/course/dullms_x

Lessons

1. Text to Numbers — /blog/ai-llms/tokenizer/1-text-to-numbers
2. Make a Tokenizer — /blog/ai-llms/tokenizer/2-coding-challenge-make-a-tokenizer
3. Preparing Text for Tokens — /blog/ai-llms/tokenizer/3-preparing-text-for-tokens
4. Tokenizing — /blog/ai-llms/tokenizer/4-tokenizing-the-time-machine
5. BPE Concepts — /blog/ai-llms/tokenizer/5-byte-pair-encoding-concepts
6. BPE Loop — /blog/ai-llms/tokenizer/6-coding-challenge-bpe-loop
7. GPT-4 Tokenizer — /blog/ai-llms/tokenizer/7-gpt4-tokenizer
8. Token Efficiency — /blog/ai-llms/tokenizer/8-coding-challenge-token-efficiency
9. The Strawberry Problem — /blog/ai-llms/tokenizer/9-the-strawberry-problem
10. Rapper Names — /blog/ai-llms/tokenizer/10-coding-challenge-rapper-names
11. BERT Tokenizer — /blog/ai-llms/tokenizer/11-bert-tokenizer
12. BERT Characters — /blog/ai-llms/tokenizer/12-coding-challenge-bert-chars
13. Token Translation — /blog/ai-llms/tokenizer/13-token-translation
14. Translators — /blog/ai-llms/tokenizer/14-coding-challenge-translator-functions
15. Token Compression — /blog/ai-llms/tokenizer/15-token-compression
16. Tokenization Across Languages — /blog/ai-llms/tokenizer/16-tokenization-across-languages
17. Zipf — /blog/ai-llms/tokenizer/17-coding-challenge-zipfs-law
18. Claude Tokenizer Variations — /blog/ai-llms/tokenizer/18-claude-tokenizer-variations

Core Concepts

Understanding the difference between tokens and embeddings is crucial for mastering LLMs.

What is a Token?

A Token is a piece of text that can be represented as an integer. It is the atomic unit that a language model processes.

Character: H e l l o → 39 68 75 75 78
Problem: >150k characters (and growing) makes for a massive, unmanageable vocabulary.
Problem: Ignores statistical regularities and semantic patterns in language.
Problem: Extremely memory-intensive, severely limiting the effective context window.
Subword: Hello → 1544 18798
Word: Hello → 15496

"Token" is a piece of text; the corresponding number is called a Token ID or Token Index.

Tokenization Comparison

Feature	Character	Subword	Word
Sequence Length	Long	Variable	Short
Vocab Size	Small	Medium	Very Large
OOV Handling	Yes	Mostly Yes	No
Semantic Meaning	No	Somewhat	Yes
Training Demands	Excessive	Low/Medium	Low

Character-level

Pros: Language-agnostic, handles misspellings, compact vocab.
Cons: Long sequences, little semantic meaning, slow learning, requires more data.

Word-level

Pros: Semantic relations, memory efficient, faster learning, human-interpretable.
Cons: Massive vocab size, language-dependent, struggles with Out-Of-Vocabulary (OOV) terms.

Byte Pair Encoding (BPE) Algorithm

BPE is the industry standard for subword tokenization. Here is the process:

Data Selection: Choosing the right training corpus.
Preprocessing: Cleaning text (removing non-ASCII, excessive whitespace, etc.).
Initialization: Starting with base characters or common subwords ("ing", "er").
Vocab Tuning: Experimenting with different vocabulary sizes.
Byte-level Merging: Merging based on bytes rather than characters (allows handling any text/encoding).
Rare Word Handling: Strategies to reduce OOV issues.
Post-training: Adjusting merges or adding special tokens.

Statistical Nature

Tokenization and embeddings are not arbitrary; they are statistical and based on training data.

Different training datasets will yield different tokenization protocols.
The process requires several decisions, parameter choices, and assumptions.
Embeddings require even more steps and choices than tokenization.

Module Overview

In this module, we explore the journey from raw text to numerical vectors:

Basic Tokenization: Splitting text into words and characters.
Vocabulary Creation: Building a unique lexicon from a corpus.
Encoding & Decoding: Implementing the mapping between text and integers.
Vectorization: Moving beyond integers to high-dimensional embeddings.

Key Take-home Points

Transformation Required: Text must be transformed into numbers before being fed into an LLM.
Token Flexibility: A chunk of text is a "token" and can be a character, subword, or full word.
Dense Representation: Embeddings are dense representations of tokens, unlike sparse one-hot encodings.
Learned Schemes: Tokenization and embeddings are learned from data; there are many ways to create these schemes (e.g., GPT vs. BERT).
Length Distribution: Token lengths follow a non-normal distribution, reflecting the Zipfian nature of language.
No Manual Splitting: With advanced BPE, there is often no need to manually process or split text before encoding, as the algorithm handles it natively.

Challenges of Tokenization

Efficiency vs. Effectiveness: Fewer tokens mean less memory but can lead to less effective generalization.
Training Overhead: More tokens require more training data but can increase information density.
Language Specificity: Statistical dependencies change across languages, requiring different tokenizers for different linguistic contexts.