Skip to content
Contact →

Claude

Updated
17 Sept 2026
2 min read
6 sections
Page 20 of 32
Deep Understanding of AI Large Language Models

Claude's Tokenizer: The Anthropic Approach 🎭

While OpenAI has standardized on cl100k_base, Anthropic uses a different vocabulary for Claude. By exploring Claude's tokenizer, we can see how different design choices affect token counts and model behavior.

Adapted from A deep understanding of AI language model mechanisms — https://udemy.com/course/dullms_x

1. Vocabulary Size

The first major difference is the size of the "dictionary":

  • GPT-4: ~100,000 tokens
  • Claude: ~65,000 tokens

A smaller vocabulary means the model has fewer "mental slots" for words, which usually results in slightly higher token counts for the same text.

2. Leading Whitespace Efficiency ⌨️

One of the most interesting features of modern BPE tokenizers is how they handle the space before a word. In Claude's tokenizer, a word and its "spaced" version are often different tokens.

python
from transformers import GPT2TokenizerFast

tokenizer = GPT2TokenizerFast.from_pretrained('Xenova/claude-tokenizer')

word1 = "hypothetical"
word2 = " hypothetical" # Note the leading space

print(f"'{word1}': {tokenizer.encode(word1)}")
print(f"'{word2}': {tokenizer.encode(word2)}")
Out · output
'hypothetical': [30678, 36881] (2 tokens)
' hypothetical': [44086] (1 token)

In this specific case, adding a space actually reduced the token count! The tokenizer has a dedicated token for " space + hypothetical", but not for the word alone.

3. Tokenizing Code & Math 🧬

Tokenizers also vary in how they "chunk" technical symbols. Let's look at how Claude handles a Python slice:

python
code = "targetActs[layeri,:,:,1]"
toks = tokenizer.encode(code)
print([tokenizer.decode(t) for t in toks])
Out · output
['target', 'Acts', '[', 'layer', 'i', ',:,', ':,', '1', ']']

Notice how Claude has a specific token for ,:, and :,. This optimization makes it much more efficient at "reading" multidimensional array notation in libraries like NumPy or PyTorch.

Summary: Why This Matters

Understanding these nuances helps you write more efficient prompts:

  1. Format Matters: Small changes in spacing or punctuation can change your token cost.
  2. Model Differences: A prompt that fits in Claude's context window might take up more (or less) space in GPT-4.
  3. Efficiency: Claude's 65k vocabulary is highly tuned for English and common code patterns, making it a powerful alternative to OpenAI's larger vocabularies.

Section Complete!

You've made it through the entire first section of the AI & LLM Tokenization series! We've covered:

  • The transition from Text to Numbers.
  • The mechanics of Byte Pair Encoding (BPE).
  • The "Strawberry Problem" and LLM limitations.
  • Statistical laws like Zipf's Law.
  • The differences between GPT, BERT, and Claude.

In the next section, we'll dive into Embeddings: how these token IDs are turned into high-dimensional vectors that represent human meaning!