Skip to content
Contact →

Inside the GPT-4 Tokenizer

Updated
17 Sept 2026
2 min read
5 sections
Page 9 of 32
Deep Understanding of AI Large Language Models

Now that we've built our own BPE loop, let's look at how the pros do it. OpenAI uses a highly optimized BPE implementation called tiktoken. Specifically, GPT-4 uses the cl100k_base encoding.

Adapted from A deep understanding of AI language model mechanisms — https://udemy.com/course/dullms_x

1. Setting Up Tiktoken

The tiktoken library is designed for speed and accuracy. You can install it via pip and load the GPT-4 encoding immediately.

python
import tiktoken

# Load the GPT-4 tokenizer
tokenizer = tiktoken.get_encoding('cl100k_base')

# Check the vocabulary size
print(f"GPT-4 Vocabulary Size: {tokenizer.n_vocab}")
Out · output
GPT-4 Vocabulary Size: 100277

2. Encoding & Decoding

Let's see how GPT-4 breaks down a specific sentence. Notice how it handles common words vs. complex subwords.

python
text = "My name is Mike and I like toothpaste-flavored chocolate."

# Convert text to IDs
tokens = tokenizer.encode(text)
print(f"Token IDs: {tokens}")

# Inspect individual tokens
for t in tokens:
  print(f"ID {t:>6} -> '{tokenizer.decode([t])}'")
Out · output
Token IDs: [5159, 836, 374, 11519, 323, 358, 1093, 26588, 57968, 12556, 76486, 18414, 13]

ID   5159 -> 'My'
ID    836 -> ' name'
ID    374 -> ' is'
ID  11519 -> ' Mike'
ID    323 -> ' and'
ID    358 -> ' I'
ID   1093 -> ' like'
ID  26588 -> ' tooth'
ID  57968 -> 'paste'
ID  12556 -> '-fl'
ID  76486 -> 'avored'
ID  18414 -> ' chocolate'
ID     13 -> '.'

3. Key Observations 🔍

  1. Whitespace Inclusion: Notice the spaces at the start of tokens like ' name' and ' is'. GPT-4 includes the space as part of the following token, which is more efficient than having a separate "space" token.
  2. Subword Splitting: The word "toothpaste-flavored" was broken into 5 pieces: tooth, paste, -fl, avored.
  3. Case Sensitivity: 'My' (ID 5159) is a different token than 'my' (ID 741).

4. Visualizing Vocabulary Distribution

What kind of tokens are in this 100,000-word vocabulary? We can plot the lengths of the first 10,000 tokens to see the distribution.

python
import matplotlib.pyplot as plt
import numpy as np

# Get lengths of first 10,000 tokens
token_lengths = [len(tokenizer.decode([i])) for i in range(10000)]

plt.figure(figsize=(10, 4))
plt.hist(token_lengths, bins=range(1, 20), color='#7ed6df', edgecolor='white')
plt.title('Token Length Distribution (First 10k Tokens)')
plt.xlabel('Number of Characters')
plt.ylabel('Frequency')
plt.show()
Out · plot
The histogram shows that GPT-4 favors tokens that are 3-6 characters long, balancing compression and granularity.

The histogram shows that GPT-4 favors tokens that are 3-6 characters long, balancing compression and granularity.

Summary

GPT-4's tokenizer is the culmination of the BPE process we learned. It's:

  • Efficient: Uses a large vocabulary (100k) to compress common words.
  • Robust: Can handle any character sequence by breaking it into subwords or bytes.
  • Context-Aware: Includes spaces and punctuation to preserve the exact structure of the text.

In the next lesson, we'll dive into Token Efficiency and see how different models compare!