Skip to content
Contact →

Tokenizing

Updated
17 Sept 2026
2 min read
6 sections
Page 6 of 32
Deep Understanding of AI Large Language Models

Tokenizing 'The Time Machine' 🕰️

In the previous lesson, we built a tokenizer for a tiny corpus. Now, let's step up the game. We'll tokenize a full book—H.G. Wells' The Time Machine—and see how our basic "word-as-token" approach handles real-world scale and complexity.

Adapted from A deep understanding of AI language model mechanisms — https://udemy.com/course/dullms_x

1. Preparing the Corpus

First, we fetch the raw text from Project Gutenberg and apply the cleaning logic we developed earlier.

python
import requests
import re
import string
import numpy as np
import matplotlib.pyplot as plt

# Get raw text
text = requests.get('https://www.gutenberg.org/files/35/35-0.txt').text

# Replace common artifacts with spaces
strings2replace = [ '\\r\\n\\r\\nâ\\x80\\x9c','â\\x80\\x9c','â\\x80\\x9d','\\r\\n','â\\x80\\x94','â\\x80\\x99','â\\x80\\x98','_', ]
for str2match in strings2replace:
  text = re.compile(r'%s'%str2match).sub(' ',text)

# Final cleaning: remove non-ASCII, numbers, and lowercase everything
text = re.sub(r'[^\\x00-\\x7F]+', ' ', text)
text = re.sub(r'\\d+','',text).lower()

# Split into words (tokens)
words = re.split(fr'[{string.punctuation}\\s]+',text)
words = [item.strip() for item in words if item.strip() and len(item)>1]

# Create the vocab
vocab = sorted(set(words))
word2idx = {w:i for i,w in enumerate(vocab)}
idx2word = {i:w for i,w in enumerate(vocab)}

print(f"Total Words: {len(words)}")
print(f"Unique Tokens: {len(vocab)}")
Out · output
Total Words: 30698
Unique Tokens: 4589

2. Analyzing the Data 📊

What does the distribution of word lengths look like? Does a book like The Time Machine follow a specific "shape"?

python
# Count characters per word
numChars = np.zeros(len(words), dtype=int)
for i, w in enumerate(words):
  numChars[i] = len(w)

# Count frequency of each word length
charCounts = np.zeros(np.max(numChars)+1, dtype=int)
for i in range(len(charCounts)):
  charCounts[i] = np.sum(numChars==i)

# Plotting the histogram
plt.figure(figsize=(10, 5))
plt.bar(range(len(charCounts)), charCounts, color='#e6b2e6', edgecolor='k')
plt.xlabel('Number of Characters')
plt.ylabel('Token Count')
plt.title('Histogram of Word Lengths in The Time Machine')
plt.show()
Out · plot
The distribution of word lengths shows that most tokens are between 2 and 7 characters long, with a steep drop-off for longer words.

The distribution of word lengths shows that most tokens are between 2 and 7 characters long, with a steep drop-off for longer words.

3. The "Uh-Oh" Moment: Out of Vocabulary (OOV) ⚠️

What happens if we try to encode a sentence containing a word that wasn't in the book?

python
sentence = 'The space aliens came to Earth to steal watermelons and staplers.'
words_new = [w.strip() for w in re.split(fr'[,.\\s]+', sentence.lower()) if w.strip()]

# This will fail!
# encoder(words_new, word2idx) 
# KeyError: 'aliens'

If a word isn't in our word2idx dictionary, the program crashes. In the real world, models must handle words they've never seen before. We solve this by adding a special Unknown Token (<UNK>).

4. Solving OOV with <|unk|>

We update our vocabulary to include a catch-all entry for unknown words.

python
# Add an entry for unknown words
word2idx_new = word2idx.copy()
idx2word_new = idx2word.copy()

word2idx_new['<|unk|>'] = len(word2idx) + 1
idx2word_new[len(idx2word) + 1] = '<|unk|>'

def encoder_new(words, encode_dict):
  return np.array([ encode_dict[w] if w in encode_dict else encode_dict['<|unk|>'] for w in words ])

# Try again
tokenidx = encoder_new(words_new, word2idx_new)
decoded = ' '.join([idx2word_new[i] for i in tokenidx])

print(f"Decoded: {decoded}")
Out · output
Decoded: the space <|unk|> came to earth to <|unk|> <|unk|> and <|unk|>
Notice how "aliens", "watermelons", and "staplers" were all mapped to &lt;|unk|&gt;. While this prevents the code from crashing, we've lost the specific meaning of those words. This limitation is why modern models use Subword Tokenization (like BPE), which we'll explore in the next lesson!

Summary

  1. Scaling up to a full book increases the vocabulary but doesn't change the fundamental logic.
  2. Word length analysis reveals the statistical patterns of human language.
  3. OOV handling is critical; using &lt;UNK&gt; tokens is the simplest way to make a tokenizer robust to new inputs.