Skip to content
Contact →

BPE Encoding Loop

Updated
17 Sept 2026
2 min read
4 sections
Page 8 of 32
Deep Understanding of AI Large Language Models

In the previous lesson, we manually merged a single pair of tokens. Now, let's automate the process. We'll write a loop that iteratively finds and merges the most frequent pairs until we reach a target vocabulary size.

Adapted from A deep understanding of AI language model mechanisms — https://udemy.com/course/dullms_x

1. Defining the Helper Functions

To make our loop clean, we'll reuse the logic for counting pairs and merging them.

python
import numpy as np

def get_pair_stats(text_tokens):
  pairs = {}
  for i in range(len(text_tokens)-1):
    pair = text_tokens[i] + text_tokens[i+1]
    pairs[pair] = pairs.get(pair, 0) + 1
  return pairs

def update_vocab(token_pairs, current_vocab):
  # Find the most frequent pair
  idx = np.argmax(list(token_pairs.values()))
  new_token = list(token_pairs.keys())[idx]
  
  # Add to vocab with the next available ID
  current_vocab[new_token] = max(current_vocab.values()) + 1
  return current_vocab, new_token

def generate_new_token_seq(prev_text, new_token):
  new_text = []
  i = 0
  while i < len(prev_text):
    if i < len(prev_text)-1 and (prev_text[i] + prev_text[i+1]) == new_token:
      new_text.append(new_token)
      i += 2
    else:
      new_text.append(prev_text[i])
      i += 1
  return new_text

2. The Main BPE Loop

Now, we run the process iteratively. We'll start with our character-level text and grow the vocabulary to 25 tokens.

python
text = 'like liker love lovely hug hugs hugging hearts'
chars = sorted(list(set(text)))
vocab = { char: i for i, char in enumerate(chars) }
updated_text = list(text)

vocab_size_target = 25

while len(vocab) < vocab_size_target:
  # 1. Get stats
  pairs = get_pair_stats(updated_text)
  
  # 2. Update vocab with the best pair
  vocab, new_token = update_vocab(pairs, vocab)
  
  # 3. Re-tokenize the text
  updated_text = generate_new_token_seq(updated_text, new_token)
  
  print(f'Vocab Size: {len(vocab)} | Added: "{new_token}"')
Out · output
Vocab Size: 17 | Added: " h"
Vocab Size: 18 | Added: " l"
Vocab Size: 19 | Added: " hu"
Vocab Size: 20 | Added: " hug"
Vocab Size: 21 | Added: "ik"
Vocab Size: 22 | Added: "ike"
Vocab Size: 23 | Added: " lo"
Vocab Size: 24 | Added: " lov"
Vocab Size: 25 | Added: " love"

3. The Resulting Tokens

After 9 iterations, our text has been transformed from 46 individual characters into a sequence of subwords and spaces.

python
print(f"Final Tokens: {updated_text}")
Out · output
Final Tokens: ['l', 'ike', ' l', 'ike', 'r', ' love', ' love', 'l', 'y', ' hug', ' hug', 's', ' hug', 'g', 'i', 'n', 'g', ' h', 'e', 'a', 'r', 't', 's']
Efficiency Check! The word "love" is now a single token (' love'). The word "hugging" is composed of ' hug', 'g', 'i', 'n', and 'g'. We've successfully balanced common word efficiency with subword flexibility.

Key Takeaway

The BPE loop is a "greedy" algorithm. It always picks the most frequent pair in the current state of the text. This simple rule allows models like GPT to handle millions of unique words using a vocabulary of only ~50,000 tokens!