Skip to content
Contact →

Token Translation: Mapping Between Models

Updated
17 Sept 2026
2 min read
5 sections
Page 15 of 32
Deep Understanding of AI Large Language Models

Every model has its own "language" (vocabulary). GPT-4's ID 25977 means "purple", but in BERT, that same ID is a placeholder for [unused25977]. If you want to take data from one model and use it in another, you can't just copy the numbers—you have to translate.

Adapted from A deep understanding of AI language model mechanisms — https://udemy.com/course/dullms_x

1. The Wrong Way ❌

A common mistake is assuming that "tokens are tokens." Let's see what happens if we take GPT-4 tokens and try to decode them with BERT.

python
import tiktoken
from transformers import BertTokenizer

gpt4 = tiktoken.get_encoding('cl100k_base')
bert = BertTokenizer.from_pretrained('bert-base-uncased')

text = "Hello, my name is Mike and I like purple."

# GPT-4 tokens
gpt_ids = gpt4.encode(text)
# [9906, 11, 856, 836, 374, 11519, 323, 358, 1093, 25977, 13]

# Decoding with BERT
print(bert.decode(gpt_ids))
Out · output
"lately [unused10] [unused851] [unused831] [unused369] decent [unused318] [unused353] ¾ olympian [unused12]"

The result is gibberish. GPT's "hello" is BERT's "lately," and everything else is a mess.

2. The Right Way: The Pivot 🔄

To translate, you must use text as the bridge.

  1. Source IDs →\to Text (Decode)
  2. Text →\to Target IDs (Encode)
python
# 1. Start with GPT IDs
gpt_ids = gpt4.encode("Hello, my name is Mike and I like purple.")

# 2. Pivot through string format
bridge_text = gpt4.decode(gpt_ids)

# 3. Re-encode for BERT
bert_ids = bert.encode(bridge_text)

print(bert.decode(bert_ids))
Out · output
"[CLS] hello, my name is mike and i like purple. [SEP]"

3. Discrepancies & "Information Loss"

Even with the correct method, translation isn't perfect. BERT might "lose" certain details that GPT preserves:

python
whitespace_text = "start\r\n\n\t\t\tend"

# GPT-4 preserves exactly what you gave it
print(f"GPT4: {repr(gpt4.decode(gpt4.encode(whitespace_text)))}")

# BERT strips most whitespace and formatting
print(f"BERT: {repr(bert.decode(bert.encode(whitespace_text), skip_special_tokens=True))}")
Out · output
GPT4: 'start\r\n\n\t\t\tend'
BERT: 'start end'

Key Takeaways

  • Vocabulary Incompatibility: No two models (usually) share the same ID-to-string mapping.
  • Granularity: GPT-4 (100k vocab) is more efficient. The same sentence might be 96 tokens in GPT but 160 tokens in BERT.
  • Sanitization: BERT uncased will convert everything to lowercase, meaning you lose casing information during the pivot!
Always decode and re-encode. Never attempt to build a manual mapping table between tokenizers—it is computationally expensive and fragile as vocabularies update.

Summary

Translation is about moving information between different numerical representations of the same underlying concept. Understanding this "pivot" is crucial when building multi-model pipelines or migrating data between different AI systems.