++++

AI

Mar 2025×10 min read

Can models talk to each other directly through token IDs? Learn why token translation is necessary and how to safely map between GPT and BERT.

Token Translation: Mapping Between Models 🌍

Driptanil DattaSoftware Developer

Token Translation: Mapping Between Models 🌍

Every model has its own "language" (vocabulary). GPT-4's ID 25977 means "purple", but in BERT, that same ID is a placeholder for [unused25977]. If you want to take data from one model and use it in another, you can't just copy the numbers—you have to translate.

1. The Wrong Way ❌

A common mistake is assuming that "tokens are tokens." Let's see what happens if we take GPT-4 tokens and try to decode them with BERT.

import tiktoken
from transformers import BertTokenizer
 
gpt4 = tiktoken.get_encoding('cl100k_base')
bert = BertTokenizer.from_pretrained('bert-base-uncased')
 
text = "Hello, my name is Mike and I like purple."
 
# GPT-4 tokens
gpt_ids = gpt4.encode(text)
# [9906, 11, 856, 836, 374, 11519, 323, 358, 1093, 25977, 13]
 
# Decoding with BERT
print(bert.decode(gpt_ids))

"lately [unused10] [unused851] [unused831] [unused369] decent [unused318] [unused353] ¾ olympian [unused12]"

The result is gibberish. GPT's "hello" is BERT's "lately," and everything else is a mess.

2. The Right Way: The Pivot 🔄

To translate, you must use text as the bridge.

1. Source IDs $\to$ Text (Decode)
2. Text $\to$ Target IDs (Encode)

# 1. Start with GPT IDs
gpt_ids = gpt4.encode("Hello, my name is Mike and I like purple.")
 
# 2. Pivot through string format
bridge_text = gpt4.decode(gpt_ids)
 
# 3. Re-encode for BERT
bert_ids = bert.encode(bridge_text)
 
print(bert.decode(bert_ids))

"[CLS] hello, my name is mike and i like purple. [SEP]"

3. Discrepancies & "Information Loss"

Even with the correct method, translation isn't perfect. BERT might "lose" certain details that GPT preserves:

whitespace_text = "start\r\n\n\t\t\tend"
 
# GPT-4 preserves exactly what you gave it
print(f"GPT4: {repr(gpt4.decode(gpt4.encode(whitespace_text)))}")
 
# BERT strips most whitespace and formatting
print(f"BERT: {repr(bert.decode(bert.encode(whitespace_text), skip_special_tokens=True))}")

GPT4: 'start\r\n\n\t\t\tend'
BERT: 'start end'

💡 Key Takeaways

• Vocabulary Incompatibility: No two models (usually) share the same ID-to-string mapping.
• Granularity: GPT-4 (100k vocab) is more efficient. The same sentence might be 96 tokens in GPT but 160 tokens in BERT.
• Sanitization: BERT uncased will convert everything to lowercase, meaning you lose casing information during the pivot!

💡 Summary

Translation is about moving information between different numerical representations of the same underlying concept. Understanding this "pivot" is crucial when building multi-model pipelines or migrating data between different AI systems.