++++

AI

Mar 2025×5 min read

What happens when you cycle text through different tokenizers? Put your translation functions to the test and see if your text survives the round trip.

Translator Fun: The Back-Translation Test 🔄

Driptanil DattaSoftware Developer

Translator Fun: The Back-Translation Test 🔄

In the previous lesson, we learned the "Text Pivot" method. Now, let's turn that into a set of robust Python functions and see if we can perform a "Lossless" translation from GPT to BERT and back again.

1. Building the Translator Library

When translating to BERT, we have to be careful: the .encode() method automatically adds [CLS] and [SEP] tokens. If we're just translating a snippet, we might want to strip those out to keep the sequence clean.

import tiktoken
from transformers import BertTokenizer
 
bert_tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
gpt4_tokenizer = tiktoken.get_encoding('cl100k_base')
 
def bert_to_gpt4(bert_toks):
  """Translates BERT IDs to GPT-4 IDs"""
  text = bert_tokenizer.decode(bert_toks)
  return gpt4_tokenizer.encode(text)
 
def gpt4_to_bert(gpt4_toks):
  """Translates GPT-4 IDs to BERT IDs (stripping special tokens)"""
  text = gpt4_tokenizer.decode(gpt4_toks)
  bert_ids = bert_tokenizer.encode(text)
  return bert_ids[1:-1] # Remove [CLS] and [SEP]

2. The Round-Trip Test

Let's see if a sentence can survive the journey: GPT-4 $\to$ BERT $\to$ GPT-4.

original_text = "I still don't have a good quote here. Now it's too late."
 
# Initial GPT-4 tokens
gpt_initial = gpt4_tokenizer.encode(original_text)
 
# Journey to BERT
bert_ids = gpt4_to_bert(gpt_initial)
print(f"BERT IDs: {bert_ids}")
 
# Journey back to GPT-4
gpt_final = bert_to_gpt4(bert_ids)
 
print(f"Final Reconstruction: '{gpt4_tokenizer.decode(gpt_final)}'")

BERT IDs: [1045, 2145, 2123, 1005, 1056, 2031, 1037, 2204, 5926, 2182, 1012, 2085, 2009, 1005, 1055, 2205, 2397, 1012]
Final Reconstruction: 'i still don't have a good quote here. now it's too late.'

3. The "Gotcha": Case Sensitivity

Notice something in the output? The word "I" became "i". Because we used bert-base-uncased, the translation process lowercased our entire string. This is a form of information loss! If you need to preserve case, you would need to use bert-base-cased.

💡 Summary

Modularizing your translation logic makes it easy to move data between models, but you must always be aware of the "lowest common denominator"—in this case, BERT's lack of casing.

Next, we'll explore Token Compression and how it affects model performance!