Translator Fun: The Back-Translation Test

In the previous lesson, we learned the "Text Pivot" method. Now, let's turn that into a set of robust Python functions and see if we can perform a "Lossless" translation from GPT to BERT and back again.

Adapted from A deep understanding of AI language model mechanisms — https://udemy.com/course/dullms_x

1. Building the Translator Library

When translating to BERT, we have to be careful: the .encode() method automatically adds [CLS] and [SEP] tokens. If we're just translating a snippet, we might want to strip those out to keep the sequence clean.

python

import tiktoken
from transformers import BertTokenizer

bert_tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
gpt4_tokenizer = tiktoken.get_encoding('cl100k_base')

def bert_to_gpt4(bert_toks):
  """Translates BERT IDs to GPT-4 IDs"""
  text = bert_tokenizer.decode(bert_toks)
  return gpt4_tokenizer.encode(text)

def gpt4_to_bert(gpt4_toks):
  """Translates GPT-4 IDs to BERT IDs (stripping special tokens)"""
  text = gpt4_tokenizer.decode(gpt4_toks)
  bert_ids = bert_tokenizer.encode(text)
  return bert_ids[1:-1] # Remove [CLS] and [SEP]

2. The Round-Trip Test

Let's see if a sentence can survive the journey: GPT-4 $\to$ BERT $\to$ GPT-4.

python

original_text = "I still don't have a good quote here. Now it's too late."

# Initial GPT-4 tokens
gpt_initial = gpt4_tokenizer.encode(original_text)

# Journey to BERT
bert_ids = gpt4_to_bert(gpt_initial)
print(f"BERT IDs: {bert_ids}")

# Journey back to GPT-4
gpt_final = bert_to_gpt4(bert_ids)

print(f"Final Reconstruction: '{gpt4_tokenizer.decode(gpt_final)}'")

Out · output

BERT IDs: [1045, 2145, 2123, 1005, 1056, 2031, 1037, 2204, 5926, 2182, 1012, 2085, 2009, 1005, 1055, 2205, 2397, 1012]
Final Reconstruction: 'i still don't have a good quote here. now it's too late.'

3. The "Gotcha": Case Sensitivity

Notice something in the output? The word "I" became "i". Because we used bert-base-uncased, the translation process lowercased our entire string. This is a form of information loss! If you need to preserve case, you would need to use bert-base-cased.

Lesson: Translation is only as "lossless" as the most restrictive tokenizer in your chain.

Summary

Modularizing your translation logic makes it easy to move data between models, but you must always be aware of the "lowest common denominator"—in this case, BERT's lack of casing.

Next, we'll explore Token Compression and how it affects model performance!