Word Embeddings

If tokenization is the process of assigning IDs to words, Embedding is the process of giving those IDs a location in a mathematical "meaning space." In this module, we move from simple integers to the complex vector representations that power large language models.

Adapted from A deep understanding of AI language model mechanisms — https://udemy.com/course/dullms_x

Lessons

• 1. Pretrained Embeddings (GloVe) — /blog/ai-llms/embeddings/1-glove-embeddings
• 2. Wikipedia vs. Twitter — /blog/ai-llms/embeddings/2-wiki-vs-twitter
• 3. GPT-2 vs. BERT — /blog/ai-llms/embeddings/3-gpt2-vs-bert
• 4. Vector Math with Tokens — /blog/ai-llms/embeddings/4-vector-math
• 5. Cosine Similarity — /blog/ai-llms/embeddings/5-cosine-similarity
• 6. GPT-2 Similarities — /blog/ai-llms/embeddings/6-gpt2-similarities
• 7. The Unembedding Layer — /blog/ai-llms/embeddings/7-unembedding-layer
• 8. Position Embeddings — /blog/ai-llms/embeddings/8-position-embeddings
• 9. Position Explorations — /blog/ai-llms/embeddings/9-position-explorations
• 10. Learning Embeddings — /blog/ai-llms/embeddings/10-loss-function
• 11. The Training Loop — /blog/ai-llms/embeddings/11-embedding-training

What is an Embedding?

An Embedding is a dense numeric representation of a token. Unlike simple integers, embeddings allow for semantic relations to be represented mathematically.

Nope! LLM does not work with tokens. Text is converted into tokens, but LLMs work with embeddings. It's all about the embeddings :)

Example: Semantic Mapping (Simplified 2D)

In a real model, these are high-dimensional, but even in 2D, we can see relationships:

• dog = [ 0.5, 0.9 ]
• cat = [ 0.4, 0.7 ]
• ladybug = [ -0.7, 0.9 ]
• hippo = [ 1.0, -1.0 ]
• mosquito = [ -0.9, -0.9 ]

A 2D visualization of semantic relationships. Tokens with similar meanings (like 'dog' and 'cat') are clustered together in the embedding space.

_Note: dog and cat are numerically closer to each other than to mosquito, reflecting their semantic similarity._

The LLM Workflow

1. Text is converted into Tokens.
2. Tokens (IDs) must be converted into Embeddings before the LLM.
3. LLMs modify the embeddings for classification and generation.

Comparison of Embedding Models

Core Concepts

High-Dimensional Meaning

An embedding is not just a number—it's a vector (a list of numbers). Each dimension in that vector represents some abstract feature of the word's meaning.

• Dimensionality: Modern models use anywhere from 50 to 12,000+ dimensions.
• Proximity: Words with similar meanings are physically closer to each other in this high-dimensional space.

Measuring Similarity

How do we mathematically decide if two words are "similar"?

• Euclidean Distance: Measuring the straight-line distance between two points.
• Cosine Similarity: Measuring the angle between two vectors. This is the gold standard for NLP because it focuses on the direction of the meaning rather than the magnitude.

Semantic Arithmetic

Because embeddings are vectors, we can perform math on them:

King - Man + Woman ≈ Queen

This demonstrates that the "concept" of royalty and gender are encoded as specific directions in the embedding space.

Position Embeddings

Transformer models (like GPT) process all tokens in a sequence simultaneously. To understand the order of words, they use Position Embeddings—special vectors added to the word embeddings to "tag" their location in the sentence.

Module Overview

In this series, we explore:

1. Static Embeddings: Understanding traditional models like GloVe.
2. Model Architectures: Comparing how BERT and GPT-2 structure their internal weights.
3. Semantic Operations: Using linear algebra to explore relationships and similarities.
4. Training & Learning: Building and training our own embedding layers from scratch.

Key Take-home Points

• Meaning is Spatial: In AI, "meaning" is represented by coordinates in a high-dimensional space.
• Context Matters: Modern models use contextual embeddings where the same token ID can have different vectors depending on the surrounding text.
• Mathematical Precision: Semantic relationships (synonyms, analogies, opposites) can be calculated using standard linear algebra.
• The Bridge: Embeddings are the bridge between the discrete world of human language and the continuous world of neural network mathematics.