4. Vector Dot Product

Vector Dot Product

There are four primary ways to multiply two vectors: the Hadamard (element-wise) product, the dot product, the cross product, and the outer product. In this lesson, we focus on the Dot Product (also known as the Scalar Product).

The dot product is arguably the single most important computational block in all of linear algebra. It is the core mathematical engine driving convolution, correlation, Fourier transforms, cosine similarity, and modern deep learning linear layers. In this lesson, we will cover the dot product from a rigorous algebraic perspective.

1. Notations and Definition

Because the dot product maps two vectors of the same size to a single number (a scalar), it is often referred to as the scalar product.

The result of a dot product is represented as a lowercase Greek letter (e.g., $\alpha$). There are four common notations representing the dot product between two column vectors $\mathbf{a}$ and $\mathbf{b}$ in $n$-dimensional space:

1. Dot Notation: $\mathbf{a} \cdot \mathbf{b}$
2. Inner Product (Angled Brackets): $\langle \mathbf{a}, \mathbf{b} \rangle$ (commonly used when working with continuous functions and general inner product spaces).
3. Transpose Matrix Notation: $\mathbf{a}^{\text{T}}\mathbf{b}$ (the standard textbook convention in linear algebra, treating vectors as single-column matrices).
4. Summation Notation: $$\mathbf{a} \cdot \mathbf{b} = \sum_{i=1}^n \mathbf{a}_i\mathbf{b}_i$$

2. The Dimensionality Constraint

Just like vector addition, the dot product is only defined between vectors that share the exact same dimensionality.

If you attempt to compute the dot product between a $5$-dimensional vector and a $3$-dimensional vector, the operation is undefined. While the first three index multiplications would succeed, you would eventually encounter "5 times nothing" at the fourth element because there are no corresponding elements left in the shorter vector.

3. Algebraic Step-by-Step Walkthrough

Let's compute the dot product of two $5$-dimensional column vectors $\mathbf{v}$ and $\mathbf{w}$:

To find the dot product $\mathbf{v}^{\text{T}}\mathbf{w}$, we multiply the corresponding elements at each index and sum them up:

4. Python Implementation: Four Ways to Compute

In data science and python programming, there are four distinct methods to compute the dot product of two vectors using NumPy. Let's compare them:

import numpy as np

# Define two 6D vectors
v1 = np.array([1, 2, 3, 4, 5, 6])
v2 = np.array([0, -4, -3, 6, 5, 1])

# Method 1: Element-wise multiply then sum
dp1 = sum(np.multiply(v1, v2))

# Method 2: Standard NumPy dot function
dp2 = np.dot(v1, v2)

# Method 3: Matrix Multiplication (np.matmul or @ operator)
dp3 = np.matmul(v1, v2)
dp3_modern = v1 @ v2  # Equivalent modern operator

# Method 4: Explicit For Loop (For learning / low-level code)
dp4 = 0
for i in range(len(v1)):
    dp4 += v1[i] * v2[i]

print(f"Method 1: {dp1}")
print(f"Method 2: {dp2}")
print(f"Method 3: {dp3} (using @: {dp3_modern})")
print(f"Method 4: {dp4}")

Method 1: 38
Method 2: 38
Method 3: 38 (using @: 38)
Method 4: 38

Why use np.dot or @?

While the for loop (Method 4) is great for understanding the mathematical formula, it is slow and inefficient in production. NumPy's np.dot (Method 2) and the @ operator (Method 3) utilize highly optimized, compiled C code (via BLAS/LAPACK libraries) that runs orders of magnitude faster through vectorization!

Key Takeaways

• Vector to Scalar: The dot product multiplies elements index-wise and sums them, mapping two vectors into a single scalar value.
• Dimensionality Bound: Vectors must share the exact same length for a dot product to be defined.
• Highly Optimized: In Python, always prefer np.dot() or the @ matrix multiplication operator over iterative for loops to leverage compiled vectorization.