9. Outer Product

Vector Outer Product

In previous lessons, we explored two methods for multiplying vectors: the Dot Product (which sums element products to yield a scalar) and the Hadamard Product (which multiplies corresponding components to yield another vector).

In this lesson, we introduce a third, incredibly powerful method of vector multiplication: the Outer Product. Instead of producing a scalar or a vector, the outer product between two vectors produces a complete matrix.

1. Mathematical Notation and Rules

Recall the differences in mathematical notation for these three vector multiplication methods:

• Dot Product (Inner Product): Written as $\mathbf{v}^{\text{T}}\mathbf{w}$. We transpose the first vector. The result is always a single scalar number.
• Hadamard Product: Written as $\mathbf{v} \odot \mathbf{w}$. It multiplies corresponding elements and results in another vector of the same size.
• Outer Product: Written as $\mathbf{v}\mathbf{w}^{\text{T}}$. We transpose the second vector. The result is always a matrix.

Dimensions and Flexibility

Unlike the dot and Hadamard products, the outer product does not require vectors of the same size!

• Let $\mathbf{v}$ be an $n$-dimensional column vector ($n \times 1$).
• Let $\mathbf{w}$ be an $m$-dimensional column vector ($m \times 1$).
• The outer product matrix $\mathbf{v}\mathbf{w}^{\text{T}}$ is defined as: $$(n \times 1) \times (1 \times m) = (n \times m)$$

The resulting matrix has $n$ rows (matching the dimension of the left vector $\mathbf{v}$) and $m$ columns (matching the dimension of the right vector $\mathbf{w}$).

2. Calculating the Outer Product

Let's compute the outer product of two vectors in $\mathbb{R}^3$:

Transposing $\mathbf{w}$ turns it into a row vector ($1 \times 3$): $$\mathbf{v}\mathbf{w}^{\text{T}} = \begin{bmatrix} v_1 \\ v_2 \\ v_3 \end{bmatrix} \begin{bmatrix} w_1 & w_2 & w_3 \end{bmatrix}$$

The resulting $3 \times 3$ matrix is populated by multiplying every element $i$ of the left vector by every element $j$ of the right vector:

3. Two Conceptual Lenses

There are two complementary ways to visualize how this matrix is constructed. Both yield the exact same result, but they provide different insights that are critical for advanced matrix factorization and machine learning algorithms.

Perspective 1: The Column Perspective

From the Column Perspective, we build the matrix column-by-column.

Each column in the resulting matrix is a vector-scalar multiplication of the entire column vector on the left ($\mathbf{v}$), scaled by a different corresponding element of the row vector on the right ($\mathbf{w}^{\text{T}}$):

• Column 1 is $\mathbf{v}$ scaled by $w_1$.
• Column 2 is $\mathbf{v}$ scaled by $w_2$.
• Column 3 is $\mathbf{v}$ scaled by $w_3$.

Because every column is a scaled version of the same vector $\mathbf{v}$, all columns of an outer product matrix are linearly dependent. In advanced linear algebra terms, this means an outer product matrix always has a rank of 1.

Perspective 2: The Row Perspective

From the Row Perspective, we build the matrix row-by-row.

Each row in the resulting matrix is a scalar-vector multiplication of the entire transposed row vector on the right ($\mathbf{w}^{\text{T}}$), scaled by a different corresponding element of the column vector on the left ($\mathbf{v}$):

• Row 1 is $\mathbf{w}^{\text{T}}$ scaled by $v_1$.
• Row 2 is $\mathbf{w}^{\text{T}}$ scaled by $v_2$.
• Row 3 is $\mathbf{w}^{\text{T}}$ scaled by $v_3$.

Again, because every row is a scaled version of the same vector $\mathbf{w}^{\text{T}}$, all rows are linearly dependent, reinforcing the rank-1 nature of the matrix.

4. Python Implementation

In NumPy, we calculate the outer product using the highly optimized np.outer() function.

To help build intuition, we also show a conceptually illustrative (but computationally inefficient) double-nested loop approach that populates each cell $M_{ij} = v_i \cdot w_j$ manually:

import numpy as np

# 1. Define two vectors (dimensions do not need to be equal)
v1 = np.array([  1, 2, 3 ])
v2 = np.array([ -1, 0, 1 ])

# 2. Compute using NumPy's optimized outer product
op_numpy = np.outer(v1, v2)

# 3. Compute manually using loops (conceptual walkthrough)
op_loop = np.zeros((len(v1), len(v2)))
for i in range(0, len(v1)):
    for j in range(0, len(v2)):
        op_loop[i, j] = v1[i] * v2[j]

print("--- Vector 1 ---")
print(v1)
print("\n--- Vector 2 ---")
print(v2)

print("\n--- NumPy Outer Product Matrix ---")
print(op_numpy)

print("\n--- Loop-Based Outer Product Matrix ---")
print(op_loop)

print(f"\nAre both results identical? {np.array_equal(op_numpy, op_loop)}")

--- Vector 1 ---
[1 2 3]

--- Vector 2 ---
[-1  0  1]

--- NumPy Outer Product Matrix ---
[[-1  0  1]
 [-2  0  2]
 [-3  0  3]]

--- Loop-Based Outer Product Matrix ---
[[-1.  0.  1.]
 [-2.  0.  2.]
 [-3.  0.  3.]]

Are both results identical? True

Key Takeaways

• Outer Product: Written as $\mathbf{v}\mathbf{w}^{\text{T}}$, which multiplies an $n \times 1$ column vector by a $1 \times m$ row vector to yield an $n \times m$ matrix.
• Flexible Dimensions: The input vectors do not need to have the same number of elements ($n \neq m$).
• Column Perspective: The matrix is constructed by columns where each column is the entire vector $\mathbf{v}$ scaled by an element $w_j$.
• Row Perspective: The matrix is constructed by rows where each row is the entire vector $\mathbf{w}^{\text{T}}$ scaled by an element $v_i$.
• Rank-1 Property: Because all rows (or columns) are scalar multiples of a single vector, an outer product matrix is always rank 1 (linearly dependent columns/rows).