15. Linear Independence

Linear Independence

In previous lessons, we learned that all possible linear combinations of two vectors can form a 2D plane. However, this is only true if those two vectors do not lie on the same line.

If they do lie on the same line, the linear combinations collapse, failing to create a second dimension. This brings us to a fundamental concept in linear algebra: Linear Independence.

The Concept: Informal Definitions & Examples

Linear independence is a property that describes an entire set of vectors, rather than any individual vector within that set.

1. Linearly Dependent Sets

A set of vectors is linearly dependent if at least one vector in the set can be expressed as a linear combination (a scaled sum) of the other vectors.

• Example A (Simple Scale): $$\mathbf{w}_1 = \begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix}, \quad \mathbf{w}_2 = \begin{bmatrix} 2 \\ 4 \\ 6 \end{bmatrix}$$ Here, the second vector is simply a scaled version of the first: $$\mathbf{w}_2 = 2\mathbf{w}_1$$ This set is linearly dependent.
• Example B (Linear Combination): Consider a set of three vectors $\{\mathbf{v}_1, \mathbf{v}_2, \mathbf{v}_3\}$. Even if it is not immediately obvious, the set is dependent if: $$\mathbf{v}_2 = 7\mathbf{v}_1 - 9\mathbf{v}_3$$

It is incorrect to say "$\mathbf{v}_2$ is the dependent vector." The property belongs to the set $\{\mathbf{v}_1, \mathbf{v}_2, \mathbf{v}_3\}$ as a whole. You could rewrite this relation to express $\mathbf{v}_1$ or $\mathbf{v}_3$ as combinations of the other two:

$$\mathbf{v}_1 = \frac{1}{7}\mathbf{v}_2 + \frac{9}{7}\mathbf{v}_3$$

2. Linearly Independent Sets

A set of vectors is linearly independent if no single vector in the set can be expressed as a linear combination of the others.

• Example C: $$\mathbf{u}_1 = \begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix}, \quad \mathbf{u}_2 = \begin{bmatrix} 2 \\ 4 \\ 7 \end{bmatrix}$$ To scale the first two components of $\mathbf{u}_1$ to $\mathbf{u}_2$, we would need a scalar of $2$ (since $1 \times 2 = 2$ and $2 \times 2 = 4$). However, $3 \times 2 = 6 \neq 7$.

Because no single scalar can map $\mathbf{u}_1$ directly to $\mathbf{u}_2$, this set is linearly independent.

Formal Algebraic Definition

The formal definition of linear dependence revolves around finding a non-trivial combination of vectors that equals the zero vector.

For a set of $n$ vectors $\{\mathbf{v}_1, \mathbf{v}_2, \dots, \mathbf{v}_n\}$, we set up the equation:

$$\mathbf{0} = \lambda_1\mathbf{v}_1 + \lambda_2\mathbf{v}_2 + \dots + \lambda_n\mathbf{v}_n, \quad \lambda_i \in \mathbb{R}$$

• Trivial Solution: Setting all coefficients to zero ($\lambda_1 = \lambda_2 = \dots = \lambda_n = 0$) always yields $\mathbf{0}$. We ignore this trivial case.
• Linear Dependence: If there exists at least one set of scalars where not all $\lambda_i$ are zero that satisfies the equation, the set is linearly dependent.
• Linear Independence: If the equation is satisfied only when every single scalar is zero ($\lambda_1 = \lambda_2 = \dots = \lambda_n = 0$), the set is linearly independent.

Derivation and Intuition

Why does the zero-vector equation define dependence? Let's rewrite the equation by isolating one of the vectors:

1. Subtract $\lambda_1\mathbf{v}_1$ from both sides: $$\lambda_1\mathbf{v}_1 = \lambda_2\mathbf{v}_2 + \dots + \lambda_n\mathbf{v}_n$$ (Note: The negative sign is absorbed into the arbitrary scalars $\lambda_i$)
2. Divide the entire equation by $\lambda_1$ (assuming $\lambda_1 \neq 0$): $$\mathbf{v}_1 = \frac{\lambda_2}{\lambda_1}\mathbf{v}_2 + \dots + \frac{\lambda_n}{\lambda_1}\mathbf{v}_n, \quad \lambda_1 \neq 0, \mathbf{v}_1 \neq \mathbf{0}$$

This algebraic derivation proves that if the zero-vector equation holds with non-zero scalars, at least one vector ($\mathbf{v}_1$) can be expressed as a linear combination of the remaining vectors.

Geometric Intuition

1. In 2D Space ($\mathbb{R}^2$)

• Co-linear Vectors (Dependent): Two vectors that lie on the same line point in the same 1D direction. Scaling one vector makes it shorter or longer, but can never change its direction to reach a point off the line. They are dependent.
• Non-colinear Vectors (Independent): Two vectors pointing in different directions are independent. By combining them, you can span the entire 2D plane.
• Three Vectors in $\mathbb{R}^2$ (Dependent): Any set of three vectors in $\mathbb{R}^2$ is guaranteed to be dependent because any two independent vectors already cover the entire 2D plane; the third vector is completely redundant.

2. In 3D Space ($\mathbb{R}^3$)

• Two Independent Vectors: Define a 2D plane passing through the origin.
• Three Vectors on the Same Plane (Dependent): If a third vector lies within the plane defined by the first two, it adds no new directional information. The set remains dependent, collapsing into a 2D subspace.
• Three Independent Vectors (Independent): If the third vector "pops out" of the plane (pointing in a new geometric dimension), the three vectors are independent and span the entire 3D ambient space.

Adding a new vector to a set maintains linear independence only if that new vector points in a different geometric dimension that is not already reachable using the existing vectors.

The Dimensionality Theorem

There is a fundamental theorem linking the number of vectors in a set ($M$) to the dimensionality of the ambient space ($\mathbb{R}^N$):

• Theorem: In $\mathbb{R}^N$, a set of $M$ vectors is guaranteed to be linearly dependent if $M > N$.
• Corollary: If $M \leq N$, the set could be linearly independent, but it is not guaranteed (e.g., two colinear vectors in $\mathbb{R}^3$).

If M > N  ==>  Always Linearly Dependent
If M <= N ==>  Could be Linearly Independent

The formal proof of this theorem involves constructing a matrix with the vectors as columns and evaluating the matrix rank, which we will study in later chapters.

Four Strategies to Determine Independence

If you need to check whether a set of vectors is linearly independent, follow these four strategies in order:

Summary Takeaways

• Set Property: Linear independence/dependence is a property of an entire set, not of individual vectors.
• Trivial Equation: Independence means that $\sum \lambda_i \mathbf{v}_i = \mathbf{0}$ has only the trivial solution ($\lambda_i = 0$).
• Dimension Generation: A set is independent if each vector points in a new geometric dimension not reachable by combining the others.
• Constraint: You can never have more independent vectors in a space than the number of its ambient dimensions ($M \leq N$).