Inner Product Spaces

Up to this point, our vector spaces have been “bare.” We can add vectors and scale them, but we have no notion of how long a vector is, or what the angle between two vectors might be. In bare linear algebra, there is no “perpendicular.”

Inner product spaces equip vector spaces with this geometric structure. This allows us to define limits, identify the “closest” approximation to a signal, and decompose data into independent components.

Geometry from Algebra

An inner product is a function that takes two vectors and returns a scalar. For $\mathbb{R}^n$, the standard inner product is the dot product $u \cdot v = \sum u_i v_i$. To generalize this to any vector space $V$ over $\mathbb{R}$ or $\mathbb{C}$, we define an inner product $\langle u, v \rangle$ as a map satisfying:

1. Conjugate Symmetry: $\langle u, v \rangle = \overline{\langle v, u \rangle}$.
2. Linearity: $\langle \alpha u + v, w \rangle = \alpha \langle u, w \rangle + \langle v, w \rangle$.
3. Positive Definiteness: $\langle v, v \rangle \geq 0$, and $\langle v, v \rangle = 0 \iff v = \mathbf{0}$.

Defining Length and Distance

The norm (length) of a vector is defined as $\|v\| = \sqrt{\langle v, v \rangle}$. This is the generalized Pythagorean theorem. The distance between two points is then simply $d(u, v) = \|u - v\|$.

The Cauchy-Schwarz Inequality

One of the most powerful results in mathematics is the Cauchy-Schwarz inequality: $|\langle u, v \rangle| \leq \|u\| \|v\|$ This ensures that the “angle” $\theta$ defined by $\cos \theta = \frac{\langle u, v \rangle}{\|u\| \|v\|}$ is always between -1 and 1 for real spaces.

Orthogonality: The Power of Perpendiculars

Two vectors are orthogonal if $\langle u, v \rangle = 0$. In terms of data, orthogonal vectors are completely “unrelated” or “uncorrelated” in the geometry defined by that inner product.

A set of vectors is orthonormal if they are all orthogonal to each other and all have length 1. Working with an orthonormal basis $\{e_1, \dots, e_n\}$ is trivial compared to a general basis because the coefficients of any vector $v$ are just the inner products: $v = \langle v, e_1 \rangle e_1 + \dots + \langle v, e_n \rangle e_n$

Gram-Schmidt: Generating Order from Chaos

The Gram-Schmidt process is a recipe for turning any basis into an orthonormal one. It works by taking the first vector, then taking the second and subtracting the part that “leaks” into the first, and so on.

Best Approximations and Projections

The most significant application of inner products is the Orthogonal Projection. Given a subspace $W$ and a vector $v$ not in $W$, the “closest” vector in $W$ to $v$ is the projection $P_W(v)$.

This is the engine behind Least Squares Regression. If we have a system $Ax = b$ that has no solution, we look for the $x$ that minimizes the error $\|Ax - b\|$. This happens when $Ax$ is the projection of $b$ onto the column space of $A$.

Fun Application: Fourier Series

Calculus students often see Fourier series as a specialized topic. In reality, it is just linear algebra on an infinite-dimensional inner product space of functions $L^2$. The functions $\sin(nx)$ and $\cos(nx)$ form an orthogonal basis. Calculating Fourier coefficients is exactly the same as calculating coordinates in $\mathbb{R}^n$ using dot products!

Exercises