Geometrical Interpretation of Least Squares

Typically, the number of frequency constraints is much greater than the number of design variables (filter taps). In these cases, we have an overdetermined system of equations (more equations than unknowns). Therefore, we cannot generally satisfy all the equations, and we are left with minimizing some error criterion to find the ``optimal compromise'' solution.

In the case of least-squares approximation, we are minimizing the Euclidean distance, which suggests the geometrical interpretation shown in Fig.E.28.

Thus, the desired vector is the vector sum of its best least-squares approximation plus an orthogonal error :

In practice, the least-squares solution can be found by minimizing the sum of squared errors:

Figure E.28 suggests that the error vector is orthogonal to the column space of the matrix , hence it must be orthogonal to each column in :

This is how the orthogonality principle can be used to derive the fact that the best least squares solution is given by

Note that the pseudo-inverse projects the vector onto the column space of .

(Note: To obtain the best numerical algorithms for least-squares solution in Matlab, it is usually better to use ``x = A b'' rather than explicitly computing the pseudo-inverse as in ``x = pinv(A) * b''.)

---

Previous: Least Squares Optimization
Next: Matlab Support for Least-Squares FIR Filter Design

Order a Hardcopy of Spectral Audio Signal Processing

---

About the Author: Julius Orion Smith III
Julius Smith's background is in electrical engineering (BS Rice 1975, PhD Stanford 1983). He is presently Professor of Music and Associate Professor (by courtesy) of Electrical Engineering at Stanford's Center for Computer Research in Music and Acoustics (CCRMA), teaching courses and pursuing research related to signal processing applied to music and audio systems. See http://ccrma.stanford.edu/~jos/ for details.