Sign in

username:

password:



Not a member?

Search Online Books



Search tips

Free Online Books

Sponsor

Industry's highest performing at the lowest power DSPs now as low as $5.00*
Start development today!
*volume pricing for 10ku

Chapters

See Also

Embedded SystemsFPGAElectronics

Spectral Audio Signal Processing
    FIR Digital Filter Design
       Optimal FIR Digital Filter Design
          Least-Squares Linear-Phase FIR Filter Design
             Least Squares Optimization

Chapter Contents:

Search Spectral Audio Signal Processing

  

Book Index | Global Index

Would you like to be notified by email when Julius Orion Smith III publishes a new entry into his blog?

  

Least Squares Optimization

$\displaystyle \hat{x} \mathrel{\stackrel{\Delta}{=}}\arg \min_x \left\Vert\,Ax-b\,\right\Vert _2 = \arg \min_x \left\Vert\,Ax-b\,\right\Vert _2^2 $

Hence we can minimize

$\displaystyle \left\Vert\,Ax-b\,\right\Vert _2^2 = (Ax-b)^T(Ax-b) $

Expanding this, we have:

$\displaystyle (Ax-b)^T(Ax-b) = (b^T-x^TA^T)(Ax-b) $

This is quadratic in $ x$, hence it has a global minimum which we can find by taking the derivative, setting it to zero, and solving for $ x$. Doing this yields:

$\displaystyle A^TAx-A^Tb=0 $

These are the famous normal equations whose solution is given by:

$\displaystyle \zbox {\hat{x} = \left[(A^TA)^{-1}A^T\right]b} $

The matrix

$\displaystyle A^\dagger \isdef (A^TA)^{-1}A^T $

is known as the (Moore-Penrose) pseudo-inverse of the matrix $ A$.

Previous: Matrix Formulation: Optimal Design, Cont'd
Next: Geometrical Interpretation of Least Squares

Order a Hardcopy of Spectral Audio Signal Processing

About the Author: Julius Orion Smith III
Julius Smith's background is in electrical engineering (BS Rice 1975, PhD Stanford 1983). He is presently Professor of Music and Associate Professor (by courtesy) of Electrical Engineering at Stanford's Center for Computer Research in Music and Acoustics (CCRMA), teaching courses and pursuing research related to signal processing applied to music and audio systems. See http://ccrma.stanford.edu/~jos/ for details.


Comments

No comments yet for this page


Add a Comment
You need to login before you can post a comment (best way to prevent spam). ( Not a member? )