Sign in

Search Online Books

Free Online Books

Sponsor

Industry's highest performing at the lowest power DSPs now as low as $5.00*
Start development today!
*volume pricing for 10ku

Chapters

Physical Audio Signal Processing
    Lumped Models
       More General Finite-Difference Methods
          Newton's Method of Nonlinear Minimization

Chapter Contents:

Search Physical Audio Signal Processing

Book Index | Global Index

Would you like to be notified by email when Julius Orion Smith III publishes a new entry into his blog?

Newton's Method of Nonlinear Minimization

Newton's method [162],[166, p. 143] finds the minimum of a nonlinear (scalar) function of several variables by locally approximating the function by a quadratic surface, and then stepping to the bottom of that ``bowl'', which generally requires a matrix inversion. Newton's method therefore requires the function to be ``close to quadratic'', and its effectiveness is directly tied to the accuracy of that assumption. For smooth functions, Newton's method gives very rapid quadratic convergence in the last stages of iteration. Quadratic convergence implies, for example, that the number of significant digits in the minimizer approximately doubles each iteration.

Newton's method may be derived as follows: Suppose we wish to minimize the real, positive function $J(\underline{x})$ with respect to $\underline{x}$ . The vector Taylor expansion [546] of $J(\underline{x})$ about $\underline{\hat{x}}$ gives

$\displaystyle J(\underline{\hat{x}}^\ast ) = J(\underline{\hat{x}}) + J^\prime(... ...ne{\hat{x}}\right) \left(\underline{\hat{x}}^\ast -\underline{\hat{x}}\right),$

for some $0\leq\lambda\leq 1$ , where $\overline{\lambda}\isdef 1-\lambda$ . It is now necessary to assume that $J''\left( \lambda\underline{\hat{x}}^\ast +\overline{\lambda}\underline{\hat{x}}\right)\approx J''(\underline{\hat{x}})$ . Differentiating with respect to $\underline{\hat{x}}^\ast$ , where $J(\underline{\hat{x}}^\ast )$ is presumed to be minimized, this becomes

$\displaystyle 0 = 0 + J^\prime(\underline{\hat{x}}) + J''(\underline{\hat{x}}) \left(\underline{\hat{x}}^\ast -\underline{\hat{x}}\right).$

Solving for $\underline{\hat{x}}^\ast$ yields

$\displaystyle \underline{\hat{x}}^\ast = \underline{\hat{x}}- [J''(\underline{\hat{x}})]^{-1} J^\prime(\underline{\hat{x}}).$

(8.13)

Applying Eq. $\,$ (7.13) iteratively, we obtain Newton's method:

$\displaystyle \underline{\hat{x}}_{n+1} = \underline{\hat{x}}_n - [J''(\underline{\hat{x}}_n)]^{-1} J^\prime(\underline{\hat{x}}_n), \quad n=1,2,\ldots\,,$

(8.14)

where $\underline{\hat{x}}_0$ is given as an initial condition.

When the $J(\underline{\hat{x}})$ is any quadratic form in $\underline{\hat{x}}$ , then $J''\left(\lambda\underline{\hat{x}}^\ast +\overline{\lambda}\underline{\hat{x}}\right)= J''(\underline{\hat{x}})$ , and Newton's method produces $\underline{\hat{x}}^\ast$ in one iteration; that is, $\underline{\hat{x}}_1=\underline{\hat{x}}^\ast$ for every $\underline{\hat{x}}_0$ .

Previous: Trapezoidal Rule
Next: Semi-Implicit Methods

Order a Hardcopy of Physical Audio Signal Processing

About the Author: Julius Orion Smith III
Julius Smith's background is in electrical engineering (BS Rice 1975, PhD Stanford 1983). He is presently Professor of Music and Associate Professor (by courtesy) of Electrical Engineering at Stanford's Center for Computer Research in Music and Acoustics (CCRMA), teaching courses and pursuing research related to signal processing applied to music and audio systems. See http://ccrma.stanford.edu/~jos/ for details.

Comments

No comments yet for this page

Add a Comment

You need to login before you can post a comment (best way to prevent spam). ( Not a member? )