Forums

Why is the original gradient of cost function increase in steepest descent algorithm

Started by fl February 17, 2013
Hi,
I read a book on adaptive filter. In part of steepest descent algorithm, it says that the gradient vector at any point of a cost function points toward the direction in which the function is increasing.

I do not find the explanation about that in that book. Could you explain it to me?
Thanks
On 2/17/13 11:05 AM, fl wrote:
> Hi, > I read a book on adaptive filter. In part of steepest descent algorithm, it says that the gradient vector at any point of a cost function points toward the direction in which the function is increasing. > > I do not find the explanation about that in that book. Could you explain it to me?
well, this is when you get your book on Calculus of Several Variables out. to visualize this, think of a function of two independent variables so you can visualize it as a 3 dimensional surface. if z = f(x,y) then the two-dimensional vector (lying on the [x,y] plane) that is [ dz/dx dz/dy ] points uphill in the direction of steepest ascent. and the negative of that vector points downhill in the direction of steepest descent. i think the reason for why it's the steepest ascent is that the magnitude of that vector exceeds the magnitude of either of its components. sqrt( |dz/dx|^2 + |dz/dy|^2 ) >= |dz/dx| and sqrt( |dz/dx|^2 + |dz/dy|^2 ) >= |dz/dy| i guess, to prove it explicitly, you need to come up with a measure of slope for all of the directions and show where that slope is the largest. it's in the calculus book, i'm pretty sure. -- r b-j rbj@audioimagination.com "Imagination is more important than knowledge."
On Sun, 17 Feb 2013 08:05:32 -0800, fl wrote:

> Hi, > I read a book on adaptive filter. In part of steepest descent algorithm, > it says that the gradient vector at any point of a cost function points > toward the direction in which the function is increasing. > > I do not find the explanation about that in that book. Could you explain > it to me? > Thanks
It's part and parcel of the formulation of a gradient vector. Dig out the book from your multivariant calculus class and review. Or look on Wikipedia: http://en.wikipedia.org/wiki/Gradient_vector -- Tim Wescott Control system and signal processing consulting www.wescottdesign.com
On Sunday, 17 February 2013 16:05:32 UTC, fl  wrote:
> Hi, > > I read a book on adaptive filter. In part of steepest descent algorithm, it says that the gradient vector at any point of a cost function points toward the direction in which the function is increasing. > > > > I do not find the explanation about that in that book. Could you explain it to me? > > Thanks
Take a step (a, b), with the length of the step, a^2 + b^2 = epsilon^2, fixed. (Obviously bigger steps produce bigger changes in a function, but this is not what interests us.) When epsilon is very small, the value of the function changes by dfx.a + dfy.b, where dfx denotes df/dx and dfy denotes df/dy. Substitute for b in terms of a and epsilon in the change equation and differentiate with respect to a. What do you find? Differentiate again to check this is the maximum change that can be achieved with a step of length epsilon. illywhacker;