On Sunday, 17 February 2013 16:05:32 UTC, fl  wrote:
> Hi,
> 
> I read a book on adaptive filter. In part of steepest descent algorithm, it says that the gradient vector at any point of a cost function points toward the direction in which the function is increasing.
> 
> 
> 
> I do not find the explanation about that in that book. Could you explain it to me?
> 
> Thanks

Take a step (a, b), with the length of the step, a^2 + b^2 = epsilon^2, fixed. (Obviously bigger steps produce bigger changes in a function, but this is not what interests us.) 

When epsilon is very small, the value of the function changes by dfx.a + dfy.b, where dfx denotes df/dx and dfy denotes df/dy.

Substitute for b in terms of a and epsilon in the change equation and differentiate with respect to a. What do you find? Differentiate again to check this is the maximum change that can be achieved with a step of length epsilon.

illywhacker; 

On Sun, 17 Feb 2013 08:05:32 -0800, fl wrote:

> Hi,
> I read a book on adaptive filter. In part of steepest descent algorithm,
> it says that the gradient vector at any point of a cost function points
> toward the direction in which the function is increasing.
> 
> I do not find the explanation about that in that book. Could you explain
> it to me?
> Thanks

It's part and parcel of the formulation of a gradient vector.  Dig out 
the book from your multivariant calculus class and review.

Or look on Wikipedia: http://en.wikipedia.org/wiki/Gradient_vector

-- 
Tim Wescott
Control system and signal processing consulting
www.wescottdesign.com

On 2/17/13 11:05 AM, fl wrote:
> Hi,
> I read a book on adaptive filter. In part of steepest descent algorithm, it says that the gradient vector at any point of a cost function points toward the direction in which the function is increasing.
>
> I do not find the explanation about that in that book. Could you explain it to me?

well, this is when you get your book on Calculus of Several Variables 
out.  to visualize this, think of a function of two independent 
variables so you can visualize it as a 3 dimensional surface.

    if  z = f(x,y)

then the two-dimensional vector (lying on the [x,y] plane) that is

        [ dz/dx  dz/dy ]

points uphill in the direction of steepest ascent.  and the negative of 
that vector points downhill in the direction of steepest descent.

i think the reason for why it's the steepest ascent is that the 
magnitude of that vector exceeds the magnitude of either of its components.

        sqrt( |dz/dx|^2 + |dz/dy|^2 )  >=  |dz/dx|
and
        sqrt( |dz/dx|^2 + |dz/dy|^2 )  >=  |dz/dy|


i guess, to prove it explicitly, you need to come up with a measure of 
slope for all of the directions and show where that slope is the 
largest.  it's in the calculus book, i'm pretty sure.


-- 

r b-j                  rbj@audioimagination.com

"Imagination is more important than knowledge."

Hi,
I read a book on adaptive filter. In part of steepest descent algorithm, it says that the gradient vector at any point of a cost function points toward the direction in which the function is increasing.

I do not find the explanation about that in that book. Could you explain it to me?
Thanks