# LMS maximum step size

Started by July 12, 2010
```Hi all,

w(n+1) = w(n) + mu*e(n)*x(n)

Can anyone tell me how to determine the maximum mu? I read some materials
that mu should be less than 2/lambda_max, where lambda_max is the maximum
eigen value of the autocorrelation matrix of the input.

Take the 50Hz noise cancellation filter in Widrow's paper as an example,
"Adaptive Noise Cancelling : Principles and Applications". How to determine
the maximum mu?

Thanks,
cfy30

```
```On Jul 12, 3:37&#2013266080;pm, "cfy30" <cfy30@n_o_s_p_a_m.yahoo.com> wrote:
> Hi all,
>
> w(n+1) = w(n) + mu*e(n)*x(n)
>
> Can anyone tell me how to determine the maximum mu? I read some materials
> that mu should be less than 2/lambda_max, where lambda_max is the maximum
> eigen value of the autocorrelation matrix of the input.
>
> Take the 50Hz noise cancellation filter in Widrow's paper as an example,
> "Adaptive Noise Cancelling : Principles and Applications". How to determine
> the maximum mu?
>
> Thanks,
> cfy30

I could tell you but then I would have to shoot you.
Best to use NLMS.

Hardy
```
```On Jul 13, 11:10&#2013266080;pm, HardySpicer <gyansor...@gmail.com> wrote:
>
> I could tell you but then I would have to shoot you.

...

> Best to use NLMS.

so now are you gonna shoot him?

r b-j

```
```On Jul 11, 10:37&#2013266080;pm, "cfy30" <cfy30@n_o_s_p_a_m.yahoo.com> wrote:
> Hi all,
>
> w(n+1) = w(n) + mu*e(n)*x(n)
>
> Can anyone tell me how to determine the maximum mu? I read some materials
> that mu should be less than 2/lambda_max, where lambda_max is the maximum
> eigen value of the autocorrelation matrix of the input.
>
> Take the 50Hz noise cancellation filter in Widrow's paper as an example,
> "Adaptive Noise Cancelling : Principles and Applications". How to determine
> the maximum mu?
>
> Thanks,
> cfy30

For the LMS algorithm:

1. Define a weight-error vector
2. Write the weight-error vector as a recursive algorithm in terms of
the auto-correlation matrix of the input
3. Perform a unitary-similarity transformation on the auto-correlation
matrix
4. Rotate the weight-error coordinate system using the eigenvector
matrix of the unitarity-similarity transformation
5. Observe what must occur for convergence
6. QED

Or, you could spend some time and look it up in one of several books:

 Clarkson, Peter M. "Optimal and Adaptive Signal Processing"
 Widrow, Bernard, and Stearns, Samuel, "Adaptive Signal Processing"
 Hatken, Simon, "Adaptive Filter Theory"
 Graupe, daniel, "Time Series Analysis"

If you look at the normalized LMS (NLMS), then the maximum mu is
limited to the sum of the number of filter coefficients. That proof I
will leave to you.

Maurice Givens
```
```
maury wrote:

> If you look at the normalized LMS (NLMS), then the maximum mu is
> limited to the sum of the number of filter coefficients.

IMO what does matter is optimim rather then maximum. You need to
minimize the total error of the process. This error is because of
numeric artifacts. It also depends on the statistics and power of the
reference signal. So the algorithm must adapt Mu in pseudo Kalman way.

DSP and Mixed Signal Design Consultant
http://www.abvolt.com
```
```On Jul 14, 9:35&#2013266080;am, Vladimir Vassilevsky <nos...@nowhere.com> wrote:
> maury wrote:
> > If you look at the normalized LMS (NLMS), then the maximum mu is
> > limited to the sum of the number of filter coefficients.
>
> IMO what does matter is optimim rather then maximum. You need to
> minimize the total error of the process. This error is because of
> numeric artifacts. It also depends on the statistics and power of the
> reference signal. So the algorithm must adapt Mu in pseudo Kalman way.
>
> DSP and Mixed Signal Design Consultanthttp://www.abvolt.com

With respect to the NLMS, the effect of the power of the reference
depends on the implmentation. For example, in a system idemtification
implementation, the power of the reference is not important since the
normalization of the LMS is performed over the signal power. The
algorithm is self-adjusting for imput power.

However!!!  The performance is very dependent on the model assumed. By
nonlinearity, I take you to mean that a linear model will have errors
if nonlinearities are present in the reference signal (the reference
being some nonlinear inner-product of the signal and system vector).
This just means the wrong model was chosen. If your model matches the
system, then the squared error will not be great. No matter what the
model, the algorithm will give the minimum squared error for the
model. This doesn't mean that the coefficients you end up with are
close to the desired coefficients.

The other important parameter is the condition of the input auto-
correlation matrix. An ill-conditioned matrix will result in not being
able to determine an optimum estimate of the system impulse response.

This is the problem with the example used by the OP. An input of 50 Hz
(single signal) will result in an ill-conditioned input auto-
correlartion matrix. There is NO optimum solution in this case. In
fact, there are an infinite number of solutions to this problem.

But, your point is well taken that the maximum mu used will be based
on the optimum performance required from the problem. That performance
may be minimum misadjustment noise, or maximum speed of convergence.
Each will ditate a different "maximum" mu.

What I think the OP was looking for, is how to determine the maximum
mu for convergence (often equated with stability). This value is
different for LMS versus NLMS.

Maurice
```
```
maury wrote:

> On Jul 14, 9:35 am, Vladimir Vassilevsky <nos...@nowhere.com> wrote:
>
>>maury wrote:
>>
>>>If you look at the normalized LMS (NLMS), then the maximum mu is
>>>limited to the sum of the number of filter coefficients.
>>
>>IMO what does matter is optimim rather then maximum. You need to
>>minimize the total error of the process. This error is because of
>>numeric artifacts. It also depends on the statistics and power of the
>>reference signal. So the algorithm must adapt Mu in pseudo Kalman way.
>>
>>DSP and Mixed Signal Design Consultanthttp://www.abvolt.com
>
>
> With respect to the NLMS, the effect of the power of the reference
> depends on the implmentation. For example, in a system idemtification
> implementation, the power of the reference is not important since the
> normalization of the LMS is performed over the signal power. The
> algorithm is self-adjusting for imput power.

Yes, but the input is catching the ambient noise and double talk besides
the reflected signal. Hence if the input power is low, the algorithm
should not crank Mu all the way up to infinity, as simple NLMS would do,
but do the exactly the opposite.

> However!!!  The performance is very dependent on the model assumed. By
> nonlinearity, I take you to mean that a linear model will have errors
> if nonlinearities are present in the reference signal (the reference
> being some nonlinear inner-product of the signal and system vector).
> This just means the wrong model was chosen. If your model matches the
> system, then the squared error will not be great.

System is given to you. Your algorithm has to work with whatever you got.

DSP and Mixed Signal Design Consultant

http://www.abvolt.com
```
```On Jul 14, 2:07&#2013266080;pm, Vladimir Vassilevsky <nos...@nowhere.com> wrote:

> System is given to you. Your algorithm has to work with whatever you got.

Kobayashi maru solution!
```
```On 7/14/2010 4:38 PM, maury wrote:
> On Jul 14, 2:07 pm, Vladimir Vassilevsky<nos...@nowhere.com>  wrote:
>
>> System is given to you. Your algorithm has to work with whatever you got.
>
> Kobayashi maru solution!

Hah! another Trekke!
--
Engineering is the art of making what you want from things you can get.
```
```On Jul 14, 8:07&#2013266080;pm, Vladimir Vassilevsky <nos...@nowhere.com> wrote:
> maury wrote:
> > On Jul 14, 9:35 am, Vladimir Vassilevsky <nos...@nowhere.com> wrote:
>
> >>maury wrote:
>
> >>>If you look at the normalized LMS (NLMS), then the maximum mu is
> >>>limited to the sum of the number of filter coefficients.
>
> >>IMO what does matter is optimim rather then maximum. You need to
> >>minimize the total error of the process. This error is because of
> >>numeric artifacts. It also depends on the statistics and power of the
> >>reference signal. So the algorithm must adapt Mu in pseudo Kalman way.
>
> >>DSP and Mixed Signal Design Consultanthttp://www.abvolt.com
>
> > With respect to the NLMS, the effect of the power of the reference
> > depends on the implmentation. For example, in a system idemtification
> > implementation, the power of the reference is not important since the
> > normalization of the LMS is performed over the signal power. The
> > algorithm is self-adjusting for imput power.
>
> Yes, but the input is catching the ambient noise and double talk besides
> &#2013266080; the reflected signal. Hence if the input power is low, the algorithm
> should not crank Mu all the way up to infinity, as simple NLMS would do,
> but do the exactly the opposite.
>
> > However!!! &#2013266080;The performance is very dependent on the model assumed. By
> > nonlinearity, I take you to mean that a linear model will have errors
> > if nonlinearities are present in the reference signal (the reference
> > being some nonlinear inner-product of the signal and system vector).
> > This just means the wrong model was chosen. If your model matches the
> > system, then the squared error will not be great.
>
> System is given to you. Your algorithm has to work with whatever you got.

What if you discovered by sheer luck that doing something wanky like
skipping a beat works wonders to your signal? Sure this is uncool,
inelegant, and can't be analyzed, only characterized. Do you scrap all
this and wake up next day pretending that nothing has happened?

-Momo
```