Energy Conservation Relation in Adaptive Filtering

Hello,

it seems that every data normalized adaptive filter or with error
non-linearity obeys a kind of an energy conservation relation. This
relation states that the norm of weight-error vector after the update plus
the norm of the apriori estimation error equals the norm of the
weight-error vector before the update plus the norm of the aposteriori
estimation error. No cross-terms appear! Due to the fact that no
cross-terms appear, this relation is the starting point for the transient
and steady-state analysis of an adaptive filter.

I was wondering: is the existence of this energy conservation relation a
convenient coincedence - an outcome of luck, or is it underpinned by some
fundamental law of adaptive filtering (data normalized or with error
non-linearities)?

For example the energy conservation relation in Newton mechanics is a
universal law and we expect it to hold everywhere. This is a law of nature.
Is the energy conservation relation of adaptive filters also a law of their
nature? Is it possible for an adaptive filter (data normalized or with
error non-linearity) not to obey an energy conservation relation?

Manolis

On 11 Apr, 08:21, "Manolis C. Tsakiris" <el01...@mail.ntua.gr> wrote:
...
> I was wondering: is the existence of this energy conservation relation a
> convenient coincedence - an outcome of luck, or is it underpinned by some
> fundamental law of adaptive filtering (data normalized or with error
> non-linearities)?

First, the 'energy' of a zero mean discrete time process is
the same as the variance of the same sequence of number.
The key, then, is to understand how one is allowed to
sum variances of different variables while discarding
the cross terms.

If one computes the variance of the sum of two random
variables one must in general account for the cross terms.
If the variables are known to be independent, the cross
terms disappear.

So the energies (or variances) of the data are directly
summable because the random variables involved are
mutually independent.

Rune

Manolis C. Tsakiris wrote:

> it seems that every data normalized adaptive filter or with error
> non-linearity obeys a kind of an energy conservation relation. This
> relation states that the norm of weight-error vector after the update plus
> the norm of the apriori estimation error equals the norm of the
> weight-error vector before the update plus the norm of the aposteriori
> estimation error. No cross-terms appear! Due to the fact that no
> cross-terms appear, this relation is the starting point for the transient
> and steady-state analysis of an adaptive filter.
> 
> I was wondering: is the existence of this energy conservation relation a
> convenient coincedence - an outcome of luck, or is it underpinned by some
> fundamental law of adaptive filtering (data normalized or with error
> non-linearities)?

As Rune Alnor noted, this has to do with the mutial independence of the 
variables.


> For example the energy conservation relation in Newton mechanics is a
> universal law and we expect it to hold everywhere. This is a law of nature.

This is a common misconception. There is no such law of nature. The 
"energy" is the artificially selected functional which is supposed to be 
invariant in a particular system. The "conservation of energy" is just a 
technical trick to solve the mechanical problems.

> Is the energy conservation relation of adaptive filters also a law of their
> nature? Is it possible for an adaptive filter (data normalized or with
> error non-linearity) not to obey an energy conservation relation?


Vladimir Vassilevsky
DSP and Mixed Signal Design Consultant
http://www.abvolt.com

"Vladimir Vassilevsky" <antispam_bogus@hotmail.com> wrote in message 
news:AaKLj.808$7Z2.404@newssvr12.news.prodigy.net...
>
>
> Manolis C. Tsakiris wrote:
>
>> it seems that every data normalized adaptive filter or with error
>> non-linearity obeys a kind of an energy conservation relation. This
>> relation states that the norm of weight-error vector after the update 
>> plus
>> the norm of the apriori estimation error equals the norm of the
>> weight-error vector before the update plus the norm of the aposteriori
>> estimation error. No cross-terms appear! Due to the fact that no
>> cross-terms appear, this relation is the starting point for the transient
>> and steady-state analysis of an adaptive filter.
>>
>> I was wondering: is the existence of this energy conservation relation a
>> convenient coincedence - an outcome of luck, or is it underpinned by some
>> fundamental law of adaptive filtering (data normalized or with error
>> non-linearities)?
>
> As Rune Alnor noted, this has to do with the mutial independence of the 
> variables.
>
>
>> For example the energy conservation relation in Newton mechanics is a
>> universal law and we expect it to hold everywhere. This is a law of 
>> nature.
>
> This is a common misconception. There is no such law of nature. The 
> "energy" is the artificially selected functional which is supposed to be 
> invariant in a particular system. The "conservation of energy" is just a 
> technical trick to solve the mechanical problems.
>
>> Is the energy conservation relation of adaptive filters also a law of 
>> their
>> nature? Is it possible for an adaptive filter (data normalized or with
>> error non-linearity) not to obey an energy conservation relation?
>
>
> Vladimir Vassilevsky
> DSP and Mixed Signal Design Consultant
> http://www.abvolt.com

In other contexts, we'd refer to "the model" and it would be understood that 
it may have simplifications for the sake of practicality and ease of 
analysis.  A common model for conservation of energy in mechanical systems 
is a frictionless model - in an electrical analog, no "resistors", eh?

Fred