# Multi input Weiner Filter

Started by January 8, 2009
```For the standard SISO Wiener filter we minimize the cost J

J=E[e^2]=E(d-W'X)^2

where W is a vector of weights and X is a vector of regressers. (d is
desired output)  Also ' denotes transpose.
We do this by diferentiating wrt the weight vector W and arrive at the
standard Wiener solution.

However, in the case where W is asymmetric  Matrix and d is a vector
(also X is a vector still) we have to differentiate wrt a Matrix and
the error is  a vector.

J=e'e = (d-W'X)'(d-W'X)

ie dJ/dW (J is still a scalar)

I have a paper that just says that the answer is the same form but
with no derivation! Differentiating wrt a matrix however is slightly
different.

------------------------------------------------------
For example...
Differentiation of a scalar wrt a vector of the quadratic form

y=x'Ax where a is a matrix and x a vector

dy/dx = Ax+A'x = 2Ax if A is symmetric.

Now in the multidimensional Wiener filter we have a term
X'WW'X which needs differentiating wrt the matrix W.

We also have  terms X'Wd and dW'X which need differentiating wrt the
matrix W.

H.
```
```On Jan 9, 8:21&#2013266080;am, HardySpicer <gyansor...@gmail.com> wrote:
> For the standard SISO Wiener filter we minimize the cost J
>
> J=E[e^2]=E(d-W'X)^2
>
> where W is a vector of weights and X is a vector of regressers. (d is
> desired output) &#2013266080;Also ' denotes transpose.
> We do this by diferentiating wrt the weight vector W and arrive at the
> standard Wiener solution.
>
> However, in the case where W is asymmetric &#2013266080;Matrix and d is a vector
> (also X is a vector still) we have to differentiate wrt a Matrix and
> the error is &#2013266080;a vector.
>
> J=e'e = (d-W'X)'(d-W'X)
>
> ie dJ/dW (J is still a scalar)
>
> I have a paper that just says that the answer is the same form but
> with no derivation! Differentiating wrt a matrix however is slightly
> different.
>
> ------------------------------------------------------
> For example...
> Differentiation of a scalar wrt a vector of the quadratic form
>
> y=x'Ax where a is a matrix and x a vector
>
> dy/dx = Ax+A'x = 2Ax if A is symmetric.
>
> Now in the multidimensional Wiener filter we have a term
> X'WW'X which needs differentiating wrt the matrix W.
>
> We also have &#2013266080;terms X'Wd and dW'X which need differentiating wrt the
> matrix W.
>
> H.

Actually I just found this result from

www.che.iitm.ac.in/~naras/ch544/matrix.pdf

That if you differentiate the norm squared

||AX+b||^2

you get
2AXX' + 2bX'

using this and substituting

||-WX+d||^2

I get

-2WXX' +2dX'=0

Now since E[XX'] = R, the correlation matrix I get

WR=E[dX']                     (1)

or

W=RdxR^-1

which is not the same result as in the paper. However, R is symmetric,
from (1)

W'R=E[Xd']

W'=R^-1 Rxd  (Rxd is the cross-correlation matrix between X and d)

which is more like it. Now does W'=W ie is the filter weight matrix
symmetric??

H.

```
```On Jan 9, 10:36&#2013266080;am, HardySpicer <gyansor...@gmail.com> wrote:
> On Jan 9, 8:21&#2013266080;am, HardySpicer <gyansor...@gmail.com> wrote:
>
>
>
> > For the standard SISO Wiener filter we minimize the cost J
>
> > J=E[e^2]=E(d-W'X)^2
>
> > where W is a vector of weights and X is a vector of regressers. (d is
> > desired output) &#2013266080;Also ' denotes transpose.
> > We do this by diferentiating wrt the weight vector W and arrive at the
> > standard Wiener solution.
>
> > However, in the case where W is asymmetric &#2013266080;Matrix and d is a vector
> > (also X is a vector still) we have to differentiate wrt a Matrix and
> > the error is &#2013266080;a vector.
>
> > J=e'e = (d-W'X)'(d-W'X)
>
> > ie dJ/dW (J is still a scalar)
>
> > I have a paper that just says that the answer is the same form but
> > with no derivation! Differentiating wrt a matrix however is slightly
> > different.
>
> > ------------------------------------------------------
> > For example...
> > Differentiation of a scalar wrt a vector of the quadratic form
>
> > y=x'Ax where a is a matrix and x a vector
>
> > dy/dx = Ax+A'x = 2Ax if A is symmetric.
>
> > Now in the multidimensional Wiener filter we have a term
> > X'WW'X which needs differentiating wrt the matrix W.
>
> > We also have &#2013266080;terms X'Wd and dW'X which need differentiating wrt the
> > matrix W.
>
> > H.
>
> Actually I just found this result from
>
> www.che.iitm.ac.in/~naras/ch544/matrix.pdf
>
> That if you differentiate the norm squared
>
> ||AX+b||^2
>
> you get
> 2AXX' + 2bX'
>
> using this and substituting
>
> ||-WX+d||^2
>
> I get
>
> -2WXX' +2dX'=0
>
> Now since E[XX'] = R, the correlation matrix I get
>
> WR=E[dX'] &#2013266080; &#2013266080; &#2013266080; &#2013266080; &#2013266080; &#2013266080; &#2013266080; &#2013266080; &#2013266080; &#2013266080; (1)
>
> or
>
> W=RdxR^-1
>
> which is not the same result as in the paper. However, R is symmetric,
> from (1)
>
> W'R=E[Xd']
>
> W'=R^-1 Rxd &#2013266080;(Rxd is the cross-correlation matrix between X and d)
>
> which is more like it. Now does W'=W ie is the filter weight matrix
> symmetric??
>
> H.

Sorry - I should have said

That if you differentiate the norm squared

||AX+b||^2

with respect to A.
```
```On Jan 9, 10:36&#2013266080;am, HardySpicer <gyansor...@gmail.com> wrote:
> On Jan 9, 8:21&#2013266080;am, HardySpicer <gyansor...@gmail.com> wrote:
>
>
>
> > For the standard SISO Wiener filter we minimize the cost J
>
> > J=E[e^2]=E(d-W'X)^2
>
> > where W is a vector of weights and X is a vector of regressers. (d is
> > desired output) &#2013266080;Also ' denotes transpose.
> > We do this by diferentiating wrt the weight vector W and arrive at the
> > standard Wiener solution.
>
> > However, in the case where W is asymmetric &#2013266080;Matrix and d is a vector
> > (also X is a vector still) we have to differentiate wrt a Matrix and
> > the error is &#2013266080;a vector.
>
> > J=e'e = (d-W'X)'(d-W'X)
>
> > ie dJ/dW (J is still a scalar)
>
> > I have a paper that just says that the answer is the same form but
> > with no derivation! Differentiating wrt a matrix however is slightly
> > different.
>
> > ------------------------------------------------------
> > For example...
> > Differentiation of a scalar wrt a vector of the quadratic form
>
> > y=x'Ax where a is a matrix and x a vector
>
> > dy/dx = Ax+A'x = 2Ax if A is symmetric.
>
> > Now in the multidimensional Wiener filter we have a term
> > X'WW'X which needs differentiating wrt the matrix W.
>
> > We also have &#2013266080;terms X'Wd and dW'X which need differentiating wrt the
> > matrix W.
>
> > H.
>
> Actually I just found this result from
>
> www.che.iitm.ac.in/~naras/ch544/matrix.pdf
>
> That if you differentiate the norm squared
>
> ||AX+b||^2
>
> you get
> 2AXX' + 2bX'
>
> using this and substituting
>
> ||-WX+d||^2
>
> I get
>
> -2WXX' +2dX'=0
>
> Now since E[XX'] = R, the correlation matrix I get
>
> WR=E[dX'] &#2013266080; &#2013266080; &#2013266080; &#2013266080; &#2013266080; &#2013266080; &#2013266080; &#2013266080; &#2013266080; &#2013266080; (1)
>
> or
>
> W=RdxR^-1
>
> which is not the same result as in the paper. However, R is symmetric,
> from (1)
>
> W'R=E[Xd']
>
> W'=R^-1 Rxd &#2013266080;(Rxd is the cross-correlation matrix between X and d)
>
> which is more like it. Now does W'=W ie is the filter weight matrix
> symmetric??
>
> H.

oops it's right. I Had minimized ||-WX+d ||^2 when it should have been
||-W'X+d ||^2. all ok!
```
```On Thu, 08 Jan 2009 11:21:21 -0800, HardySpicer wrote:

> For the standard SISO Wiener filter we minimize the cost J
>
> J=E[e^2]=E(d-W'X)^2
>
> where W is a vector of weights and X is a vector of regressers. (d is
> desired output)  Also ' denotes transpose. We do this by diferentiating
> wrt the weight vector W and arrive at the standard Wiener solution.
>
> However, in the case where W is asymmetric  Matrix and d is a vector
> (also X is a vector still) we have to differentiate wrt a Matrix and the
> error is  a vector.
>
> J=e'e = (d-W'X)'(d-W'X)
>
> ie dJ/dW (J is still a scalar)
>
> I have a paper that just says that the answer is the same form but with
> no derivation! Differentiating wrt a matrix however is slightly
> different.
>
> ------------------------------------------------------ For example...
> Differentiation of a scalar wrt a vector of the quadratic form
>
> y=x'Ax where a is a matrix and x a vector
>
>
> dy/dx = Ax+A'x = 2Ax if A is symmetric.
>
>
>
>
> Now in the multidimensional Wiener filter we have a term X'WW'X which
> needs differentiating wrt the matrix W.
>
> We also have  terms X'Wd and dW'X which need differentiating wrt the
> matrix W.
>
>
> H.

This is a solved problem if you bear in mind that a steady state Kalman
filter is really just a Wiener filter with a fancy name.

Then web search accordingly.

--
Tim Wescott
Control systems and communications consulting
http://www.wescottdesign.com

Need to learn how to apply control theory in your embedded system?
"Applied Control Theory for Embedded Systems" by Tim Wescott
Elsevier/Newnes, http://www.wescottdesign.com/actfes/actfes.html
```
```>This is a solved problem if you bear in mind that a steady state Kalman
>filter is really just a Wiener filter with a fancy name.
>
>Then web search accordingly.
>
>--
>Tim Wescott

I'm OK with the adaptive filtering, but I'm still puzzled. What makes
"Kalman" a fancier name than "Weiner"? :-\

Steve

```
```On Jan 9, 4:08&#2013266080;pm, Tim Wescott <t...@seemywebsite.com> wrote:
> On Thu, 08 Jan 2009 11:21:21 -0800, HardySpicer wrote:
> > For the standard SISO Wiener filter we minimize the cost J
>
> > J=E[e^2]=E(d-W'X)^2
>
> > where W is a vector of weights and X is a vector of regressers. (d is
> > desired output) &#2013266080;Also ' denotes transpose. We do this by diferentiating
> > wrt the weight vector W and arrive at the standard Wiener solution.
>
> > However, in the case where W is asymmetric &#2013266080;Matrix and d is a vector
> > (also X is a vector still) we have to differentiate wrt a Matrix and the
> > error is &#2013266080;a vector.
>
> > J=e'e = (d-W'X)'(d-W'X)
>
> > ie dJ/dW (J is still a scalar)
>
> > I have a paper that just says that the answer is the same form but with
> > no derivation! Differentiating wrt a matrix however is slightly
> > different.
>
> > ------------------------------------------------------ For example...
> > Differentiation of a scalar wrt a vector of the quadratic form
>
> > y=x'Ax where a is a matrix and x a vector
>
> > dy/dx = Ax+A'x = 2Ax if A is symmetric.
>
> > Now in the multidimensional Wiener filter we have a term X'WW'X which
> > needs differentiating wrt the matrix W.
>
> > We also have &#2013266080;terms X'Wd and dW'X which need differentiating wrt the
> > matrix W.
>
> > H.
>
> This is a solved problem if you bear in mind that a steady state Kalman
> filter is really just a Wiener filter with a fancy name.
>
> Then web search accordingly.
>
> --
> Tim Wescott
> Control systems and communications consultinghttp://www.wescottdesign.com
>
> Need to learn how to apply control theory in your embedded system?
> "Applied Control Theory for Embedded Systems" by Tim Wescott
> Elsevier/Newnes,http://www.wescottdesign.com/actfes/actfes.html

Trouble with Kalman filters is that you need to solve a Ricatti
equation for the gain matrix.
With Polynomial or LMS type adaptive Wiener filters, the computation
is far less.

H.
```