Hi, I'm learning digital communication now. I use Proakis' "Digital Communications" as my text book. Now several terms confused me, which are projection, correlation and covariance. These terms appeared in the book many times, but with different formula. Can any body help me to understand them? thanks very much.
how to understand projection, correlation & covariance
Started by ●April 7, 2009
Reply by ●April 7, 20092009-04-07
On Apr 7, 9:16 am, chenyong20...@gmail.com wrote:> Hi, > > I'm learning digital communication now. I use Proakis' "Digital > Communications" as my text book. Now several terms confused me, which > are projection, correlation and covariance. These terms appeared in > the book many times, but with different formula. Can any body help me > to understand them? thanks very much.Chenyong, your question is rather generic. Can you point out the specific formulae that are confusing you? It will be easier to clarify specific instances. Thanks, Dilip.
Reply by ●April 7, 20092009-04-07
On 7 Apr, 15:16, chenyong20...@gmail.com wrote:> Hi, > > I'm learning digital communication now. I use Proakis' "Digital > Communications" as my text book. Now several terms confused me, which > are projection, correlation and covariance. These terms appeared in > the book many times, but with different formula.As others already said, it's hard to answer generic questions. Having said that, it might be helpful to you that the terms you mention are generic concepts, which can be used in different contexts. For instance, correlation and covariance are statistical terms that are all but omnipresent. Chances are they are used in almost every context where statistics is used. Different formulas could also be explained by that one uses different estimators for the same statistic. You also already know the concept of projections from basic vector calculus, where projections are computed in terms of inner products between vectors. It's the same general idea in complex-valued N-D space, but the expressions become a bit more involved, particularly if you project onto planes instead of vectors. Rune
Reply by ●April 8, 20092009-04-08
On 4��7��, ����9ʱ42��, Rune Allnor <all...@tele.ntnu.no> wrote:> On 7 Apr, 15:16, chenyong20...@gmail.com wrote: > > > Hi, > > > I'm learning digital communication now. I use Proakis' "Digital > > Communications" as my text book. Now several terms confused me, which > > are projection, correlation and covariance. These terms appeared in > > the book many times, but with different formula. > > As others already said, it's hard to answer generic questions. > > Having said that, it might be helpful to you that the terms you > mention are generic concepts, which can be used in different > contexts. For instance, correlation and covariance are statistical > terms that are all but omnipresent. Chances are they are used > in almost every context where statistics is used. Different formulas > could also be explained by that one uses different estimators > for the same statistic. > > You also already know the concept of projections from > basic vector calculus, where projections are computed > in terms of inner products between vectors. It's the same > general idea in complex-valued N-D space, but the > expressions become a bit more involved, particularly > if you project onto planes instead of vectors. > > RuneHi Dilip & Rune, thanks for your reply. Well, I'm learning Proakis' book "Digital Communications". This book is full of mathematics. There are problems when I'm reading chapter 5 "Optimum Receivers for the additive white gaussian noise channel". 1. Projection. "Suppose the received signal r(t) is passed through a parallel bank of N cross correlators which basically compute the projection of r(t) onto the N basis functions {fn(t)}, ..." Question: the book gives the formula as integral[r(t)*fk(t)] from 0 to T. Does projection f1 onto f2 means integral[f1*f2]? I can't find this definition in book and wiki. 2. Covariance Question: The book gives covariance in one part as E(n1*n2), but I found another formula in the book as E[(x-mx)*(y-my)]. This confused me. Can you help me to understand their difference? Thanks very much.
Reply by ●April 8, 20092009-04-08
On 8 Apr, 13:55, chenyong20...@gmail.com wrote:> On 4��7��, ����9ʱ42��, Rune Allnor <all...@tele.ntnu.no> wrote: > > > > > > > On 7 Apr, 15:16, chenyong20...@gmail.com wrote: > > > > Hi, > > > > I'm learning digital communication now. I use Proakis' "Digital > > > Communications" as my text book. Now several terms confused me, which > > > are projection, correlation and covariance. These terms appeared in > > > the book many times, but with different formula. > > > As others already said, it's hard to answer generic questions. > > > Having said that, it might be helpful to you that the terms you > > mention are generic concepts, which can be used in different > > contexts. For instance, correlation and covariance are statistical > > terms that are all but omnipresent. Chances are they are used > > in almost every context where statistics is used. Different formulas > > could also be explained by that one uses different estimators > > for the same statistic. > > > You also already know the concept of projections from > > basic vector calculus, where projections are computed > > in terms of inner products between vectors. It's the same > > general idea in complex-valued N-D space, but the > > expressions become a bit more involved, particularly > > if you project onto planes instead of vectors. > > > Rune > > Hi Dilip & Rune, > > thanks for your reply. Well, I'm learning Proakis' book "Digital > Communications". This book is full of mathematics. There are problems > when I'm reading chapter 5 "Optimum Receivers for the additive white > gaussian noise channel". > > 1. Projection. > "Suppose the received signal r(t) is passed through a parallel bank of > N cross correlators which basically compute the projection of r(t) > onto the N basis functions {fn(t)}, ..." > Question: the book gives the formula as integral[r(t)*fk(t)] from 0 to > T. Does projection f1 onto f2 means integral[f1*f2]?Yes, although 'inner product' is a more common term for the integral. But as you remember for vector calculus, an inner product between vectors a and b is the projection of a onto b. Or vice versa.> I can't find this > definition in book and wiki. > > 2. Covariance > Question: The book gives covariance in one part as E(n1*n2), but I > found another formula in the book as E[(x-mx)*(y-my)]. This confused > me. Can you help me to understand their difference?Covariance is defined as E[(x-mx)*(y-my)] while correlation is defined as E[x*y], the difference being whether the mean is subtracted or not. When it comes to communications, radar and sonar, one deals with signals that have zero mean. Authors often forget about the subtract-the-mean terms, since these are zero anyway. It is very common for authors to forget about the formal difference between correlation and covariance, as well as the special case of zero-mean signals. Lots of people use these terms and definitions interchangeably. It is just human nature, so you will have to keep your eyes open and look for these kinds of things, and maybe make the effort to be more precise in your own writings. Rune
Reply by ●April 8, 20092009-04-08
On 2009-04-08 09:49:46 -0300, Rune Allnor <allnor@tele.ntnu.no> said:> On 8 Apr, 13:55, chenyong20...@gmail.com wrote: >> On 4月7日, 下午9时42分, Rune Allnor <all...@tele. > ntnu.no> wrote: >> >> >> >> >> >>> On 7 Apr, 15:16, chenyong20...@gmail.com wrote: >> >>>> Hi, >> >>>> I'm learning digital communication now. I use Proakis' "Digital >>>> Communications" as my text book. Now several terms confused me, which >>>> are projection, correlation and covariance. These terms appeared in >>>> the book many times, but with different formula. >> >>> As others already said, it's hard to answer generic questions. >> >>> Having said that, it might be helpful to you that the terms you >>> mention are generic concepts, which can be used in different >>> contexts. For instance, correlation and covariance are statistical >>> terms that are all but omnipresent. Chances are they are used >>> in almost every context where statistics is used. Different formulas >>> could also be explained by that one uses different estimators >>> for the same statistic. >> >>> You also already know the concept of projections from >>> basic vector calculus, where projections are computed >>> in terms of inner products between vectors. It's the same >>> general idea in complex-valued N-D space, but the >>> expressions become a bit more involved, particularly >>> if you project onto planes instead of vectors. >> >>> Rune >> >> Hi Dilip & Rune, >> >> thanks for your reply. Well, I'm learning Proakis' book "Digital >> Communications". This book is full of mathematics. There are problems >> when I'm reading chapter 5 "Optimum Receivers for the additive white >> gaussian noise channel". >> >> 1. Projection. >> "Suppose the received signal r(t) is passed through a parallel bank of >> N cross correlators which basically compute the projection of r(t) >> onto the N basis functions {fn(t)}, ..." >> Question: the book gives the formula as integral[r(t)*fk(t)] from 0 to >> T. Does projection f1 onto f2 means integral[f1*f2]? > > Yes, although 'inner product' is a more common term for the > integral. But as you remember for vector calculus, an inner > product between vectors a and b is the projection of a onto b. > Or vice versa. > >> I can't find this >> definition in book and wiki. >> >> 2. Covariance >> Question: The book gives covariance in one part as E(n1*n2), but I >> found another formula in the book as E[(x-mx)*(y-my)]. This confused >> me. Can you help me to understand their difference? > > Covariance is defined as E[(x-mx)*(y-my)] while correlation is > defined as E[x*y], the difference being whether the mean is > subtracted or not. When it comes to communications, radar > and sonar, one deals with signals that have zero mean. > Authors often forget about the subtract-the-mean terms, > since these are zero anyway.Correlation is a rescaled covariance. Correlations are between -1 and +1. The scaling is by the product of the two standard deviations. In the case of a time series with both autocovariance and autocorrelations the scale factor is just the autocovariance at time zero. The appearance of formulae withut a mean being subtracted is a pretty sure sign that an assumption of zero mean was stated shortly before, or at the beginning of the book or paper, the formulae. Stronger coffee in the morning is called for as you got the zero mean assumption correct but fell on your face over the scaling of correlations. Maybe too much ice on the roads even if spring is coming!> It is very common for authors to forget about the formal > difference between correlation and covariance, as well as > the special case of zero-mean signals. Lots of people > use these terms and definitions interchangeably. It is > just human nature, so you will have to keep your eyes > open and look for these kinds of things, and maybe > make the effort to be more precise in your own writings.Correlation seems to be much more common in everyday speech and means appening together. It is a qualitative notion in speech with the value between +1 and -1 of little import. Covariance is more of an arithmetical notion.> Rune
Reply by ●April 8, 20092009-04-08
On 4月8日, 下午9时15分, Gordon Sande <g.sa...@worldnet.att.net> wrote:> On 2009-04-08 09:49:46 -0300, Rune Allnor <all...@tele.ntnu.no> said: > > > > > On 8 Apr, 13:55, chenyong20...@gmail.com wrote: > >> On 4月7日, 下午9时42分, Rune Allnor <all...@tele. > > ntnu.no> wrote: > > >>> On 7 Apr, 15:16, chenyong20...@gmail.com wrote: > > >>>> Hi, > > >>>> I'm learning digital communication now. I use Proakis' "Digital > >>>> Communications" as my text book. Now several terms confused me, which > >>>> are projection, correlation and covariance. These terms appeared in > >>>> the book many times, but with different formula. > > >>> As others already said, it's hard to answer generic questions. > > >>> Having said that, it might be helpful to you that the terms you > >>> mention are generic concepts, which can be used in different > >>> contexts. For instance, correlation and covariance are statistical > >>> terms that are all but omnipresent. Chances are they are used > >>> in almost every context where statistics is used. Different formulas > >>> could also be explained by that one uses different estimators > >>> for the same statistic. > > >>> You also already know the concept of projections from > >>> basic vector calculus, where projections are computed > >>> in terms of inner products between vectors. It's the same > >>> general idea in complex-valued N-D space, but the > >>> expressions become a bit more involved, particularly > >>> if you project onto planes instead of vectors. > > >>> Rune > > >> Hi Dilip & Rune, > > >> thanks for your reply. Well, I'm learning Proakis' book "Digital > >> Communications". This book is full of mathematics. There are problems > >> when I'm reading chapter 5 "Optimum Receivers for the additive white > >> gaussian noise channel". > > >> 1. Projection. > >> "Suppose the received signal r(t) is passed through a parallel bank of > >> N cross correlators which basically compute the projection of r(t) > >> onto the N basis functions {fn(t)}, ..." > >> Question: the book gives the formula as integral[r(t)*fk(t)] from 0 to > >> T. Does projection f1 onto f2 means integral[f1*f2]? > > > Yes, although 'inner product' is a more common term for the > > integral. But as you remember for vector calculus, an inner > > product between vectors a and b is the projection of a onto b. > > Or vice versa. > > >> I can't find this > >> definition in book and wiki. > > >> 2. Covariance > >> Question: The book gives covariance in one part as E(n1*n2), but I > >> found another formula in the book as E[(x-mx)*(y-my)]. This confused > >> me. Can you help me to understand their difference? > > > Covariance is defined as E[(x-mx)*(y-my)] while correlation is > > defined as E[x*y], the difference being whether the mean is > > subtracted or not. When it comes to communications, radar > > and sonar, one deals with signals that have zero mean. > > Authors often forget about the subtract-the-mean terms, > > since these are zero anyway. > > Correlation is a rescaled covariance. Correlations are > between -1 and +1. The scaling is by the product of the two > standard deviations. In the case of a time series with > both autocovariance and autocorrelations the scale factor > is just the autocovariance at time zero. > > The appearance of formulae withut a mean being subtracted > is a pretty sure sign that an assumption of zero mean was > stated shortly before, or at the beginning of the book or > paper, the formulae. > > Stronger coffee in the morning is called for as you got the > zero mean assumption correct but fell on your face over > the scaling of correlations. Maybe too much ice on the roads > even if spring is coming! > > > It is very common for authors to forget about the formal > > difference between correlation and covariance, as well as > > the special case of zero-mean signals. Lots of people > > use these terms and definitions interchangeably. It is > > just human nature, so you will have to keep your eyes > > open and look for these kinds of things, and maybe > > make the effort to be more precise in your own writings. > > Correlation seems to be much more common in everyday speech > and means appening together. It is a qualitative notion in > speech with the value between +1 and -1 of little import. > Covariance is more of an arithmetical notion. > > > RuneHi Rune and Gordon, I got your reply and have some clear sight. Thanks for both of you very much.
Reply by ●April 8, 20092009-04-08
On 8 Apr, 15:15, Gordon Sande <g.sa...@worldnet.att.net> wrote:> On 2009-04-08 09:49:46 -0300, Rune Allnor <all...@tele.ntnu.no> said: > > > > > > > On 8 Apr, 13:55, chenyong20...@gmail.com wrote: > >> On 4月7日, 下午9时42分, Rune Allnor <all...@tele. > > ntnu.no> wrote: > > >>> On 7 Apr, 15:16, chenyong20...@gmail.com wrote: > > >>>> Hi, > > >>>> I'm learning digital communication now. I use Proakis' "Digital > >>>> Communications" as my text book. Now several terms confused me, which > >>>> are projection, correlation and covariance. These terms appeared in > >>>> the book many times, but with different formula. > > >>> As others already said, it's hard to answer generic questions. > > >>> Having said that, it might be helpful to you that the terms you > >>> mention are generic concepts, which can be used in different > >>> contexts. For instance, correlation and covariance are statistical > >>> terms that are all but omnipresent. Chances are they are used > >>> in almost every context where statistics is used. Different formulas > >>> could also be explained by that one uses different estimators > >>> for the same statistic. > > >>> You also already know the concept of projections from > >>> basic vector calculus, where projections are computed > >>> in terms of inner products between vectors. It's the same > >>> general idea in complex-valued N-D space, but the > >>> expressions become a bit more involved, particularly > >>> if you project onto planes instead of vectors. > > >>> Rune > > >> Hi Dilip & Rune, > > >> thanks for your reply. Well, I'm learning Proakis' book "Digital > >> Communications". This book is full of mathematics. There are problems > >> when I'm reading chapter 5 "Optimum Receivers for the additive white > >> gaussian noise channel". > > >> 1. Projection. > >> "Suppose the received signal r(t) is passed through a parallel bank of > >> N cross correlators which basically compute the projection of r(t) > >> onto the N basis functions {fn(t)}, ..." > >> Question: the book gives the formula as integral[r(t)*fk(t)] from 0 to > >> T. Does projection f1 onto f2 means integral[f1*f2]? > > > Yes, although 'inner product' is a more common term for the > > integral. But as you remember for vector calculus, an inner > > product between vectors a and b is the projection of a onto b. > > Or vice versa. > > >> I can't find this > >> definition in book and wiki. > > >> 2. Covariance > >> Question: The book gives covariance in one part as E(n1*n2), but I > >> found another formula in the book as E[(x-mx)*(y-my)]. This confused > >> me. Can you help me to understand their difference? > > > Covariance is defined as E[(x-mx)*(y-my)] while correlation is > > defined as E[x*y], the difference being whether the mean is > > subtracted or not. When it comes to communications, radar > > and sonar, one deals with signals that have zero mean. > > Authors often forget about the subtract-the-mean terms, > > since these are zero anyway. > > Correlation is a rescaled covariance.I have some recollection that the 'correlation' is defined differently, depending on the source. However, my recollections have been proven wrong in the past, so I have to check the sources after I get back home.> Correlations are > between -1 and +1. The scaling is by the product of the two > standard deviations. In the case of a time series with > both autocovariance and autocorrelations the scale factor > is just the autocovariance at time zero.This is common in statistics literature, yes. I'm pretty sure I have seen things be defined differently in DSP texts - the normalization is not too common there. But again, I'll have to check my sources.> The appearance of formulae withut a mean being subtracted > is a pretty sure sign that an assumption of zero mean was > stated shortly before, or at the beginning of the book or > paper, the formulae.Sure. But you need to keep an eye out for that assumption.> Stronger coffee in the morning is called for as you got the > zero mean assumption correct but fell on your face over > the scaling of correlations. Maybe too much ice on the roads > even if spring is coming!True, driving on icey roads with spike-less winter tyres is no joke. That 0.5 mm sheet of ice on the asphalt can really mess up your day. Coffee is not a solution, though, as you don't make fewer mistakes, just make mistakes for different reasons. Rune
Reply by ●April 9, 20092009-04-09
On 8 Apr, 17:07, Rune Allnor <all...@tele.ntnu.no> wrote:> On 8 Apr, 15:15, Gordon Sande <g.sa...@worldnet.att.net> wrote:> > On 2009-04-08 09:49:46 -0300, Rune Allnor <all...@tele.ntnu.no> said:> > > Covariance is defined as E[(x-mx)*(y-my)] while correlation is > > > defined as �E[x*y],> > Correlation is a rescaled covariance. > > I have some recollection that the 'correlation' is defined > differently, depending on the source. However, my recollections > have been proven wrong in the past, so I have to check > the sources after I get back home. > > > Correlations are > > between -1 and +1.It seems it might be weeks before I get the oportunity to check my own books, so we might save some time if somebody could help check what the texts below say about the difference between correlation and covariance: Proakis & Manolakis: "Digital Signal Processing" (preferably the 2nd edition (1992'ish), since that was the one I first used), and Therrien: "Discrete Random Signals and Statistical Signal Processing" (1992) I am 99.9% sure Therrien describes the difference as I did in the post quoted above. For completeness, it would be interesting to see what is said on the matter in Kay: "Modern Spectral Estimation: Theory and Application" (1988) Marple: "Digital Spectral Analysis: With Applications" (1987) (can't check the Marple book myself, since I only saw a library copy decades ago) Bendat & Piersol: "Random Data" (2000) The one book where I am pretty sure to have seen the [-1, 1] scaling of the correlation mentioned, is Shumway: "Applied Statistical Time Series Analysis " (1988) which, as I (possibly wrongly) recall, talks about this quantity as the correlation *coefficient*, which is something else than correlation - much the same as the 'directional cosine' between two vectors is different from the 'inner product' between the same vectors, although one might say they express similar relations. So I'd appreciate if anyone who might have one or more of these books available could check out what they say about the difference between correlation and covariance. Rune
Reply by ●April 14, 20092009-04-14
On Apr 9, 4:21�am, Rune Allnor <all...@tele.ntnu.no> wrote:> On 8 Apr, 17:07, Rune Allnor <all...@tele.ntnu.no> wrote: > > > On 8 Apr, 15:15, Gordon Sande <g.sa...@worldnet.att.net> wrote: > > > On 2009-04-08 09:49:46 -0300, Rune Allnor <all...@tele.ntnu.no> said: > > > > Covariance is defined as E[(x-mx)*(y-my)] while correlation is > > > > defined as �E[x*y], > > > Correlation is a rescaled covariance. > > > I have some recollection that the 'correlation' is defined > > differently, depending on the source. However, my recollections > > have been proven wrong in the past, so I have to check > > the sources after I get back home. > > > > Correlations are > > > between -1 and +1. > > It seems it might be weeks before I get the oportunity > to check my own books, so we might save some time > if somebody could help check what the texts below > say about the difference between correlation and > covariance: > > Proakis & Manolakis: "Digital Signal Processing" > > (preferably the 2nd edition (1992'ish), since that was > the one I first used), and > > Therrien: "Discrete Random Signals and Statistical > � �Signal Processing" (1992) > > I am 99.9% sure Therrien describes the difference as > I did in the post quoted above. > > For completeness, it would be interesting to see > what is said on the matter in > > Kay: "Modern Spectral Estimation: Theory and > � �Application" (1988) > > Marple: "Digital Spectral Analysis: With Applications" > � � (1987) > > (can't check the Marple book myself, since I only saw > a library copy decades ago) > > Bendat & Piersol: "Random Data" (2000) > > The one book where I am pretty sure to have seen > the [-1, 1] scaling of the correlation mentioned, is > > Shumway: "Applied Statistical Time Series Analysis " > � � (1988) > > which, as I (possibly wrongly) recall, talks about this > quantity as the correlation *coefficient*, which is > something else than correlation - much the same as > the 'directional cosine' between two vectors is different > from the 'inner product' between the same vectors, > although one might say they express similar relations. > > So I'd appreciate if anyone who might have one or more > of these books available could check out what they say > about the difference between correlation and covariance. > > RuneI'm pretty sure that the scaling that produces the range -1 to +1 is referred to as the correlation coefficient. Cheers, David
