Rigorous definition of the Spectral Density of a random signal?

Carlos Moreno wrote:
> [...]
> I feel so frustrated that I haven't been able to
> communicate what's in my mind!!  

Carlos,

Although I haven't read all of the dozens of
articles in this thread, would the following
describe the "question in your head"?  ...:

   Let X(t) be a zero-mean, IID, continuous-time process
   with variance of sigma^2. Now we know that since this
   function is IID, it has a white PSD and therefore its
   autocorrelation function at lag 0, R_XX(0), should be
   b*delta(tau), where delta(tau) is the usual Dirac delta
   function and b is some constant.

   However, we also know that the autocorrelation function R_XX(tau)
   is defined to be E[X(t)*X(t+tau)], and therefore that R_XX(0)
   is E[X^2] (where we have dropped the dependence on t since the
   process is IID). Therefore in this sense R_XX(0) = E[X^2] = sigma^2,
   which is not infinite.

Is this the contradiction you speak of?

PS: I'm a little put off that you haven't responded to my post on
the units of PSD. Did you see it?
-- 
%  Randy Yates                  % "...the answer lies within your soul
%% Fuquay-Varina, NC            %       'cause no one knows which side
%%% 919-577-9882                %                   the coin will fall."
%%%% <yates@ieee.org>           %  'Big Wheels', *Out of the Blue*, ELO
http://home.earthlink.net/~yatescr

Hi Randy,

Randy Yates wrote:
> 
> Carlos,
> 
> Although I haven't read all of the dozens of
> articles in this thread, would the following
> describe the "question in your head"?  ...:
> 
>   Let X(t) be a zero-mean, IID, continuous-time process
>   with variance of sigma^2. Now we know that since this
>   function is IID, it has a white PSD and therefore its
>   autocorrelation function at lag 0, R_XX(0), should be
>   b*delta(tau), where delta(tau) is the usual Dirac delta
>   function and b is some constant.
> 
>   However, we also know that the autocorrelation function R_XX(tau)
>   is defined to be E[X(t)*X(t+tau)], and therefore that R_XX(0)
>   is E[X^2] (where we have dropped the dependence on t since the
>   process is IID). Therefore in this sense R_XX(0) = E[X^2] = sigma^2,
>   which is not infinite.
> 
> Is this the contradiction you speak of?

Maybe.  As a matter of fact, this may be at the heart of
what I perceive as a contradiction.  What surprises me is
that you show the two "contradictory" derivations, but
don't mention why the contradiction, or if it is indeed
a contradiction (what I mean is that it might be an
*apparent* contradiction).

In fact, now that you "equate" the results of those two
different approaches, I guess one problem (maybe part of
the same problem?) is that I'm not sure I understand the
justification of defining the PSD and Rxx as a Fourier
transform pair.  I mean, I remember from my Signals and
Systems course, understanding the intuitive interpretation
of it:  the more high-frequency contents, the less correlated
close samples would be -- in particular, for white noise,
with strong frequency contents going to infinity, samples
arbitrarily close are still uncorrelated.  That made (and
still makes) perfect sense to me.

Fine.  So, why the contradiction?  (or the "apparent"
contradiction of results?).

In fact, how would Rxx(0) be related to the average
electrical power of the signal?  (the average "watts"
that a voltage signal would dissipate on a 1-ohm resistor).
I seem to see clearly how the E{x^2} is related to the
average power of a voltage signal, but not so sure that
I understand the link with Rxx.

> PS: I'm a little put off that you haven't responded to my post on
> the units of PSD. Did you see it?

Yes!!  And I apologize for letting it go unnoticed!!
The thing is that I read it together with the other
messages that finally made me decide that I was starting
to sound like a troll, and thus decided to witdraw...
My sincere apologies!

Ironically, that was *the* message that best seemed to
address my initial point  (I mean, it was the one message
that did not give me that feeling of "this guy didn't
understand what I was asking"  -- not that I'm saying
that the others didn't understand; but almost all the
other messages (the initial ones, at least) gave me, at
some extent, the impression that they were going off a
tangent).

When you showed me that the units of the PSD are indeed
watts per herz, that seemed like a very precise attempt
at convincing me of what the PSD is... (still, after
reading that message, I could not seem ro reconcile
ideas that seemed contradictory, and then again, that
contributed to my decision that "maybe I should go
re-study these concepts before I continue to bother
these guys"  :-))

So, I'm hoping that you won't be too mad at me and will
be willing to elaborate a bit on this contradiction of
Rxx(0) being infinite while E{x^2} being finite.

Thanks!!

Carlos
--

Hi Carlos,

Carlos Moreno wrote:
> 
> Hi Randy,
> 
> Randy Yates wrote:
> 
>>
>> Carlos,
>>
>> Although I haven't read all of the dozens of
>> articles in this thread, would the following
>> describe the "question in your head"?  ...:
>>
>>   Let X(t) be a zero-mean, IID, continuous-time process
>>   with variance of sigma^2. Now we know that since this
>>   function is IID, it has a white PSD and therefore its
>>   autocorrelation function at lag 0, R_XX(0), should be
>>   b*delta(tau), where delta(tau) is the usual Dirac delta
>>   function and b is some constant.
>>
>>   However, we also know that the autocorrelation function R_XX(tau)
>>   is defined to be E[X(t)*X(t+tau)], and therefore that R_XX(0)
>>   is E[X^2] (where we have dropped the dependence on t since the
>>   process is IID). Therefore in this sense R_XX(0) = E[X^2] = sigma^2,
>>   which is not infinite.
>>
>> Is this the contradiction you speak of?
> 
> 
> Maybe.  As a matter of fact, this may be at the heart of
> what I perceive as a contradiction.  What surprises me is
> that you show the two "contradictory" derivations, but
> don't mention why the contradiction, or if it is indeed
> a contradiction (what I mean is that it might be an
> *apparent* contradiction).

It is indeed a contradiction to me, and I cannot resolve
the contradiction. It is one I have had for years.

> In fact, now that you "equate" the results of those two
> different approaches, I guess one problem (maybe part of
> the same problem?) is that I'm not sure I understand the
> justification of defining the PSD and Rxx as a Fourier
> transform pair.

The justification of the definition? Are you asking how
Wiener and Khinchine proved the theorem that the PSD
is the Fourier transform of the autocorrelation function?
That I cannot say - it is an interesting question, though.

>  I mean, I remember from my Signals and
> Systems course, understanding the intuitive interpretation
> of it:  the more high-frequency contents, the less correlated
> close samples would be -- in particular, for white noise,
> with strong frequency contents going to infinity, samples
> arbitrarily close are still uncorrelated.  That made (and
> still makes) perfect sense to me.
> 
> Fine.  So, why the contradiction?  (or the "apparent"
> contradiction of results?).

Because Rxx(tau) != F^(-1)[Sxx(w)], and they should be.

> In fact, how would Rxx(0) be related to the average
> electrical power of the signal?  (the average "watts"
> that a voltage signal would dissipate on a 1-ohm resistor).
> I seem to see clearly how the E{x^2} is related to the
> average power of a voltage signal, but not so sure that
> I understand the link with Rxx.

Rxx(0) = E[X^2(t)], by definition. Remember, Rxx(tau) is
defined to be E[X(t)*X(t+tau)], so when tau = 0, this
yields E[X^2(t)].

>> PS: I'm a little put off that you haven't responded to my post on
>> the units of PSD. Did you see it?
> 
> 
> Yes!!  And I apologize for letting it go unnoticed!!
> The thing is that I read it together with the other
> messages that finally made me decide that I was starting
> to sound like a troll, and thus decided to witdraw...
> My sincere apologies!

Acknowledged. Thanks, Carlos.

> Ironically, that was *the* message that best seemed to
> address my initial point  (I mean, it was the one message
> that did not give me that feeling of "this guy didn't
> understand what I was asking"  -- not that I'm saying
> that the others didn't understand; but almost all the
> other messages (the initial ones, at least) gave me, at
> some extent, the impression that they were going off a
> tangent).
> 
> When you showed me that the units of the PSD are indeed
> watts per herz, that seemed like a very precise attempt
> at convincing me of what the PSD is... (still, after
> reading that message, I could not seem ro reconcile
> ideas that seemed contradictory, and then again, that
> contributed to my decision that "maybe I should go
> re-study these concepts before I continue to bother
> these guys"  :-))
> 
> So, I'm hoping that you won't be too mad at me and will
> be willing to elaborate a bit on this contradiction of
> Rxx(0) being infinite while E{x^2} being finite.

Carlos, I am indeed not mad with you. Thanks for asking
these questions - it helps remind me and others of defintions
and concepts that tend to fade out unless you revisit them
often.

Regarding the contradiction, I hope my responses to you previously
in this post have clarified.

PS: Go to sci.math, where I posed this very question (in fact,
I had cut and paste it here in my last message to you). As of
a few minutes ago, I hadn't yet gotten any responses.
-- 
%  Randy Yates                  % "...the answer lies within your soul
%% Fuquay-Varina, NC            %       'cause no one knows which side
%%% 919-577-9882                %                   the coin will fall."
%%%% <yates@ieee.org>           %  'Big Wheels', *Out of the Blue*, ELO
http://home.earthlink.net/~yatescr

Carlos Moreno wrote:

   ...
> 
> I'm really hoping that someone will be able and
> willing to make me understand what's happening!!
> (so that I can leave you guys alone AND sleep
> peacefully at the same time  :-)) -- I'm serious!!
> I'm having nightmares about this!!!  :-(((
> 
> Cheers,
> 
> Carlos
> -- 

By applying correct geometric reasoning to a flawed figure, I can prove
that two lines which intersect are parallel. So what?

Consider a line of LEDs a foot apart. Each draws 10 milliwatts. It's not
absurd to say that if the line extends from -infinity to infinity, then
the power represented by them in the aggregate is infinite. Absurdity
lies in the belief that a line of such extent must follow rules that
apply to things that can be.

Jerry
-- 
Engineering is the art of making what you want from things you can get.
�����������������������������������������������������������������������

Randy Yates wrote:

> It is indeed a contradiction to me, and I cannot resolve
> the contradiction. It is one I have had for years.

Hmmm...  This thread is becoming interesting!!  :-)

>> In fact, now that you "equate" the results of those two
>> different approaches, I guess one problem (maybe part of
>> the same problem?) is that I'm not sure I understand the
>> justification of defining the PSD and Rxx as a Fourier
>> transform pair.
> 
> The justification of the definition? Are you asking how
> Wiener and Khinchine proved the theorem that the PSD
> is the Fourier transform of the autocorrelation function?

Well, I wasn't looking for something as rigorous as a proof
of the theorem.  Just an interpretation of the definition
that helps me understand it better (as opposed to an
interpretation *of the results* or an interpretation of
the consequences of such definition -- that, I think I
understand relatively well).

>> Fine.  So, why the contradiction?  (or the "apparent"
>> contradiction of results?).
> 
> Because Rxx(tau) != F^(-1)[Sxx(w)], and they should be.

Ok.

>> In fact, how would Rxx(0) be related to the average
>> electrical power of the signal?  (the average "watts"
>> that a voltage signal would dissipate on a 1-ohm resistor).
>> I seem to see clearly how the E{x^2} is related to the
>> average power of a voltage signal, but not so sure that
>> I understand the link with Rxx.
> 
> 
> Rxx(0) = E[X^2(t)], by definition. 

Huh??

I thought Rxx(0) was defined as the time-domain correlation:

             oo
            /
           |
Rxx(T) =  |   X^2(t) dt
           |
          /
           -oo

Notice that in that case, this represents the total energy
of the signal...  Hmmm, though something confuses me again:
this would be the energy of *one particular* realization.
(so, I think this brings me back to my original doubt:
what the hell does PSD and Rxx represent for a random
process?)

> Carlos, I am indeed not mad with you. 

I never actually thought you'd be mad, of course!  :-)

> PS: Go to sci.math, where I posed this very question 

Still unanswered.  We'll wait.  Maybe in that reference I
was pointed to (Priestly, vol. 1) such proof/justification
is given?

Thanks,

Carlos
--

Jerry Avins wrote:

> By applying correct geometric reasoning to a flawed figure, I can prove
> that two lines which intersect are parallel. So what?
> 
> Consider a line of LEDs a foot apart. Each draws 10 milliwatts. It's not
> absurd to say that if the line extends from -infinity to infinity, then
> the power represented by them in the aggregate is infinite. Absurdity
> lies in the belief that a line of such extent must follow rules that
> apply to things that can be.

Jerry,

You seem to keep missing my point.  What you're saying is more
or less equivalent to deriving an incorrect property of sinusoidal
signals and claiming that that's because complex numbers do not
exist in nature, and thus it is absurd to pretend that we'll get
a meaningful result when applied to something that exists in
nature, etc. etc.

Complex numbers *are* an absurdity, the way you define absurdity.

It is a mathematical abstraction that does not correspond to anything
that exist in our universe (at least nothing thay we humans are aware
of).

Complex numbers simply follow *the mathematical rules* established
by their purely mathematical definition.  Following **those**
mathematical definition, rules and axioms, you derive other
results.

Provided the definition that i^2 is equal to -1, Euler derived,
following that definition and following the usual axioms of
mathematics/algebra/calculus, that e^(x+iy) = e^x (cosy + i siny).

You could claim that anything that you derive from there is plain
wrong, since the imaginary unit does not exist in nature.

Same thing with the delta -- you claim that none of what I've said
makes sense because white noise can not exist in nature!

Then why do you justify that results derived by applying the
properties and the definition of Dirac's delta are indeed correct?? 
Direc's delta most certainly can not exist in nature;  however,
the results that you obtain using that mathematical abstraction
are verifiable mathematically (and well, from the practical point
of view, they also have relevance, since they *very closely*
approximate the way certain phisical systems behave).  You convert
a differential equation to its integral form, and applying the
definition of the delta (a definition that has no place in the real
world), which implies that the integral from -epsilon to epsilon
is 1 (for every epsilon > 0), then you obtain results that do not
contradict any other result obtained using mathematical constructs.

I insist:  it does not make sense to use the argument "white noise
can not exist in nature" to justify a contradiction that arises
in mathematical terms -- two different ways of analytically
calculating something that should be equivalent, yield different
results;   your argument would be valid if I claimed that in an
actual experiment, I expect to obtain whatever this or whatever
that, and I obtain something different from what the formulas tell
me...  Then, your argument *might* be valid  (emphasis on the
*might*;  the conclusions you draw from properties of complex
numbers, Dirac's delta, etc., *are indeed* verifiable in practice,
with an accuracy given by the precision of the model, the
implementation, and the precision of the measurements).


Carlos
--

Carlos Moreno wrote:
[...]

My feeling is that you're mixing things that
seems to be the same, but they're not.

The PDS of a stochastic process is the "power"
or "energy" the process can have in terms of
probability, not in terms of physical energy.

In case of white noise, it means that there
is no limitation to the possible sequence.
If not white, than the process is limited in
the "possible" realization and this is reflected
in the PDS as well.

The energy or power you calculate for a possible
realization is something else, it does not relate
with the PDS is any physical sense.
It just happens they can be exchanged, but they
should not.

bye,

-- 

piergiorgio

Piergiorgio Sartor wrote:
> Carlos Moreno wrote:
> [...]
> 
> My feeling is that you're mixing things that
> seems to be the same, but they're not.
> 
> The PDS of a stochastic process is the "power"
> or "energy" the process can have in terms of
> probability, not in terms of physical energy.

Hmm...  Interesting  (i.e., interesting how for
so many years I've misunderstood all these
concepts  :-)  I mean, :-( ).

I would ask you to define "power the process can
have in terms of probability"...  But I guess
this falls again in the "I-better-go-do-my-
homework" category  (homework not in the sense
that this question is part of some homework I
got;  but in the sense that I have my own work
to do in investigating these things)

> In case of white noise, it means that there
> is no limitation to the possible sequence.

Can you elaborate??  No limitation to the possible
sequence meaning what exactly?  (what kind of
limitation -- or lack thereof -- are we talking
about?)

> The energy or power you calculate for a possible
> realization is something else, it does not relate
> with the PDS is any physical sense.

I'm now confused by why are the units of the PSD
so conveniently Watts per Hz?

Thanks,

Carlos
--

Carlos Moreno wrote:

> Jerry Avins wrote:
> 
>> By applying correct geometric reasoning to a flawed figure, I can prove
>> that two lines which intersect are parallel. So what?
>>
>> Consider a line of LEDs a foot apart. Each draws 10 milliwatts. It's not
>> absurd to say that if the line extends from -infinity to infinity, then
>> the power represented by them in the aggregate is infinite. Absurdity
>> lies in the belief that a line of such extent must follow rules that
>> apply to things that can be.
> 
> 
> Jerry,
> 
> You seem to keep missing my point.   ...

It's worse than that: We're missing each other's points. Mine are

1. When integrating from -infinity to +infinity that which exists only
locally in time, the result is of no practical interest. Maybe that's too
strong. Put another way, integrating a function over all frequencies that
represents power and is valid only within a limited band can yield the
surprising conclusion that troubles you.

2. The assumption that successive samples of a signal are statistically
independent is only valid when the bandwidth of the signal is less than
half the sample rate. Your assumed properties of the signal should assure
you that the samples you hypothesize, and the calculations you perform on
them, are worthless.

Jerry
-- 
Engineering is the art of making what you want from things you can get.
�����������������������������������������������������������������������

Carlos Moreno wrote:
> Randy Yates wrote:
>>
>> Rxx(0) = E[X^2(t)], by definition. 
> 
> 
> Huh??
> 
> I thought Rxx(0) was defined as the time-domain correlation:
> 
>             oo
>            /
>           |
> Rxx(T) =  |   X^2(t) dt
>           |
>          /
>           -oo
> 
> Notice that in that case, this represents the total energy
> of the signal...  Hmmm, though something confuses me again:
> this would be the energy of *one particular* realization.
> (so, I think this brings me back to my original doubt:
> what the hell does PSD and Rxx represent for a random
> process?)

Yup, there's that definition too. But the autocorrelation function
which the Wiener-Khinchine theorem utilizes is the probabilistic
one.

>> PS: Go to sci.math, where I posed this very question 
> 
> 
> Still unanswered.  We'll wait.  Maybe in that reference I
> was pointed to (Priestly, vol. 1) such proof/justification
> is given?

Maybe. If you find it there, please let me know.
-- 
%  Randy Yates                  % "...the answer lies within your soul
%% Fuquay-Varina, NC            %       'cause no one knows which side
%%% 919-577-9882                %                   the coin will fall."
%%%% <yates@ieee.org>           %  'Big Wheels', *Out of the Blue*, ELO
http://home.earthlink.net/~yatescr