need help understanding meaning of FFT

Hi Everyone,

I observed an unexpected result in doing some FFT work recently, that
I think relates to understanding the fundamentals of an FFT I'm hoping
someone can help me understand better. I'm not a DSP expert.

I captured five waveforms sequentially using an oscilloscope. Aside
from capturing these 5 waveforms with a short time delay between each,
these 5 waveforms are otherwise identical. I captured 5 waveforms with
the intent of averaging them to reduce the random noise and cause any
spurs to stand out more clearly from the background random noise.

When I take an FFT in matlab and plot abs(2*Y) (that is, the
magnitude) for each of the individual captures, there are a couple
dozen spurs (spikes indicating sinusoidal noise) standing above a
random noise floor. When I look at the averaged spectrum, some of
these spurs are now gone. I would have expected this, but when I went
back to analyze some specific spurs, I found that in several cases,
there would be a spur standing clearly above the noise floor in the
exact same frequency bin for EACH of the five individual captures, but
NOT in the averaged spectrum.

For example, the following spur is peaking at the same position in the
FFT for individual captures...

Y1(363540) =     5.754953052662731e-015 +4.485425416719249e-015i
Y2(363540) =     6.933769335163839e-015 -4.932545982083068e-015i
Y3(363540) =    -1.212685676778186e-015 -7.810350064989210e-015i
Y4(363540) =    -8.210562239595824e-015 +2.188852638554278e-015i
Y5(363540) =     2.012052774970021e-017 +7.208534215893042e-015i

but is buried in the noise when averaged...

avg_Y(363540) =     6.571189998404520e-016 +2.279832448188584e-016i

I don't doubt the math here, but I'm looking for some understanding
what this says about my data. How can a spur be present in all five
individual captures, but absent from the average of them? I know
there's a thing called destructive interference that means the
amplitudes and phases can add to cancel each other, and perhaps that's
what's going on here, but I'm simply trying to find out which
frequencies are spurs in the original signal.

If I analyze the instantaneous signal I get one answer, and if I
analyze the averaged signal I get another answer. Is it that I cannot
use the averaged signal if I want to answer the question "where are
the spurs in this signal?".

Any intuition or deeper understanding appreciated. Best regards, -GGK

ggk wrote:
> 
> Hi Everyone,
> 
> I observed an unexpected result in doing some FFT work recently, that
> I think relates to understanding the fundamentals of an FFT I'm hoping
> someone can help me understand better. I'm not a DSP expert.
> 
> I captured five waveforms sequentially using an oscilloscope. Aside
> from capturing these 5 waveforms with a short time delay between each,
> these 5 waveforms are otherwise identical. I captured 5 waveforms with
> the intent of averaging them to reduce the random noise and cause any
> spurs to stand out more clearly from the background random noise.
> 
> When I take an FFT in matlab and plot abs(2*Y) (that is, the
> magnitude) for each of the individual captures, there are a couple
> dozen spurs (spikes indicating sinusoidal noise) standing above a
> random noise floor. When I look at the averaged spectrum, some of
> these spurs are now gone. I would have expected this, but when I went
> back to analyze some specific spurs, I found that in several cases,
> there would be a spur standing clearly above the noise floor in the
> exact same frequency bin for EACH of the five individual captures, but
> NOT in the averaged spectrum.
> 
> For example, the following spur is peaking at the same position in the
> FFT for individual captures...
> 
> Y1(363540) =     5.754953052662731e-015 +4.485425416719249e-015i
> Y2(363540) =     6.933769335163839e-015 -4.932545982083068e-015i
> Y3(363540) =    -1.212685676778186e-015 -7.810350064989210e-015i
> Y4(363540) =    -8.210562239595824e-015 +2.188852638554278e-015i
> Y5(363540) =     2.012052774970021e-017 +7.208534215893042e-015i

I'm probably just missing something obvious. You have negative numbers
yet you said these were from an abs() procedure. Also these numbers
are extremely close to zero. So close that it is hard to imagine you
would be able to derive anything meaningful from them.

-jim 



> 
> but is buried in the noise when averaged...
> 
> avg_Y(363540) =     6.571189998404520e-016 +2.279832448188584e-016i
> 
> I don't doubt the math here, but I'm looking for some understanding
> what this says about my data. How can a spur be present in all five
> individual captures, but absent from the average of them? I know
> there's a thing called destructive interference that means the
> amplitudes and phases can add to cancel each other, and perhaps that's
> what's going on here, but I'm simply trying to find out which
> frequencies are spurs in the original signal.
> 
> If I analyze the instantaneous signal I get one answer, and if I
> analyze the averaged signal I get another answer. Is it that I cannot
> use the averaged signal if I want to answer the question "where are
> the spurs in this signal?".
> 
> Any intuition or deeper understanding appreciated. Best regards, -GGK


----== Posted via Pronews.Com - Unlimited-Unrestricted-Secure Usenet News==----
http://www.pronews.com The #1 Newsgroup Service in the World! >100,000 Newsgroups
---= - Total Privacy via Encryption =---

On 8 Aug, 18:16, ggk <ggkm...@comcast.net> wrote:
> Hi Everyone,
>
> I observed an unexpected result in doing some FFT work recently, that
> I think relates to understanding the fundamentals of an FFT I'm hoping
> someone can help me understand better. I'm not a DSP expert.
>
> I captured five waveforms sequentially using an oscilloscope. Aside
> from capturing these 5 waveforms with a short time delay between each,
> these 5 waveforms are otherwise identical. I captured 5 waveforms with
> the intent of averaging them to reduce the random noise and cause any
> spurs to stand out more clearly from the background random noise.
>
> When I take an FFT in matlab and plot abs(2*Y) (that is, the
> magnitude) for each of the individual captures, there are a couple
> dozen spurs (spikes indicating sinusoidal noise) standing above a
> random noise floor. When I look at the averaged spectrum, some of
> these spurs are now gone. I would have expected this, but when I went
> back to analyze some specific spurs, I found that in several cases,
> there would be a spur standing clearly above the noise floor in the
> exact same frequency bin for EACH of the five individual captures, but
> NOT in the averaged spectrum.
>
> For example, the following spur is peaking at the same position in the
> FFT for individual captures...
>
> Y1(363540) = &#4294967295; &#4294967295; 5.754953052662731e-015 +4.485425416719249e-015i
> Y2(363540) = &#4294967295; &#4294967295; 6.933769335163839e-015 -4.932545982083068e-015i
> Y3(363540) = &#4294967295; &#4294967295;-1.212685676778186e-015 -7.810350064989210e-015i
> Y4(363540) = &#4294967295; &#4294967295;-8.210562239595824e-015 +2.188852638554278e-015i
> Y5(363540) = &#4294967295; &#4294967295; 2.012052774970021e-017 +7.208534215893042e-015i

The magnitudes are pretty small, 1e-15 isn't a very strong signal.
What is the energy in the total signal?

Try to compute sum(Y.*conj(Y1)) and post the result here.

> but is buried in the noise when averaged...
>
> avg_Y(363540) = &#4294967295; &#4294967295; 6.571189998404520e-016 +2.279832448188584e-016i
>
> I don't doubt the math here, but I'm looking for some understanding
> what this says about my data. How can a spur be present in all five
> individual captures, but absent from the average of them? I know
> there's a thing called destructive interference that means the
> amplitudes and phases can add to cancel each other, and perhaps that's
> what's going on here, but I'm simply trying to find out which
> frequencies are spurs in the original signal.

Coherent averaging of complex numbers works like this:

z1 =  1
z2 =  i
z3 = -1
z4 = -i

(z1+z2+z3+z4)/4 = 0

So even if the magnitudes of each number equals 1, the
coherent average equals 0.

*In*coherent averaging is to average the magnitudes (ignoring
phases), in which case

(|z1|+|z2|+|z3|+|z4|)/4 = 1.

> If I analyze the instantaneous signal I get one answer, and if I
> analyze the averaged signal I get another answer. Is it that I cannot
> use the averaged signal if I want to answer the question "where are
> the spurs in this signal?".

If you want to use coherent averaging you need a fixed time
reference in order to match up the phases. If that's not an
option you might want to use cross spectra.

Rune

On Aug 8, 12:42 pm, jim <"sjedgingN0sp"@m...@mwt.net> wrote:
> ggk wrote:
>
>
> > If I analyze the instantaneous signal I get one answer, and if I
> > analyze the averaged signal I get another answer. Is it that I cannot
> > use the averaged signal if I want to answer the question "where are
> > the spurs in this signal?".
>

Hint,

Imagine if you captured one sequence that went

50,100,50,100,50,100,...

then your second sequence, which didn't have synchronized start phase,
went

100,50,100,50,100,50,...

Now average them (75,75,75,...) and think about where the spur went.

If you want to "average the fact that spurs are there" in a quick and
dirty way, consider averaging the abs of the FFT points.  This
effectively removes the unknown phase angle from consideration.

 - Kenn

Jim,

I should clarify. The data I listed above is the raw FFT data, but
when I plot the FFT to identify the spurs I only plot the magnitude
(that is, I plot 2*abs(Y)). I just wanted to show the raw FFT data in
case it helped. Also, the numbers are close to zero but the
measurement is fairly sensitive, and the noise floor is actually an
order of magnitude lower.

Rune,

I agree the energy is fairly small. It just so happens to be the
nature of the phenomenon I'm trying to analyze. Not sure what the
total energy is. Let's see, if I type sum(Y1.*conj(Y1)), it returns
4.86e-024.

Per your definitions I am using coherent averaging to evaluate the
average FFT. I think you and Ken are pointing to the same root cause
of this... lack of a fixed time reference.  If coherent averaging
isn't taking me where I want to go, can you talk about "cross
spectra"... what is it, how to do it, how it would help determine the
spur frequencies present?

Ken,

Thanks for the mental exercise to hightlight the point. I may not have
got it without that! So if I understand what you and Rune are saying,
is that because the 5 waveforms are not connected in real-time, they
lack a consistent time reference between the files, which means when
you throw in the phase information (the imaginary number in the FFT
freq bin), things don't line up. Thus, I shouldn't use coherent
averaging, but rather ... what? If I have 3 frequency bins to average,
should I use...

a1 + b1i
a2 + b2i
a3 + b3i

--> ( abs(a1 + b1i) + abs(a2 + b2i) + abs(a3 + b3i)) / 3

... for example?

Thanks so much! -GGK

On Aug 8, 1:21 pm, ggk <ggkm...@comcast.net> wrote:
> Jim,
>
> I should clarify. The data I listed above is the raw FFT data, but
> when I plot the FFT to identify the spurs I only plot the magnitude
> (that is, I plot 2*abs(Y)). I just wanted to show the raw FFT data in
> case it helped. Also, the numbers are close to zero but the
> measurement is fairly sensitive, and the noise floor is actually an
> order of magnitude lower.
>
> Rune,
>
> I agree the energy is fairly small. It just so happens to be the
> nature of the phenomenon I'm trying to analyze. Not sure what the
> total energy is. Let's see, if I type sum(Y1.*conj(Y1)), it returns
> 4.86e-024.
>
> Per your definitions I am using coherent averaging to evaluate the
> average FFT. I think you and Ken are pointing to the same root cause
> of this... lack of a fixed time reference.  If coherent averaging
> isn't taking me where I want to go, can you talk about "cross
> spectra"... what is it, how to do it, how it would help determine the
> spur frequencies present?
>
> Ken,
>
> Thanks for the mental exercise to hightlight the point. I may not have
> got it without that! So if I understand what you and Rune are saying,
> is that because the 5 waveforms are not connected in real-time, they
> lack a consistent time reference between the files, which means when
> you throw in the phase information (the imaginary number in the FFT
> freq bin), things don't line up. Thus, I shouldn't use coherent
> averaging, but rather ... what? If I have 3 frequency bins to average,
> should I use...
>
> a1 + b1i
> a2 + b2i
> a3 + b3i
>
> --> ( abs(a1 + b1i) + abs(a2 + b2i) + abs(a3 + b3i)) / 3
>
> ... for example?
>
> Thanks so much! -GGK

If you have 3 captures to average, just average the co-located bins'
magnitude (a.k.a absolute value), not the phase.  Taking the absolute
value of a complex number is more or less the same thing as taking
x*conj(x), as suggested earlier   - that's just some alegbra to show
that (abs(x))^2 = x*conj(x).

In matlab, you'd do something like
p=(abs(fft(a))+abs(fft(b))+abs(fft(c)))/3;
where a,b,c are the three equal-length input signal captures from your
scope. Then look at the value of, for example, p(363540), to find the
average amplitude in bin 363540 across all three captures. If you then
want to average adjacent bins (say bins 363540 through 363543) you can
do this as a simple arithmetic average, since the values are real.

This will get you to the point where you know that the spur is there -
trying to extract, demodulate, or measure it accurately will be a
little more work.

 - Kenn

On Aug 8, 4:21�pm, kennheinr...@sympatico.ca wrote:
> On Aug 8, 1:21 pm, ggk <ggkm...@comcast.net> wrote:
>
>
>
>
>
> > Jim,
>
> > I should clarify. The data I listed above is the raw FFT data, but
> > when I plot the FFT to identify the spurs I only plot the magnitude
> > (that is, I plot 2*abs(Y)). I just wanted to show the raw FFT data in
> > case it helped. Also, the numbers are close to zero but the
> > measurement is fairly sensitive, and the noise floor is actually an
> > order of magnitude lower.
>
> > Rune,
>
> > I agree the energy is fairly small. It just so happens to be the
> > nature of the phenomenon I'm trying to analyze. Not sure what the
> > total energy is. Let's see, if I type sum(Y1.*conj(Y1)), it returns
> > 4.86e-024.
>
> > Per your definitions I am using coherent averaging to evaluate the
> > average FFT. I think you and Ken are pointing to the same root cause
> > of this... lack of a fixed time reference. &#4294967295;If coherent averaging
> > isn't taking me where I want to go, can you talk about "cross
> > spectra"... what is it, how to do it, how it would help determine the
> > spur frequencies present?
>
> > Ken,
>
> > Thanks for the mental exercise to hightlight the point. I may not have
> > got it without that! So if I understand what you and Rune are saying,
> > is that because the 5 waveforms are not connected in real-time, they
> > lack a consistent time reference between the files, which means when
> > you throw in the phase information (the imaginary number in the FFT
> > freq bin), things don't line up. Thus, I shouldn't use coherent
> > averaging, but rather ... what? If I have 3 frequency bins to average,
> > should I use...
>
> > a1 + b1i
> > a2 + b2i
> > a3 + b3i
>
> > --> ( abs(a1 + b1i) + abs(a2 + b2i) + abs(a3 + b3i)) / 3
>
> > ... for example?
>
> > Thanks so much! -GGK
>
> If you have 3 captures to average, just average the co-located bins'
> magnitude (a.k.a absolute value), not the phase. &#4294967295;Taking the absolute
> value of a complex number is more or less the same thing as taking
> x*conj(x), as suggested earlier &#4294967295; - that's just some alegbra to show
> that (abs(x))^2 = x*conj(x).
>
> In matlab, you'd do something like
> p=(abs(fft(a))+abs(fft(b))+abs(fft(c)))/3;
> where a,b,c are the three equal-length input signal captures from your
> scope. Then look at the value of, for example, p(363540), to find the
> average amplitude in bin 363540 across all three captures. If you then
> want to average adjacent bins (say bins 363540 through 363543) you can
> do this as a simple arithmetic average, since the values are real.
>
> This will get you to the point where you know that the spur is there -
> trying to extract, demodulate, or measure it accurately will be a
> little more work.
>
> &#4294967295;- Kenn- Hide quoted text -
>
> - Show quoted text -

Generally for averaging FFTs, you average the magnitude squared for
each frequency rather than the magnitude.

Dirk

Thanks so much Kenn,

I think my first step here is to identify where the spur frequencies
are located, with high accuracy. Seems like your scheme above is the
perfect answer.

Best regards, -GGK

On Aug 8, 1:43&#4294967295;pm, dbell <bellda2...@cox.net> wrote:
> On Aug 8, 4:21&#4294967295;pm, kennheinr...@sympatico.ca wrote:
>
>
>
>
>
> > On Aug 8, 1:21 pm, ggk <ggkm...@comcast.net> wrote:
>
> > > Jim,
>
> > > I should clarify. The data I listed above is the raw FFT data, but
> > > when I plot the FFT to identify the spurs I only plot the magnitude
> > > (that is, I plot 2*abs(Y)). I just wanted to show the raw FFT data in
> > > case it helped. Also, the numbers are close to zero but the
> > > measurement is fairly sensitive, and the noise floor is actually an
> > > order of magnitude lower.
>
> > > Rune,
>
> > > I agree the energy is fairly small. It just so happens to be the
> > > nature of the phenomenon I'm trying to analyze. Not sure what the
> > > total energy is. Let's see, if I type sum(Y1.*conj(Y1)), it returns
> > > 4.86e-024.
>
> > > Per your definitions I am using coherent averaging to evaluate the
> > > average FFT. I think you and Ken are pointing to the same root cause
> > > of this... lack of a fixed time reference. &#4294967295;If coherent averaging
> > > isn't taking me where I want to go, can you talk about "cross
> > > spectra"... what is it, how to do it, how it would help determine the
> > > spur frequencies present?
>
> > > Ken,
>
> > > Thanks for the mental exercise to hightlight the point. I may not have
> > > got it without that! So if I understand what you and Rune are saying,
> > > is that because the 5 waveforms are not connected in real-time, they
> > > lack a consistent time reference between the files, which means when
> > > you throw in the phase information (the imaginary number in the FFT
> > > freq bin), things don't line up. Thus, I shouldn't use coherent
> > > averaging, but rather ... what? If I have 3 frequency bins to average,
> > > should I use...
>
> > > a1 + b1i
> > > a2 + b2i
> > > a3 + b3i
>
> > > --> ( abs(a1 + b1i) + abs(a2 + b2i) + abs(a3 + b3i)) / 3
>
> > > ... for example?
>
> > > Thanks so much! -GGK
>
> > If you have 3 captures to average, just average the co-located bins'
> > magnitude (a.k.a absolute value), not the phase. &#4294967295;Taking the absolute
> > value of a complex number is more or less the same thing as taking
> > x*conj(x), as suggested earlier &#4294967295; - that's just some alegbra to show
> > that (abs(x))^2 = x*conj(x).
>
> > In matlab, you'd do something like
> > p=(abs(fft(a))+abs(fft(b))+abs(fft(c)))/3;
> > where a,b,c are the three equal-length input signal captures from your
> > scope. Then look at the value of, for example, p(363540), to find the
> > average amplitude in bin 363540 across all three captures. If you then
> > want to average adjacent bins (say bins 363540 through 363543) you can
> > do this as a simple arithmetic average, since the values are real.
>
> > This will get you to the point where you know that the spur is there -
> > trying to extract, demodulate, or measure it accurately will be a
> > little more work.
>
> > &#4294967295;- Kenn- Hide quoted text -
>
> > - Show quoted text -
>
> Generally for averaging FFTs, you average the magnitude squared for
> each frequency rather than the magnitude.
>
> Dirk- Hide quoted text -
>
> - Show quoted text -

Thanks Dirk, can anyone explain the various tradeoffs in choosing to
average as a magnitude versus magnitude squared (I assume root-sum-of-
squares approach)? When is one more appropriate than another?

On Aug 8, 3:44&#4294967295;pm, ggk <ggkm...@comcast.net> wrote:
> Thanks so much Kenn,
>
> I think my first step here is to identify where the spur frequencies
> are located, with high accuracy. Seems like your scheme above is the
> perfect answer.
>
> Best regards, -GGK

The spectrum of a signal is defined as:

 S_{xx}(f) = E[ | X(f) |^2 ]

as Dirk suggested.