What is it I don't undersatand about FFT (&/or SCILAB)?

Started by Richard Owlett November 17, 2006
I'm interested in the general issue of how the spectrum of a signal 
changes over time.

In thread titled "FFT not showing high frequency oscillation", provided 
data that made a nice clean test case. I posed myself a "simple" 
question, "Is the parasitic oscillation changing over time?"

I imported his data into Excel for some experimentation.
Then decided to use SCILAB whose facilities are more appropriate to the 
size of data I will eventually be interested in.

My code (using SCILAB 4.0 under WinXP Pro)

sheets = readxls('E:\parasitic.xls')
s1=sheets(1);               // get the first sheet
dwave=s1(:,2);
y=size(dwave,'r');
k=max(abs(dwave([2:y])));
dwave=dwave([2:y])/k;       // first row is label
                             // normalize array

for i=1:19
     j=500*(i-1);           // data ~10,000 points
                            // oscillations are ~1500 points wide
     array_of_ffts(i,:)=abs(fft(dwave([j+1:j+500]))');
end

fft_n=[1:19];
frequency=[1:500];
plot3d(20*fft_n,frequency,array_of_ffts,flag=[0,4,4])

This shows very large peaks on left and small peaks on right.
I was expecting symmetry.
Is this an artifact of not windowing the data?
[I have SCILAB specific problems applying windowing]

What obvious thing am I missing ;(

"Richard Owlett" <rowlett@atlascomm.net> wrote in message 
news:12lrk8a68i4l7a5@corp.supernews.com...
> I'm interested in the general issue of how the spectrum of a signal > changes over time. > > In thread titled "FFT not showing high frequency oscillation", provided > data that made a nice clean test case. I posed myself a "simple" question, > "Is the parasitic oscillation changing over time?" > > I imported his data into Excel for some experimentation. > Then decided to use SCILAB whose facilities are more appropriate to the > size of data I will eventually be interested in. > > My code (using SCILAB 4.0 under WinXP Pro) > > sheets = readxls('E:\parasitic.xls') > s1=sheets(1); // get the first sheet > dwave=s1(:,2); > y=size(dwave,'r'); > k=max(abs(dwave([2:y]))); > dwave=dwave([2:y])/k; // first row is label > // normalize array > > for i=1:19 > j=500*(i-1); // data ~10,000 points > // oscillations are ~1500 points wide > array_of_ffts(i,:)=abs(fft(dwave([j+1:j+500]))'); > end > > fft_n=[1:19]; > frequency=[1:500]; > plot3d(20*fft_n,frequency,array_of_ffts,flag=[0,4,4]) > > This shows very large peaks on left and small peaks on right. > I was expecting symmetry. > Is this an artifact of not windowing the data? > [I have SCILAB specific problems applying windowing] > > What obvious thing am I missing ;( >
The only thing I can think is to subtract the average from your 500 sample sets before doing the FFTs or don't plot the DC component. Best of Luck - Mike
Richard Owlett wrote:
> I'm interested in the general issue of how the spectrum of a signal > changes over time.
...
> My code (using SCILAB 4.0 under WinXP Pro)
...
> for i=1:19 > j=500*(i-1); // data ~10,000 points > // oscillations are ~1500 points wide > array_of_ffts(i,:)=abs(fft(dwave([j+1:j+500]))'); > end
Does the oscillation burst cross fft apertures? If so, you may want to overlap your fft apertures, so that you can see if the window crossings are interfering with the signal evolution over larger time spans. IMHO. YMMV. -- rhn A.T nicholson d.0.t C-o-M
Richard Owlett <rowlett@atlascomm.net> writes:

> I'm interested in the general issue of how the spectrum of a signal > changes over time. > > In thread titled "FFT not showing high frequency oscillation", > provided data that made a nice clean test case. I posed myself a > "simple" question, "Is the parasitic oscillation changing over time?" > > I imported his data into Excel for some experimentation. > Then decided to use SCILAB whose facilities are more appropriate to > the size of data I will eventually be interested in. > > My code (using SCILAB 4.0 under WinXP Pro) > > sheets = readxls('E:\parasitic.xls') > s1=sheets(1); // get the first sheet > dwave=s1(:,2); > y=size(dwave,'r'); > k=max(abs(dwave([2:y]))); > dwave=dwave([2:y])/k; // first row is label > // normalize array > > for i=1:19 > j=500*(i-1); // data ~10,000 points > // oscillations are ~1500 points wide > array_of_ffts(i,:)=abs(fft(dwave([j+1:j+500]))'); > end > > fft_n=[1:19]; > frequency=[1:500]; > plot3d(20*fft_n,frequency,array_of_ffts,flag=[0,4,4]) > > This shows very large peaks on left and small peaks on right. > I was expecting symmetry. > Is this an artifact of not windowing the data? > [I have SCILAB specific problems applying windowing]
Richard, I don't know SciLab, but if it works like Matlab, your FFT data vector is arranged with the frequencies as follows. n*Fs/N, 0 <= n < N-1, N being the FFT size and Fs being the sample rate of the data. When you get to n = N/2 and above, you've covered the positive frequencies and begin the negative frequencies, i.e., n*Fs/N == (n-N)*Fs/N, n > N/2. So what you could be seeing in your plot is right - just rearranged. You would be seeing the negative data beginning halfway and extending to the right. In Matlab, there is a function to rearrange the data to what you would expect called fftshift(). Don't know if it's in scilab or not. -- % Randy Yates % "My Shangri-la has gone away, fading like %% Fuquay-Varina, NC % the Beatles on 'Hey Jude'" %%% 919-577-9882 % %%%% <yates@ieee.org> % 'Shangri-La', *A New World Record*, ELO http://home.earthlink.net/~yatescr
Mike Yarwood wrote:

> "Richard Owlett" <rowlett@atlascomm.net> wrote in message > news:12lrk8a68i4l7a5@corp.supernews.com... > >>I'm interested in the general issue of how the spectrum of a signal >>changes over time. >> >>In thread titled "FFT not showing high frequency oscillation", provided >>data that made a nice clean test case. I posed myself a "simple" question, >>"Is the parasitic oscillation changing over time?" >> >>I imported his data into Excel for some experimentation. >>Then decided to use SCILAB whose facilities are more appropriate to the >>size of data I will eventually be interested in. >> >>My code (using SCILAB 4.0 under WinXP Pro) >> >>sheets = readxls('E:\parasitic.xls') >>s1=sheets(1); // get the first sheet >>dwave=s1(:,2); >>y=size(dwave,'r'); >>k=max(abs(dwave([2:y]))); >>dwave=dwave([2:y])/k; // first row is label >> // normalize array >> >>for i=1:19 >> j=500*(i-1); // data ~10,000 points >> // oscillations are ~1500 points wide >> array_of_ffts(i,:)=abs(fft(dwave([j+1:j+500]))'); >>end >> >>fft_n=[1:19]; >>frequency=[1:500]; >>plot3d(20*fft_n,frequency,array_of_ffts,flag=[0,4,4]) >> >>This shows very large peaks on left and small peaks on right. >>I was expecting symmetry. >>Is this an artifact of not windowing the data? >>[I have SCILAB specific problems applying windowing] >> >>What obvious thing am I missing ;( >> > > The only thing I can think is to subtract the average from your 500 sample > sets before doing the FFTs or don't plot the DC component. > > Best of Luck - Mike > >
YEAH BUT ;) If problem was the DC component -- which *DOES* exist -- was the problem? Would it not show up only in first bin? or With a FFT length of 500 and a rectangular window, over how many bins would a DC component be spread?
Randy Yates wrote:

> Richard Owlett <rowlett@atlascomm.net> writes: > > >>I'm interested in the general issue of how the spectrum of a signal >>changes over time. >> >>In thread titled "FFT not showing high frequency oscillation", >>provided data that made a nice clean test case. I posed myself a >>"simple" question, "Is the parasitic oscillation changing over time?" >> >>I imported his data into Excel for some experimentation. >>Then decided to use SCILAB whose facilities are more appropriate to >>the size of data I will eventually be interested in. >> >>My code (using SCILAB 4.0 under WinXP Pro) >> >>sheets = readxls('E:\parasitic.xls') >>s1=sheets(1); // get the first sheet >>dwave=s1(:,2); >>y=size(dwave,'r'); >>k=max(abs(dwave([2:y]))); >>dwave=dwave([2:y])/k; // first row is label >> // normalize array >> >>for i=1:19 >> j=500*(i-1); // data ~10,000 points >> // oscillations are ~1500 points wide >> array_of_ffts(i,:)=abs(fft(dwave([j+1:j+500]))'); >>end >> >>fft_n=[1:19]; >>frequency=[1:500]; >>plot3d(20*fft_n,frequency,array_of_ffts,flag=[0,4,4]) >> >>This shows very large peaks on left and small peaks on right. >>I was expecting symmetry. >>Is this an artifact of not windowing the data? >>[I have SCILAB specific problems applying windowing] > > > Richard, > > I don't know SciLab, but if it works like Matlab, your FFT data vector > is arranged with the frequencies as follows. > > n*Fs/N, 0 <= n < N-1, N being the FFT size and Fs being the sample > rate of the data. When you get to n = N/2 and above, you've covered > the positive frequencies and begin the negative frequencies, i.e., > n*Fs/N == (n-N)*Fs/N, n > N/2. > > So what you could be seeing in your plot is right - just rearranged. > You would be seeing the negative data beginning halfway and > extending to the right.
That's why I was expecting symmetry.
> > In Matlab, there is a function to rearrange the data to what you would > expect called fftshift(). Don't know if it's in scilab or not.
Similar function is available. I'll have to drive the FFT with a function with a known transform to orient myself. I still have "gut feel" that something is wrong somewhere.
Richard Owlett skrev:
> I'm interested in the general issue of how the spectrum of a signal > changes over time. > > In thread titled "FFT not showing high frequency oscillation", provided > data that made a nice clean test case. I posed myself a "simple" > question, "Is the parasitic oscillation changing over time?" > > I imported his data into Excel for some experimentation. > Then decided to use SCILAB whose facilities are more appropriate to the > size of data I will eventually be interested in.
...
> This shows very large peaks on left and small peaks on right. > I was expecting symmetry. > Is this an artifact of not windowing the data?
First: This has nothing to do with not windowing the data. As others have suggested, at least part of the issue is due to the DC component. The first thing to do is to remove it from the data set to be plotted. Matlab code: X = fft(x,Nfft); plot(X(2:Nfft)); This should give a symmetrical spectrum for real-valued x. If not, check whether the whole spectrum is plotted, or if only the range [0..Fs/2] is plotted. Rune
"Richard Owlett" <rowlett@atlascomm.net> wrote in message 
news:12mh0dabp43dn2a@news.supernews.com...
> Mike Yarwood wrote: > >> "Richard Owlett" <rowlett@atlascomm.net> wrote in message >> news:12lrk8a68i4l7a5@corp.supernews.com... >> >>>I'm interested in the general issue of how the spectrum of a signal >>>changes over time. >>> >>>In thread titled "FFT not showing high frequency oscillation", provided >>>data that made a nice clean test case. I posed myself a "simple" >>>question, "Is the parasitic oscillation changing over time?" >>> >>>I imported his data into Excel for some experimentation. >>>Then decided to use SCILAB whose facilities are more appropriate to the >>>size of data I will eventually be interested in. >>> >>>My code (using SCILAB 4.0 under WinXP Pro) >>> >>>sheets = readxls('E:\parasitic.xls') >>>s1=sheets(1); // get the first sheet >>>dwave=s1(:,2); >>>y=size(dwave,'r'); >>>k=max(abs(dwave([2:y]))); >>>dwave=dwave([2:y])/k; // first row is label >>> // normalize array >>> >>>for i=1:19 >>> j=500*(i-1); // data ~10,000 points >>> // oscillations are ~1500 points wide >>> array_of_ffts(i,:)=abs(fft(dwave([j+1:j+500]))'); >>>end >>> >>>fft_n=[1:19]; >>>frequency=[1:500]; >>>plot3d(20*fft_n,frequency,array_of_ffts,flag=[0,4,4]) >>> >>>This shows very large peaks on left and small peaks on right. >>>I was expecting symmetry. >>>Is this an artifact of not windowing the data? >>>[I have SCILAB specific problems applying windowing] >>> >>>What obvious thing am I missing ;( >>> >> >> The only thing I can think is to subtract the average from your 500 >> sample sets before doing the FFTs or don't plot the DC component. >> >> Best of Luck - Mike >> >> > > YEAH BUT ;) > > If problem was the DC component -- which *DOES* exist -- was the problem? > > Would it not show up only in first bin? > or > With a FFT length of 500 and a rectangular window, over how many bins > would a DC component be spread? >
If I fft his data, and choose an FFT length that is very close to the period of one of his two and a bit low frequency cycles then the DC component is small. If you look at it in 500 sample chunks you will see the DC component varying from chunk to chunk and quite high in places. It only shows up in the left most bin but that's the only asymetry I can find. If you plot linear instead of log it looks very messy. I can't find anything else that might be unexpected about the plot.
Mike Yarwood wrote:

> "Richard Owlett" <rowlett@atlascomm.net> wrote in message > news:12mh0dabp43dn2a@news.supernews.com... > >>Mike Yarwood wrote: >> >> >>>"Richard Owlett" <rowlett@atlascomm.net> wrote in message >>>news:12lrk8a68i4l7a5@corp.supernews.com... >>> >>> >>>>I'm interested in the general issue of how the spectrum of a signal >>>>changes over time. >>>> >>>>In thread titled "FFT not showing high frequency oscillation", provided >>>>data that made a nice clean test case. I posed myself a "simple" >>>>question, "Is the parasitic oscillation changing over time?" >>>> >>>>I imported his data into Excel for some experimentation. >>>>Then decided to use SCILAB whose facilities are more appropriate to the >>>>size of data I will eventually be interested in. >>>> >>>>My code (using SCILAB 4.0 under WinXP Pro) >>>> >>>>sheets = readxls('E:\parasitic.xls') >>>>s1=sheets(1); // get the first sheet >>>>dwave=s1(:,2); >>>>y=size(dwave,'r'); >>>>k=max(abs(dwave([2:y]))); >>>>dwave=dwave([2:y])/k; // first row is label >>>> // normalize array >>>> >>>>for i=1:19 >>>> j=500*(i-1); // data ~10,000 points >>>> // oscillations are ~1500 points wide >>>> array_of_ffts(i,:)=abs(fft(dwave([j+1:j+500]))'); >>>>end >>>> >>>>fft_n=[1:19]; >>>>frequency=[1:500]; >>>>plot3d(20*fft_n,frequency,array_of_ffts,flag=[0,4,4]) >>>> >>>>This shows very large peaks on left and small peaks on right. >>>>I was expecting symmetry. >>>>Is this an artifact of not windowing the data? >>>>[I have SCILAB specific problems applying windowing] >>>> >>>>What obvious thing am I missing ;( >>>> >>> >>>The only thing I can think is to subtract the average from your 500 >>>sample sets before doing the FFTs or don't plot the DC component. >>> >>>Best of Luck - Mike >>> >>> >> >>YEAH BUT ;) >> >>If problem was the DC component -- which *DOES* exist -- was the problem? >> >>Would it not show up only in first bin? >>or >>With a FFT length of 500 and a rectangular window, over how many bins >>would a DC component be spread? >> > > If I fft his data, and choose an FFT length that is very close to the period > of one of his two and a bit low frequency cycles then the DC component is > small. If you look at it in 500 sample chunks you will see the DC component > varying from chunk to chunk and quite high in places. It only shows up in > the left most bin but that's the only asymetry I can find. If you plot > linear instead of log it looks very messy. I can't find anything else that > might be unexpected about the plot. >
Thank you. I'm going to have to look closer. I have a *WEIRD* work schedule, so may not get to it until next week.