Hi All, I am collecting data from a sensor hooked up via USB. The sensor is output= ting complex data and I am trying to plot its PSD. I am getting confused b= ecause I am plotting its PSD in 2 different applications, and in one of the= methods I get HUGE spikes at one of the FFT bins, but in the other it is s= mooth. =20 The first way to plot was to use an application developed in GNU radio whic= h in real-time collects data from the sensor and displays its real-time PSD= . Using this, the spectrum looks "normal", as in the spectrum looks smooth= . The second way was to pull the data into Octave. I did a PSD estimation us= ing the pwelch function available in Octave, and I notice huge spikes. The exact same data that is going in to produce the PSD using the GNU radio= module is being fed into the pwelch function in Octave. I'm not sure what= is causing the huge spike. Should it be there? I know that the data does= *not* have a DC offset, because it is removed in the time-domain. Is ther= e anything obvious with respect to huge spikes in the PSD of a signal? I've posted images of the spectrum with and without spikes. See links here= : Spectrum with spike: http://imgur.com/Sg9EU Spectrum without spike: http://imgur.com/ikCZ8 Again, I am 100% sure that the data going into both of these spectrum calcu= lators is the same... so that must mean that my Octave pwelch calculation i= s wrong? but I am using the pwelch built into Octave. Is there some pre-p= rocessing going on in the other PSD calculator that I don't know about? Any hints are greatly appreciated, Thanks!!
Spikes in Spectrum
Started by ●January 5, 2013
Reply by ●January 5, 20132013-01-05
dsp student wrote: <snip>> > Any hints are greatly appreciated, Thanks!! >pwelch() seems to have lots and lots of options. I'd start with a simple magnitude* plot of the FFT of the data and go from there. *mag(real,imag) = sqrt((real*real)+(imag*imag)); -- Les Cargill
Reply by ●January 5, 20132013-01-05
On Sat, 5 Jan 2013 14:28:44 -0800 (PST), dsp student <Kiran.Karra@gmail.com> wrote:>Hi All, >I am collecting data from a sensor hooked up via USB. The sensor is output= >ting complex data and I am trying to plot its PSD. I am getting confused b= >ecause I am plotting its PSD in 2 different applications, and in one of the= > methods I get HUGE spikes at one of the FFT bins, but in the other it is s= >mooth. =20 > >The first way to plot was to use an application developed in GNU radio whic= >h in real-time collects data from the sensor and displays its real-time PSD= >. Using this, the spectrum looks "normal", as in the spectrum looks smooth= >. > >The second way was to pull the data into Octave. I did a PSD estimation us= >ing the pwelch function available in Octave, and I notice huge spikes. > >The exact same data that is going in to produce the PSD using the GNU radio= > module is being fed into the pwelch function in Octave. I'm not sure what= > is causing the huge spike. Should it be there? I know that the data does= > *not* have a DC offset, because it is removed in the time-domain. Is ther= >e anything obvious with respect to huge spikes in the PSD of a signal? > >I've posted images of the spectrum with and without spikes. See links here= >: >Spectrum with spike: http://imgur.com/Sg9EU >Spectrum without spike: http://imgur.com/ikCZ8 > >Again, I am 100% sure that the data going into both of these spectrum calcu= >lators is the same... so that must mean that my Octave pwelch calculation i= >s wrong? but I am using the pwelch built into Octave. Is there some pre-p= >rocessing going on in the other PSD calculator that I don't know about? > >Any hints are greatly appreciated, >Thanks!!The spike at around bin 2048 is most certainly due to a DC offset in the data. The big spike near positive Nyquist is a concern, though. Do you know what algorithm(s) are used to display the "smooth" plot using the GNU radio app? It appears to me that there is some filtering going on since it seems that there is a "peak hold" function associated with the green trace and the "average" box is checked. This could mean that the spike seen in the pwelch plot is spurious and got filtered out in the GNU radio plot. Same thing with the "hair" on the left side of the plot; those smaller spikes could have been filtered, or "smoothed" by the GNU radio software and are still apparent using pwelch. If you don't know exactly what the GNU radio software is doing (I don't) to generate its plot it's hard to say why they look different other than speculation. What sort of format is the data in? Two's complement integer? How many bits? It could be that there's a difference in format interpretation as well, as even a difference in interpretation of signed/unsigned or the format of signed can have screwy results like that that are data dependent. If one of the plots is interpretating the data in a way other than how it was encoded there's no telling what it may look like. You could also try breaking the data up into smaller vectors, maybe N = 512 instead of N = 4096 and see if the smaller vectors all look the same in the frequency domain or whether that big spike is really transient. Just some thoughts that may or may not be useful to you. Let us know what you sort out. Eric Jacobsen Anchor Hill Communications http://www.anchorhill.com
Reply by ●January 6, 20132013-01-06
Hi, I think what you have just discovered is the concept of "resolution bandwidth" RBW. A hardware "spectrum analyzer" - a box with a coax input and a screen - has a button that allows you to set the RBW. Depending on what you set, the result will look totally different. The catch is, there is no single "correct" way to calculate the spectrum. The instrument uses a narrow-band filter (with a width of RBH Hz), sweeps it across the spectrum, averages the power that passes through the filter and plots one dot on the screen. For a narrow RBW (similar to your pwelsh result) - the average energy in the filter is low due to the narrow bandwidth. - the overall spectrum drops down - a sine wave is infinitely narrow and passes through the filter. For a wide RBW (the GNU radio result): - the average energy in the filter is high because of the wide bandwidth - therefore, the average spectrum is at higher values - the sine wave still goes through the filter unattenuated, but in comparison it looks smaller. Check your "spike-free plot". You can still see them at the same location, they are just smaller. In Matlab etc, this concept seems very abstract, but in the lab with hardware instruments it's one of the most basic things. You want to bring out narrow-band details, turn down the RBW on the analyzer. Maybe I should shoot a video, it would explain better than 1000 words.
Reply by ●January 6, 20132013-01-06
Hi All, Taking all three of the suggestions above has helped me to resolve the issu= e. So here is what I have found: First of all, there is a detrend option in the pwelch function. The option= s that I played with were: 'short' and 'long'. The 'short' option removes = the mean for each window that it calculates the PSD for, while the 'long' o= ption removes the mean for the entire sequence before proceeding to calcula= te the PSD. Second, I played with the FFT Size. I began to think of the FFT Size as a = way to control the RBW, as mentioned in mnentwig's post, because as the FFT= size gets bigger, you have less resolution per frequency bin. =20 The first thing that I did was turn up the FFT Size in the GNU Radio plot..= and I found spikes! Here is the link to the screenshot for the gnu-radio:= http://imgur.com/hB2d1 Then, I tried combinations of small FFT's and large FFT's with different de= trend options. Here is a size-256 FFT with detrend to long: http://imgur.com/3HMxi Here is a size-256 FFT with detrend to short:http://imgur.com/2rSzt Here is a size-1024 FFT with detrend to long: http://imgur.com/VdNa6 Here is a size-1024 FFT with detrend to short: http://imgur.com/Vhju3 One conclusion to draw from the detrend option is, that my DC offset is var= ying slowly over time? =20 Secondly, is it correct to correlate RBW with FFT Size? Of course this doe= sn't completely correlate with your definition, which is an actual filter p= assing over the data to smooth it out before plotting, but it seems like a = very easy way to control RBW? Thanks again everybody.
Reply by ●January 6, 20132013-01-06
On Sun, 6 Jan 2013 06:46:45 -0800 (PST), dsp student <Kiran.Karra@gmail.com> wrote:>Hi All, >Taking all three of the suggestions above has helped me to resolve the issu= >e. So here is what I have found: > >First of all, there is a detrend option in the pwelch function. The option= >s that I played with were: 'short' and 'long'. The 'short' option removes = >the mean for each window that it calculates the PSD for, while the 'long' o= >ption removes the mean for the entire sequence before proceeding to calcula= >te the PSD. > >Second, I played with the FFT Size. I began to think of the FFT Size as a = >way to control the RBW, as mentioned in mnentwig's post, because as the FFT= > size gets bigger, you have less resolution per frequency bin. =20 > >The first thing that I did was turn up the FFT Size in the GNU Radio plot..= > and I found spikes! Here is the link to the screenshot for the gnu-radio:= > http://imgur.com/hB2d1 > >Then, I tried combinations of small FFT's and large FFT's with different de= >trend options. > >Here is a size-256 FFT with detrend to long: http://imgur.com/3HMxi >Here is a size-256 FFT with detrend to short:http://imgur.com/2rSzt > >Here is a size-1024 FFT with detrend to long: http://imgur.com/VdNa6 >Here is a size-1024 FFT with detrend to short: http://imgur.com/Vhju3 > >One conclusion to draw from the detrend option is, that my DC offset is var= >ying slowly over time? =20 > >Secondly, is it correct to correlate RBW with FFT Size? Of course this doe= >sn't completely correlate with your definition, which is an actual filter p= >assing over the data to smooth it out before plotting, but it seems like a = >very easy way to control RBW? > >Thanks again everybody.RBW for spectrum analysis done with DFT/FFT processing is affected by the sample rate, the DFT/FFT length, any window functions applied to the data, and probably a few other things. Sounds like you're on the right track. You've also touched on the idea that the if certain statistics or signal components (e.g., the DC offset) aren't stable over the length of the analysis window it can certainly affect things. That's the usual time-frequency uncertainty issue in that you can't really know both aspects through a FT. As usual, interpretation of the output can be a minefield when things aren't straightforward and it sounds like you're on your way to learning some of the important issues. Eric Jacobsen Anchor Hill Communications http://www.anchorhill.com
Reply by ●January 6, 20132013-01-06
dsp student wrote:> Hi All,<snip>> > One conclusion to draw from the detrend option is, that my DC offset > is varying slowly over time? >Hard to say, really. If you have to manage data varying at sample rate, then there may be DC offset from using FFT. DC offset may be only zero at specific points in the time domain. A sine wave will have exactly zero DC offset fairly infrequently.> Secondly, is it correct to correlate RBW with FFT Size? Of course > this doesn't completely correlate with your definition, which is an > actual filter passing over the data to smooth it out before plotting, > but it seems like a very easy way to control RBW? >it may be a matter of tuning FFT size as well as choosing windowing, and possibly judicious use of filtering.> Thanks again everybody. >-- Les Cargill
Reply by ●January 6, 20132013-01-06
On Saturday, January 5, 2013 2:28:44 PM UTC-8, dsp student wrote: ...> > Any hints are greatly appreciated, > > Thanks!!Manufacturers of spectrum analyzers have long been a source of good application notes. A good review of analog swept spectrum analyzers with a more accurate and detailed presentation that in this thread so far: Agilent Spectrum Analysis Basics Application Note 150 5952-0292.pdf Available at: http://www.hep.ucl.ac.uk/lc/T474/Documentation/agilent_spectrum_analysis_basics.pdf Example usage of an example insrument: FFT Applications for TDS Oscilloscopes 55W_8815-2_0.pdf Available at: http://www.electron.frba.utn.edu.ar/~jcecconi/Bibliografia/06%20-%20Osciloscopios%20de%20Almacenamiento%20Digital/FFT%20Applications%20for%20TDS%20Oscilloscopes.pdf In the digital domain a common parameter used to describe DFT/window resolution is ENBW (effective noise bandwidth). Look in a reference like the one below. A good review of DFT/window application: Spectrum and spectral density estimation by the Discrete Fourier transform (DFT), including a comprehensive list of window functions and some new flat-top windows. GH_FFT.pdf Available at: http://www.rssd.esa.int/SP/LISAPATHFINDER/docs/Data_Analysis/GH_FFT.pdf Dale B. Dalrymple
Reply by ●January 7, 20132013-01-07
>> Secondly, is it correct to correlate RBW with FFT Size?As a rule of thumb, yes. Not necessarily if the code tries something clever, like padding to the next power-of-two FFT size. If you like, you can try "my" spectrum analyzer. http://www.dsprelated.com/showcode/239.php You have to apply windowing yourself, or provide a signal that is inherently cyclic. The code takes the power spectrum via FFT and then convolves with a well-defined RBW filter (for a start, use 30 kHz "brickwall" aka boxcar or Dirichlet). It processes the whole dataset all at once, the more data the better (up to a point, say 10M..100M points).
Reply by ●January 7, 20132013-01-07
On Monday, January 7, 2013 6:40:03 AM UTC-8, mnentwig wrote:> >> Secondly, is it correct to correlate RBW with FFT Size?=20 > > As a rule of thumb, yes. Not necessarily if the code tries something > clever, like padding to the next power-of-two FFT size. > ...The resolution that results from DFT calculations is determined by window s= ize and type. The window size may be different from the DFT size due to zer= o-fill or 'time domain aloiasing'. The common windows references give the e= ffective bandwidth in units of Fs/window-size. When the transform and windo= w size are the same, this unit is the seperation between adjacent FFT outpu= ts.> You have to apply windowing yourself, or provide a signal that is > inherently cyclic. > ...This sentence structure looks like a comparison of alternatives, it isn't. = You can't calculate a DFT (or via FFT) without applying an effective window= the size of the number of data samples. If you do nothing else, it is the = rectangular window. If you have a discrete sequence that is inherently peri= odic and wish to apply the FFT to calculate samples of the discrete values = of the Fourier Transform of the sequence, the sequence must be periodic in = an integer number of sample periods, the window size must be an integer mul= tiple of the period, and the discrete transform size must be an integer mul= tiple of the window size. The integer most commonly used as each such 'mult= iple' is 1 and the window type is rectangular. Dale B. Dalrymple
