Help please: I have a time series which is highly oversampled. After low pass filtering, I have about 17,000 data points. In that span I have a very low frequency signal (about 1/50th of a period), two more moderate frequency signals (can be between 1 and 8 periods) and the sum of those two signals. From sweep to sweep the two signals will both move together in frequency space (by very small amounts) and their relative intensities can change dramatically. What I need is a high accuracy method of determining the frequency of these two signals. The near DC component and the sum frequency are pretty much irrelevent to me (as both are typically low amplitude, anyhow). What I have done so far: average subtract the raw data Low pass filter the result (there is high frequency noise on it) chop the zones which are in the filter's group delay zone Apply a raised cosine window zero pad (or not) dft.... I know that zero padding my dft interpolates the psd, but I don't think that buys me anything in terms of ability to determine the absolute value of frequencies. I've tried to use a zero padded dft and then determine the centriod of the peak, but that hasn't seemed to help either. Some things that I can't do: I can't get any more data. It comes in sets of 20,000 only. (well, actually that may be the only thing I can think of that I can't do.) I would really, really appreciate any help on this one. I am by no means an expert in dsp (I'm learning as I go), but if you can give me even a hint, I can try to work from there. Thanks in advance, Dan K
Resolving frequencies in an oversampled spectrum
Started by ●October 13, 2006
Reply by ●October 13, 20062006-10-13
Dan K wrote:> I have a time series which is highly oversampled. After low pass > filtering, I have about 17,000 data points. In that span I have a very > low frequency signal (about 1/50th of a period), two more moderate > frequency signals (can be between 1 and 8 periods) and the sum of those > two signals. From sweep to sweep the two signals will both move > together in frequency space (by very small amounts) and their relative > intensities can change dramatically. What I need is a high accuracy > method of determining the frequency of these two signals.If you can describe your signals as the models with the unknown parameters, you can apply the Proni algorithm or some other method of the parametric estimation. For example, if your signal is described by a model Y = A sin(Wt + Fi) with A, W, Fi as the unknown parameters, you can solve for the values which provide the best fit with your data. This is a very accurate method although it takes a lot of computation. Vladimir Vassilevsky DSP and Mixed Signal Design Consultant http://www.abvolt.com
Reply by ●October 13, 20062006-10-13
Dan K wrote:> Help please: > > I have a time series which is highly oversampled. After low pass > filtering, I have about 17,000 data points.??? Filtering doesn't change the number of data points. Do we have a terminology problem?> In that span I have a very > low frequency signal (about 1/50th of a period), two more moderate > frequency signals (can be between 1 and 8 periods) and the sum of those > two signals. From sweep to sweep the two signals will both move > together in frequency space (by very small amounts) and their relative > intensities can change dramatically. What I need is a high accuracy > method of determining the frequency of these two signals. The near DC > component and the sum frequency are pretty much irrelevent to me (as > both are typically low amplitude, anyhow).I would call 1/50th of a period "near DC", but I suspect you don't. Where does the sum frequency come from? If from a nonlinearity, expect a difference frequency too.> What I have done so far: > average subtract the raw data > Low pass filter the result (there is high frequency noise on it)Averaging is a low-pass operation. Maybe there is an efficient way to combine those two filters.> chop the zones which are in the filter's group delay zoneI don't know what that means.> Apply a raised cosine window > zero pad (or not) > dft.... > > I know that zero padding my dft interpolates the psd, but I don't think > that buys me anything in terms of ability to determine the absolute > value of frequencies. I've tried to use a zero padded dft and then > determine the centriod of the peak, but that hasn't seemed to help > either. > > Some things that I can't do: > I can't get any more data. It comes in sets of 20,000 only. > (well, actually that may be the only thing I can think of that I can't > do.) > > I would really, really appreciate any help on this one. I am by no > means an expert in dsp (I'm learning as I go), but if you can give me > even a hint, I can try to work from there.A naive (straightforward might be a better term) approach to frequency determination requires either substantial /a priori/ information about the signal, or at least nearly a complete cycle's worth of samples. The math for extracting information from too few samples gets pretty sophisticated, especially when other signals or noise is also present. Jerry -- "The rights of the best of men are secured only as the rights of the vilest and most abhorrent are protected." - Chief Justice Charles Evans Hughes, 1927 ���������������������������������������������������������������������
Reply by ●October 13, 20062006-10-13
Jerry Avins <jya@ieee.org> wrote in news:t4Gdncasq9Btnq3YnZ2dnUVZ_uidnZ2d@rcn.net:> ??? Filtering doesn't change the number of data points. Do we have a > terminology problem?The first and last N/2 points are often rendered useless after an N-point filtering operation, and are often thrown out. -- Scott Reverse name to reply
Reply by ●October 13, 20062006-10-13
Scott Seidman wrote:> Jerry Avins <jya@ieee.org> wrote in > news:t4Gdncasq9Btnq3YnZ2dnUVZ_uidnZ2d@rcn.net: > >> ??? Filtering doesn't change the number of data points. Do we have a >> terminology problem? > > > The first and last N/2 points are often rendered useless after an N-point > filtering operation, and are often thrown out.Right. My bad. He goes from 10,000 down to 17,000 with the "telomere" losses of two filters. A more efficient filter than averaging may save some of that loss. Jerry -- "The rights of the best of men are secured only as the rights of the vilest and most abhorrent are protected." - Chief Justice Charles Evans Hughes, 1927 ���������������������������������������������������������������������
Reply by ●October 13, 20062006-10-13
>I would really, really appreciate any help on this one. I am by no >means an expert in dsp (I'm learning as I go), but if you can give me >even a hint, I can try to work from there.So here's a hint - very accurate estimates of frequency can be made by observing the spectral phase change in a target bin between two (or more) adjacent (or overlapping) FFTs. The point-sizes of the FFTs will depend on the spectral resolution required to resolve your two frequencies into separate bins (preferably more than two bins apart). Note that this is quite different from choosing a resolution to match the frequency accuracy you require - the FFTs can therefore be quite modest in size in many instances. Using this technique I have measured frequencies to an accuracy of better than 0.0001% using FFTs wih point-sizes as small as 32. I wrote a paper on this some years ago which detailed how the phase differences need to be interpreted for varying overlap factors, and which highlighted the difference in behaviour between odd and even bins. I do not have this paper to hand right now - maybe I should resurrect it and publish it on the web if it is of any general interest. I do seem to remember that the principles and necessary correction factors were quite simple, and easy to apply 'blind' to the spectrum - resulting in a modified 'spectrum' which now held accurate frequency values rather than spectrum magnitudes. Jeff
Reply by ●October 13, 20062006-10-13
On Oct 13, 3:19 pm, "Jeff Caunter" <jeffcaun...@sparkysworld.co.uk> wrote:> >I would really, really appreciate any help on this one. I am by no > >means an expert in dsp (I'm learning as I go), but if you can give me > >even a hint, I can try to work from there.So here's a hint - very accurate estimates of frequency can be made by > observing the spectral phase change in a target bin between two (or more) > adjacent (or overlapping) FFTs. The point-sizes of the FFTs will depend on > the spectral resolution required to resolve your two frequencies into > separate bins (preferably more than two bins apart). Note that this is > quite different from choosing a resolution to match the frequency accuracy > you require - the FFTs can therefore be quite modest in size in many > instances. > > Using this technique I have measured frequencies to an accuracy of better > than 0.0001% using FFTs wih point-sizes as small as 32. I wrote a paper on > this some years ago which detailed how the phase differences need to be > interpreted for varying overlap factors, and which highlighted the > difference in behaviour between odd and even bins. I do not have this > paper to hand right now - maybe I should resurrect it and publish it on > the web if it is of any general interest. I do seem to remember that the > principles and necessary correction factors were quite simple, and easy to > apply 'blind' to the spectrum - resulting in a modified 'spectrum' which > now held accurate frequency values rather than spectrum magnitudes.This sounds similar to the algorithm commonly used by phase vocoders. IMHO. YMMV. -- rhn A.T nicholson d.0.t C-o-M
Reply by ●October 14, 20062006-10-14
Vladimir Vassilevsky skrev:> Dan K wrote: > > > I have a time series which is highly oversampled. After low pass > > filtering, I have about 17,000 data points. In that span I have a very > > low frequency signal (about 1/50th of a period), two more moderate > > frequency signals (can be between 1 and 8 periods) and the sum of those > > two signals. From sweep to sweep the two signals will both move > > together in frequency space (by very small amounts) and their relative > > intensities can change dramatically. What I need is a high accuracy > > method of determining the frequency of these two signals. > > If you can describe your signals as the models with the unknown > parameters, you can apply the Proni algorithm or some other method of > the parametric estimation....> This is a very accurate method although it takes a lot of computation.I'd go with Prony's method. Certain variations of the algorithm are very tolerant for noise. Don't worry about the computational load; with some skill in setting up the problem, it will not be a problem. Rune
Reply by ●October 14, 20062006-10-14
Jeff Caunter wrote: (snip)> So here's a hint - very accurate estimates of frequency can be made by > observing the spectral phase change in a target bin between two (or more) > adjacent (or overlapping) FFTs. The point-sizes of the FFTs will depend on > the spectral resolution required to resolve your two frequencies into > separate bins (preferably more than two bins apart). Note that this is > quite different from choosing a resolution to match the frequency accuracy > you require - the FFTs can therefore be quite modest in size in many > instances.How hard would it be to FFT an entire CD? I have done external sorts on 2GB files before, which is larger than a CD. It shouldn't be hard to get the data into bit reversed form for easy FFT processing. -- glen
Reply by ●October 14, 20062006-10-14
Vladimir Vassilevsky wrote:> > > Dan K wrote: > >> I have a time series which is highly oversampled. After low pass >> filtering, I have about 17,000 data points. In that span I have a very >> low frequency signal (about 1/50th of a period), two more moderate >> frequency signals (can be between 1 and 8 periods) and the sum of those >> two signals. From sweep to sweep the two signals will both move >> together in frequency space (by very small amounts) and their relative >> intensities can change dramatically. What I need is a high accuracy >> method of determining the frequency of these two signals. > > If you can describe your signals as the models with the unknown > parameters, you can apply the Proni algorithm or some other method of > the parametric estimation. > > For example, if your signal is described by a model Y = A sin(Wt + Fi) > with A, W, Fi as the unknown parameters, you can solve for the values > which provide the best fit with your data. > > This is a very accurate method although it takes a lot of computation. > > > Vladimir Vassilevsky > > DSP and Mixed Signal Design Consultant > > http://www.abvolt.comHi Dan, I agree with Vladimir. If your signal model is sinusoidal, you can probably try using a high resolution parameteric estimation method like ESPRIT (Estimation of Signal Parameters via Rotational Invariance Technique) where the parameters you are trying to estimate are the different frequencies. It basically involves partitioning your data into two overlap sets, e.g. x0 - x10000 and x9000 - x17000. Then you will see that the second set of data is just a phase shifted version of the first set. From this relationship, you can obtain a general eigen equation such that the frequency estimates are contained inside the eigenvalue matrix. I found that this technique works reasonably well (measure by the variance of the estimates) starting at about SNR = 15 dB and improve as the SNR increases. For detail, I suggest you look at some advanced DSP book like: Manolakis, Ingle, Kogon, "Statistical and adaptive signal processing : spectral estimation, signal modeling, adaptive filtering, and array processing", 2005. This is an e-book so you should be able to view it from an university's library website. Hope this helps, cf
