My raw signal is sampled once per second. The two effects I wish to investigate have rep rates of once per 24.0000 hours once per ~23.8833 hours Initially I'll have data for 30 days. Eventually will have data for >= 360 days. How safe am I decimating fft input to once per .1 hours?
How much may I decimate?
Started by ●February 16, 2006
Reply by ●February 16, 20062006-02-16
Hello Richard,> My raw signal is sampled once per second. > The two effects I wish to investigate have rep rates of > once per 24.0000 hours > once per ~23.8833 hours > > Initially I'll have data for 30 days. > Eventually will have data for >= 360 days. > > How safe am I decimating fft input to once per .1 hours? >Sounds like riding on the edge but could work if you only need to detect the presence of the events. Why don't you run two filters with these 'frequencies'? Regards, Joerg http://www.analogconsultants.com
Reply by ●February 17, 20062006-02-17
Richard Owlett wrote:> My raw signal is sampled once per second. > The two effects I wish to investigate have rep rates of > once per 24.0000 hours > once per ~23.8833 hours > > Initially I'll have data for 30 days. > Eventually will have data for >= 360 days. > > How safe am I decimating fft input to once per .1 hours?The necessary sample rate is set by the highest frequency in your data, not by the frequencies you're interested in. The sample frequency must exceed the highest noticeable component by a factor exceeding 2, and 3 is easier to work with. How strong a component is "noticeable" depends on the application. If you are certain that your data contain no higher frequencies than 1/day (either inherently or because you filtered it) you can be quite comfortable with one sample every 6 hours. How did you arrive at your figure of 6 minutes? You know all that. Why do you ask? Jerry P.S. Where does 23.8833 hours arise? (A sidereal day is 23.934 hours.) -- Engineering is the art of making what you want from things you can get. �����������������������������������������������������������������������
Reply by ●February 17, 20062006-02-17
Joerg wrote:> Hello Richard, > >> My raw signal is sampled once per second. >> The two effects I wish to investigate have rep rates of >> once per 24.0000 hours >> once per ~23.8833 hours >> >> Initially I'll have data for 30 days. >> Eventually will have data for >= 360 days. >> >> How safe am I decimating fft input to once per .1 hours? >> > > Sounds like riding on the edge but could work if you only need to detect > the presence of the events. > > Why don't you run two filters with these 'frequencies'?Primarily cause I just didn't think of it. Thinking in terms of a spectrum analyzer screen is part of me. Besides which, in spite of efforts of many in this group, I just do not grok digital filters ;/> > Regards, Joerg > > http://www.analogconsultants.com
Reply by ●February 17, 20062006-02-17
Jerry Avins wrote:> Richard Owlett wrote: > >> My raw signal is sampled once per second. >> The two effects I wish to investigate have rep rates of >> once per 24.0000 hours >> once per ~23.8833 hours >> >> Initially I'll have data for 30 days. >> Eventually will have data for >= 360 days. >> >> How safe am I decimating fft input to once per .1 hours? > > > The necessary sample rate is set by the highest frequency in your data, > not by the frequencies you're interested in. The sample frequency must > exceed the highest noticeable component by a factor exceeding 2, and 3 > is easier to work with. How strong a component is "noticeable" depends > on the application. > > If you are certain that your data contain no higher frequencies than > 1/day (either inherently or because you filtered it) you can be quite > comfortable with one sample every 6 hours. How did you arrive at your > figure of 6 minutes? > > You know all that. Why do you ask?I keep forgetting that frequency resolution is inverse of window width ;}> > Jerry > > P.S. Where does 23.8833 hours arise? (A sidereal day is 23.934 hours.)The time between exact GPS satellite configurations reoccurring. I thought I had read that it was 23 hours 53 minutes which I had converted to decimal for convenience. But if sidereal day is 23 hours 56 minutes, I'm wondering if I misremembered or if there was a typo in post I read. And yes, that "24.0000 hours" should have been "24.000 +- .005 hours" [the solar day].
Reply by ●February 17, 20062006-02-17
Hello Richard,> > Primarily cause I just didn't think of it. > Thinking in terms of a spectrum analyzer screen is part of me. >FFT wins if you have lots of bins to look at, a dozen or so. But for two bins almost any kind of DFT would require less horses from the processor. Sometimes the FFT even doesn't win with a dozen bins. For example when the bin centers don't fall on even numbers unless your FFT granularity is extremely fine, turning the FFT routine into a MIPS burner. Bin averaging doesn't always work (depending on the BW you are after).> Besides which, in spite of efforts of many in this group, I just do not > grok digital filters ;/ >But, but, you don't have to start at Maxwell's theorem anymore these days. There are lots of cheat sheet SW routine. I can't share mine because it probably won't do you any good (it speaketh German and also requires the machine to switch to clean DOS ...). Regards, Joerg http://www.analogconsultants.com
Reply by ●February 17, 20062006-02-17
Joerg wrote:> Hello Richard, > >> >> Primarily cause I just didn't think of it. >> Thinking in terms of a spectrum analyzer screen is part of me. >> > > FFT wins if you have lots of bins to look at, a dozen or so. But for two > bins almost any kind of DFT would require less horses from the processor.What is "DFT"? OK, I Goggled and was referred to answers.com which stated "The DFT can be computed efficiently in practice using a fast Fourier transform (FFT) algorithm." *OBVIOUSLY* I'm missing something ;]> > Sometimes the FFT even doesn't win with a dozen bins. For example when > the bin centers don't fall on even numbers unless your FFT granularity > is extremely fine, turning the FFT routine into a MIPS burner. Bin > averaging doesn't always work (depending on the BW you are after).I doubt granularity is a problem. Effect has ~24 *HOUR* period while sampled once per *SECOND*> > >> Besides which, in spite of efforts of many in this group, I just do >> not grok digital filters ;/ >> > > But, but, you don't have to start at Maxwell's theorem anymore these > days. There are lots of cheat sheet SW routine.Not only am I dense. I am also CHEAP ;) Anything I do that is DSP related is strictly as a hobby/avocation.> I can't share mine > because it probably won't do you any good (it speaketh German and also > requires the machine to switch to clean DOS ...).German would be a problem. BUT I like DOS.> > Regards, Joerg > > http://www.analogconsultants.com
Reply by ●February 17, 20062006-02-17
Hello Richard,> > What is "DFT"?Discrete Fourier Transform.> OK, I Goggled and was referred to answers.com which stated "The DFT can > be computed efficiently in practice using a fast Fourier transform (FFT) > algorithm." > *OBVIOUSLY* I'm missing something ;] >The FFT is, in a nutshell, the calculation of 128, 256, 1024 or whatever binary number of DFTs at the same time while greatly reducing the math overhead by exploiting symmetries and repetitions. A DFT or derivative thereof can be much more practical when you are only looking at very few frequencies. You can do things that are next to impossible with an FFT: Use different windows between them, different sample quantities, modified algorithms and so on. An example would be music where the distance between tones is not equidistant like the FFT requires but arranged in a Fibonacci/Octave pattern. There are also lots of digital filter algorithms you can use. If you do a lot of DSP work it pays to buy a text book like Haddad/Parsons that explains them. Just don't expect those types of books to be complete. Many are, for example, missing wave digital filters.>> >> Sometimes the FFT even doesn't win with a dozen bins. For example when >> the bin centers don't fall on even numbers unless your FFT granularity >> is extremely fine, turning the FFT routine into a MIPS burner. Bin >> averaging doesn't always work (depending on the BW you are after). > > I doubt granularity is a problem. Effect has ~24 *HOUR* period while > sampled once per *SECOND* >Yes, but you said you wanted to throttle down the FFT input to 0.1hrs.>> I can't share mine because it probably won't do you any good (it >> speaketh German and also requires the machine to switch to clean DOS >> ...). > > German would be a problem. > BUT I like DOS. >It's Professor Mildenberger's program. I am using a licensed copy in German that I bought many moons ago and I am not sure whether they translated it into English. Should have been created in English in the first place. Anyway, at the end of this page you'll find the download (nowadays free AFAIK): http://www.ite.fh-wiesbaden.de/~mildeno/ Often a different language isn't too hard since the commands repeat all the time. The beam field simulator I started with talked Swedish (which I do not speak) and my first parts database program was in French, where pretty much the only thing I could say would be to order a beer in a pub. Regards, Joerg http://www.analogconsultants.com
Reply by ●February 17, 20062006-02-17
Richard Owlett wrote: ...> What is "DFT"? > OK, I Goggled and was referred to answers.com which stated "The DFT can > be computed efficiently in practice using a fast Fourier transform (FFT) > algorithm." > *OBVIOUSLY* I'm missing something ;]The FFT is a fast way to compute a /complete/ DFT. It's not the fastest way to get a small subset of the bins. ... Jerry -- Engineering is the art of making what you want from things you can get. �����������������������������������������������������������������������
Reply by ●February 17, 20062006-02-17
Jerry Avins wrote:> Richard Owlett wrote: > > ... > >> What is "DFT"? >> OK, I Goggled and was referred to answers.com which stated "The DFT >> can be computed efficiently in practice using a fast Fourier transform >> (FFT) algorithm." >> *OBVIOUSLY* I'm missing something ;] > > > The FFT is a fast way to compute a /complete/ DFT. It's not the fastest > way to get a small subset of the bins. >I have glimmer of understanding. I am interested in effects whose characteristic periods are on of order of minutes to years. we'll see if scilab chokes when i obtain real data ;]
