When is FFT of experimental numerical data logically valid/legitamate

In another thread (Re: FIR Bandpass filtering (C code)) the OP was doing 
FFT of short segments of a much larger financial data stream and was 
told FFT was inherently meaningless.

I did not understand the stated logic.

I'll pose another data set.
GPS position data from single frequency receiver.

Such data is known to have periodic components to the position errors.
(eg diurnal variation due to solar wind effects on ionosphere)
Are white noise components.
There are also anomalous impulse variations.

As an educational experience I intend to "discover" the above ;)
My data set will be a years worth of data sampled faster than every 30 
seconds -- 1 second data available but ;0>

That data would seem to violate the same constraints (as I read them) 
but my gut feel that FFT would be an appropriate way to analyze the data.

I'm interested in spotting the anomalous impulses as well as describing 
the periodic variations.

"Richard Owlett" <rowlett@atlascomm.net> wrote in message 
news:1312p625qb09r34@news.supernews.com...
> In another thread (Re: FIR Bandpass filtering (C code)) the OP was doing 
> FFT of short segments of a much larger financial data stream and was told 
> FFT was inherently meaningless.
>
> I did not understand the stated logic.
>
> I'll pose another data set.
> GPS position data from single frequency receiver.
>
> Such data is known to have periodic components to the position errors.
> (eg diurnal variation due to solar wind effects on ionosphere)
> Are white noise components.
> There are also anomalous impulse variations.
>
> As an educational experience I intend to "discover" the above ;)
> My data set will be a years worth of data sampled faster than every 30 
> seconds -- 1 second data available but ;0>
>
> That data would seem to violate the same constraints (as I read them) but 
> my gut feel that FFT would be an appropriate way to analyze the data.
>
> I'm interested in spotting the anomalous impulses as well as describing 
> the periodic variations.

If the periodic variations are diurnal then how much data is needed to 
resolve the period?  If the period is 24 hours then you want to have 
resolution at least 1/24 hours == 0.0000115Hz.  This resolution requires a 
data set that's 24 hours long.  To really do a decent job of determining the 
period you'd need a data set that's quite a bit longer.

If you don't like this assertion then consider this:

If the data set is 1 hour long then the frequency resolution will be 1/1 
hour or 1/3600 seconds or .000277Hz.  Not nearly enough to resolve what you 
want to resolve.

Consider the closely related Fourier Series:
Assume, as we do, that the data set is one period of a periodic waveform.
In that case the fundamental frequency is the reciprocal of the record 
length.
If the frequency you want to detect is lower than that then there are *no* 
Fourier Series terms clustered around it.  The frequency domain is 
under-resolved for your purposes.

Fred 

Fred Marshall wrote:

> "Richard Owlett" <rowlett@atlascomm.net> wrote in message 
> news:1312p625qb09r34@news.supernews.com...
> 
>>In another thread (Re: FIR Bandpass filtering (C code)) the OP was doing 
>>FFT of short segments of a much larger financial data stream and was told 
>>FFT was inherently meaningless.
>>
>>I did not understand the stated logic.
>>
>>I'll pose another data set.
>>GPS position data from single frequency receiver.
>>
>>Such data is known to have periodic components to the position errors.
>>(eg diurnal variation due to solar wind effects on ionosphere)
>>Are white noise components.
>>There are also anomalous impulse variations.
>>
>>As an educational experience I intend to "discover" the above ;)
>>My data set will be a years worth of data sampled faster than every 30 
>>seconds -- 1 second data available but ;0>
>>
>>That data would seem to violate the same constraints (as I read them) but 
>>my gut feel that FFT would be an appropriate way to analyze the data.
>>
>>I'm interested in spotting the anomalous impulses as well as describing 
>>the periodic variations.
> 
> 
> If the periodic variations are diurnal then how much data is needed to 
> resolve the period?  If the period is 24 hours then you want to have 
> resolution at least 1/24 hours == 0.0000115Hz.  This resolution requires a 
> data set that's 24 hours long.  To really do a decent job of determining the 
> period you'd need a data set that's quite a bit longer.
> 

How about if it is 365 times as long?
I *DID* specify that I had one _YEAR_ of data ;)


> If you don't like this assertion then consider this:
> 
> If the data set is 1 hour long 

*BUT* it is 365*24 hours long ;)!


> then the frequency resolution will be 1/1 
> hour or 1/3600 seconds or .000277Hz.  Not nearly enough to resolve what you 
> want to resolve.
> 
> Consider the closely related Fourier Series:
> Assume, as we do, that the data set is one period of a periodic waveform.
> In that case the fundamental frequency is the reciprocal of the record 
> length.
> If the frequency you want to detect is lower than that then there are *no* 
> Fourier Series terms clustered around it.  The frequency domain is 
> under-resolved for your purposes.
> 
> Fred 
> 
> 

"Richard Owlett" <rowlett@atlascomm.net> wrote in message 
news:13130ulljg28827@news.supernews.com...
> Fred Marshall wrote:
>
>> "Richard Owlett" <rowlett@atlascomm.net> wrote in message 
>> news:1312p625qb09r34@news.supernews.com...
>>
>>>In another thread (Re: FIR Bandpass filtering (C code)) the OP was doing 
>>>FFT of short segments of a much larger financial data stream and was told 
>>>FFT was inherently meaningless.
>>>
>>>I did not understand the stated logic.
>>>
>>>I'll pose another data set.
>>>GPS position data from single frequency receiver.
>>>
>>>Such data is known to have periodic components to the position errors.
>>>(eg diurnal variation due to solar wind effects on ionosphere)
>>>Are white noise components.
>>>There are also anomalous impulse variations.
>>>
>>>As an educational experience I intend to "discover" the above ;)
>>>My data set will be a years worth of data sampled faster than every 30 
>>>seconds -- 1 second data available but ;0>
>>>
>>>That data would seem to violate the same constraints (as I read them) but 
>>>my gut feel that FFT would be an appropriate way to analyze the data.
>>>
>>>I'm interested in spotting the anomalous impulses as well as describing 
>>>the periodic variations.
>>
>>
>> If the periodic variations are diurnal then how much data is needed to 
>> resolve the period?  If the period is 24 hours then you want to have 
>> resolution at least 1/24 hours == 0.0000115Hz.  This resolution requires 
>> a data set that's 24 hours long.  To really do a decent job of 
>> determining the period you'd need a data set that's quite a bit longer.
>>
>
> How about if it is 365 times as long?
> I *DID* specify that I had one _YEAR_ of data ;)
>
>
>> If you don't like this assertion then consider this:
>>
>> If the data set is 1 hour long
>
> *BUT* it is 365*24 hours long ;)!

I misunderstood you then.  I thought the question was about short FFTs. To 
be clear, not only does the data set have to be long but also the 
transforms.

Fred 

On Apr 2, 12:40 pm, "Fred Marshall" <fmarshallx@remove_the_x.acm.org>
wrote:
> If the periodic variations are diurnal then how much data is needed to
> resolve the period?  If the period is 24 hours then you want to have
> resolution at least 1/24 hours == 0.0000115Hz.  This resolution requires a
> data set that's 24 hours long.

Doesn't the length of the data set needed depend on
the signal-to-noise ratio?  If the noise is zero,
then only 3 or so points may be enough to determine
an arbitrary sinusoid.  But if the signal is well
below the noise per bin floor in a 24 hour data set,
then it may require a very large multiple of 24 hours
of data to resolve the period above some statistical
likelihood for an accuracy bound of 1.15e-05 Hz.


IMHO. YMMV.
--
rhn A.T nicholson d.0.t C-o-M

On Apr 2, 3:17 pm, Richard Owlett <rowl...@atlascomm.net> wrote:
> In another thread (Re: FIR Bandpass filtering (C code)) the OP was doing
> FFT of short segments of a much larger financial data stream and was
> told FFT was inherently meaningless.
>
> I did not understand the stated logic.
>

Seems that they are considering signals of form:

  x[n] = \sum_k c_k (u_k)^n + noise.

A one-sided Z transform reveals that these signals can be
seen as a bunch of poles in the complex plane, located at
u_k.  Now, the FFT is an evaluation of the Z plane on the
unit circle.

Now, cos(\Omega_k n) show up as u_k = \exp(+ j \Omega_k)
and \exp(- j \Omega_k).  So the samples of the FFT fall close
to the locations of the poles.  But what if u_k are not complex
roots of unity?

I am not sure whether the argument is on the problem of
estimation of u_k or filtering out known u_k, they are two
very different problems.

Signals of this type are studied under "exponential analysis".

Hope that helps,
Julius

julius wrote:
> On Apr 2, 3:17 pm, Richard Owlett <rowl...@atlascomm.net> wrote:
> 
>>In another thread (Re: FIR Bandpass filtering (C code)) the OP was doing
>>FFT of short segments of a much larger financial data stream and was
>>told FFT was inherently meaningless.
>>
>>I did not understand the stated logic.
>>
> 
> 
> Seems that they are considering signals of form:
> 
>   x[n] = \sum_k c_k (u_k)^n + noise.
> 

I'm not sure I understand your *notation* .

I read it as

Nth element of X equals sum over all k of c-sub-k times u-sub-k raised 
to Nth power and a noise term. I presume c and u may be complex. I don't 
know if that is equivalent to model I am presuming.

My hypothesis is that I am dealing with a system that may be modeled by

(sinusoids) + (type1 perturbations) + (type2 perturbations) + noise

All "type1 perturbations" would be unidirectional with finite integral.
All "type2 perturbations" would be step functions in either direction.



> A one-sided Z transform reveals that these signals can be
> seen as a bunch of poles in the complex plane, located at
> u_k.  Now, the FFT is an evaluation of the Z plane on the
> unit circle.
> 
> Now, cos(\Omega_k n) show up as u_k = \exp(+ j \Omega_k)
> and \exp(- j \Omega_k).  So the samples of the FFT fall close
> to the locations of the poles.  But what if u_k are not complex
> roots of unity?
> 
> I am not sure whether the argument is on the problem of
> estimation of u_k or filtering out known u_k, they are two
> very different problems.
> 
> Signals of this type are studied under "exponential analysis".
> 
> Hope that helps,

nope ;/
I'm math challenged and think in physical terms. (Had no understanding 
of 1st semester calculus until 3rd semester physics. Poor math 
instructor (a pure mathematician) was babbling when his time with 
freshman engineers was over.

> Julius
> 

On 2 Apr, 22:17, Richard Owlett <rowl...@atlascomm.net> wrote:
> In another thread (Re: FIR Bandpass filtering (C code)) the OP was doing
> FFT of short segments of a much larger financial data stream and was
> told FFT was inherently meaningless.
>
> I did not understand the stated logic.

The argument is based on the fact that Fourier analysis as known
from DSP requires certain integrals or sums to be finite. The
types of signals which satisfy these requirements are known
as either "(finite) power signal" or "(finite) energy signal".
The data in question in the mentioned thread is neither, so
the proposed approach, using FIR filters to isolate a Fourier
component, don't make sense.

> I'll pose another data set.
> GPS position data from single frequency receiver.
>
> Such data is known to have periodic components to the position errors.
> (eg diurnal variation due to solar wind effects on ionosphere)
> Are white noise components.
> There are also anomalous impulse variations.

Same thing. A reciever might move along a meridian from the
south pole to the north pole. The logged positions will show
a general linear trend (assuming constant velocity) from
-90 deg to +90 degrees. It is not possible to use FIR or
IIR filters to isolate periodic fluctuations in such data.
If the sensor is located in one spot, though, you can
subtract the mean lat and long positions to isolate the
variations.

Rune

Rune Allnor wrote:
> On 2 Apr, 22:17, Richard Owlett <rowl...@atlascomm.net> wrote:
> 
>>In another thread (Re: FIR Bandpass filtering (C code)) the OP was doing
>>FFT of short segments of a much larger financial data stream and was
>>told FFT was inherently meaningless.
>>
>>I did not understand the stated logic.
> 
> 
> The argument is based on the fact that Fourier analysis as known
> from DSP requires certain integrals or sums to be finite.

Which ones? Point me to a reference if simpler. Would it be in Lyons' 
book? [will have to search for book - friends cleaned house when I was 
in hospital;]

> The
> types of signals which satisfy these requirements are known
> as either "(finite) power signal" or "(finite) energy signal".
> The data in question in the mentioned thread is neither, so
> the proposed approach, using FIR filters to isolate a Fourier
> component, don't make sense.
> 
> 
>>I'll pose another data set.
>>GPS position data from single frequency receiver.
>>
>>Such data is known to have periodic components to the position errors.
>>(eg diurnal variation due to solar wind effects on ionosphere)
>>Are white noise components.
>>There are also anomalous impulse variations.
> 
> 
> Same thing. A reciever might move along a meridian from the
> south pole to the north pole. The logged positions will show
> a general linear trend (assuming constant velocity) from
> -90 deg to +90 degrees. It is not possible to use FIR or
> IIR filters to isolate periodic fluctuations in such data.

Would first derivative of raw data satisfy restriction?


> If the sensor is located in one spot, though, you can
> subtract the mean lat and long positions to isolate the
> variations.

That's closer to my case.

My hypothesis is that I am dealing with a system that may be modeled by

(sinusoids) + (type1 perturbations) + (type2 perturbations) + noise

All "type1 perturbations" would be unidirectional with finite integral.
  [offsets due to tropospheric effects, choice of satellites chosen to
   emphasize effects - ie use only satellites west of zenith]
All "type2 perturbations" would be step functions in either direction.
  [eg slippage along fault line]

The sinusoids and noise have zero net integral.
The perturbations do not.

I'm interested in "type1 perturbations" ;)
I live in tornado alley and have read survey articles of NOAA using 
similar effect to measure total water vapor *vertically above* measuring 
station. Can approaching weather systems be "seen"? There are good data 
sets available that may allow "predicting" events that occurred in 
*PAST* week.



> 
> Rune
> 

On 3 Apr, 18:47, Richard Owlett <rowl...@atlascomm.net> wrote:
> Rune Allnor wrote:
> > On 2 Apr, 22:17, Richard Owlett <rowl...@atlascomm.net> wrote:
>
> >>In another thread (Re: FIR Bandpass filtering (C code)) the OP was doing
> >>FFT of short segments of a much larger financial data stream and was
> >>told FFT was inherently meaningless.
>
> >>I did not understand the stated logic.
>
> > The argument is based on the fact that Fourier analysis as known
> > from DSP requires certain integrals or sums to be finite.
>
> Which ones? Point me to a reference if simpler. Would it be in Lyons'
> book? [will have to search for book - friends cleaned house when I was
> in hospital;]

I'm away from home now, so I can't look it up. The only place
I *know* I saw this was in a book by Stremler on communications
systems, in the late '80s. The book by Proakis & Manolakis *might*
have mentioned this...

> > The
> > types of signals which satisfy these requirements are known
> > as either "(finite) power signal" or "(finite) energy signal".
> > The data in question in the mentioned thread is neither, so
> > the proposed approach, using FIR filters to isolate a Fourier
> > component, don't make sense.
>
> >>I'll pose another data set.
> >>GPS position data from single frequency receiver.
>
> >>Such data is known to have periodic components to the position errors.
> >>(eg diurnal variation due to solar wind effects on ionosphere)
> >>Are white noise components.
> >>There are also anomalous impulse variations.
>
> > Same thing. A reciever might move along a meridian from the
> > south pole to the north pole. The logged positions will show
> > a general linear trend (assuming constant velocity) from
> > -90 deg to +90 degrees. It is not possible to use FIR or
> > IIR filters to isolate periodic fluctuations in such data.
>
> Would first derivative of raw data satisfy restriction?

Formally, maybe. The problem with diffrentiators is that they
amplify noise.

> > If the sensor is located in one spot, though, you can
> > subtract the mean lat and long positions to isolate the
> > variations.
>
> That's closer to my case.
>
> My hypothesis is that I am dealing with a system that may be modeled by
>
> (sinusoids) + (type1 perturbations) + (type2 perturbations) + noise
>
> All "type1 perturbations" would be unidirectional with finite integral.
>   [offsets due to tropospheric effects, choice of satellites chosen to
>    emphasize effects - ie use only satellites west of zenith]
> All "type2 perturbations" would be step functions in either direction.
>   [eg slippage along fault line]
>
> The sinusoids and noise have zero net integral.
> The perturbations do not.
>
> I'm interested in "type1 perturbations" ;)
> I live in tornado alley and have read survey articles of NOAA using
> similar effect to measure total water vapor *vertically above* measuring
> station. Can approaching weather systems be "seen"? There are good data
> sets available that may allow "predicting" events that occurred in
> *PAST* week.

20 years ago I worked at a site where similar questions
were asked: Can deviations from a "normal" state be explained
in terms of step functions in governing variables?

I don't know what the answer is, but the guys who worked
on those sorts of analyses used a completely different set
of tools than "mere" FIR filters...

Rune