Astrodot-

>> Thanks for the response. I didn't describe my problem well. The data is actually a list of times of discrete events.
> The number of events is small, let's say a hundred or so, but they span a very long time in the following sense: the
> events are generated by a periodic process with period 10^-10 of the total observation time. I.e. if the period is 1,
> my data consists of a list of 100 times between 0 and 10^10.
>
> My plan was to bin the data with a bin size less than 1 and then do an FFT of this binned data. However, I would need
> on the order of 10^10 bins (virtually all of which would contain a 0 and one hundred of which would contain a 1).
> Doing an FFT of this size is impossible for me, though.
>
> Because the data is so sparse, the windows you suggest would each contain at most 1 event and an FFT of the events
> that arrived during one window would therefore contain no information.
>
> Hopefully I explained the problem better but if I didn't do a good job let me know if you have any questions.
> Thanks again for your help!

If your data is truly this sparse and your events are discrete (I take this to mean exist/not exist, or 0 and 1), then
FFT may be the wrong tool. An underlying frequency -- if it exists -- would be in the uHz to nHz range. With such
sparsity, you're probably better off just measuring intervals between events and looking for patterns, for example,
intervals that are multiples of others, etc.

-Jeff

> --- In a..., "Jeff Brower" wrote:
>>
>> Astrodot-
>>
>> > I am interested in detecting whether or not there is any
>> > periodicity in a bunch of very long time series. The lengths
>> > of the time series can be extremely long (e.g. 10^10 bins)
>> > and the periodic signals that might be present have periods
>> > of 1-10 bins or so. The data is very sparse.
>> >
>> > Because of the length of the time series a FFT would
>> > concentrate the power in a signal in a very very tiny
>> > frequency range.
>> >
>> > Is it possible to somehow sacrifice this frequency
>> > resolution for power in the following sense? I don't
>> > really care what the actual frequency of the periodicity
>> > is. I just want to know if it exists. I would rather more
>> > power spread out into a bunch of frequency bins than less
>> > power in a single bin.
>>
>> Suggest an STFFT analysis:
>>
>> -small analysis framesize (you mentioned 10 pts?)
>>
>> -4096 pt or larger FFT size, zero fill remaining pts
>>
>> -Hann window, 50% overlap
>>
>> -display using 3-D Spectrograph or Spectrogram type methods
>>
>> -Jeff

> Thanks for the response. I didn't describe my problem well. The data is actually a list of times of discrete events.
The number of events is small, let's say a hundred or so, but they span a very long time in the following sense: the events are generated by a periodic process with period 10^-10 of the total observation time. I.e. if the period is 1, my data consists of a list of 100 times between 0 and 10^10.

My plan was to bin the data with a bin size less than 1 and then do an FFT of this binned data. However, I would need on the order of 10^10 bins (virtually all of which would contain a 0 and one hundred of which would contain a 1). Doing an FFT of this size is impossible for me, though.

Because the data is so sparse, the windows you suggest would each contain at most 1 event and an FFT of the events that arrived during one window would therefore contain no information.

Hopefully I explained the problem better but if I didn't do a good job let me know if you have any questions.
Thanks again for your help!
(sorry about mis-sent post)

--- In a..., "Jeff Brower" wrote:
>
> Astrodot-
>
> > I am interested in detecting whether or not there is any
> > periodicity in a bunch of very long time series. The lengths
> > of the time series can be extremely long (e.g. 10^10 bins)
> > and the periodic signals that might be present have periods
> > of 1-10 bins or so. The data is very sparse.
> >
> > Because of the length of the time series a FFT would
> > concentrate the power in a signal in a very very tiny
> > frequency range.
> >
> > Is it possible to somehow sacrifice this frequency
> > resolution for power in the following sense? I don't
> > really care what the actual frequency of the periodicity
> > is. I just want to know if it exists. I would rather more
> > power spread out into a bunch of frequency bins than less
> > power in a single bin.
>
> Suggest an STFFT analysis:
>
> -small analysis framesize (you mentioned 10 pts?)
>
> -4096 pt or larger FFT size, zero fill remaining pts
>
> -Hann window, 50% overlap
>
> -display using 3-D Spectrograph or Spectrogram type methods
>
> -Jeff
>

Astrodot-

> I am interested in detecting whether or not there is any
> periodicity in a bunch of very long time series. The lengths
> of the time series can be extremely long (e.g. 10^10 bins)
> and the periodic signals that might be present have periods
> of 1-10 bins or so. The data is very sparse.
>
> Because of the length of the time series a FFT would
> concentrate the power in a signal in a very very tiny
> frequency range.
>
> Is it possible to somehow sacrifice this frequency
> resolution for power in the following sense? I don't
> really care what the actual frequency of the periodicity
> is. I just want to know if it exists. I would rather more
> power spread out into a bunch of frequency bins than less
> power in a single bin.

Suggest an STFFT analysis:

-small analysis framesize (you mentioned 10 pts?)

-4096 pt or larger FFT size, zero fill remaining pts

-Hann window, 50% overlap

-display using 3-D Spectrograph or Spectrogram type methods

-Jeff

Hi,

I am interested in detecting whether or not there is any periodicity in a bunch of very long time series. The lengths of the time series can be extremely long (e.g. 10^10 bins) and the periodic signals that might be present have periods of 1-10 bins or so. The data is very sparse.

Because of the length of the time series a FFT would concentrate the power in a signal in a very very tiny frequency range.

Is it possible to somehow sacrifice this frequency resolution for power in the following sense? I don't really care what the actual frequency of the periodicity is. I just want to know if it exists. I would rather more power spread out into a bunch of frequency bins than less power in a single bin.

Has anyone come across this sort of problem or have any thoughts on the issue?

Any ideas are much appreciated.

Thanks,

Alex