Spectral estimation with the Sliding DFT?

I was reading the paper "The Sliding DFT" by Eric Jacobsen and Richard
Lyons and have a couple of questions for you guys out there..

1)Is it possible to reduce the variance in estimating amplitude, frequency
and phase of a sinusoid by somehow averaging the sucessive outputs of each
bin and subtracting out some kind of linear phase term?

In the paper the author(s) say...
"We calculate S_k(n) by phase shifting the previous S_k(n − 1)
components, subtract the x(n − N) sample, and add the current
x(n) sample."

Id like to think of the dropping of x(n-N) and the adding of x(n) as some
kind of innovation, which we should be able to incorporate usefully in an
estimator.

2) What could one say about S_k(n) - S_k(n − 1)? Could we say anything
useful or not? Could one say the spectral phase in bin 'y' rotated 'x'
amount of radians in addition to a linear phase term?  Could we make
accurate realative measurements even though the variance of an absolute
estimate may not be so good?

On Wed, 13 Jun 2012 15:49:33 -0500, "westocl" <cweston_@n_o_s_p_a_m.hotmail.com>
wrote:

>1)Is it possible to reduce the variance in estimating amplitude, frequency
>and phase of a sinusoid by somehow averaging the sucessive outputs of each
>bin and subtracting out some kind of linear phase term?

A better way to do that might be with the "synchronous" form described here:
http://groups.google.com/groups?safe=images&ie=UTF-8&oe=UTF-8&as_ugroup=comp.dsp&as_umsgid=ft6uutcn4kq9u8uqvvrn1vh4lhi3tkkcgq@4ax.com&lr=&hl=en

In the synchronous form the reference time is fixed while the DFT window is slid
across the signal. In the traditional form the reference time moves with the DFT
window.

>2) What could one say about S_k(n) - S_k(n &minus; 1)? Could we say anything
>useful or not? Could one say the spectral phase in bin 'y' rotated 'x'
>amount of radians in addition to a linear phase term?  Could we make
>accurate realative measurements even though the variance of an absolute
>estimate may not be so good?

Well, since S_k(n) = S_k(n - 1) after some phase-shifting and addition of a
constant, S_k(n) - S_k(n - 1) should be representative of nothing more than that
phase shifting and that constant.

Indeed, in the synchronous form 

	S_k(n) = S_k(n - 1) + [x(n-N)-x(n)]*exp(-j2PIkn/N),

so

	S_k(n) - S_k(n - 1) = [x(n-N)-x(n)]*exp(-j2PIkn/N) exactly.

Greg

westocl <cweston_@n_o_s_p_a_m.hotmail.com> wrote:

(snip)

> 1)Is it possible to reduce the variance in estimating amplitude, 
> frequency and phase of a sinusoid by somehow averaging the 
> sucessive outputs of each bin and subtracting out some kind 
> of linear phase term?

The general rule is that averaging reduces the variance by a
factor or N. (Standard Deviation by sqrt(N).)

That assumes no correlations in the sampling, though, which
might not be a good assumption here.

-- glen

On Wed, 13 Jun 2012 22:26:25 +0000 (UTC), glen herrmannsfeldt
<gah@ugcs.caltech.edu> wrote:

>westocl <cweston_@n_o_s_p_a_m.hotmail.com> wrote:
>
>(snip)
>
>> 1)Is it possible to reduce the variance in estimating amplitude, 
>> frequency and phase of a sinusoid by somehow averaging the 
>> sucessive outputs of each bin and subtracting out some kind 
>> of linear phase term?
>
>The general rule is that averaging reduces the variance by a
>factor or N. (Standard Deviation by sqrt(N).)
>
>That assumes no correlations in the sampling, though, which
>might not be a good assumption here.

Exactly, the slide to the next window of an N-point transform still
has N-1 inputs in common with the previous output, so correlation is
high.  


Eric Jacobsen
Anchor Hill Communications
www.anchorhill.com

>On Wed, 13 Jun 2012 22:26:25 +0000 (UTC), glen herrmannsfeldt
><gah@ugcs.caltech.edu> wrote:
>
>>westocl <cweston_@n_o_s_p_a_m.hotmail.com> wrote:
>>
>>(snip)
>>
>>> 1)Is it possible to reduce the variance in estimating amplitude, 
>>> frequency and phase of a sinusoid by somehow averaging the 
>>> sucessive outputs of each bin and subtracting out some kind 
>>> of linear phase term?
>>
>>The general rule is that averaging reduces the variance by a
>>factor or N. (Standard Deviation by sqrt(N).)
>>
>>That assumes no correlations in the sampling, though, which
>>might not be a good assumption here.
>
>Exactly, the slide to the next window of an N-point transform still
>has N-1 inputs in common with the previous output, so correlation is
>high.  
>
>
>Eric Jacobsen
>Anchor Hill Communications
>www.anchorhill.com


>>The general rule is that averaging reduces the variance by a
>>factor or N. (Standard Deviation by sqrt(N).)
>>
>>That assumes no correlations in the sampling, though, which
>>might not be a good assumption here.
>
>Exactly, the slide to the next window of an N-point transform still
>has N-1 inputs in common with the previous output, so correlation is
>high. 


Agreed, but one must get something with the non correlated portion, and
when we consider a 'Real Time' type stream of data, that little somehting
has to rack up  pretty quickly.

It would be a neat idea to do a paper on tho...

Eric, wanna do that paper, so i can just read it and not have to go through
the math and all myself?? :)





A
>

Eric Jacobsen <eric.jacobsen@ieee.org> wrote:

>>westocl <cweston_@n_o_s_p_a_m.hotmail.com> wrote:

>>(snip)
>>> 1)Is it possible to reduce the variance in estimating amplitude, 
>>> frequency and phase of a sinusoid by somehow averaging the 
>>> sucessive outputs of each bin and subtracting out some kind 
>>> of linear phase term?

(snip, then I wrote)
>>The general rule is that averaging reduces the variance by a
>>factor or N. (Standard Deviation by sqrt(N).)

>>That assumes no correlations in the sampling, though, which
>>might not be a good assumption here.

> Exactly, the slide to the next window of an N-point transform still
> has N-1 inputs in common with the previous output, so correlation is
> high.  

Hmmm. 

If you look at the DFT result as the (complex) weighted average of N
input points, then it should still apply.

Well, start with the 0 bin, which is just the average of N input points.

In the case of a sliding transform, the result is just a moving
average, which has the usual properties of an N point average (mean).

If you combine the results from different transforms, such as
shifted transforms, then you have more points in the average.

It would seem easier, though, to just do a longer transform.

If you use the transform after a shift of just one, then you have N+1
input points and, with proper weighting, should be able to get the
variance as you would expect for N+1 points. Because of the correlation,
you don't get 2N points worth of averaging.

-- glen

>On Wed, 13 Jun 2012 15:49:33 -0500, "westocl"
<cweston_@n_o_s_p_a_m.hotmail.com>
>wrote:
>
>>1)Is it possible to reduce the variance in estimating amplitude,
frequency
>>and phase of a sinusoid by somehow averaging the sucessive outputs of
each
>>bin and subtracting out some kind of linear phase term?
>
>A better way to do that might be with the "synchronous" form described
here:
>http://groups.google.com/groups?safe=images&ie=UTF-8&oe=UTF-8&as_ugroup=comp.dsp&as_umsgid=ft6uutcn4kq9u8uqvvrn1vh4lhi3tkkcgq@4ax.com&lr=&hl=en
>
>In the synchronous form the reference time is fixed while the DFT window
is slid
>across the signal. In the traditional form the reference time moves with
the DFT
>window.
>
>>2) What could one say about S_k(n) - S_k(n &minus; 1)? Could we say anything
>>useful or not? Could one say the spectral phase in bin 'y' rotated 'x'
>>amount of radians in addition to a linear phase term?  Could we make
>>accurate realative measurements even though the variance of an absolute
>>estimate may not be so good?
>
>Well, since S_k(n) = S_k(n - 1) after some phase-shifting and addition of
a
>constant, S_k(n) - S_k(n - 1) should be representative of nothing more
than that
>phase shifting and that constant.
>
>Indeed, in the synchronous form 
>
>	S_k(n) = S_k(n - 1) + [x(n-N)-x(n)]*exp(-j2PIkn/N),
>
>so
>
>	S_k(n) - S_k(n - 1) = [x(n-N)-x(n)]*exp(-j2PIkn/N) exactly.
>
>Greg


>A better way to do that might be with the "synchronous" form described
here:
>http://groups.google.com/groups?safe=images&ie=UTF-8&oe=UTF-8&as_ugroup=comp.dsp&as_umsgid=ft6uutcn4kq9u8uqvvrn1vh4lhi3tkkcgq@4ax.com&lr=&hl=en


Hey Greg, 

That link is broken, id like to look more into it, could you email me the
link at westocl@yahoo.com?

Thanks

>Eric Jacobsen <eric.jacobsen@ieee.org> wrote:
>
>>>westocl <cweston_@n_o_s_p_a_m.hotmail.com> wrote:
>
>>>(snip)
>>>> 1)Is it possible to reduce the variance in estimating amplitude, 
>>>> frequency and phase of a sinusoid by somehow averaging the 
>>>> sucessive outputs of each bin and subtracting out some kind 
>>>> of linear phase term?
>
>(snip, then I wrote)
>>>The general rule is that averaging reduces the variance by a
>>>factor or N. (Standard Deviation by sqrt(N).)
>
>>>That assumes no correlations in the sampling, though, which
>>>might not be a good assumption here.
>
>> Exactly, the slide to the next window of an N-point transform still
>> has N-1 inputs in common with the previous output, so correlation is
>> high.  
>
>Hmmm. 
>
>If you look at the DFT result as the (complex) weighted average of N
>input points, then it should still apply.
>
>Well, start with the 0 bin, which is just the average of N input points.
>
>In the case of a sliding transform, the result is just a moving
>average, which has the usual properties of an N point average (mean).
>
>If you combine the results from different transforms, such as
>shifted transforms, then you have more points in the average.
>
>It would seem easier, though, to just do a longer transform.
>
>If you use the transform after a shift of just one, then you have N+1
>input points and, with proper weighting, should be able to get the
>variance as you would expect for N+1 points. Because of the correlation,
>you don't get 2N points worth of averaging.
>
>-- glen
>

>It would seem easier, though, to just do a longer transform.

I think the result would be differnt however... if one does some kind of
'sliding averaging' i would think the spectral leakage would average also,
because we consistently have a new beginning and end point on the data.

On Wed, 13 Jun 2012 19:26:13 -0500, "westocl"
<cweston_@n_o_s_p_a_m.hotmail.com> wrote:

>>Eric Jacobsen <eric.jacobsen@ieee.org> wrote:
>>
>>>>westocl <cweston_@n_o_s_p_a_m.hotmail.com> wrote:
>>
>>>>(snip)
>>>>> 1)Is it possible to reduce the variance in estimating amplitude, 
>>>>> frequency and phase of a sinusoid by somehow averaging the 
>>>>> sucessive outputs of each bin and subtracting out some kind 
>>>>> of linear phase term?
>>
>>(snip, then I wrote)
>>>>The general rule is that averaging reduces the variance by a
>>>>factor or N. (Standard Deviation by sqrt(N).)
>>
>>>>That assumes no correlations in the sampling, though, which
>>>>might not be a good assumption here.
>>
>>> Exactly, the slide to the next window of an N-point transform still
>>> has N-1 inputs in common with the previous output, so correlation is
>>> high.  
>>
>>Hmmm. 
>>
>>If you look at the DFT result as the (complex) weighted average of N
>>input points, then it should still apply.
>>
>>Well, start with the 0 bin, which is just the average of N input points.
>>
>>In the case of a sliding transform, the result is just a moving
>>average, which has the usual properties of an N point average (mean).
>>
>>If you combine the results from different transforms, such as
>>shifted transforms, then you have more points in the average.
>>
>>It would seem easier, though, to just do a longer transform.
>>
>>If you use the transform after a shift of just one, then you have N+1
>>input points and, with proper weighting, should be able to get the
>>variance as you would expect for N+1 points. Because of the correlation,
>>you don't get 2N points worth of averaging.
>>
>>-- glen
>>
>
>>It would seem easier, though, to just do a longer transform.
>
>I think the result would be differnt however... if one does some kind of
>'sliding averaging' i would think the spectral leakage would average also,
>because we consistently have a new beginning and end point on the data.
>

Each sliding window DFT output is just the DFT of the current window.
Doing what you suggest could be useful for situations where each bin
output (or the bin output of interest) is taken as a sample in a
highly oversampled system and processed accordingly (like an input to
a PLL or something).   One could get a better estimate of frequency by
measuring the phase over time of a particular bin output, but I think
it would compare to the better frequency estimate from a longer
transform taken over the same time.

There'd be processing complexity tradeoffs, of course, as well as
latency, etc., for other outputs that may be important.

I think what you propose would work, at least I can't see why it
wouldn't, but whether it's the best way to do it in a given
application probably depends on a lot of other things.


Eric Jacobsen
Anchor Hill Communications
www.anchorhill.com

On Wed, 13 Jun 2012 23:30:46 GMT, eric.jacobsen@ieee.org (Eric
Jacobsen) wrote:

>On Wed, 13 Jun 2012 22:26:25 +0000 (UTC), glen herrmannsfeldt
><gah@ugcs.caltech.edu> wrote:
>
>>westocl <cweston_@n_o_s_p_a_m.hotmail.com> wrote:
>>
>>(snip)
>>
>>> 1)Is it possible to reduce the variance in estimating amplitude, 
>>> frequency and phase of a sinusoid by somehow averaging the 
>>> sucessive outputs of each bin and subtracting out some kind 
>>> of linear phase term?
>>
>>The general rule is that averaging reduces the variance by a
>>factor or N. (Standard Deviation by sqrt(N).)
>>
>>That assumes no correlations in the sampling, though, which
>>might not be a good assumption here.
>
>Exactly, the slide to the next window of an N-point transform still
>has N-1 inputs in common with the previous output, so correlation is
>high.  

Hi Eric,
   At first I thought of the obvious way to minimize the 
variance of spectral magnitude estimation by averaging.
But you and Glen, smartly, mentioned that averaging 
successive SDFT outputs are too-highly correlated and the 
variance reduction effects of averaging could not be achieved. 

To add my 1.5 cents to this discussion, I'd guess that if 
the SDFT's are N-point SDFTs, then westocl could achieve 
variance reduction if he averages SDFT spec magnitudes 
for every (N/2)th-in-time SDFT spec magnitude.

That is, if the SDFTs are 128-point SDFTs, then he could 
average a sequence of SDFT spec magnitudes from every 
64th-in-time SDFT spec magnitude.

See Ya',
[-Rick-]