Estimating the Spectrum of a Non-Stationary Process

I'm working with some folks who are trying to estimate the spectrum of a
signal that has continuous noise and a deterministic signal of
relatively small temporal extent. Their current method is just to take
the squared magnitude of an FFT to perform the estimate. I suggested
that they average FFT squared magnitudes (over several FFTs) in order to
mitigate the well-known phenomenom of the variance of such spectral
estimates when using a single FFT.

However, since the deterministic component of the signal is of short
temporal extent, they don't feel (and I think they are right) that they
can afford to average FFTs (periodograms) since that would "smear" and
delay their estimates and hinder detection.

They also cannot afford to take smaller FFTs and average those together
since the SNR is low and the processing gain of a large FFT is required.

Is there a technique that will allow the variance of the spectral
estimates to be reduced in such a case?
-- 
Randy Yates
Digital Signal Labs
http://www.digitalsignallabs.com

>I'm working with some folks who are trying to estimate the spectrum of a
>signal that has continuous noise and a deterministic signal of
>relatively small temporal extent. Their current method is just to take
>the squared magnitude of an FFT to perform the estimate. I suggested
>that they average FFT squared magnitudes (over several FFTs) in order to
>mitigate the well-known phenomenom of the variance of such spectral
>estimates when using a single FFT.
>
>However, since the deterministic component of the signal is of short
>temporal extent, they don't feel (and I think they are right) that they
>can afford to average FFTs (periodograms) since that would "smear" and
>delay their estimates and hinder detection.
>
>They also cannot afford to take smaller FFTs and average those together
>since the SNR is low and the processing gain of a large FFT is required.
>
>Is there a technique that will allow the variance of the spectral
>estimates to be reduced in such a case?
>-- 
>Randy Yates
>Digital Signal Labs
>http://www.digitalsignallabs.com
>

By averageing the spectrum of the signal over a long period of time, I
would think you'd be left with the shape of the continuous noise.  Could
you then attenuate that spectrum from the individual FFTs and be left with
the spectrum of desired signal?

If all you cared about was the shape of the spectrum, I think that would
work.

Hi,

thinking very simple: How about a noise gate / squelch function? Suppressed
parts of the signal go as zero samples into the FFT, and don't contribute
to the FFT output.

On 3/25/2013 11:00 AM, Randy Yates wrote:
> I'm working with some folks who are trying to estimate the spectrum of a
> signal that has continuous noise and a deterministic signal of
> relatively small temporal extent. Their current method is just to take
> the squared magnitude of an FFT to perform the estimate. I suggested
> that they average FFT squared magnitudes (over several FFTs) in order to
> mitigate the well-known phenomenom of the variance of such spectral
> estimates when using a single FFT.
>
> However, since the deterministic component of the signal is of short
> temporal extent, they don't feel (and I think they are right) that they
> can afford to average FFTs (periodograms) since that would "smear" and
> delay their estimates and hinder detection.
>
> They also cannot afford to take smaller FFTs and average those together
> since the SNR is low and the processing gain of a large FFT is required.
>
> Is there a technique that will allow the variance of the spectral
> estimates to be reduced in such a case?

1. How different are the signal and the noise? I.e. what is the criterion?

2. If the signal has characteristic duration T, then the FFT window size 
should be T and the FFT should be performed at every T/2 or more often.

Vladimir Vassilevsky
DSP and Mixed Signal Designs
www.abvolt.com

On Mon, 25 Mar 2013 12:00:14 -0400, Randy Yates
<yates@digitalsignallabs.com> wrote:

>I'm working with some folks who are trying to estimate the spectrum of a
>signal that has continuous noise and a deterministic signal of
>relatively small temporal extent. Their current method is just to take
>the squared magnitude of an FFT to perform the estimate. I suggested
>that they average FFT squared magnitudes (over several FFTs) in order to
>mitigate the well-known phenomenom of the variance of such spectral
>estimates when using a single FFT.
>
>However, since the deterministic component of the signal is of short
>temporal extent, they don't feel (and I think they are right) that they
>can afford to average FFTs (periodograms) since that would "smear" and
>delay their estimates and hinder detection.
>
>They also cannot afford to take smaller FFTs and average those together
>since the SNR is low and the processing gain of a large FFT is required.
>
>Is there a technique that will allow the variance of the spectral
>estimates to be reduced in such a case?
>-- 
>Randy Yates
>Digital Signal Labs
>http://www.digitalsignallabs.com

Does their detection algorithm lend itself to averaging or some other
combining method?   In other words, if the detection method has or can
be made to have a "soft" output, then averaging the detector/estimator
outputs over a few FFTs (even if they overlap) may have benefit.


Eric Jacobsen
Anchor Hill Communications
http://www.anchorhill.com

eric.jacobsen@ieee.org (Eric Jacobsen) writes:

> On Mon, 25 Mar 2013 12:00:14 -0400, Randy Yates
> <yates@digitalsignallabs.com> wrote:
>
>>I'm working with some folks who are trying to estimate the spectrum of a
>>signal that has continuous noise and a deterministic signal of
>>relatively small temporal extent. Their current method is just to take
>>the squared magnitude of an FFT to perform the estimate. I suggested
>>that they average FFT squared magnitudes (over several FFTs) in order to
>>mitigate the well-known phenomenom of the variance of such spectral
>>estimates when using a single FFT.
>>
>>However, since the deterministic component of the signal is of short
>>temporal extent, they don't feel (and I think they are right) that they
>>can afford to average FFTs (periodograms) since that would "smear" and
>>delay their estimates and hinder detection.
>>
>>They also cannot afford to take smaller FFTs and average those together
>>since the SNR is low and the processing gain of a large FFT is required.
>>
>>Is there a technique that will allow the variance of the spectral
>>estimates to be reduced in such a case?
>>-- 
>>Randy Yates
>>Digital Signal Labs
>>http://www.digitalsignallabs.com
>
> Does their detection algorithm lend itself to averaging or some other
> combining method?   In other words, if the detection method has or can
> be made to have a "soft" output, then averaging the detector/estimator
> outputs over a few FFTs (even if they overlap) may have benefit.

I don't know - haven't been privvy to that algorithm. 
-- 
Randy Yates
Digital Signal Labs
http://www.digitalsignallabs.com

Vladimir Vassilevsky <nospam@nowhere.com> writes:

> On 3/25/2013 11:00 AM, Randy Yates wrote:
>> I'm working with some folks who are trying to estimate the spectrum of a
>> signal that has continuous noise and a deterministic signal of
>> relatively small temporal extent. Their current method is just to take
>> the squared magnitude of an FFT to perform the estimate. I suggested
>> that they average FFT squared magnitudes (over several FFTs) in order to
>> mitigate the well-known phenomenom of the variance of such spectral
>> estimates when using a single FFT.
>>
>> However, since the deterministic component of the signal is of short
>> temporal extent, they don't feel (and I think they are right) that they
>> can afford to average FFTs (periodograms) since that would "smear" and
>> delay their estimates and hinder detection.
>>
>> They also cannot afford to take smaller FFTs and average those together
>> since the SNR is low and the processing gain of a large FFT is required.
>>
>> Is there a technique that will allow the variance of the spectral
>> estimates to be reduced in such a case?
>
> 1. How different are the signal and the noise? I.e. what is the
> criterion?

Hey Vlad,

I don't know how they are doing the detection, and I also am not privvy
to the signal details.

> 2. If the signal has characteristic duration T, then the FFT window
> size should be T and the FFT should be performed at every T/2 or more
> often.

That is actually a very good idea, for this and another reason I haven't
even mentioned.

One problem they're having in the implementation is that the data
transfer time is dwarfing the FFT time, but they have been doing
non-overlapping FFTs. This could very well improve the transfer time as
well as spectral estimates!
-- 
Randy Yates
Digital Signal Labs
http://www.digitalsignallabs.com

On Mon, 25 Mar 2013 12:00:14 -0400, Randy Yates wrote:

> I'm working with some folks who are trying to estimate the spectrum of a
> signal that has continuous noise and a deterministic signal of
> relatively small temporal extent. Their current method is just to take
> the squared magnitude of an FFT to perform the estimate. I suggested
> that they average FFT squared magnitudes (over several FFTs) in order to
> mitigate the well-known phenomenom of the variance of such spectral
> estimates when using a single FFT.
> 
> However, since the deterministic component of the signal is of short
> temporal extent, they don't feel (and I think they are right) that they
> can afford to average FFTs (periodograms) since that would "smear" and
> delay their estimates and hinder detection.
> 
> They also cannot afford to take smaller FFTs and average those together
> since the SNR is low and the processing gain of a large FFT is required.
> 
> Is there a technique that will allow the variance of the spectral
> estimates to be reduced in such a case?

Are they trying to get the spectrum of the deterministic signal without 
the noise, or are they trying to get a spectral estimate of signal + 
noise?

If they can synchronize to the deterministic signal, getting a bunch of 
samples of it and averaging them would pound the noise down -- then they 
could just do an FFT of what's left.

-- 
My liberal friends think I'm a conservative kook.
My conservative friends think I'm a liberal kook.
Why am I not happy that they have found common ground?

Tim Wescott, Communications, Control, Circuits & Software
http://www.wescottdesign.com

Tim Wescott <tim@seemywebsite.com> writes:

> On Mon, 25 Mar 2013 12:00:14 -0400, Randy Yates wrote:
>
>> I'm working with some folks who are trying to estimate the spectrum of a
>> signal that has continuous noise and a deterministic signal of
>> relatively small temporal extent. Their current method is just to take
>> the squared magnitude of an FFT to perform the estimate. I suggested
>> that they average FFT squared magnitudes (over several FFTs) in order to
>> mitigate the well-known phenomenom of the variance of such spectral
>> estimates when using a single FFT.
>> 
>> However, since the deterministic component of the signal is of short
>> temporal extent, they don't feel (and I think they are right) that they
>> can afford to average FFTs (periodograms) since that would "smear" and
>> delay their estimates and hinder detection.
>> 
>> They also cannot afford to take smaller FFTs and average those together
>> since the SNR is low and the processing gain of a large FFT is required.
>> 
>> Is there a technique that will allow the variance of the spectral
>> estimates to be reduced in such a case?
>
> Are they trying to get the spectrum of the deterministic signal without 
> the noise, or are they trying to get a spectral estimate of signal + 
> noise?

They are trying to detect the presence of the deterministic signal. If
you know of a magical algorithm that will remove the noise from a
signal, I'd like to see it!

> If they can synchronize to the deterministic signal, getting a bunch of 
> samples of it and averaging them would pound the noise down -- then they 
> could just do an FFT of what's left.

Chicken/egg - this is the detection stage - they don't even know they
HAVE a signal...
-- 
Randy Yates
Digital Signal Labs
http://www.digitalsignallabs.com

I would be tempted to try the following.

Lets assume the noise portion of the signal has been present for a long time, without the deterministic signal. You could then average many fft frames together and get an accurate spectral estimate of the noise. You would want to have a finite number of past frames in the average, such that the estimate is continuously evolving. 

Now assume that you take the Euclidian distance between the magnitude of the current fft frame and the stored noise estimate. In the non-deterministic-signal case, this will have some Gaussian distribution around the averaged noise estimate, but when the deterministic  signal is present you will get a larger-than-expected variance in the Euclidian distance. Assuming your frames are sized according the the temporal duration of the signal, you might get 2 or 3 frames with this excess variance relative to the stored noise average spectrum, and this could be used as your signal-present flag.

Of course one problem is that the signal will affect the noise estimate. If the signal does not occur very often, you might get away with ignoring this error. If the signal burst happens frequently, you might need a more complicated strategy where you don't update the noise estimate when you think the signal is present. This leads to an initial startup chicken-and-egg problem, but you could probably work around this by using a timeout procedure. 

Bob