noise reduction with long fft's

On Tuesday, April 28, 2015 at 5:47:08 AM UTC-7, Rick Lyons wrote:

> Hi Bob,
>   If by "vector averaging" you mean separately averaging 
> the real and imaginary parts of the corresponding DFT bins 
> of multiple DFTs, I thought that process is only useful 
> if two conditions are satisfied:
.
> [1] The signal of interest is periodic, and 
> [2] The onset of A/D conversion is synchronized 
>     with the periodic signal of interest.
.
> It seems to me that such a process is rarely useful 
> because "typical" signals of interest are usually 
> not periodic.  (Except, maybe, radar signals.)
.
> [-Rick-]

Rick

I started with PSD calculated from a noise signal. It has a large error from the noise process PSD no matter how large the fft. The estimate of process PSD can be improved by averaging many PSD calculations. Synchronization does not apply.

For non-periodic or slowly varying period or unknown period signals, power spectrum has been averaged to improve signal estimates despite lack of complete knowledge or periodicity.

Where there is periodicity and an integer number of samples per period, you can average synchronized data blocks as either time samples or dft coefficients to produce signal gain against noise. This is sometimes called 'coherent integration'. Averaging in the time domain requires only a single fft calculation. The order of dft and sum can be reversed since both are linear when no power calculation is  performed in between.

The synchronization problem is commonly solved in applications like roller balancing in many types of industrial mills, balancing shafts of turbine engines (stationary generators and aircraft engines) and helicopter rotor track and balancing by generation of a tach signal on the hardware itself. There really are a lot of places where synchronous machinery vibration analysis is done.

Dale B Dalrymple

I think the ultimate is to use an evolutionary algorithm to fit the data set with as many a*sin(f*t+c) functions you want/need.
Yes, I know it is slow, in the 10's of seconds range.  However for things like astronomical data sets it is a much better idea than using fft's that throw up all kinds of mathematical difficulties.

Tikhonov, as part of his programme of formulating a framework of regularization methods for ill-defined and ambiguously-defined mathematical devised a regularization for the Fourier Transform. This can be found under "Tikhonov Regularization", searched on in conjunction with "Fourier Transform".

Noise removal is a natural outgrowth of the process, and the same basic approach is also applied to deconvolution, yielding the Wiener method as a special case.