Noise filter and fourier

How to remove noise from a signal.

I have available a zero signal mainly consisting of noise and a signal 
during a measurement. The real signal consists theoretically of one or a 
few (overlapping) more or less gaussian shaped peaks. In general the 
signal is collected in 64 time samples. The peak widths are then equal 
to between 20 and 40 time samples. The noise has essentially a Poisson 
distribution (counting process). A more or less "true" signal can be 
obtianed from a measurement with high signal.

I understand that a neat filtering method is based on the convolution of 
the measurement signal and the zero signal using Fourier 
transformations. The FT of the zero signal thereby (after some 
transformation I presume) acting as a filter, but how?

Can anybody give me steps needed in terms understandable for somebody 
not ignorant of numerical math but certainly not an expert in signal 
processing.

Many thanks,
Janwillem

P.S. Any source examples in e.g. Pascal Fortran C or Mathematica most 
welcome

e-mail replace xyz -> jwe

"Janwillem van Dijk" <xyz.van.dijk@hccnet.nl> wrote in message 
news:41836bdb$0$162$3a628fcd@reader1.nntp.hccnet.nl...
> How to remove noise from a signal.
>
> I have available a zero signal mainly consisting of noise and a signal 
> during a measurement. The real signal consists theoretically of one or a 
> few (overlapping) more or less gaussian shaped peaks. In general the 
> signal is collected in 64 time samples. The peak widths are then equal to 
> between 20 and 40 time samples. The noise has essentially a Poisson 
> distribution (counting process). A more or less "true" signal can be 
> obtianed from a measurement with high signal.
>
> I understand that a neat filtering method is based on the convolution of 
> the measurement signal and the zero signal using Fourier transformations. 
> The FT of the zero signal thereby (after some transformation I presume) 
> acting as a filter, but how?
>
> Can anybody give me steps needed in terms understandable for somebody not 
> ignorant of numerical math but certainly not an expert in signal 
> processing.

I don't know exactly what you mean by a "zero signal" but I will assume that 
you meant it is the best replica you have of the signal that you want to 
detect.
But then you say that the "real signal has: peak widths are then equal to 
between 20 and 40 time samples"????

So, it's not clear what your starting reference is and what you're trying to 
detect.  Actually, I guess you didn't say you wanted to detect - so that's 
an assumption.  What are you wanting to do?
- filter an obviously present signal so that you can extract parameters?
- detect a not-so obviously present signal?
- both?

The process with the FFT you describe sounds like a matched filter which has 
optimum properties in the presence of white noise.  Here's how that goes:

Cross-correlate the signal reference with any / all received or measured 
signals.
Signals that match the reference will be "enhanced" relative to the noise.
The ideal output that results is the cross-correlation of the reference with 
itself.  Sometimes called the "ambiguity function" - you will want to ponder 
what this looks like and how it affects what you do with the results.
Cross-correlation is very closely related to convolution - as you probably 
know.

Where does an FFT come into this?
You can efficiently compute a temporal convolution like this:
FFT a signal replica and save this array.
FFT measurements (in an array length the same as the replica above) in 
succeeding array blocks.
Multiply each measurement by the replica FFT you have saved.
IFFT the product.
For correlation, time-reverse the replica before the FFT - I believe that's 
all it takes (check this out).

Now, you may have many replicas depending on what the signal of interest 
looks like.
For example, in sonar and radar there would be a replica for each Doppler 
shift and you may choose to quantize the replica parameter into a discrete 
set of values and, thus, replicas.
In your case, there may be a replica for each width of the signal shape you 
described.

Fred

Fred Marshall wrote:
> "Janwillem van Dijk" <xyz.van.dijk@hccnet.nl> wrote in message 
> news:41836bdb$0$162$3a628fcd@reader1.nntp.hccnet.nl...
> 
>>How to remove noise from a signal.
>>
>>I have available a zero signal mainly consisting of noise and a signal 
>>during a measurement. The real signal consists theoretically of one or a 
>>few (overlapping) more or less gaussian shaped peaks. In general the 
>>signal is collected in 64 time samples. The peak widths are then equal to 
>>between 20 and 40 time samples. The noise has essentially a Poisson 
>>distribution (counting process). A more or less "true" signal can be 
>>obtianed from a measurement with high signal.
>>
>>I understand that a neat filtering method is based on the convolution of 
>>the measurement signal and the zero signal using Fourier transformations. 
>>The FT of the zero signal thereby (after some transformation I presume) 
>>acting as a filter, but how?
>>
>>Can anybody give me steps needed in terms understandable for somebody not 
>>ignorant of numerical math but certainly not an expert in signal 
>>processing.
> 
> 
> I don't know exactly what you mean by a "zero signal" but I will assume that 
> you meant it is the best replica you have of the signal that you want to 
> detect.
> But then you say that the "real signal has: peak widths are then equal to 
> between 20 and 40 time samples"????
> 
> So, it's not clear what your starting reference is and what you're trying to 
> detect.  Actually, I guess you didn't say you wanted to detect - so that's 
> an assumption.  What are you wanting to do?
> - filter an obviously present signal so that you can extract parameters?
> - detect a not-so obviously present signal?
> - both?
> 
> The process with the FFT you describe sounds like a matched filter which has 
> optimum properties in the presence of white noise.  Here's how that goes:
> 
> Cross-correlate the signal reference with any / all received or measured 
> signals.
> Signals that match the reference will be "enhanced" relative to the noise.
> The ideal output that results is the cross-correlation of the reference with 
> itself.  Sometimes called the "ambiguity function" - you will want to ponder 
> what this looks like and how it affects what you do with the results.
> Cross-correlation is very closely related to convolution - as you probably 
> know.
> 
> Where does an FFT come into this?
> You can efficiently compute a temporal convolution like this:
> FFT a signal replica and save this array.
> FFT measurements (in an array length the same as the replica above) in 
> succeeding array blocks.
> Multiply each measurement by the replica FFT you have saved.
> IFFT the product.
> For correlation, time-reverse the replica before the FFT - I believe that's 
> all it takes (check this out).
> 
> Now, you may have many replicas depending on what the signal of interest 
> looks like.
> For example, in sonar and radar there would be a replica for each Doppler 
> shift and you may choose to quantize the replica parameter into a discrete 
> set of values and, thus, replicas.
> In your case, there may be a replica for each width of the signal shape you 
> described.
> 
> Fred
> 
> 
> 
> 
Thanks for the answers, I will have a look at that. Sounds easy enough.

What I meant with zero-signal versus measurement signal is the response 
of the instrument witha blank sample versus the response with either a
standard sample giving relatively high signal for calibration purposus 
or a sample to be analysed with in general a rather low signal.

Hope this helps in understanding my question.
Janwillem

"Janwillem van Dijk" <xyz.van.dijk@hccnet.nl> wrote in message 
news:4184abaf$0$161$3a628fcd@reader2.nntp.hccnet.nl...
> Fred Marshall wrote:
>> "Janwillem van Dijk" <xyz.van.dijk@hccnet.nl> wrote in message 
>> news:41836bdb$0$162$3a628fcd@reader1.nntp.hccnet.nl...
>>
>>>How to remove noise from a signal.
>>>
>>>I have available a zero signal mainly consisting of noise and a signal 
>>>during a measurement. The real signal consists theoretically of one or a 
>>>few (overlapping) more or less gaussian shaped peaks. In general the 
>>>signal is collected in 64 time samples. The peak widths are then equal to 
>>>between 20 and 40 time samples. The noise has essentially a Poisson 
>>>distribution (counting process). A more or less "true" signal can be 
>>>obtianed from a measurement with high signal.
>>>
>>>I understand that a neat filtering method is based on the convolution of 
>>>the measurement signal and the zero signal using Fourier transformations. 
>>>The FT of the zero signal thereby (after some transformation I presume) 
>>>acting as a filter, but how?
>>>
>>>Can anybody give me steps needed in terms understandable for somebody not 
>>>ignorant of numerical math but certainly not an expert in signal 
>>>processing.
>>
>>
>> I don't know exactly what you mean by a "zero signal" but I will assume 
>> that you meant it is the best replica you have of the signal that you 
>> want to detect.
>> But then you say that the "real signal has: peak widths are then equal to 
>> between 20 and 40 time samples"????
>>
>> So, it's not clear what your starting reference is and what you're trying 
>> to detect.  Actually, I guess you didn't say you wanted to detect - so 
>> that's an assumption.  What are you wanting to do?
>> - filter an obviously present signal so that you can extract parameters?
>> - detect a not-so obviously present signal?
>> - both?
>>
>> The process with the FFT you describe sounds like a matched filter which 
>> has optimum properties in the presence of white noise.  Here's how that 
>> goes:
>>
>> Cross-correlate the signal reference with any / all received or measured 
>> signals.
>> Signals that match the reference will be "enhanced" relative to the 
>> noise.
>> The ideal output that results is the cross-correlation of the reference 
>> with itself.  Sometimes called the "ambiguity function" - you will want 
>> to ponder what this looks like and how it affects what you do with the 
>> results.
>> Cross-correlation is very closely related to convolution - as you 
>> probably know.
>>
>> Where does an FFT come into this?
>> You can efficiently compute a temporal convolution like this:
>> FFT a signal replica and save this array.
>> FFT measurements (in an array length the same as the replica above) in 
>> succeeding array blocks.
>> Multiply each measurement by the replica FFT you have saved.
>> IFFT the product.
>> For correlation, time-reverse the replica before the FFT - I believe 
>> that's all it takes (check this out).
>>
>> Now, you may have many replicas depending on what the signal of interest 
>> looks like.
>> For example, in sonar and radar there would be a replica for each Doppler 
>> shift and you may choose to quantize the replica parameter into a 
>> discrete set of values and, thus, replicas.
>> In your case, there may be a replica for each width of the signal shape 
>> you described.
>>
>> Fred
>>
>>
>>
>>
> Thanks for the answers, I will have a look at that. Sounds easy enough.
>
> What I meant with zero-signal versus measurement signal is the response of 
> the instrument witha blank sample versus the response with either a
> standard sample giving relatively high signal for calibration purposus or 
> a sample to be analysed with in general a rather low signal.
>
> Hope this helps in understanding my question.
> Janwillem

Janwillem,

Oh!  Well, if the "zero signal" is the instrument output with no signal 
present, then that would probably be of no use for any processing unless it 
has structure - i.e. if it has sinusoidal components that are measureable 
above the noise.

So, perhaps the "standard sample" version is a good replica....

Fred