Using DSP to separate noise from signal

Hi -

I'm working with some basic over air coms at around 1KHz frequency. For
the time, let's say there's no  modulation, just simple tone codes. (beat
the drums!) 
Let's (also) say received signal is exposed to broadband noise bursts that
are spread across from (pick 2 numbers) 600Hz to 1300Hz. These burst can
be higher than the signal being received.

My ADC is sampling at 15Khz. Ahead of that, a 5th order LPF with an Fc of
2500Hz. I want to use DSP processing to see what energy levels are at the
FFT bins. 

Considering the nature and amplitude of the noise, what would be the best
method to filter the noise? Additionally, is it at all possible to somehow
differentiate this broadband noise at the same frequency from the signal
of interest?

What would be the ideal FIR to use, and what about the specific FFT
method? I read an article where it recommended an SDFT (short discrete
Fourier transform) and employing overlap and add, etc.

Again - objectives are simple, but under the conditions of the noise, the
solution may not be simple.

Can you please suggest some methodology? I should also mention I have
access to the CMSIS DSP library which has a pretty diverse range of tools
in it, so ideally I would like to use that.

Thanks for any suggestions!





---------------------------------------
Posted through http://www.DSPRelated.com

On Mon, 16 Nov 2015 16:36:17 -0600, grog wrote:

> Hi -
> 
> I'm working with some basic over air coms at around 1KHz frequency. For
> the time, let's say there's no  modulation, just simple tone codes.
> (beat the drums!)
> Let's (also) say received signal is exposed to broadband noise bursts
> that are spread across from (pick 2 numbers) 600Hz to 1300Hz. These
> burst can be higher than the signal being received.
> 
> My ADC is sampling at 15Khz. Ahead of that, a 5th order LPF with an Fc
> of 2500Hz. I want to use DSP processing to see what energy levels are at
> the FFT bins.
> 
> Considering the nature and amplitude of the noise, what would be the
> best method to filter the noise? Additionally, is it at all possible to
> somehow differentiate this broadband noise at the same frequency from
> the signal of interest?
> 
> What would be the ideal FIR to use, and what about the specific FFT
> method? I read an article where it recommended an SDFT (short discrete
> Fourier transform) and employing overlap and add, etc.
> 
> Again - objectives are simple, but under the conditions of the noise,
> the solution may not be simple.
> 
> Can you please suggest some methodology? I should also mention I have
> access to the CMSIS DSP library which has a pretty diverse range of
> tools in it, so ideally I would like to use that.
> 
> Thanks for any suggestions!

So, the reason that "DSP" doesn't have an 'M' is because there is no 
magic.

If you're going to separate noise from signal, there has to be something 
that distinguishes the noise from the signal.  It has to be in a 
different frequency range, or at a different time, or something.

Unless these broadband noise bursts are very short duration, I think 
you're probably out of luck, because while you may be able to identify 
when noise is happening, I very much doubt you'll be able to filter out 
the noise.

If the noise bursts ARE of short duration, and they're reliably much 
louder than the speech, then you can just watch for a high-energy pulse 
and mute the radio for its duration.

-- 

Tim Wescott
Wescott Design Services
http://www.wescottdesign.com

Well if the tones carry the message and the tone freqs are known, you can build narrow band filters to select the tones.  Consider that when you measure a power level of noise, that power is spead across the measurement bandwidth.  As you narrow the filter, the power goes down.  If you can make and use a very narrow filter and the desired signal fits inside the filter, the noise power can be very low.
You actually should think of it in terms of Density.  The noise has a Density and the signal, if it is a tone, could have a much higher Density.
So you need to consider the exact characteristics of your desired signal .
There is nothing about this unique to DSP.
Mark

If you listen, can you hear the tones when the noise is present?

Bob

grog <108385@DSPRelated> wrote:

>I'm working with some basic over air coms at around 1KHz frequency. For
>the time, let's say there's no  modulation, just simple tone codes. 
>Let's (also) say received signal is exposed to broadband noise bursts that
>are spread across from (pick 2 numbers) 600Hz to 1300Hz. These burst can
>be higher than the signal being received.

>My ADC is sampling at 15Khz. Ahead of that, a 5th order LPF with an Fc of
>2500Hz. 

Okay so far

>I want to use DSP processing to see what energy levels are at the
>FFT bins. 

First question, why is it a given that you are using an FFT?

>Considering the nature and amplitude of the noise, what would be the best
>method to filter the noise? 

Why is it a given that you wish to filter the noise?

> Additionally, is it at all possible to somehow
> differentiate this broadband noise at the same frequency from the signal
> of interest?

Yes, if you can find some characteristic that is not shared between
the noise and the signal of interest -- for example, the signal of
interest is of steady magnitude over some interval, and the impulse
noise is not.

>What would be the ideal FIR to use, 

Why an FIR?

>and what about the specific FFT
>method? 

Why an FFT?

>[snip] and employing overlap and add, etc.

First, determine if your application even benefits from using
a discrete Fourier transform.  Then determine if an FFT implementation
of the DFT is the right way to go (it often, but not always is),
and after that, look for a particular FFT architecture.  But
only after you've verified performance, either on the bench or
in simlation, is it profitable to worry about specific FFT architecture
tradeoff.

That is, separate your problem into algorithm development
and implementation and try not to commingle the two.

As for implementation, at these low frequencies, you can likely do it all 
in software even on a slow computer, and possibly without even crafting the
bestest slickest FFT implementation.

Good luck.

Steve

On 16.11.15 23.36, grog wrote:
> I'm working with some basic over air coms at around 1KHz frequency. For
> the time, let's say there's no  modulation, just simple tone codes. (beat
> the drums!)

This is obviously amplitude modulation. So you no longer have one frequency.

> Let's (also) say received signal is exposed to broadband noise bursts that
> are spread across from (pick 2 numbers) 600Hz to 1300Hz. These burst can
> be higher than the signal being received.
>
> My ADC is sampling at 15Khz. Ahead of that, a 5th order LPF with an Fc of
> 2500Hz. I want to use DSP processing to see what energy levels are at the
> FFT bins.

???
Do you want a pretty picture or do you want noise filtering?

> Considering the nature and amplitude of the noise, what would be the best
> method to filter the noise? Additionally, is it at all possible to somehow
> differentiate this broadband noise at the same frequency from the signal
> of interest?

To differentiate between noise and signal you need to know some 
properties of either the noise or, more likely, your signal. Now you can 
apply this model to your samples and divide into signal and noise.

E.g. if your signal is small band, like a slow amplitude modulation, 
then you can strip all frequencies that do not fit in that window. If 
the window is sufficiently narrow your broadband noise will no longer 
have enough spectral density to put much energy into this window, 
although the total energy of the noise is far beyond your signal.

Furthermore the spectrum of amplitude modulation is symmetric to the 
carrier. If you enforce this symmetry, all frequency components that do 
not fit into this symmetry are also noise. This gives you another factor 
of two.
If you /exactly/ know your carrier frequency, then you usually also know 
the phase. At least after some synchronization. Again you can strip a 
factor two. (This is also the trick behind lock-in amplifiers.)

These are just examples. I used similar methods some time ago to measure 
the loudspeaker response (at reasonable levels) outdoor while trucks are 
passing the road. The latter was invisible in the result, because my 
filter reduced the noise by about 4 orders of magnitude.

> What would be the ideal FIR to use, and what about the specific FFT
> method? I read an article where it recommended an SDFT (short discrete
> Fourier transform) and employing overlap and add, etc.

First of all you need properties that your signal /must/ have. Once you 
have them, you could define a transformation that extract this properties.
If your properties are defined in the frequency domain, than FFT is 
likely helpful no more no less.

So what are the known properties of your signal?

> Again - objectives are simple, but under the conditions of the noise, the
> solution may not be simple.

Have you redundancy in your signal? Can you apply forward error correction?


Marcel

On Mon, 16 Nov 2015 16:36:17 -0600, "grog"
<108385@DSPRelated> wrote:

>Hi -
>
>I'm working with some basic over air coms at around 1KHz frequency. For
>the time, let's say there's no  modulation, just simple tone codes. (beat
>the drums!) 
>Let's (also) say received signal is exposed to broadband noise bursts that
>are spread across from (pick 2 numbers) 600Hz to 1300Hz. These burst can
>be higher than the signal being received.
>
>My ADC is sampling at 15Khz. Ahead of that, a 5th order LPF with an Fc of
>2500Hz. I want to use DSP processing to see what energy levels are at the
>FFT bins. 
>
>Considering the nature and amplitude of the noise, what would be the best
>method to filter the noise? Additionally, is it at all possible to somehow
>differentiate this broadband noise at the same frequency from the signal
>of interest?
>
>What would be the ideal FIR to use, and what about the specific FFT
>method? I read an article where it recommended an SDFT (short discrete
>Fourier transform) and employing overlap and add, etc.
>
>Again - objectives are simple, but under the conditions of the noise, the
>solution may not be simple.
>
>Can you please suggest some methodology? I should also mention I have
>access to the CMSIS DSP library which has a pretty diverse range of tools
>in it, so ideally I would like to use that.
>
>Thanks for any suggestions!

It's not clear what variables you can control, but if you
can insure that the tone frequency falls exactly on a
spectral line, that's an excellent 'filter' all by itself.

To reduce the in-band noise you can use waveform averaging
(or the equivalent coherent spectral averaging) if you have
a sync signal that is not subject to the channel noise.  For
every doubling of the number of frames averaged, you get 3
dB S/N improvement.  This is how it's possible to recover
EEG signals that are responses to auditory or visual stimuli
and are totally buried in neural background noise.  

The downside of averaging is that it takes time, and needs
that sync signal.  A plain old FFT does essentially the same
thing for each spectral line, averaging over the number of
tone cycles  in one processing frame.  You don't need a
separate sync signal, but for best results (without
windowing) the frame must hold an integer number of cycles.

Best regards,


Bob Masta
 
              DAQARTA  v8.00
   Data AcQuisition And Real-Time Analysis
              www.daqarta.com
Scope, Spectrum, Spectrogram, Sound Level Meter
 Frequency Counter, Pitch Track, Pitch-to-MIDI 
FREE 8-channel Signal Generator, DaqMusiq generator    
          Science with your sound card!

On 11/16/2015 7:42 PM, makolber@yahoo.com wrote:
> Well if the tones carry the message and the tone freqs are known, you can build narrow band filters to select the tones.  Consider that when you measure a power level of noise, that power is spead across the measurement bandwidth.  As you narrow the filter, the power goes down.  If you can make and use a very narrow filter and the desired signal fits inside the filter, the noise power can be very low.
> You actually should think of it in terms of Density.  The noise has a Density and the signal, if it is a tone, could have a much higher Density.
> So you need to consider the exact characteristics of your desired signal .
> There is nothing about this unique to DSP.
> Mark

The detection is not just to say the tones are present when they are, 
but to say they are *not* present when they are not.  Rather than just 
filter the tone from all other background, it would be useful to compare 
the signal at the tone frequency to the signal at other frequencies. 
Broadband noise will raise both the tone and non-tone frequencies.  When 
you see significantly more energy at the tone frequency then you can say 
the tone is detected.

-- 

Rick

>On 11/16/2015 7:42 PM, makolber@yahoo.com wrote:
>> Well if the tones carry the message and the tone freqs are known, you
can
>build narrow band filters to select the tones.  Consider that when you
>measure a power level of noise, that power is spead across the
measurement
>bandwidth.  As you narrow the filter, the power goes down.  If you can
make
>and use a very narrow filter and the desired signal fits inside the
filter,
>the noise power can be very low.
>> You actually should think of it in terms of Density.  The noise has a
>Density and the signal, if it is a tone, could have a much higher
Density.
>> So you need to consider the exact characteristics of your desired
signal
>.
>> There is nothing about this unique to DSP.
>> Mark
>
>The detection is not just to say the tones are present when they are, 
>but to say they are *not* present when they are not.  Rather than just 
>filter the tone from all other background, it would be useful to compare

>the signal at the tone frequency to the signal at other frequencies. 
>Broadband noise will raise both the tone and non-tone frequencies.  When

>you see significantly more energy at the tone frequency then you can say

>the tone is detected.
>
>-- 
>
>Rick


Hi,

Thanks EVERYONE for responding to my questions. Lots of useful information
to "filter" through (no DSP pun intended) 
I realize there's no magic, and more than one way to reach the
objectives.

Regards...





---------------------------------------
Posted through http://www.DSPRelated.com

Am 16.11.2015 um 23:36 schrieb grog:
> Hi -
>
> I'm working with some basic over air coms at around 1KHz frequency. For
> the time, let's say there's no  modulation, just simple tone codes. (beat
> the drums!)
> Let's (also) say received signal is exposed to broadband noise bursts that
> are spread across from (pick 2 numbers) 600Hz to 1300Hz. These burst can
> be higher than the signal being received.
>
> My ADC is sampling at 15Khz. Ahead of that, a 5th order LPF with an Fc of
> 2500Hz. I want to use DSP processing to see what energy levels are at the
> FFT bins.
>
> Considering the nature and amplitude of the noise, what would be the best
> method to filter the noise? Additionally, is it at all possible to somehow
> differentiate this broadband noise at the same frequency from the signal
> of interest?
>
> What would be the ideal FIR to use, and what about the specific FFT
> method? I read an article where it recommended an SDFT (short discrete
> Fourier transform) and employing overlap and add, etc.
>
> Again - objectives are simple, but under the conditions of the noise, the
> solution may not be simple.
>
> Can you please suggest some methodology? I should also mention I have
> access to the CMSIS DSP library which has a pretty diverse range of tools
> in it, so ideally I would like to use that.
>
> Thanks for any suggestions!
>
>
>
>
>
> ---------------------------------------
> Posted through http://www.DSPRelated.com
>

Well you could of course do some background estimation of your noise 
floor and define a narrowband threshold relative to the noise floor for 
a detection. But I do not know if that gets you anywhere. If in the 
narrowband sense the noise is as high than the signal than you are lost.

But you mention short bursts right. So maybe the solution is not 
filtering but using forward error correction and burst error mitigation 
methods like interleaving to increase the quality of the transmission 
channel. This means of course more protocol overhead.