Line noise removal

Hi, I need some help on removing the line noise (50Hz) from my data. I know
the method conventionally adopted is to fit a sinusoidal function and
subtract from the original data --- the so called notch filter, I have two
more questions with this:

1. In my data, not only 50Hz, but the frequencies nearest to it also have
abnormal values, this may be caused by the leakage of the power in 50 Hz (I
used the simple periodogram), but after I subtract the 50 Hz, it's
neighbors   are still there. So I am thinking whether the 50 Hz is stable
or not (I took only 4 seconds of data). And do have to remove the
neighbors?

2. The notch filter seems not very good I think, is there any way to remove
this line spectrum while preserving the spectral continuum?


Thank you for your attention, any suggestions will be welcome!

On 11/28/2010 12:06 PM, acat wrote:
> Hi, I need some help on removing the line noise (50Hz) from my data. I know
> the method conventionally adopted is to fit a sinusoidal function and
> subtract from the original data --- the so called notch filter, I have two
> more questions with this:
>
> 1. In my data, not only 50Hz, but the frequencies nearest to it also have
> abnormal values, this may be caused by the leakage of the power in 50 Hz (I
> used the simple periodogram), but after I subtract the 50 Hz, it's
> neighbors   are still there. So I am thinking whether the 50 Hz is stable
> or not (I took only 4 seconds of data). And do have to remove the
> neighbors?
>
> 2. The notch filter seems not very good I think, is there any way to remove
> this line spectrum while preserving the spectral continuum?
>
>
> Thank you for your attention, any suggestions will be welcome!
>
>

As you might imagine: "It depends".....

Subtracting a sinusoid isn't the same thing as a notch filter for a 
bunch of reasons.
- the 50Hz may have sidebands due to frequency modulation because it 
isn't stable
or
- the 50Hz may have sidebands due to amplitude modulation because its 
amplitude isn't stable.

Depending on the data, you might use a notch filter of wide enough 
bandwidth to eliminate all these.  It depends on the frequency content 
of the desired signal.

If the 50Hz frequency is stable then you might be able to do something 
like this:

1) Build a 50Hz bandpass filter with variable gain and with a variable 
delay at its input or output.  This filter will have some total 
(adjustable) delay at 50Hz.

2) Build a delay element with approximately the same delay as the filter.

3) Feed the signal into both.

4) Subtract the bandpass filter output from the delay element output.

5) Adjust the delay in the bandpass filter path until the delays match 
and the 50Hz is max reduced.

6) Adjust the gain in the bandpass filter path until the 50Hz is canceled.

This is sorta equivalent to an adaptive line canceller except that 
you're setting the frequency response to be fixed and messing with only 
the gain and phase/delay.
Because you want to cancel out amplitude variations in the 50Hz 
sinusoid, the delays should be equal.

Now, others probably have better methods and this one depends on the 
character of the "noise" but it could work well enough.....

Fred

On 11/28/2010 12:06 PM, acat wrote:
> Hi, I need some help on removing the line noise (50Hz) from my data. I know
> the method conventionally adopted is to fit a sinusoidal function and
> subtract from the original data --- the so called notch filter, I have two
> more questions with this:
>
> 1. In my data, not only 50Hz, but the frequencies nearest to it also have
> abnormal values, this may be caused by the leakage of the power in 50 Hz (I
> used the simple periodogram), but after I subtract the 50 Hz, it's
> neighbors   are still there. So I am thinking whether the 50 Hz is stable
> or not (I took only 4 seconds of data). And do have to remove the
> neighbors?

What is the source of power line noise?  Power line frequencies are 
usually pretty darn stable in most areas of most 1st-world countries. 
Unless you're taking your data in a remote community or spot that is run 
off of a small generating plant you should be pretty close to dead-on.

How are you sampling the data?  Could your data collection have jitter? 
  This would appear as phase jitter in your power line signal.

How are you taking the periodogram?  If your four seconds of data don't 
capture an exact integer number of cycles of 50Hz, and if take a FFT, 
and if you don't window the data, then you'll see spectral leakage as an 
artifact.

> 2. The notch filter seems not very good I think, is there any way to remove
> this line spectrum while preserving the spectral continuum?

If you are capturing data, then processing it after capture, you can 
take it's discrete Fourier transform (i.e. FFT), zero out as many bins 
around 50Hz as you think is appropriate, then take the IFFT.  That'll 
stomp your 50Hz signal dead.  It also may well give you some ringing, 
particularly if you're not windowing things correctly (see my response, 
above).  Making a filter that sneaks up a bit on zeroing out just one 
bin may be a good idea.

HTH.

-- 

Tim Wescott
Wescott Design Services
http://www.wescottdesign.com

Do you need to implement control loops in software?
"Applied Control Theory for Embedded Systems" was written for you.
See details at http://www.wescottdesign.com/actfes/actfes.html

>Hi, I need some help on removing the line noise (50Hz) from my data. I
know
>the method conventionally adopted is to fit a sinusoidal function and
>subtract from the original data --- the so called notch filter, I have
two
>more questions with this:
>
>1. In my data, not only 50Hz, but the frequencies nearest to it also have
>abnormal values, this may be caused by the leakage of the power in 50 Hz
(I
>used the simple periodogram), but after I subtract the 50 Hz, it's
>neighbors   are still there. So I am thinking whether the 50 Hz is stable
>or not (I took only 4 seconds of data). And do have to remove the
>neighbors?
>
>2. The notch filter seems not very good I think, is there any way to
remove
>this line spectrum while preserving the spectral continuum?
>
>
>Thank you for your attention, any suggestions will be welcome!

As Fred said, a notch filter is something very different from subtracting
the 50Hz. If the 50Hz is mains interference, it should be a narrow spectral
line. If the coupling into your signal is stable, the amplitude of the 50Hz
interference should also be stable, too. It should be possible to phase
lock a 50Hz sine wave to the 50Hz in your signal, and then amplitude lock
it to the 50Hz in your signal, and then subtract it from your signal. That
kind of approach can work extremely well for a long period of signal, as
you can make the bandwidth of the loops very narrow, and get a really good
match for the interference. For short bursts of signal you might want to
try an interative approach to estimating the 50Hz component.

Steve

On Nov 28, 9:06&#4294967295;pm, "acat" <newthinker@n_o_s_p_a_m.126.com> wrote:
> Hi, I need some help on removing the line noise (50Hz) from my data. I know
> the method conventionally adopted is to fit a sinusoidal function and
> subtract from the original data

No, it isn't.

> --- the so called notch filter, I have two
> more questions with this:

The notch filter doesn't subtract anything from the data.
It attenuates the annoying part of the spectrum.

> 1. In my data, not only 50Hz, but the frequencies nearest to it also have
> abnormal values, this may be caused by the leakage of the power in 50 Hz (I
> used the simple periodogram), but after I subtract the 50 Hz, it's
> neighbors &#4294967295; are still there. So I am thinking whether the 50 Hz is stable
> or not (I took only 4 seconds of data). And do have to remove the
> neighbors?

*Subtracting* a complex number from another complex number
is the same operation as adding the two complex numbers with
an 180 degrees phase difference.

For this to work, you need to know two parameters of
the spectrum:

1) The magnitude of the annoying component(s)
2) The phase of the annoying component(s)

Since you know neither (no, you don't), you actually
add two random complex numbers.

> 2. The notch filter seems not very good I think, is there any way to remove
> this line spectrum while preserving the spectral continuum?

No.

You can *attenuate* the annoying spectrum components by using
a regular filter. Either use a notch filter, which in the ideal
world multiplies the magnitude by 0, in which case the 'spectral
continuum' is not preserved, or use a narrow-band band-stop
filter, in which case the annoying band is not 'removed'.

Rune

Thank you all for the kind replies. 
(1) I realize the 'notch filter' is indeed not appropriate for what I was
doing to the data. I used multitaper method (Thomson) to estimate the 50 Hz
noise, the sampling rate was initially 2000 Hz, but I downsampled it to 250
Hz. The estimated amplitude and phase were used to construct the estimate
and subtracted from the original data. 

(2) As Tim Wescott put it, I think the 50 Hz should be stable even the
amplitude may drift in my recording. Another idea that came to me is that
the clock of my DAQ computer may be exactly the same as the clock of the
power plant. So the spectral line may not be on the frequency sampling
grid. That will cause some leakage as I just removed the 50 Hz. So I will
try remove 50 Hz prior to downsampling. 

(3) Is there any reference on this topic?

(4) I intended to put all the index of annoying frequencies to 0, is this
way a little unnatural?



>On Nov 28, 9:06=A0pm, "acat" <newthinker@n_o_s_p_a_m.126.com> wrote:
>> Hi, I need some help on removing the line noise (50Hz) from my data. I
kn=
>ow
>> the method conventionally adopted is to fit a sinusoidal function and
>> subtract from the original data
>
>No, it isn't.
>
>> --- the so called notch filter, I have two
>> more questions with this:
>
>The notch filter doesn't subtract anything from the data.
>It attenuates the annoying part of the spectrum.
>
>> 1. In my data, not only 50Hz, but the frequencies nearest to it also
have
>> abnormal values, this may be caused by the leakage of the power in 50 Hz
=
>(I
>> used the simple periodogram), but after I subtract the 50 Hz, it's
>> neighbors =A0 are still there. So I am thinking whether the 50 Hz is
stab=
>le
>> or not (I took only 4 seconds of data). And do have to remove the
>> neighbors?
>
>*Subtracting* a complex number from another complex number
>is the same operation as adding the two complex numbers with
>an 180 degrees phase difference.
>
>For this to work, you need to know two parameters of
>the spectrum:
>
>1) The magnitude of the annoying component(s)
>2) The phase of the annoying component(s)
>
>Since you know neither (no, you don't), you actually
>add two random complex numbers.
>
>> 2. The notch filter seems not very good I think, is there any way to
remo=
>ve
>> this line spectrum while preserving the spectral continuum?
>
>No.
>
>You can *attenuate* the annoying spectrum components by using
>a regular filter. Either use a notch filter, which in the ideal
>world multiplies the magnitude by 0, in which case the 'spectral
>continuum' is not preserved, or use a narrow-band band-stop
>filter, in which case the annoying band is not 'removed'.
>
>Rune
>

On Nov 28, 8:06&#4294967295;pm, "acat" <newthinker@n_o_s_p_a_m.126.com> wrote:
> Hi, I need some help on removing the line noise (50Hz) from my data. I know
> the method conventionally adopted is to fit a sinusoidal function and
> subtract from the original data --- the so called notch filter, I have two
> more questions with this:
>
> 1. In my data, not only 50Hz, but the frequencies nearest to it also have
> abnormal values, this may be caused by the leakage of the power in 50 Hz (I
> used the simple periodogram), but after I subtract the 50 Hz, it's
> neighbors &#4294967295; are still there. So I am thinking whether the 50 Hz is stable
> or not (I took only 4 seconds of data). And do have to remove the
> neighbors?
>
> 2. The notch filter seems not very good I think, is there any way to remove
> this line spectrum while preserving the spectral continuum?
>
> Thank you for your attention, any suggestions will be welcome!

You might consider an 'adaptive' filter. This can be done most easily
if you have a reference of the 'noise' (in this case your 50 Hz
without the signal) that you can use as one input. The filter then
adapts to eliminate anything in the signal that is most similar to the
noise.

Chris
==============================
Chris Bore
BORES Signal Processing
www.bores.com

On Nov 29, 9:50&#4294967295;am, "acat" <newthinker@n_o_s_p_a_m.126.com> wrote:

> (3) Is there any reference on this topic?

The literature on DSP.

> (4) I intended to put all the index of annoying frequencies to 0, is this
> way a little unnatural?

No and yes: The *intention* is not unnatural; that's what
every newbie and neophyte want to do.

The problem is that there is a theorem which is very central
to DSP and that I don't remember the name of, off the top of
my head, that states that one can not set a continuous
frequency band to zero. That will cause all sorts of havoc,
e.g. that the system will be impossible to implement.

Rune

On Nov 28, 3:06&#4294967295;pm, "acat" <newthinker@n_o_s_p_a_m.126.com> wrote:
> Hi, I need some help on removing the line noise (50Hz) from my data. I know
> the method conventionally adopted is to fit a sinusoidal function and
> subtract from the original data --- the so called notch filter, I have two
> more questions with this:
>
> 1. In my data, not only 50Hz, but the frequencies nearest to it also have
> abnormal values, this may be caused by the leakage of the power in 50 Hz (I
> used the simple periodogram), but after I subtract the 50 Hz, it's
> neighbors &#4294967295; are still there. So I am thinking whether the 50 Hz is stable
> or not (I took only 4 seconds of data). And do have to remove the
> neighbors?
>
> 2. The notch filter seems not very good I think, is there any way to remove
> this line spectrum while preserving the spectral continuum?
>
> Thank you for your attention, any suggestions will be welcome!

Besides, what all of the others have said, you also may have harmonics
of the 50 Hz, for example 100, 150, 200, 250, etc. Look at the
spectrum and see the nature of your interference. You may need several
notch filters or even a comb filter. It just depends on your
interference.

Clay

On 11/29/2010 02:07 AM, Rune Allnor wrote:
> On Nov 29, 9:50 am, "acat"<newthinker@n_o_s_p_a_m.126.com>  wrote:
>
>> (3) Is there any reference on this topic?
>
> The literature on DSP.
>
>> (4) I intended to put all the index of annoying frequencies to 0, is this
>> way a little unnatural?
>
> No and yes: The *intention* is not unnatural; that's what
> every newbie and neophyte want to do.
>
> The problem is that there is a theorem which is very central
> to DSP and that I don't remember the name of, off the top of
> my head, that states that one can not set a continuous
> frequency band to zero. That will cause all sorts of havoc,
> e.g. that the system will be impossible to implement.

In this case, practically, what would happen is that you'd bring the 
average of the 50Hz signal to zero, but in many places that would leave 
a lot of "50 Hz-ish" signal there to trouble you.

Hence my suggestion to zero out some bins, but to taper the filtering 
effect.

-- 

Tim Wescott
Wescott Design Services
http://www.wescottdesign.com

Do you need to implement control loops in software?
"Applied Control Theory for Embedded Systems" was written for you.
See details at http://www.wescottdesign.com/actfes/actfes.html