Reply by acat November 30, 20102010-11-30
Okay, thanks again for all the advices, I will try them now!

Best

>On 11/29/2010 02:07 AM, Rune Allnor wrote: >> On Nov 29, 9:50 am, "acat"<newthinker@n_o_s_p_a_m.126.com> wrote: >> >>> (3) Is there any reference on this topic? >> >> The literature on DSP. >> >>> (4) I intended to put all the index of annoying frequencies to 0, is
this
>>> way a little unnatural? >> >> No and yes: The *intention* is not unnatural; that's what >> every newbie and neophyte want to do. >> >> The problem is that there is a theorem which is very central >> to DSP and that I don't remember the name of, off the top of >> my head, that states that one can not set a continuous >> frequency band to zero. That will cause all sorts of havoc, >> e.g. that the system will be impossible to implement. > >In this case, practically, what would happen is that you'd bring the >average of the 50Hz signal to zero, but in many places that would leave >a lot of "50 Hz-ish" signal there to trouble you. > >Hence my suggestion to zero out some bins, but to taper the filtering >effect. > >-- > >Tim Wescott >Wescott Design Services >http://www.wescottdesign.com > >Do you need to implement control loops in software? >"Applied Control Theory for Embedded Systems" was written for you. >See details at http://www.wescottdesign.com/actfes/actfes.html >
Reply by Vladimir Vassilevsky November 29, 20102010-11-29

Rune Allnor wrote:

> The problem is that there is a theorem which is very central > to DSP and that I don't remember the name of, off the top of > my head, that states that one can not set a continuous > frequency band to zero. That will cause all sorts of havoc, > e.g. that the system will be impossible to implement.
Paley-Wiener criterion of causality. Vladimir Vassilevsky DSP and Mixed Signal Design Consultant http://www.abvolt.com
Reply by John November 29, 20102010-11-29
On Nov 28, 3:06&#4294967295;pm, "acat" <newthinker@n_o_s_p_a_m.126.com> wrote:
> Hi, I need some help on removing the line noise (50Hz) from my data. I know > the method conventionally adopted is to fit a sinusoidal function and > subtract from the original data --- the so called notch filter, I have two > more questions with this: > > 1. In my data, not only 50Hz, but the frequencies nearest to it also have > abnormal values, this may be caused by the leakage of the power in 50 Hz (I > used the simple periodogram), but after I subtract the 50 Hz, it's > neighbors &#4294967295; are still there. So I am thinking whether the 50 Hz is stable > or not (I took only 4 seconds of data). And do have to remove the > neighbors? > > 2. The notch filter seems not very good I think, is there any way to remove > this line spectrum while preserving the spectral continuum? > > Thank you for your attention, any suggestions will be welcome!
Have you tried running your hardware on a battery?
Reply by Tim Wescott November 29, 20102010-11-29
On 11/28/2010 12:06 PM, acat wrote:
> Hi, I need some help on removing the line noise (50Hz) from my data. I know > the method conventionally adopted is to fit a sinusoidal function and > subtract from the original data --- the so called notch filter, I have two > more questions with this: > > 1. In my data, not only 50Hz, but the frequencies nearest to it also have > abnormal values, this may be caused by the leakage of the power in 50 Hz (I > used the simple periodogram), but after I subtract the 50 Hz, it's > neighbors are still there. So I am thinking whether the 50 Hz is stable > or not (I took only 4 seconds of data). And do have to remove the > neighbors? > > 2. The notch filter seems not very good I think, is there any way to remove > this line spectrum while preserving the spectral continuum?
I just realized that no one mentioned the obligatory caveat: DSP isn't magic. Often the best way to remove artifacts in your collected data is to get grease under your fingernails and change the way you're collecting data. If there's a way you can get what you need without the 50Hz hum -- try it, it may be cheaper and better than trying to remove the noise after the fact. Consider also that power line hum is usually rich in harmonics, so you'll have 50Hz, 100Hz, 150Hz, etc. -- often up into the kHz. -- Tim Wescott Wescott Design Services http://www.wescottdesign.com Do you need to implement control loops in software? "Applied Control Theory for Embedded Systems" was written for you. See details at http://www.wescottdesign.com/actfes/actfes.html
Reply by Tim Wescott November 29, 20102010-11-29
On 11/29/2010 02:07 AM, Rune Allnor wrote:
> On Nov 29, 9:50 am, "acat"<newthinker@n_o_s_p_a_m.126.com> wrote: > >> (3) Is there any reference on this topic? > > The literature on DSP. > >> (4) I intended to put all the index of annoying frequencies to 0, is this >> way a little unnatural? > > No and yes: The *intention* is not unnatural; that's what > every newbie and neophyte want to do. > > The problem is that there is a theorem which is very central > to DSP and that I don't remember the name of, off the top of > my head, that states that one can not set a continuous > frequency band to zero. That will cause all sorts of havoc, > e.g. that the system will be impossible to implement.
In this case, practically, what would happen is that you'd bring the average of the 50Hz signal to zero, but in many places that would leave a lot of "50 Hz-ish" signal there to trouble you. Hence my suggestion to zero out some bins, but to taper the filtering effect. -- Tim Wescott Wescott Design Services http://www.wescottdesign.com Do you need to implement control loops in software? "Applied Control Theory for Embedded Systems" was written for you. See details at http://www.wescottdesign.com/actfes/actfes.html
Reply by Clay November 29, 20102010-11-29
On Nov 28, 3:06&#4294967295;pm, "acat" <newthinker@n_o_s_p_a_m.126.com> wrote:
> Hi, I need some help on removing the line noise (50Hz) from my data. I know > the method conventionally adopted is to fit a sinusoidal function and > subtract from the original data --- the so called notch filter, I have two > more questions with this: > > 1. In my data, not only 50Hz, but the frequencies nearest to it also have > abnormal values, this may be caused by the leakage of the power in 50 Hz (I > used the simple periodogram), but after I subtract the 50 Hz, it's > neighbors &#4294967295; are still there. So I am thinking whether the 50 Hz is stable > or not (I took only 4 seconds of data). And do have to remove the > neighbors? > > 2. The notch filter seems not very good I think, is there any way to remove > this line spectrum while preserving the spectral continuum? > > Thank you for your attention, any suggestions will be welcome!
Besides, what all of the others have said, you also may have harmonics of the 50 Hz, for example 100, 150, 200, 250, etc. Look at the spectrum and see the nature of your interference. You may need several notch filters or even a comb filter. It just depends on your interference. Clay
Reply by Rune Allnor November 29, 20102010-11-29
On Nov 29, 9:50&#4294967295;am, "acat" <newthinker@n_o_s_p_a_m.126.com> wrote:

> (3) Is there any reference on this topic?
The literature on DSP.
> (4) I intended to put all the index of annoying frequencies to 0, is this > way a little unnatural?
No and yes: The *intention* is not unnatural; that's what every newbie and neophyte want to do. The problem is that there is a theorem which is very central to DSP and that I don't remember the name of, off the top of my head, that states that one can not set a continuous frequency band to zero. That will cause all sorts of havoc, e.g. that the system will be impossible to implement. Rune
Reply by Chris Bore November 29, 20102010-11-29
On Nov 28, 8:06&#4294967295;pm, "acat" <newthinker@n_o_s_p_a_m.126.com> wrote:
> Hi, I need some help on removing the line noise (50Hz) from my data. I know > the method conventionally adopted is to fit a sinusoidal function and > subtract from the original data --- the so called notch filter, I have two > more questions with this: > > 1. In my data, not only 50Hz, but the frequencies nearest to it also have > abnormal values, this may be caused by the leakage of the power in 50 Hz (I > used the simple periodogram), but after I subtract the 50 Hz, it's > neighbors &#4294967295; are still there. So I am thinking whether the 50 Hz is stable > or not (I took only 4 seconds of data). And do have to remove the > neighbors? > > 2. The notch filter seems not very good I think, is there any way to remove > this line spectrum while preserving the spectral continuum? > > Thank you for your attention, any suggestions will be welcome!
You might consider an 'adaptive' filter. This can be done most easily if you have a reference of the 'noise' (in this case your 50 Hz without the signal) that you can use as one input. The filter then adapts to eliminate anything in the signal that is most similar to the noise. Chris ============================== Chris Bore BORES Signal Processing www.bores.com
Reply by acat November 29, 20102010-11-29
Thank you all for the kind replies. 
(1) I realize the 'notch filter' is indeed not appropriate for what I was
doing to the data. I used multitaper method (Thomson) to estimate the 50 Hz
noise, the sampling rate was initially 2000 Hz, but I downsampled it to 250
Hz. The estimated amplitude and phase were used to construct the estimate
and subtracted from the original data. 

(2) As Tim Wescott put it, I think the 50 Hz should be stable even the
amplitude may drift in my recording. Another idea that came to me is that
the clock of my DAQ computer may be exactly the same as the clock of the
power plant. So the spectral line may not be on the frequency sampling
grid. That will cause some leakage as I just removed the 50 Hz. So I will
try remove 50 Hz prior to downsampling. 

(3) Is there any reference on this topic?

(4) I intended to put all the index of annoying frequencies to 0, is this
way a little unnatural?



>On Nov 28, 9:06=A0pm, "acat" <newthinker@n_o_s_p_a_m.126.com> wrote: >> Hi, I need some help on removing the line noise (50Hz) from my data. I
kn=
>ow >> the method conventionally adopted is to fit a sinusoidal function and >> subtract from the original data > >No, it isn't. > >> --- the so called notch filter, I have two >> more questions with this: > >The notch filter doesn't subtract anything from the data. >It attenuates the annoying part of the spectrum. > >> 1. In my data, not only 50Hz, but the frequencies nearest to it also
have
>> abnormal values, this may be caused by the leakage of the power in 50 Hz
=
>(I >> used the simple periodogram), but after I subtract the 50 Hz, it's >> neighbors =A0 are still there. So I am thinking whether the 50 Hz is
stab=
>le >> or not (I took only 4 seconds of data). And do have to remove the >> neighbors? > >*Subtracting* a complex number from another complex number >is the same operation as adding the two complex numbers with >an 180 degrees phase difference. > >For this to work, you need to know two parameters of >the spectrum: > >1) The magnitude of the annoying component(s) >2) The phase of the annoying component(s) > >Since you know neither (no, you don't), you actually >add two random complex numbers. > >> 2. The notch filter seems not very good I think, is there any way to
remo=
>ve >> this line spectrum while preserving the spectral continuum? > >No. > >You can *attenuate* the annoying spectrum components by using >a regular filter. Either use a notch filter, which in the ideal >world multiplies the magnitude by 0, in which case the 'spectral >continuum' is not preserved, or use a narrow-band band-stop >filter, in which case the annoying band is not 'removed'. > >Rune >
Reply by Rune Allnor November 28, 20102010-11-28
On Nov 28, 9:06&#4294967295;pm, "acat" <newthinker@n_o_s_p_a_m.126.com> wrote:
> Hi, I need some help on removing the line noise (50Hz) from my data. I know > the method conventionally adopted is to fit a sinusoidal function and > subtract from the original data
No, it isn't.
> --- the so called notch filter, I have two > more questions with this:
The notch filter doesn't subtract anything from the data. It attenuates the annoying part of the spectrum.
> 1. In my data, not only 50Hz, but the frequencies nearest to it also have > abnormal values, this may be caused by the leakage of the power in 50 Hz (I > used the simple periodogram), but after I subtract the 50 Hz, it's > neighbors &#4294967295; are still there. So I am thinking whether the 50 Hz is stable > or not (I took only 4 seconds of data). And do have to remove the > neighbors?
*Subtracting* a complex number from another complex number is the same operation as adding the two complex numbers with an 180 degrees phase difference. For this to work, you need to know two parameters of the spectrum: 1) The magnitude of the annoying component(s) 2) The phase of the annoying component(s) Since you know neither (no, you don't), you actually add two random complex numbers.
> 2. The notch filter seems not very good I think, is there any way to remove > this line spectrum while preserving the spectral continuum?
No. You can *attenuate* the annoying spectrum components by using a regular filter. Either use a notch filter, which in the ideal world multiplies the magnitude by 0, in which case the 'spectral continuum' is not preserved, or use a narrow-band band-stop filter, in which case the annoying band is not 'removed'. Rune