Forums

notch iir implementation

Started by inna-s December 1, 2006
I have a similar problem described by Par in 2005.
(Only different frequencies and sample size).
 
I have signal of 10000 length that has undesired frequencies:
f=0.0003, 0.0006 (f \in [0 1]) I want to get rid off. 
I build notch IIR and apply it. 
To test that it works I first created 
a synthetic signal (linear sum of harmonics) and apply filter on this
data. 
1. I know that my IIR design is perfect.
2. I also know that to get zero linear phase I apply filter forw and back

sequentially. 

If my synthetic signal is long enough about 50000, I get very good
results
in the middle part of the signal from [5000 - 45000]. But I don't deal
well
at the bounds 5000 samples from both sides. 
It is clear that since my real signal is 10000, I will not succeed to 
apply my filter to real data. I should get rid off the boundary effects.

Please can you advise me how to resolve these boundary effects.
In the old post people advised to use superposition, but I did not 
catch an idea. The reference to filtfilt of matlab  was given there 
as well and it was claimed that some IEEE paper is mentioned in filtfilt
doc. I checked the Matlab documents, but today there is no any reference
to  
a paper. Does anyone know what is this paper? My idea was to make
padding;
repeat the same signal a few times, but I understood from old post that 
this is a wrong technique... 

I also know from simulation that these boundary effects are related to the
bandwidth of the notch filter. The more narrow filter, the more area is 
occupied by the boundary effects. I am curious if there is also some 
math. that relates desired frequency, bandwidth etc. and the boundary
effects zone. 

Also if this problem has a standard name please can you correct me, 
as I will try to make search using a right terminology. 

Thanks in advance,
Inna.






inna wrote:
...
> I have signal of 10000 length that has undesired frequencies: > f=0.0003, 0.0006 (f \in [0 1]) I want to get rid off. > I build notch IIR and apply it. > To test that it works I first created > a synthetic signal (linear sum of harmonics) and apply filter on this > data. > 1. I know that my IIR design is perfect. > 2. I also know that to get zero linear phase I apply filter forw and back > > sequentially. > > If my synthetic signal is long enough about 50000, I get very good > results > in the middle part of the signal from [5000 - 45000]. But I don't deal > well > at the bounds 5000 samples from both sides. > It is clear that since my real signal is 10000, I will not succeed to > apply my filter to real data. I should get rid off the boundary effects. > > Please can you advise me how to resolve these boundary effects. > In the old post people advised to use superposition, but I did not > catch an idea.
If the sine signal has constant amplitude, phase and frequency, you can try to estimate these parameters and then subtract a synthesized sine. Regards, Andor
> > >inna wrote: >... >> I have signal of 10000 length that has undesired frequencies: >> f=0.003, 0.006 (f \in [0 1]) I want to get rid off. >> I build notch IIR and apply it. >> To test that it works I first created >> a synthetic signal (linear sum of harmonics) and apply filter on this >> data. >> 1. I know that my IIR design is perfect. >> 2. I also know that to get zero linear phase I apply filter forw and
back
>> >> sequentially. >> >> If my synthetic signal is long enough about 50000, I get very good >> results >> in the middle part of the signal from [5000 - 45000]. But I don't deal >> well >> at the bounds 5000 samples from both sides. >> It is clear that since my real signal is 10000, I will not succeed to >> apply my filter to real data. I should get rid off the boundary
effects.
>> >> Please can you advise me how to resolve these boundary effects. >> In the old post people advised to use superposition, but I did not >> catch an idea. > >If the sine signal has constant amplitude, phase and frequency, you can >try to estimate these parameters and then subtract a synthesized sine. > >Regards, >Andor > >
Yes, you are right, I used this method as well; it is ok for additive models, but my real signal is more complicated. I am more interested now to understand the constraints and pitfalls of IIR filtering. This method works quite well, but needs currently a signal of length 50000. I used already filtfilt.m and don't find it improves something for me.
> > >inna wrote: >... >> I have signal of 10000 length that has undesired frequencies: >> f=0.003, 0.006 (f \in [0 1]) I want to get rid off. >> I build notch IIR and apply it. >> To test that it works I first created >> a synthetic signal (linear sum of harmonics) and apply filter on this >> data. >> 1. I know that my IIR design is perfect. >> 2. I also know that to get zero linear phase I apply filter forw and
back
>> >> sequentially. >> >> If my synthetic signal is long enough about 50000, I get very good >> results >> in the middle part of the signal from [5000 - 45000]. But I don't deal >> well >> at the bounds 5000 samples from both sides. >> It is clear that since my real signal is 10000, I will not succeed to >> apply my filter to real data. I should get rid off the boundary
effects.
>> >> Please can you advise me how to resolve these boundary effects. >> In the old post people advised to use superposition, but I did not >> catch an idea. > >If the sine signal has constant amplitude, phase and frequency, you can >try to estimate these parameters and then subtract a synthesized sine. > >Regards, >Andor > >
Yes, you are right, I used this method as well; it is ok for additive models, but my real signal is more complicated. I am more interested now to understand the constraints and pitfalls of IIR filtering. This method works quite well, but needs currently a signal of length 50000. I used already filtfilt.m and don't find it improves something for me.
inna-s skrev:
> I have a similar problem described by Par in 2005. > (Only different frequencies and sample size). > > I have signal of 10000 length that has undesired frequencies: > f=0.0003, 0.0006 (f \in [0 1]) I want to get rid off. > I build notch IIR and apply it. > To test that it works I first created > a synthetic signal (linear sum of harmonics) and apply filter on this > data. > 1. I know that my IIR design is perfect. > 2. I also know that to get zero linear phase I apply filter forw and back > > sequentially. > > If my synthetic signal is long enough about 50000, I get very good > results > in the middle part of the signal from [5000 - 45000]. But I don't deal > well > at the bounds 5000 samples from both sides. > It is clear that since my real signal is 10000, I will not succeed to > apply my filter to real data. I should get rid off the boundary effects.
These are very narrow bands, and it is not surprising that the end effects last for a few thousand samples. Why not design a HP filter with stop band 0 - 0.001 and passband 0.002 - 0.5 ? Rune
If you have a waveform that is more complicated than a sine wave, IIR filters 
are not the right choice.  If you have a signal that is only 10,000 points in 
length, an IIR filter is not the right choice. I think you have performed that 
experiment and shown that it isn't the proper choice. 

IMHO, the proper choice for problems with a short signal length, say less than 
16.000.000 points, is the FFT.  Perform a forward FFT, excise the offending 
signal elements, then perform an Inverse FFT.  It will probably be necessary 
to taper the ends of the spectrum next to the excised components using a ramp 
or half cosine taper with a few points length, maybe nine, before the IFFT to 
prevent undesirable signal components.

In article <1164995848.064811.177320@j72g2000cwa.googlegroups.com>, "Rune 
Allnor" <allnor@tele.ntnu.no> wrote:
> >inna-s skrev: >> I have a similar problem described by Par in 2005. >> (Only different frequencies and sample size). >> >> I have signal of 10000 length that has undesired frequencies: >> f=0.0003, 0.0006 (f \in [0 1]) I want to get rid off. >> I build notch IIR and apply it. >> To test that it works I first created >> a synthetic signal (linear sum of harmonics) and apply filter on this >> data. >> 1. I know that my IIR design is perfect. >> 2. I also know that to get zero linear phase I apply filter forw and back >> >> sequentially. >> >> If my synthetic signal is long enough about 50000, I get very good >> results >> in the middle part of the signal from [5000 - 45000]. But I don't deal >> well >> at the bounds 5000 samples from both sides. >> It is clear that since my real signal is 10000, I will not succeed to >> apply my filter to real data. I should get rid off the boundary effects. > >These are very narrow bands, and it is not surprising that >the end effects last for a few thousand samples. > >Why not design a HP filter with stop band 0 - 0.001 and >passband 0.002 - 0.5 ? > >Rune >
John Herman skrev:
> If you have a waveform that is more complicated than a sine wave, IIR filters > are not the right choice.
This is a bold claim...
> If you have a signal that is only 10,000 points in > length, an IIR filter is not the right choice.
..and so is this. Could you elaborate, please?
> I think you have performed that > experiment and shown that it isn't the proper choice.
Do you mean me personally, or DSP'ers in general? If I have, it is without my own knowledge. Could you point to any excercise that support such a claim? Preferably one you know I have run through, though.
> IMHO, the proper choice for problems with a short signal length, say less than > 16.000.000 points, is the FFT.
OK. And what do you consider to be a long signal?
> Perform a forward FFT, excise the offending > signal elements, then perform an Inverse FFT.
How many "offending" elements do you expect to find in a spectrum that contain tens of millions of points? How do you intend to automize such a process?
> It will probably be necessary > to taper the ends of the spectrum next to the excised components using a ramp > or half cosine taper with a few points length, maybe nine, before the IFFT to > prevent undesirable signal components.
Nine points out of 16 millions; sure, that'll do the trick... Rune