I have a similar problem described by Par in 2005. (Only different frequencies and sample size). I have signal of 10000 length that has undesired frequencies: f=0.0003, 0.0006 (f \in [0 1]) I want to get rid off. I build notch IIR and apply it. To test that it works I first created a synthetic signal (linear sum of harmonics) and apply filter on this data. 1. I know that my IIR design is perfect. 2. I also know that to get zero linear phase I apply filter forw and back sequentially. If my synthetic signal is long enough about 50000, I get very good results in the middle part of the signal from [5000 - 45000]. But I don't deal well at the bounds 5000 samples from both sides. It is clear that since my real signal is 10000, I will not succeed to apply my filter to real data. I should get rid off the boundary effects. Please can you advise me how to resolve these boundary effects. In the old post people advised to use superposition, but I did not catch an idea. The reference to filtfilt of matlab was given there as well and it was claimed that some IEEE paper is mentioned in filtfilt doc. I checked the Matlab documents, but today there is no any reference to a paper. Does anyone know what is this paper? My idea was to make padding; repeat the same signal a few times, but I understood from old post that this is a wrong technique... I also know from simulation that these boundary effects are related to the bandwidth of the notch filter. The more narrow filter, the more area is occupied by the boundary effects. I am curious if there is also some math. that relates desired frequency, bandwidth etc. and the boundary effects zone. Also if this problem has a standard name please can you correct me, as I will try to make search using a right terminology. Thanks in advance, Inna.

# notch iir implementation

Started by ●December 1, 2006

Reply by ●December 1, 20062006-12-01

inna wrote: ...> I have signal of 10000 length that has undesired frequencies: > f=0.0003, 0.0006 (f \in [0 1]) I want to get rid off. > I build notch IIR and apply it. > To test that it works I first created > a synthetic signal (linear sum of harmonics) and apply filter on this > data. > 1. I know that my IIR design is perfect. > 2. I also know that to get zero linear phase I apply filter forw and back > > sequentially. > > If my synthetic signal is long enough about 50000, I get very good > results > in the middle part of the signal from [5000 - 45000]. But I don't deal > well > at the bounds 5000 samples from both sides. > It is clear that since my real signal is 10000, I will not succeed to > apply my filter to real data. I should get rid off the boundary effects. > > Please can you advise me how to resolve these boundary effects. > In the old post people advised to use superposition, but I did not > catch an idea.If the sine signal has constant amplitude, phase and frequency, you can try to estimate these parameters and then subtract a synthesized sine. Regards, Andor

Reply by ●December 1, 20062006-12-01

> > >inna wrote: >... >> I have signal of 10000 length that has undesired frequencies: >> f=0.003, 0.006 (f \in [0 1]) I want to get rid off. >> I build notch IIR and apply it. >> To test that it works I first created >> a synthetic signal (linear sum of harmonics) and apply filter on this >> data. >> 1. I know that my IIR design is perfect. >> 2. I also know that to get zero linear phase I apply filter forw andback>> >> sequentially. >> >> If my synthetic signal is long enough about 50000, I get very good >> results >> in the middle part of the signal from [5000 - 45000]. But I don't deal >> well >> at the bounds 5000 samples from both sides. >> It is clear that since my real signal is 10000, I will not succeed to >> apply my filter to real data. I should get rid off the boundaryeffects.>> >> Please can you advise me how to resolve these boundary effects. >> In the old post people advised to use superposition, but I did not >> catch an idea. > >If the sine signal has constant amplitude, phase and frequency, you can >try to estimate these parameters and then subtract a synthesized sine. > >Regards, >Andor > >Yes, you are right, I used this method as well; it is ok for additive models, but my real signal is more complicated. I am more interested now to understand the constraints and pitfalls of IIR filtering. This method works quite well, but needs currently a signal of length 50000. I used already filtfilt.m and don't find it improves something for me.

Reply by ●December 1, 20062006-12-01

> > >inna wrote: >... >> I have signal of 10000 length that has undesired frequencies: >> f=0.003, 0.006 (f \in [0 1]) I want to get rid off. >> I build notch IIR and apply it. >> To test that it works I first created >> a synthetic signal (linear sum of harmonics) and apply filter on this >> data. >> 1. I know that my IIR design is perfect. >> 2. I also know that to get zero linear phase I apply filter forw andback>> >> sequentially. >> >> If my synthetic signal is long enough about 50000, I get very good >> results >> in the middle part of the signal from [5000 - 45000]. But I don't deal >> well >> at the bounds 5000 samples from both sides. >> It is clear that since my real signal is 10000, I will not succeed to >> apply my filter to real data. I should get rid off the boundaryeffects.>> >> Please can you advise me how to resolve these boundary effects. >> In the old post people advised to use superposition, but I did not >> catch an idea. > >If the sine signal has constant amplitude, phase and frequency, you can >try to estimate these parameters and then subtract a synthesized sine. > >Regards, >Andor > >Yes, you are right, I used this method as well; it is ok for additive models, but my real signal is more complicated. I am more interested now to understand the constraints and pitfalls of IIR filtering. This method works quite well, but needs currently a signal of length 50000. I used already filtfilt.m and don't find it improves something for me.

Reply by ●December 1, 20062006-12-01

inna-s skrev:> I have a similar problem described by Par in 2005. > (Only different frequencies and sample size). > > I have signal of 10000 length that has undesired frequencies: > f=0.0003, 0.0006 (f \in [0 1]) I want to get rid off. > I build notch IIR and apply it. > To test that it works I first created > a synthetic signal (linear sum of harmonics) and apply filter on this > data. > 1. I know that my IIR design is perfect. > 2. I also know that to get zero linear phase I apply filter forw and back > > sequentially. > > If my synthetic signal is long enough about 50000, I get very good > results > in the middle part of the signal from [5000 - 45000]. But I don't deal > well > at the bounds 5000 samples from both sides. > It is clear that since my real signal is 10000, I will not succeed to > apply my filter to real data. I should get rid off the boundary effects.These are very narrow bands, and it is not surprising that the end effects last for a few thousand samples. Why not design a HP filter with stop band 0 - 0.001 and passband 0.002 - 0.5 ? Rune

Reply by ●December 8, 20062006-12-08

If you have a waveform that is more complicated than a sine wave, IIR filters are not the right choice. If you have a signal that is only 10,000 points in length, an IIR filter is not the right choice. I think you have performed that experiment and shown that it isn't the proper choice. IMHO, the proper choice for problems with a short signal length, say less than 16.000.000 points, is the FFT. Perform a forward FFT, excise the offending signal elements, then perform an Inverse FFT. It will probably be necessary to taper the ends of the spectrum next to the excised components using a ramp or half cosine taper with a few points length, maybe nine, before the IFFT to prevent undesirable signal components. In article <1164995848.064811.177320@j72g2000cwa.googlegroups.com>, "Rune Allnor" <allnor@tele.ntnu.no> wrote:> >inna-s skrev: >> I have a similar problem described by Par in 2005. >> (Only different frequencies and sample size). >> >> I have signal of 10000 length that has undesired frequencies: >> f=0.0003, 0.0006 (f \in [0 1]) I want to get rid off. >> I build notch IIR and apply it. >> To test that it works I first created >> a synthetic signal (linear sum of harmonics) and apply filter on this >> data. >> 1. I know that my IIR design is perfect. >> 2. I also know that to get zero linear phase I apply filter forw and back >> >> sequentially. >> >> If my synthetic signal is long enough about 50000, I get very good >> results >> in the middle part of the signal from [5000 - 45000]. But I don't deal >> well >> at the bounds 5000 samples from both sides. >> It is clear that since my real signal is 10000, I will not succeed to >> apply my filter to real data. I should get rid off the boundary effects. > >These are very narrow bands, and it is not surprising that >the end effects last for a few thousand samples. > >Why not design a HP filter with stop band 0 - 0.001 and >passband 0.002 - 0.5 ? > >Rune >

Reply by ●December 8, 20062006-12-08

John Herman skrev:> If you have a waveform that is more complicated than a sine wave, IIR filters > are not the right choice.This is a bold claim...> If you have a signal that is only 10,000 points in > length, an IIR filter is not the right choice...and so is this. Could you elaborate, please?> I think you have performed that > experiment and shown that it isn't the proper choice.Do you mean me personally, or DSP'ers in general? If I have, it is without my own knowledge. Could you point to any excercise that support such a claim? Preferably one you know I have run through, though.> IMHO, the proper choice for problems with a short signal length, say less than > 16.000.000 points, is the FFT.OK. And what do you consider to be a long signal?> Perform a forward FFT, excise the offending > signal elements, then perform an Inverse FFT.How many "offending" elements do you expect to find in a spectrum that contain tens of millions of points? How do you intend to automize such a process?> It will probably be necessary > to taper the ends of the spectrum next to the excised components using a ramp > or half cosine taper with a few points length, maybe nine, before the IFFT to > prevent undesirable signal components.Nine points out of 16 millions; sure, that'll do the trick... Rune