notch iir implementation

I have a similar problem described by Par in 2005.
(Only different frequencies and sample size).
 
I have signal of 10000 length that has undesired frequencies:
f=0.0003, 0.0006 (f \in [0 1]) I want to get rid off. 
I build notch IIR and apply it. 
To test that it works I first created 
a synthetic signal (linear sum of harmonics) and apply filter on this
data. 
1. I know that my IIR design is perfect.
2. I also know that to get zero linear phase I apply filter forw and back

sequentially. 

If my synthetic signal is long enough about 50000, I get very good
results
in the middle part of the signal from [5000 - 45000]. But I don't deal
well
at the bounds 5000 samples from both sides. 
It is clear that since my real signal is 10000, I will not succeed to 
apply my filter to real data. I should get rid off the boundary effects.

Please can you advise me how to resolve these boundary effects.
In the old post people advised to use superposition, but I did not 
catch an idea. The reference to filtfilt of matlab  was given there 
as well and it was claimed that some IEEE paper is mentioned in filtfilt
doc. I checked the Matlab documents, but today there is no any reference
to  
a paper. Does anyone know what is this paper? My idea was to make
padding;
repeat the same signal a few times, but I understood from old post that 
this is a wrong technique... 

I also know from simulation that these boundary effects are related to the
bandwidth of the notch filter. The more narrow filter, the more area is 
occupied by the boundary effects. I am curious if there is also some 
math. that relates desired frequency, bandwidth etc. and the boundary
effects zone. 

Also if this problem has a standard name please can you correct me, 
as I will try to make search using a right terminology. 

Thanks in advance,
Inna.

inna wrote:
...
> I have signal of 10000 length that has undesired frequencies:
> f=0.0003, 0.0006 (f \in [0 1]) I want to get rid off.
> I build notch IIR and apply it.
> To test that it works I first created
> a synthetic signal (linear sum of harmonics) and apply filter on this
> data.
> 1. I know that my IIR design is perfect.
> 2. I also know that to get zero linear phase I apply filter forw and back
>
> sequentially.
>
> If my synthetic signal is long enough about 50000, I get very good
> results
> in the middle part of the signal from [5000 - 45000]. But I don't deal
> well
> at the bounds 5000 samples from both sides.
> It is clear that since my real signal is 10000, I will not succeed to
> apply my filter to real data. I should get rid off the boundary effects.
>
> Please can you advise me how to resolve these boundary effects.
> In the old post people advised to use superposition, but I did not
> catch an idea.

If the sine signal has constant amplitude, phase and frequency, you can
try to estimate these parameters and then subtract a synthesized sine.

Regards,
Andor

>
>
>inna wrote:
>...
>> I have signal of 10000 length that has undesired frequencies:
>> f=0.003, 0.006 (f \in [0 1]) I want to get rid off.
>> I build notch IIR and apply it.
>> To test that it works I first created
>> a synthetic signal (linear sum of harmonics) and apply filter on this
>> data.
>> 1. I know that my IIR design is perfect.
>> 2. I also know that to get zero linear phase I apply filter forw and
back
>>
>> sequentially.
>>
>> If my synthetic signal is long enough about 50000, I get very good
>> results
>> in the middle part of the signal from [5000 - 45000]. But I don't deal
>> well
>> at the bounds 5000 samples from both sides.
>> It is clear that since my real signal is 10000, I will not succeed to
>> apply my filter to real data. I should get rid off the boundary
effects.
>>
>> Please can you advise me how to resolve these boundary effects.
>> In the old post people advised to use superposition, but I did not
>> catch an idea.
>
>If the sine signal has constant amplitude, phase and frequency, you can
>try to estimate these parameters and then subtract a synthesized sine.
>
>Regards,
>Andor
>
>
Yes, you are right, I used this method as well; it is ok for additive
models, but my real signal is more complicated. 
I am more interested now to understand the constraints and pitfalls 
of IIR filtering. This method works quite well, but needs currently a
signal of length 50000. I used already filtfilt.m and don't find it
improves something for me. 
   

>
>
>inna wrote:
>...
>> I have signal of 10000 length that has undesired frequencies:
>> f=0.003, 0.006 (f \in [0 1]) I want to get rid off.
>> I build notch IIR and apply it.
>> To test that it works I first created
>> a synthetic signal (linear sum of harmonics) and apply filter on this
>> data.
>> 1. I know that my IIR design is perfect.
>> 2. I also know that to get zero linear phase I apply filter forw and
back
>>
>> sequentially.
>>
>> If my synthetic signal is long enough about 50000, I get very good
>> results
>> in the middle part of the signal from [5000 - 45000]. But I don't deal
>> well
>> at the bounds 5000 samples from both sides.
>> It is clear that since my real signal is 10000, I will not succeed to
>> apply my filter to real data. I should get rid off the boundary
effects.
>>
>> Please can you advise me how to resolve these boundary effects.
>> In the old post people advised to use superposition, but I did not
>> catch an idea.
>
>If the sine signal has constant amplitude, phase and frequency, you can
>try to estimate these parameters and then subtract a synthesized sine.
>
>Regards,
>Andor
>
>
Yes, you are right, I used this method as well; it is ok for additive
models, but my real signal is more complicated. 
I am more interested now to understand the constraints and pitfalls 
of IIR filtering. This method works quite well, but needs currently a
signal of length 50000. I used already filtfilt.m and don't find it
improves something for me. 
   

inna-s skrev:
> I have a similar problem described by Par in 2005.
> (Only different frequencies and sample size).
>
> I have signal of 10000 length that has undesired frequencies:
> f=0.0003, 0.0006 (f \in [0 1]) I want to get rid off.
> I build notch IIR and apply it.
> To test that it works I first created
> a synthetic signal (linear sum of harmonics) and apply filter on this
> data.
> 1. I know that my IIR design is perfect.
> 2. I also know that to get zero linear phase I apply filter forw and back
>
> sequentially.
>
> If my synthetic signal is long enough about 50000, I get very good
> results
> in the middle part of the signal from [5000 - 45000]. But I don't deal
> well
> at the bounds 5000 samples from both sides.
> It is clear that since my real signal is 10000, I will not succeed to
> apply my filter to real data. I should get rid off the boundary effects.

These are very narrow bands, and it is not surprising that
the end effects last for a few thousand samples.

Why not design a HP filter with stop band 0 - 0.001 and
passband 0.002 - 0.5 ?

Rune

If you have a waveform that is more complicated than a sine wave, IIR filters 
are not the right choice.  If you have a signal that is only 10,000 points in 
length, an IIR filter is not the right choice. I think you have performed that 
experiment and shown that it isn't the proper choice. 

IMHO, the proper choice for problems with a short signal length, say less than 
16.000.000 points, is the FFT.  Perform a forward FFT, excise the offending 
signal elements, then perform an Inverse FFT.  It will probably be necessary 
to taper the ends of the spectrum next to the excised components using a ramp 
or half cosine taper with a few points length, maybe nine, before the IFFT to 
prevent undesirable signal components.

In article <1164995848.064811.177320@j72g2000cwa.googlegroups.com>, "Rune 
Allnor" <allnor@tele.ntnu.no> wrote:
>
>inna-s skrev:
>> I have a similar problem described by Par in 2005.
>> (Only different frequencies and sample size).
>>
>> I have signal of 10000 length that has undesired frequencies:
>> f=0.0003, 0.0006 (f \in [0 1]) I want to get rid off.
>> I build notch IIR and apply it.
>> To test that it works I first created
>> a synthetic signal (linear sum of harmonics) and apply filter on this
>> data.
>> 1. I know that my IIR design is perfect.
>> 2. I also know that to get zero linear phase I apply filter forw and back
>>
>> sequentially.
>>
>> If my synthetic signal is long enough about 50000, I get very good
>> results
>> in the middle part of the signal from [5000 - 45000]. But I don't deal
>> well
>> at the bounds 5000 samples from both sides.
>> It is clear that since my real signal is 10000, I will not succeed to
>> apply my filter to real data. I should get rid off the boundary effects.
>
>These are very narrow bands, and it is not surprising that
>the end effects last for a few thousand samples.
>
>Why not design a HP filter with stop band 0 - 0.001 and
>passband 0.002 - 0.5 ?
>
>Rune
>

John Herman skrev:
> If you have a waveform that is more complicated than a sine wave, IIR filters
> are not the right choice.

This is a bold claim...

> If you have a signal that is only 10,000 points in
> length, an IIR filter is not the right choice.

..and so is this. Could you elaborate, please?

> I think you have performed that
> experiment and shown that it isn't the proper choice.

Do you mean me personally, or DSP'ers in general? If I have,
it is without my own knowledge. Could you point to any excercise
that support such a claim? Preferably one you know I have run
through, though.

> IMHO, the proper choice for problems with a short signal length, say less than
> 16.000.000 points, is the FFT.

OK. And what do you consider to be a long signal?

> Perform a forward FFT, excise the offending
> signal elements, then perform an Inverse FFT.

How many "offending" elements do you expect to find in a
spectrum that contain tens of millions of points? How do you intend
to automize such a process?

> It will probably be necessary
> to taper the ends of the spectrum next to the excised components using a ramp
> or half cosine taper with a few points length, maybe nine, before the IFFT to
> prevent undesirable signal components.

Nine points out of 16 millions; sure, that'll do the trick...

Rune