IIR filter on a fixed set of data points

I have a fixed 3600 data points(samples) and I want to apply a notch 2nd
order Butterworth IIR Notch filter to remove the 1/4Fs frequency, the
results start to converge(or filter start to work) after about 250 or so
samples. The issue is the it is a continuous stream of data and I need to
remove the harmonics on the initial 250 samples. What can I do to filter
the whole data set?


thank you,
Andrew

ahgu <honestgold@n_o_s_p_a_m.gmail.com> wrote:

> I have a fixed 3600 data points(samples) and I want to apply a notch 2nd
> order Butterworth IIR Notch filter to remove the 1/4Fs frequency, the
> results start to converge(or filter start to work) after about 250 or so
> samples. The issue is the it is a continuous stream of data and I need to
> remove the harmonics on the initial 250 samples. What can I do to filter
> the whole data set?

Most obvious to me is that you can also filter it in the other
direction.  (That is, from the end toward the beginning.)

That is easy if you have the whole thing stored, not easy at
all if you have to do it in streaming mode.

-- glen

On 11/07/2010 05:50 PM, ahgu wrote:
> I have a fixed 3600 data points(samples) and I want to apply a notch 2nd
> order Butterworth IIR Notch filter to remove the 1/4Fs frequency, the
> results start to converge(or filter start to work) after about 250 or so
> samples. The issue is the it is a continuous stream of data and I need to
> remove the harmonics on the initial 250 samples. What can I do to filter
> the whole data set?

If you can handle the delay, wait until the whole data set is taken then 
either apply the filter once in each direction, use a FIR filter, or 
detect the amplitude and phase of the 1/4 Fs tone using synchronous 
techniques and subtract it out.

If you _can't_ handle the delay then a Kalman filter is indicated, or 
some other time-varying filter that'll trade off a fast "attack" for 
initial noise sensitivity* but give you lower noise later once it 
'knows' more about what the signal's really doing.

* You can't have instantaneous acquisition and zero noise sensitivity -- 
that just can't be done.

-- 

Tim Wescott
Wescott Design Services
http://www.wescottdesign.com

Do you need to implement control loops in software?
"Applied Control Theory for Embedded Systems" was written for you.
See details at http://www.wescottdesign.com/actfes/actfes.html

Tim Wescott <tim@seemywebsite.com> writes:

> On 11/07/2010 05:50 PM, ahgu wrote:
>> I have a fixed 3600 data points(samples) and I want to apply a notch 2nd
>> order Butterworth IIR Notch filter to remove the 1/4Fs frequency, the
>> results start to converge(or filter start to work) after about 250 or so
>> samples. The issue is the it is a continuous stream of data and I need to
>> remove the harmonics on the initial 250 samples. What can I do to filter
>> the whole data set?
>
> If you can handle the delay, wait until the whole data set is taken
> then either apply the filter once in each direction, use a FIR filter,
> or detect the amplitude and phase of the 1/4 Fs tone using synchronous
> techniques and subtract it out.

Is there a textbook treatment of this "two-way IIR" process? I've
seen this many times but not yet grokked it.

If you use an FIR, you'll still have the startup delay, no? The
subtraction technique seems to be the most optimum if, as you've
stated, he can accept the delay.
-- 
Randy Yates                      % "Remember the good old 1980's, when 
Digital Signal Labs              %  things were so uncomplicated?"
mailto://yates@ieee.org          % 'Ticket To The Moon' 
http://www.digitalsignallabs.com % *Time*, Electric Light Orchestra

On Mon, 08 Nov 2010 01:43:02 -0500, Randy Yates <yates@ieee.org>
wrote:

>Tim Wescott <tim@seemywebsite.com> writes:
>
>> On 11/07/2010 05:50 PM, ahgu wrote:
>>> I have a fixed 3600 data points(samples) and I want to apply a notch 2nd
>>> order Butterworth IIR Notch filter to remove the 1/4Fs frequency, the
>>> results start to converge(or filter start to work) after about 250 or so
>>> samples. The issue is the it is a continuous stream of data and I need to
>>> remove the harmonics on the initial 250 samples. What can I do to filter
>>> the whole data set?
>>
>> If you can handle the delay, wait until the whole data set is taken
>> then either apply the filter once in each direction, use a FIR filter,
>> or detect the amplitude and phase of the 1/4 Fs tone using synchronous
>> techniques and subtract it out.
>
>Is there a textbook treatment of this "two-way IIR" process? I've
>seen this many times but not yet grokked it.
>
>If you use an FIR, you'll still have the startup delay, no? The
>subtraction technique seems to be the most optimum if, as you've
>stated, he can accept the delay.

Hi Randy,
  Assuming I understand the discussion here, I think 
that "two-way IIR" process is the "Zero-Phase Filtering" 
topic in Section 13.12 of the 2nd Edition of my book.

See Ya',
[-Rick-]

Rick Lyons <R.Lyons@_BOGUS_ieee.org> writes:

> Hi Randy,
>   Assuming I understand the discussion here, I think 
> that "two-way IIR" process is the "Zero-Phase Filtering" 
> topic in Section 13.12 of the 2nd Edition of my book.

Thanks Rick - hit the spot!
-- 
Randy Yates                      % "Maybe one day I'll feel her cold embrace,
Digital Signal Labs              %                    and kiss her interface, 
mailto://yates@ieee.org          %            til then, I'll leave her alone."
http://www.digitalsignallabs.com %        'Yours Truly, 2095', *Time*, ELO   

On 11/07/2010 10:43 PM, Randy Yates wrote:
> Tim Wescott<tim@seemywebsite.com>  writes:
>
>> On 11/07/2010 05:50 PM, ahgu wrote:
>>> I have a fixed 3600 data points(samples) and I want to apply a notch 2nd
>>> order Butterworth IIR Notch filter to remove the 1/4Fs frequency, the
>>> results start to converge(or filter start to work) after about 250 or so
>>> samples. The issue is the it is a continuous stream of data and I need to
>>> remove the harmonics on the initial 250 samples. What can I do to filter
>>> the whole data set?
>>
>> If you can handle the delay, wait until the whole data set is taken
>> then either apply the filter once in each direction, use a FIR filter,
>> or detect the amplitude and phase of the 1/4 Fs tone using synchronous
>> techniques and subtract it out.
>
> Is there a textbook treatment of this "two-way IIR" process? I've
> seen this many times but not yet grokked it.
>
> If you use an FIR, you'll still have the startup delay, no?

Argh -- so it is.

> The
> subtraction technique seems to be the most optimum if, as you've
> stated, he can accept the delay.

The subtraction technique would be the most optimum if the frequency of 
the interfering signal stays dead-on throughout the data vector.

If it changes -- even just to warble the phase -- then you need 
something that implements a broader filter.

If you need a broader filter, then you're back to some filter that takes 
the 'unknown-ness' of the samples beyond the edges of your signal vector 
into account -- the Kalman filter that I suggested would do this, but 
only from one direction.  I'm sure there are "FIR-ish" ways to do this 
in one thump -- but maybe not without a 3200 x 3200 matrix multiply.

-- 

Tim Wescott
Wescott Design Services
http://www.wescottdesign.com

Do you need to implement control loops in software?
"Applied Control Theory for Embedded Systems" was written for you.
See details at http://www.wescottdesign.com/actfes/actfes.html

Tim Wescott <tim@seemywebsite.com> wrote:
> On 11/07/2010 10:43 PM, Randy Yates wrote:
>> Tim Wescott<tim@seemywebsite.com>  writes:

>>> On 11/07/2010 05:50 PM, ahgu wrote:
>>>> I have a fixed 3600 data points(samples) and I want to apply a notch 2nd
>>>> order Butterworth IIR Notch filter to remove the 1/4Fs frequency, the
>>>> results start to converge(or filter start to work) after about 250 or so
>>>> samples. The issue is the it is a continuous stream of data and I need to
>>>> remove the harmonics on the initial 250 samples. What can I do to filter
>>>> the whole data set?

It seems that the real answer should be to have 250 samples before
and/or after the ones you really want.

>> If you use an FIR, you'll still have the startup delay, no?
 
> Argh -- so it is.
 
>> The
>> subtraction technique seems to be the most optimum if, as you've
>> stated, he can accept the delay.

After posting to the previous thread, I was remembering that sampling
theory pretty much assumes an infinite duration.  The error due to
finite duration is easily determined, but that is no excuse for
not storing some samples before and after the ones you really want.

If you didn't do it this time, then you know better next time.

-- glen

On 11/08/2010 12:34 PM, glen herrmannsfeldt wrote:
> Tim Wescott<tim@seemywebsite.com>  wrote:
>> On 11/07/2010 10:43 PM, Randy Yates wrote:
>>> Tim Wescott<tim@seemywebsite.com>   writes:
>
>>>> On 11/07/2010 05:50 PM, ahgu wrote:
>>>>> I have a fixed 3600 data points(samples) and I want to apply a notch 2nd
>>>>> order Butterworth IIR Notch filter to remove the 1/4Fs frequency, the
>>>>> results start to converge(or filter start to work) after about 250 or so
>>>>> samples. The issue is the it is a continuous stream of data and I need to
>>>>> remove the harmonics on the initial 250 samples. What can I do to filter
>>>>> the whole data set?
>
> It seems that the real answer should be to have 250 samples before
> and/or after the ones you really want.
>
>>> If you use an FIR, you'll still have the startup delay, no?
>
>> Argh -- so it is.
>
>>> The
>>> subtraction technique seems to be the most optimum if, as you've
>>> stated, he can accept the delay.
>
> After posting to the previous thread, I was remembering that sampling
> theory pretty much assumes an infinite duration.  The error due to
> finite duration is easily determined, but that is no excuse for
> not storing some samples before and after the ones you really want.
>
> If you didn't do it this time, then you know better next time.

OTOH, it's easy to imagine valid reasons that a sample may need to be of 
finite duration -- data storage or transmission speed limits leap to mind.

At base it's the same set of constraints that one has with any signal 
processing problem -- to get the best answer from the available data.

-- 

Tim Wescott
Wescott Design Services
http://www.wescottdesign.com

Do you need to implement control loops in software?
"Applied Control Theory for Embedded Systems" was written for you.
See details at http://www.wescottdesign.com/actfes/actfes.html

ahgu wrote:

> I have a fixed 3600 data points(samples) and I want to apply a notch 2nd
> order Butterworth IIR Notch filter to remove the 1/4Fs frequency, the
> results start to converge(or filter start to work) after about 250 or so
> samples. The issue is the it is a continuous stream of data and I need to
> remove the harmonics on the initial 250 samples. What can I do to filter
> the whole data set?

Aghu,

I am curious why don't you do what every imbecile in the world would do: 
take the FFT of the whole array, zero the unwanted components, take 
inverse FFT ?

VLV