Minimal lag, non-causal smoothing filter

I want to smooth time-series data with ideally zero-lag.  The filter can be
non-causal.  I am currently using a butterworth filter and then advancing
the output in time.  However, since the butterworth filter's lag is not
constant, the output series does not line up as desired with the input.  My
goal here is to highlight the prominent turning points in the time series
-- points where signal changes abruptly.  Anyone know of a good filter for
this purpose?  Or if there is a better way to detect abrupt changes to a
signal other than measuring acceleration (to minimize noise) please let me
know.

On May 27, 7:33=A0am, "dspr39" <mark3422011@n_o_s_p_a_m.hotmail.com>
wrote:
> I want to smooth time-series data with ideally zero-lag. =A0The filter ca=
n be
> non-causal. =A0I am currently using a butterworth filter and then advanci=
ng
> the output in time. =A0However, since the butterworth filter's lag is not
> constant, the output series does not line up as desired with the input. =
=A0My
> goal here is to highlight the prominent turning points in the time series
> -- points where signal changes abruptly. =A0Anyone know of a good filter =
for
> this purpose? =A0Or if there is a better way to detect abrupt changes to =
a
> signal other than measuring acceleration (to minimize noise) please let m=
e
> know.

If your filter isn't causal, you can have any lag you want, even
negative. The only way the real world lets your filter produce outputs
today from inputs you will get tomorrow is by screwing around with
what your calendar says. What do you really want?

Jerry
--
Engineering is the art of making what you want from things you can get.

>On May 27, 7:33=A0am, "dspr39" <mark3422011@n_o_s_p_a_m.hotmail.com>
>wrote:
>> I want to smooth time-series data with ideally zero-lag. =A0The filter
ca=
>n be
>> non-causal. =A0I am currently using a butterworth filter and then
advanci=
>ng
>> the output in time. =A0However, since the butterworth filter's lag is
not
>> constant, the output series does not line up as desired with the input.
=
>=A0My
>> goal here is to highlight the prominent turning points in the time
series
>> -- points where signal changes abruptly. =A0Anyone know of a good filter
=
>for
>> this purpose? =A0Or if there is a better way to detect abrupt changes to
=
>a
>> signal other than measuring acceleration (to minimize noise) please let
m=
>e
>> know.
>
>If your filter isn't causal, you can have any lag you want, even
>negative. The only way the real world lets your filter produce outputs
>today from inputs you will get tomorrow is by screwing around with
>what your calendar says. What do you really want?
>
>Jerry
>--
>Engineering is the art of making what you want from things you can get.
>

I am trying to identify places in my time series where the signal changes
abruptly with respect to time -- time series is a 2D vector of position vs.
time.  I attempted to do this by taking the second derivative of position
to measure acceleration -- places where the signal (position) changes
abruptly are indicated as having higher acceleration.  However, the noise
in the signal also produces high acceleration values.  I'm post-processing
so it's fine for the filter to look ahead at future values.  

Here is a screenshot of the places in my time series that I am trying to
identify.  Obviously, I can do this by hand, but I would like a filter to
automate the process:

http://img29.imageshack.us/img29/2889/capturepab.png

Note that there are places where the signal changes abruptly due to noise
(noise meaning that signal does not continue in direction of abrupt change,
but reverts back to previous path), I wish to filter out or not select
those locations in the time series.  I tried to use a butterworth filter to
smooth out those locations before taking the 2nd first-order difference to
measure acceleration, but approach did not work well for reasons mentioned
in 1st post.

On 05/27/2011 07:33 AM, dspr39 wrote:
>> On May 27, 7:33=A0am, "dspr39"<mark3422011@n_o_s_p_a_m.hotmail.com>
>> wrote:
>>> I want to smooth time-series data with ideally zero-lag. =A0The filter
> ca=
>> n be
>>> non-causal. =A0I am currently using a butterworth filter and then
> advanci=
>> ng
>>> the output in time. =A0However, since the butterworth filter's lag is
> not
>>> constant, the output series does not line up as desired with the input.
> =
>> =A0My
>>> goal here is to highlight the prominent turning points in the time
> series
>>> -- points where signal changes abruptly. =A0Anyone know of a good filter
> =
>> for
>>> this purpose? =A0Or if there is a better way to detect abrupt changes to
> =
>> a
>>> signal other than measuring acceleration (to minimize noise) please let
> m=
>> e
>>> know.
>>
>> If your filter isn't causal, you can have any lag you want, even
>> negative. The only way the real world lets your filter produce outputs
>> today from inputs you will get tomorrow is by screwing around with
>> what your calendar says. What do you really want?
>>
>> Jerry
>> --
>> Engineering is the art of making what you want from things you can get.
>>
>
> I am trying to identify places in my time series where the signal changes
> abruptly with respect to time -- time series is a 2D vector of position vs.
> time.  I attempted to do this by taking the second derivative of position
> to measure acceleration -- places where the signal (position) changes
> abruptly are indicated as having higher acceleration.  However, the noise
> in the signal also produces high acceleration values.  I'm post-processing
> so it's fine for the filter to look ahead at future values.
>
> Here is a screenshot of the places in my time series that I am trying to
> identify.  Obviously, I can do this by hand, but I would like a filter to
> automate the process:
>
> http://img29.imageshack.us/img29/2889/capturepab.png
>
> Note that there are places where the signal changes abruptly due to noise
> (noise meaning that signal does not continue in direction of abrupt change,
> but reverts back to previous path), I wish to filter out or not select
> those locations in the time series.  I tried to use a butterworth filter to
> smooth out those locations before taking the 2nd first-order difference to
> measure acceleration, but approach did not work well for reasons mentioned
> in 1st post.

I'm not sure why you latched onto the Butterworth filter as the way to 
go.  It's not a _bad_ choice of all the filters that you may blindly 
choose, but it's almost certainly not the _best_ choice if you started 
with a knowledge of the underlying process, and there are other filters 
that would work as well.

Try running your data through the Butterworth filter forward and then 
backwards.  The result will have zero delay (you can think of this as 
the delay from the backwards filter generating 'advance' that exactly 
cancels the delay).  Or choose some FIR smoothing filter.

When you say your data is 2D, do you mean that you have X and Y position 
data?  Or that you have a vector of sample times matched with a vector 
of positions along one physical axis?  If it's the latter, are the 
sample times evenly spaced, or is your position being sampled 
irregularly?  Simply filtering the position data with a shift-invariant 
filter when the sample times are irregular presents its own set of less 
than optimal behavior, although things still may work.

-- 

Tim Wescott
Wescott Design Services
http://www.wescottdesign.com

Do you need to implement control loops in software?
"Applied Control Theory for Embedded Systems" was written for you.
See details at http://www.wescottdesign.com/actfes/actfes.html

>On 05/27/2011 07:33 AM, dspr39 wrote:
>>> On May 27, 7:33=A0am, "dspr39"<mark3422011@n_o_s_p_a_m.hotmail.com>
>>> wrote:
>>>> I want to smooth time-series data with ideally zero-lag. =A0The
filter
>> ca=
>>> n be
>>>> non-causal. =A0I am currently using a butterworth filter and then
>> advanci=
>>> ng
>>>> the output in time. =A0However, since the butterworth filter's lag is
>> not
>>>> constant, the output series does not line up as desired with the
input.
>> =
>>> =A0My
>>>> goal here is to highlight the prominent turning points in the time
>> series
>>>> -- points where signal changes abruptly. =A0Anyone know of a good
filter
>> =
>>> for
>>>> this purpose? =A0Or if there is a better way to detect abrupt changes
to
>> =
>>> a
>>>> signal other than measuring acceleration (to minimize noise) please
let
>> m=
>>> e
>>>> know.
>>>
>>> If your filter isn't causal, you can have any lag you want, even
>>> negative. The only way the real world lets your filter produce outputs
>>> today from inputs you will get tomorrow is by screwing around with
>>> what your calendar says. What do you really want?
>>>
>>> Jerry
>>> --
>>> Engineering is the art of making what you want from things you can
get.
>>>
>>
>> I am trying to identify places in my time series where the signal
changes
>> abruptly with respect to time -- time series is a 2D vector of position
vs.
>> time.  I attempted to do this by taking the second derivative of
position
>> to measure acceleration -- places where the signal (position) changes
>> abruptly are indicated as having higher acceleration.  However, the
noise
>> in the signal also produces high acceleration values.  I'm
post-processing
>> so it's fine for the filter to look ahead at future values.
>>
>> Here is a screenshot of the places in my time series that I am trying
to
>> identify.  Obviously, I can do this by hand, but I would like a filter
to
>> automate the process:
>>
>> http://img29.imageshack.us/img29/2889/capturepab.png
>>
>> Note that there are places where the signal changes abruptly due to
noise
>> (noise meaning that signal does not continue in direction of abrupt
change,
>> but reverts back to previous path), I wish to filter out or not select
>> those locations in the time series.  I tried to use a butterworth filter
to
>> smooth out those locations before taking the 2nd first-order difference
to
>> measure acceleration, but approach did not work well for reasons
mentioned
>> in 1st post.
>
>I'm not sure why you latched onto the Butterworth filter as the way to 
>go.  It's not a _bad_ choice of all the filters that you may blindly 
>choose, but it's almost certainly not the _best_ choice if you started 
>with a knowledge of the underlying process, and there are other filters 
>that would work as well.
>
>Try running your data through the Butterworth filter forward and then 
>backwards.  The result will have zero delay (you can think of this as 
>the delay from the backwards filter generating 'advance' that exactly 
>cancels the delay).  Or choose some FIR smoothing filter.
>
>When you say your data is 2D, do you mean that you have X and Y position 
>data?  Or that you have a vector of sample times matched with a vector 
>of positions along one physical axis?  If it's the latter, are the 
>sample times evenly spaced, or is your position being sampled 
>irregularly?  Simply filtering the position data with a shift-invariant 
>filter when the sample times are irregular presents its own set of less 
>than optimal behavior, although things still may work.
>
>-- 
>
>Tim Wescott
>Wescott Design Services
>http://www.wescottdesign.com
>
>Do you need to implement control loops in software?
>"Applied Control Theory for Embedded Systems" was written for you.
>See details at http://www.wescottdesign.com/actfes/actfes.html
>

By 2D vector, I meant a vector of sample times matched with a vector of 1D
positions along one physical axis -- think of time vs. distance from a
fixed point.  Butterworth filter might not have been the best choice.  I
was simply looking for a filter to attenuate high-frequency noise as much
as possible so that it's not amplified when I take the 2nd derivative of
position to get acceleration.  Honestly, my entire approach is probably not
ideal.  I'm basically trying to do time-series feature extraction.  I
simply want to mark the places in my time series where the signal takes a
significant turn, that would be indicated by a change in velocity, which is
acceleration.  To mark these places requires knowledge of future and past
position state...filtering came to mind.  If anyone knows of a better
approach, please let me know.

On 05/27/2011 11:07 AM, dspr39 wrote:
>> On 05/27/2011 07:33 AM, dspr39 wrote:
>>>> On May 27, 7:33=A0am, "dspr39"<mark3422011@n_o_s_p_a_m.hotmail.com>
>>>> wrote:
>>>>> I want to smooth time-series data with ideally zero-lag. =A0The
> filter
>>> ca=
>>>> n be
>>>>> non-causal. =A0I am currently using a butterworth filter and then
>>> advanci=
>>>> ng
>>>>> the output in time. =A0However, since the butterworth filter's lag is
>>> not
>>>>> constant, the output series does not line up as desired with the
> input.
>>> =
>>>> =A0My
>>>>> goal here is to highlight the prominent turning points in the time
>>> series
>>>>> -- points where signal changes abruptly. =A0Anyone know of a good
> filter
>>> =
>>>> for
>>>>> this purpose? =A0Or if there is a better way to detect abrupt changes
> to
>>> =
>>>> a
>>>>> signal other than measuring acceleration (to minimize noise) please
> let
>>> m=
>>>> e
>>>>> know.
>>>>
>>>> If your filter isn't causal, you can have any lag you want, even
>>>> negative. The only way the real world lets your filter produce outputs
>>>> today from inputs you will get tomorrow is by screwing around with
>>>> what your calendar says. What do you really want?
>>>>
>>>> Jerry
>>>> --
>>>> Engineering is the art of making what you want from things you can
> get.
>>>>
>>>
>>> I am trying to identify places in my time series where the signal
> changes
>>> abruptly with respect to time -- time series is a 2D vector of position
> vs.
>>> time.  I attempted to do this by taking the second derivative of
> position
>>> to measure acceleration -- places where the signal (position) changes
>>> abruptly are indicated as having higher acceleration.  However, the
> noise
>>> in the signal also produces high acceleration values.  I'm
> post-processing
>>> so it's fine for the filter to look ahead at future values.
>>>
>>> Here is a screenshot of the places in my time series that I am trying
> to
>>> identify.  Obviously, I can do this by hand, but I would like a filter
> to
>>> automate the process:
>>>
>>> http://img29.imageshack.us/img29/2889/capturepab.png
>>>
>>> Note that there are places where the signal changes abruptly due to
> noise
>>> (noise meaning that signal does not continue in direction of abrupt
> change,
>>> but reverts back to previous path), I wish to filter out or not select
>>> those locations in the time series.  I tried to use a butterworth filter
> to
>>> smooth out those locations before taking the 2nd first-order difference
> to
>>> measure acceleration, but approach did not work well for reasons
> mentioned
>>> in 1st post.
>>
>> I'm not sure why you latched onto the Butterworth filter as the way to
>> go.  It's not a _bad_ choice of all the filters that you may blindly
>> choose, but it's almost certainly not the _best_ choice if you started
>> with a knowledge of the underlying process, and there are other filters
>> that would work as well.
>>
>> Try running your data through the Butterworth filter forward and then
>> backwards.  The result will have zero delay (you can think of this as
>> the delay from the backwards filter generating 'advance' that exactly
>> cancels the delay).  Or choose some FIR smoothing filter.
>>
>> When you say your data is 2D, do you mean that you have X and Y position
>> data?  Or that you have a vector of sample times matched with a vector
>> of positions along one physical axis?  If it's the latter, are the
>> sample times evenly spaced, or is your position being sampled
>> irregularly?  Simply filtering the position data with a shift-invariant
>> filter when the sample times are irregular presents its own set of less
>> than optimal behavior, although things still may work.
>>
>> --
>>
>> Tim Wescott
>> Wescott Design Services
>> http://www.wescottdesign.com
>>
>> Do you need to implement control loops in software?
>> "Applied Control Theory for Embedded Systems" was written for you.
>> See details at http://www.wescottdesign.com/actfes/actfes.html
>>
>
> By 2D vector, I meant a vector of sample times matched with a vector of 1D
> positions along one physical axis -- think of time vs. distance from a
> fixed point.  Butterworth filter might not have been the best choice.  I
> was simply looking for a filter to attenuate high-frequency noise as much
> as possible so that it's not amplified when I take the 2nd derivative of
> position to get acceleration.  Honestly, my entire approach is probably not
> ideal.  I'm basically trying to do time-series feature extraction.  I
> simply want to mark the places in my time series where the signal takes a
> significant turn, that would be indicated by a change in velocity, which is
> acceleration.  To mark these places requires knowledge of future and past
> position state...filtering came to mind.  If anyone knows of a better
> approach, please let me know.

Filtering in general is certainly a good approach -- in fact, few things 
that you can do on this that work can't be accurately described as 
"filtering".  I was only questioning the use of a Butterworth.  Even 
there, it may work well enough once you sweep it through in both directions.

But:  how much does your sampling rate vary?  If you've got a big 
variation in the interval from sample to sample then your filtering 
needs to be considerably more fancy than any fixed-coefficient thing.

-- 

Tim Wescott
Wescott Design Services
http://www.wescottdesign.com

Do you need to implement control loops in software?
"Applied Control Theory for Embedded Systems" was written for you.
See details at http://www.wescottdesign.com/actfes/actfes.html

dspr39 schrieb:
>> On 05/27/2011 07:33 AM, dspr39 wrote:
>>>> On May 27, 7:33=A0am, "dspr39"<mark3422011@n_o_s_p_a_m.hotmail.com>
>>>> wrote:
>>>>> I want to smooth time-series data with ideally zero-lag. =A0The
> filter
>>> ca=
>>>> n be
>>>>> non-causal. =A0I am currently using a butterworth filter and then
>>> advanci=
>>>> ng
>>>>> the output in time. =A0However, since the butterworth filter's lag is
>>> not
>>>>> constant, the output series does not line up as desired with the
> input.
>>> =
>>>> =A0My
>>>>> goal here is to highlight the prominent turning points in the time
>>> series
>>>>> -- points where signal changes abruptly. =A0Anyone know of a good
> filter
>>> =
>>>> for
>>>>> this purpose? =A0Or if there is a better way to detect abrupt changes
> to
>>> =
>>>> a
>>>>> signal other than measuring acceleration (to minimize noise) please
> let
>>> m=
>>>> e
>>>>> know.
>>>>
>>>> If your filter isn't causal, you can have any lag you want, even
>>>> negative. The only way the real world lets your filter produce outputs
>>>> today from inputs you will get tomorrow is by screwing around with
>>>> what your calendar says. What do you really want?
>>>>
>>>> Jerry
>>>> --
>>>> Engineering is the art of making what you want from things you can
> get.
>>>>
>>>
>>> I am trying to identify places in my time series where the signal
> changes
>>> abruptly with respect to time -- time series is a 2D vector of position
> vs.
>>> time.  I attempted to do this by taking the second derivative of
> position
>>> to measure acceleration -- places where the signal (position) changes
>>> abruptly are indicated as having higher acceleration.  However, the
> noise
>>> in the signal also produces high acceleration values.  I'm
> post-processing
>>> so it's fine for the filter to look ahead at future values.
>>>
>>> Here is a screenshot of the places in my time series that I am trying
> to
>>> identify.  Obviously, I can do this by hand, but I would like a filter
> to
>>> automate the process:
>>>
>>> http://img29.imageshack.us/img29/2889/capturepab.png
>>>
>>> Note that there are places where the signal changes abruptly due to
> noise
>>> (noise meaning that signal does not continue in direction of abrupt
> change,
>>> but reverts back to previous path), I wish to filter out or not select
>>> those locations in the time series.  I tried to use a butterworth filter
> to
>>> smooth out those locations before taking the 2nd first-order difference
> to
>>> measure acceleration, but approach did not work well for reasons
> mentioned
>>> in 1st post.
>>
>> I'm not sure why you latched onto the Butterworth filter as the way to
>> go.  It's not a _bad_ choice of all the filters that you may blindly
>> choose, but it's almost certainly not the _best_ choice if you started
>> with a knowledge of the underlying process, and there are other filters
>> that would work as well.
>>
>> Try running your data through the Butterworth filter forward and then
>> backwards.  The result will have zero delay (you can think of this as
>> the delay from the backwards filter generating 'advance' that exactly
>> cancels the delay).  Or choose some FIR smoothing filter.
>>
>> When you say your data is 2D, do you mean that you have X and Y position
>> data?  Or that you have a vector of sample times matched with a vector
>> of positions along one physical axis?  If it's the latter, are the
>> sample times evenly spaced, or is your position being sampled
>> irregularly?  Simply filtering the position data with a shift-invariant
>> filter when the sample times are irregular presents its own set of less
>> than optimal behavior, although things still may work.
>>
>> --
>>
>> Tim Wescott
>> Wescott Design Services
>> http://www.wescottdesign.com
>>
>> Do you need to implement control loops in software?
>> "Applied Control Theory for Embedded Systems" was written for you.
>> See details at http://www.wescottdesign.com/actfes/actfes.html
>>
>
> By 2D vector, I meant a vector of sample times matched with a vector of 1D
> positions along one physical axis -- think of time vs. distance from a
> fixed point.  Butterworth filter might not have been the best choice.  I
> was simply looking for a filter to attenuate high-frequency noise as much
> as possible so that it's not amplified when I take the 2nd derivative of
> position to get acceleration.  Honestly, my entire approach is probably not
> ideal.  I'm basically trying to do time-series feature extraction.  I
> simply want to mark the places in my time series where the signal takes a
> significant turn, that would be indicated by a change in velocity, which is
> acceleration.  To mark these places requires knowledge of future and past
> position state...filtering came to mind.  If anyone knows of a better
> approach, please let me know.

I am not sure you want to use a time-invariant filter at all. What you 
are trying to do sounds to me like target tracking with maneuver 
detection. Google should give you some results on that. I am not an 
expert but I would try to model the data in state-space and use a Kalman 
filter to get a smoothed estimate of position, velocity and 
accelaration. This requires some a-priori knowledge about the initial 
state of your target and the expected measurement accuracies which you 
may or may not have.

Best regards,

Sebastian

On 05/27/2011 01:36 PM, Sebastian Doht wrote:
> dspr39 schrieb:
>>> On 05/27/2011 07:33 AM, dspr39 wrote:
>>>>> On May 27, 7:33=A0am, "dspr39"<mark3422011@n_o_s_p_a_m.hotmail.com>
>>>>> wrote:
>>>>>> I want to smooth time-series data with ideally zero-lag. =A0The
>> filter
>>>> ca=
>>>>> n be
>>>>>> non-causal. =A0I am currently using a butterworth filter and then
>>>> advanci=
>>>>> ng
>>>>>> the output in time. =A0However, since the butterworth filter's lag is
>>>> not
>>>>>> constant, the output series does not line up as desired with the
>> input.
>>>> =
>>>>> =A0My
>>>>>> goal here is to highlight the prominent turning points in the time
>>>> series
>>>>>> -- points where signal changes abruptly. =A0Anyone know of a good
>> filter
>>>> =
>>>>> for
>>>>>> this purpose? =A0Or if there is a better way to detect abrupt changes
>> to
>>>> =
>>>>> a
>>>>>> signal other than measuring acceleration (to minimize noise) please
>> let
>>>> m=
>>>>> e
>>>>>> know.
>>>>>
>>>>> If your filter isn't causal, you can have any lag you want, even
>>>>> negative. The only way the real world lets your filter produce outputs
>>>>> today from inputs you will get tomorrow is by screwing around with
>>>>> what your calendar says. What do you really want?
>>>>>
>>>>> Jerry
>>>>> --
>>>>> Engineering is the art of making what you want from things you can
>> get.
>>>>>
>>>>
>>>> I am trying to identify places in my time series where the signal
>> changes
>>>> abruptly with respect to time -- time series is a 2D vector of position
>> vs.
>>>> time. I attempted to do this by taking the second derivative of
>> position
>>>> to measure acceleration -- places where the signal (position) changes
>>>> abruptly are indicated as having higher acceleration. However, the
>> noise
>>>> in the signal also produces high acceleration values. I'm
>> post-processing
>>>> so it's fine for the filter to look ahead at future values.
>>>>
>>>> Here is a screenshot of the places in my time series that I am trying
>> to
>>>> identify. Obviously, I can do this by hand, but I would like a filter
>> to
>>>> automate the process:
>>>>
>>>> http://img29.imageshack.us/img29/2889/capturepab.png
>>>>
>>>> Note that there are places where the signal changes abruptly due to
>> noise
>>>> (noise meaning that signal does not continue in direction of abrupt
>> change,
>>>> but reverts back to previous path), I wish to filter out or not select
>>>> those locations in the time series. I tried to use a butterworth filter
>> to
>>>> smooth out those locations before taking the 2nd first-order difference
>> to
>>>> measure acceleration, but approach did not work well for reasons
>> mentioned
>>>> in 1st post.
>>>
>>> I'm not sure why you latched onto the Butterworth filter as the way to
>>> go. It's not a _bad_ choice of all the filters that you may blindly
>>> choose, but it's almost certainly not the _best_ choice if you started
>>> with a knowledge of the underlying process, and there are other filters
>>> that would work as well.
>>>
>>> Try running your data through the Butterworth filter forward and then
>>> backwards. The result will have zero delay (you can think of this as
>>> the delay from the backwards filter generating 'advance' that exactly
>>> cancels the delay). Or choose some FIR smoothing filter.
>>>
>>> When you say your data is 2D, do you mean that you have X and Y position
>>> data? Or that you have a vector of sample times matched with a vector
>>> of positions along one physical axis? If it's the latter, are the
>>> sample times evenly spaced, or is your position being sampled
>>> irregularly? Simply filtering the position data with a shift-invariant
>>> filter when the sample times are irregular presents its own set of less
>>> than optimal behavior, although things still may work.
>>>
>>> --
>>>
>>> Tim Wescott
>>> Wescott Design Services
>>> http://www.wescottdesign.com
>>>
>>> Do you need to implement control loops in software?
>>> "Applied Control Theory for Embedded Systems" was written for you.
>>> See details at http://www.wescottdesign.com/actfes/actfes.html
>>>
>>
>> By 2D vector, I meant a vector of sample times matched with a vector
>> of 1D
>> positions along one physical axis -- think of time vs. distance from a
>> fixed point. Butterworth filter might not have been the best choice. I
>> was simply looking for a filter to attenuate high-frequency noise as much
>> as possible so that it's not amplified when I take the 2nd derivative of
>> position to get acceleration. Honestly, my entire approach is probably
>> not
>> ideal. I'm basically trying to do time-series feature extraction. I
>> simply want to mark the places in my time series where the signal takes a
>> significant turn, that would be indicated by a change in velocity,
>> which is
>> acceleration. To mark these places requires knowledge of future and past
>> position state...filtering came to mind. If anyone knows of a better
>> approach, please let me know.
>
> I am not sure you want to use a time-invariant filter at all. What you
> are trying to do sounds to me like target tracking with maneuver
> detection. Google should give you some results on that. I am not an
> expert but I would try to model the data in state-space and use a Kalman
> filter to get a smoothed estimate of position, velocity and
> accelaration. This requires some a-priori knowledge about the initial
> state of your target and the expected measurement accuracies which you
> may or may not have.

A Kalman may not be the best filter if you don't know the target 
dynamics beforehand.  In fact, if want you can choose what filter you 
want the Kalman to settle out to in steady state, and specify your 
target dynamics and measurement noise so you get that filter in the end.

But if the OP _does_ know the target dynamics, _exactly_, then a Kalman 
may work.

We need to know whether the sampling is irregular, at any rate.

-- 

Tim Wescott
Wescott Design Services
http://www.wescottdesign.com

Do you need to implement control loops in software?
"Applied Control Theory for Embedded Systems" was written for you.
See details at http://www.wescottdesign.com/actfes/actfes.html

Tim Wescott schrieb:
> On 05/27/2011 01:36 PM, Sebastian Doht wrote:
>> dspr39 schrieb:
>>>> On 05/27/2011 07:33 AM, dspr39 wrote:
>>>>>> On May 27, 7:33=A0am, "dspr39"<mark3422011@n_o_s_p_a_m.hotmail.com>
>>>>>> wrote:
>>>>>>> I want to smooth time-series data with ideally zero-lag. =A0The
>>> filter
>>>>> ca=
>>>>>> n be
>>>>>>> non-causal. =A0I am currently using a butterworth filter and then
>>>>> advanci=
>>>>>> ng
>>>>>>> the output in time. =A0However, since the butterworth filter's
>>>>>>> lag is
>>>>> not
>>>>>>> constant, the output series does not line up as desired with the
>>> input.
>>>>> =
>>>>>> =A0My
>>>>>>> goal here is to highlight the prominent turning points in the time
>>>>> series
>>>>>>> -- points where signal changes abruptly. =A0Anyone know of a good
>>> filter
>>>>> =
>>>>>> for
>>>>>>> this purpose? =A0Or if there is a better way to detect abrupt
>>>>>>> changes
>>> to
>>>>> =
>>>>>> a
>>>>>>> signal other than measuring acceleration (to minimize noise) please
>>> let
>>>>> m=
>>>>>> e
>>>>>>> know.
>>>>>>
>>>>>> If your filter isn't causal, you can have any lag you want, even
>>>>>> negative. The only way the real world lets your filter produce
>>>>>> outputs
>>>>>> today from inputs you will get tomorrow is by screwing around with
>>>>>> what your calendar says. What do you really want?
>>>>>>
>>>>>> Jerry
>>>>>> --
>>>>>> Engineering is the art of making what you want from things you can
>>> get.
>>>>>>
>>>>>
>>>>> I am trying to identify places in my time series where the signal
>>> changes
>>>>> abruptly with respect to time -- time series is a 2D vector of
>>>>> position
>>> vs.
>>>>> time. I attempted to do this by taking the second derivative of
>>> position
>>>>> to measure acceleration -- places where the signal (position) changes
>>>>> abruptly are indicated as having higher acceleration. However, the
>>> noise
>>>>> in the signal also produces high acceleration values. I'm
>>> post-processing
>>>>> so it's fine for the filter to look ahead at future values.
>>>>>
>>>>> Here is a screenshot of the places in my time series that I am trying
>>> to
>>>>> identify. Obviously, I can do this by hand, but I would like a filter
>>> to
>>>>> automate the process:
>>>>>
>>>>> http://img29.imageshack.us/img29/2889/capturepab.png
>>>>>
>>>>> Note that there are places where the signal changes abruptly due to
>>> noise
>>>>> (noise meaning that signal does not continue in direction of abrupt
>>> change,
>>>>> but reverts back to previous path), I wish to filter out or not select
>>>>> those locations in the time series. I tried to use a butterworth
>>>>> filter
>>> to
>>>>> smooth out those locations before taking the 2nd first-order
>>>>> difference
>>> to
>>>>> measure acceleration, but approach did not work well for reasons
>>> mentioned
>>>>> in 1st post.
>>>>
>>>> I'm not sure why you latched onto the Butterworth filter as the way to
>>>> go. It's not a _bad_ choice of all the filters that you may blindly
>>>> choose, but it's almost certainly not the _best_ choice if you started
>>>> with a knowledge of the underlying process, and there are other filters
>>>> that would work as well.
>>>>
>>>> Try running your data through the Butterworth filter forward and then
>>>> backwards. The result will have zero delay (you can think of this as
>>>> the delay from the backwards filter generating 'advance' that exactly
>>>> cancels the delay). Or choose some FIR smoothing filter.
>>>>
>>>> When you say your data is 2D, do you mean that you have X and Y
>>>> position
>>>> data? Or that you have a vector of sample times matched with a vector
>>>> of positions along one physical axis? If it's the latter, are the
>>>> sample times evenly spaced, or is your position being sampled
>>>> irregularly? Simply filtering the position data with a shift-invariant
>>>> filter when the sample times are irregular presents its own set of less
>>>> than optimal behavior, although things still may work.
>>>>
>>>> --
>>>>
>>>> Tim Wescott
>>>> Wescott Design Services
>>>> http://www.wescottdesign.com
>>>>
>>>> Do you need to implement control loops in software?
>>>> "Applied Control Theory for Embedded Systems" was written for you.
>>>> See details at http://www.wescottdesign.com/actfes/actfes.html
>>>>
>>>
>>> By 2D vector, I meant a vector of sample times matched with a vector
>>> of 1D
>>> positions along one physical axis -- think of time vs. distance from a
>>> fixed point. Butterworth filter might not have been the best choice. I
>>> was simply looking for a filter to attenuate high-frequency noise as
>>> much
>>> as possible so that it's not amplified when I take the 2nd derivative of
>>> position to get acceleration. Honestly, my entire approach is probably
>>> not
>>> ideal. I'm basically trying to do time-series feature extraction. I
>>> simply want to mark the places in my time series where the signal
>>> takes a
>>> significant turn, that would be indicated by a change in velocity,
>>> which is
>>> acceleration. To mark these places requires knowledge of future and past
>>> position state...filtering came to mind. If anyone knows of a better
>>> approach, please let me know.
>>
>> I am not sure you want to use a time-invariant filter at all. What you
>> are trying to do sounds to me like target tracking with maneuver
>> detection. Google should give you some results on that. I am not an
>> expert but I would try to model the data in state-space and use a Kalman
>> filter to get a smoothed estimate of position, velocity and
>> accelaration. This requires some a-priori knowledge about the initial
>> state of your target and the expected measurement accuracies which you
>> may or may not have.
>
> A Kalman may not be the best filter if you don't know the target
> dynamics beforehand. In fact, if want you can choose what filter you
> want the Kalman to settle out to in steady state, and specify your
> target dynamics and measurement noise so you get that filter in the end.
>
> But if the OP _does_ know the target dynamics, _exactly_, then a Kalman
> may work.
>
> We need to know whether the sampling is irregular, at any rate.
>

That's why I wrote:

 >> This requires some a-priori knowledge about the initial
 >> state of your target and the expected measurement accuracies which 
 >> you may or may not have.

If it is the best filter depends mostly if one can come up with an 
initial state at some t = 0 that is close to the reality and if the 
sensor characteristics (covariance) are known.

If not than a Kalman filter is just as happy guessing as choosing some 
arbitrary Butterworth filter...

On 05/27/2011 02:04 PM, Sebastian Doht wrote:
> Tim Wescott schrieb:
>> On 05/27/2011 01:36 PM, Sebastian Doht wrote:
>>> dspr39 schrieb:
>>>>> On 05/27/2011 07:33 AM, dspr39 wrote:
>>>>>>> On May 27, 7:33=A0am, "dspr39"<mark3422011@n_o_s_p_a_m.hotmail.com>
>>>>>>> wrote:
>>>>>>>> I want to smooth time-series data with ideally zero-lag. =A0The
>>>> filter
>>>>>> ca=
>>>>>>> n be
>>>>>>>> non-causal. =A0I am currently using a butterworth filter and then
>>>>>> advanci=
>>>>>>> ng
>>>>>>>> the output in time. =A0However, since the butterworth filter's
>>>>>>>> lag is
>>>>>> not
>>>>>>>> constant, the output series does not line up as desired with the
>>>> input.
>>>>>> =
>>>>>>> =A0My
>>>>>>>> goal here is to highlight the prominent turning points in the time
>>>>>> series
>>>>>>>> -- points where signal changes abruptly. =A0Anyone know of a good
>>>> filter
>>>>>> =
>>>>>>> for
>>>>>>>> this purpose? =A0Or if there is a better way to detect abrupt
>>>>>>>> changes
>>>> to
>>>>>> =
>>>>>>> a
>>>>>>>> signal other than measuring acceleration (to minimize noise) please
>>>> let
>>>>>> m=
>>>>>>> e
>>>>>>>> know.
>>>>>>>
>>>>>>> If your filter isn't causal, you can have any lag you want, even
>>>>>>> negative. The only way the real world lets your filter produce
>>>>>>> outputs
>>>>>>> today from inputs you will get tomorrow is by screwing around with
>>>>>>> what your calendar says. What do you really want?
>>>>>>>
>>>>>>> Jerry
>>>>>>> --
>>>>>>> Engineering is the art of making what you want from things you can
>>>> get.
>>>>>>>
>>>>>>
>>>>>> I am trying to identify places in my time series where the signal
>>>> changes
>>>>>> abruptly with respect to time -- time series is a 2D vector of
>>>>>> position
>>>> vs.
>>>>>> time. I attempted to do this by taking the second derivative of
>>>> position
>>>>>> to measure acceleration -- places where the signal (position) changes
>>>>>> abruptly are indicated as having higher acceleration. However, the
>>>> noise
>>>>>> in the signal also produces high acceleration values. I'm
>>>> post-processing
>>>>>> so it's fine for the filter to look ahead at future values.
>>>>>>
>>>>>> Here is a screenshot of the places in my time series that I am trying
>>>> to
>>>>>> identify. Obviously, I can do this by hand, but I would like a filter
>>>> to
>>>>>> automate the process:
>>>>>>
>>>>>> http://img29.imageshack.us/img29/2889/capturepab.png
>>>>>>
>>>>>> Note that there are places where the signal changes abruptly due to
>>>> noise
>>>>>> (noise meaning that signal does not continue in direction of abrupt
>>>> change,
>>>>>> but reverts back to previous path), I wish to filter out or not
>>>>>> select
>>>>>> those locations in the time series. I tried to use a butterworth
>>>>>> filter
>>>> to
>>>>>> smooth out those locations before taking the 2nd first-order
>>>>>> difference
>>>> to
>>>>>> measure acceleration, but approach did not work well for reasons
>>>> mentioned
>>>>>> in 1st post.
>>>>>
>>>>> I'm not sure why you latched onto the Butterworth filter as the way to
>>>>> go. It's not a _bad_ choice of all the filters that you may blindly
>>>>> choose, but it's almost certainly not the _best_ choice if you started
>>>>> with a knowledge of the underlying process, and there are other
>>>>> filters
>>>>> that would work as well.
>>>>>
>>>>> Try running your data through the Butterworth filter forward and then
>>>>> backwards. The result will have zero delay (you can think of this as
>>>>> the delay from the backwards filter generating 'advance' that exactly
>>>>> cancels the delay). Or choose some FIR smoothing filter.
>>>>>
>>>>> When you say your data is 2D, do you mean that you have X and Y
>>>>> position
>>>>> data? Or that you have a vector of sample times matched with a vector
>>>>> of positions along one physical axis? If it's the latter, are the
>>>>> sample times evenly spaced, or is your position being sampled
>>>>> irregularly? Simply filtering the position data with a shift-invariant
>>>>> filter when the sample times are irregular presents its own set of
>>>>> less
>>>>> than optimal behavior, although things still may work.
>>>>>
>>>>> --
>>>>>
>>>>> Tim Wescott
>>>>> Wescott Design Services
>>>>> http://www.wescottdesign.com
>>>>>
>>>>> Do you need to implement control loops in software?
>>>>> "Applied Control Theory for Embedded Systems" was written for you.
>>>>> See details at http://www.wescottdesign.com/actfes/actfes.html
>>>>>
>>>>
>>>> By 2D vector, I meant a vector of sample times matched with a vector
>>>> of 1D
>>>> positions along one physical axis -- think of time vs. distance from a
>>>> fixed point. Butterworth filter might not have been the best choice. I
>>>> was simply looking for a filter to attenuate high-frequency noise as
>>>> much
>>>> as possible so that it's not amplified when I take the 2nd
>>>> derivative of
>>>> position to get acceleration. Honestly, my entire approach is probably
>>>> not
>>>> ideal. I'm basically trying to do time-series feature extraction. I
>>>> simply want to mark the places in my time series where the signal
>>>> takes a
>>>> significant turn, that would be indicated by a change in velocity,
>>>> which is
>>>> acceleration. To mark these places requires knowledge of future and
>>>> past
>>>> position state...filtering came to mind. If anyone knows of a better
>>>> approach, please let me know.
>>>
>>> I am not sure you want to use a time-invariant filter at all. What you
>>> are trying to do sounds to me like target tracking with maneuver
>>> detection. Google should give you some results on that. I am not an
>>> expert but I would try to model the data in state-space and use a Kalman
>>> filter to get a smoothed estimate of position, velocity and
>>> accelaration. This requires some a-priori knowledge about the initial
>>> state of your target and the expected measurement accuracies which you
>>> may or may not have.
>>
>> A Kalman may not be the best filter if you don't know the target
>> dynamics beforehand. In fact, if want you can choose what filter you
>> want the Kalman to settle out to in steady state, and specify your
>> target dynamics and measurement noise so you get that filter in the end.
>>
>> But if the OP _does_ know the target dynamics, _exactly_, then a Kalman
>> may work.
>>
>> We need to know whether the sampling is irregular, at any rate.
>>
>
> That's why I wrote:
>
>  >> This requires some a-priori knowledge about the initial
>  >> state of your target and the expected measurement accuracies which
>  >> you may or may not have.
>
> If it is the best filter depends mostly if one can come up with an
> initial state at some t = 0 that is close to the reality and if the
> sensor characteristics (covariance) are known.
>
> If not than a Kalman filter is just as happy guessing as choosing some
> arbitrary Butterworth filter...

The nice thing about the old standbys like the Butterworth (or a boxcar 
filter that just averages around a point) are that they are pretty 
robust when the underlying process changes.  You can't guarantee that 
with a Kalman.  An H-infinity filter, yes.  But not a Kalman.

-- 

Tim Wescott
Wescott Design Services
http://www.wescottdesign.com

Do you need to implement control loops in software?
"Applied Control Theory for Embedded Systems" was written for you.
See details at http://www.wescottdesign.com/actfes/actfes.html