On Thu, 15 Mar 2012 19:58:28 -0500, andrew66 wrote:>>On Thu, 15 Mar 2012 15:36:07 -0500, andrew66 wrote: >> >>> I have a signal that generally lasts about 6 seconds and has a lot of >>> Low-Frequency, < 10 Hz, noise in it. I would like to use a filter to >>> obtain a reliable answer for the mean value of the signal, as soon as >>> possible following the start of the data stream. >>> >>> I use a 2nd order discrete-time Low Pass Butterworth Filter with a 10 > Hz >>> sample rate, and I can change the cutoff frequency on the fly, via a >>> lookup table that has the filter coefficients in it. >>> >>> The output of the 0.2Hz cutoff version of this filter is smooth and I >>> can readily calculate a slope or average value from it. But, it takes >>> too long to respond to the input signal. >>> >>> A lowpass filter with a higher cutoff frequency, on the other hand, > such >>> as a 1 Hz cutoff version, responds more quickly of course, but it is >>> more difficult to pick an average slope or value from it. >>> >>> My question is, are there DSP filtering techniques that would be more >>> effective for this application? By “more effective”, I simply mean > it >>> can filter out the low noise but do it more quickly. For example, do >>> you know what they use for a rumble filter in real-time audio >>> applications? >> >>Yes maybe, and no. >> >>Yes there are DSP filtering techniques that might be more effective for >>this, but you haven't said what your constraints are. And no, I don't >>know what 'they' use for a rumble filter in real-time audio >>applications. >> >>You talk about getting a mean value, but you also talk about getting >>average and slope. And, you haven't said anything about the expected >>characteristics of the signal, or the noise. >> >>To answer your question _as stated_, without those troublesome >>implications of more depth, and _assuming_ that the noise is purely >>additive and of relatively high frequency, the best estimate of the mean >>value is a moving average filter: add up all the samples so far, divide >>by the number of samples, and there you are. >> >>You can speed this up by using a recursive formula: >> >>x_mean = (x - x_mean)/n >> >>where x is your signal, x_mean is your estimate of the signal mean, and >>n > >>is the number of samples so far, starting at 1. >> >>But you probably don't actually want the mean of the signal, because if >>you did then you wouldn't be asking about slope, and you're probably >>smart enough to know how to do an average. >> >>So what do you really want? The mean "for a while"? How long is "a >>while"? What made you decide that "a while" is long enough? >> >>And do you really want the slope? Why? >> >>Normally, the way that you go about solving this in a structured manner >>is to decide what the structure of your noise and your signal are (i.e., >>is it a constant, is it a ramp, if it is moving what is the >>_mathematical_ description of its motion, etc.), then you use that >>knowledge to design a filter that has the best chance of getting the >>right answer. >> >>Anything else is just tossing rocks in the dark, and hoping you'll hit >>something tasty. >> >>-- >>My liberal friends think I'm a conservative kook. My conservative >>friends think I'm a liberal kook. Why am I not happy that they have >>found common ground? >> >>Tim Wescott, Communications, Control, Circuits & Software >>http://www.wescottdesign.com >> > Thank you for your reply and excellent questions. I would like to begin > my response with a more thorough definition of the signal. I am > sampling the voltage output of a ratiometric bridge sensor. Over the 6 > second sample window, the base signal starts at about 500mV for the > first 1 - 2 seconds, when it jumps to 1 - 2 V, and then steadily ramps > toward 2 - 4 V for the remainder of the window. The ramp slope varies > from acquisition to acquisition. During this time, there is significant > LF ringing noise present: 1/2 to 3 Hz noise bandwidth, and the p-p > noise amplitude can be as much as 500 mV. > > The reason I mentioned slope, AND mean, AND average, is that I would > take any of these that would yield a reliable measurement. If the > measurement time window is large, say, half the sample window, then I > would take either the voltage slope between the beginning and end times, > or the average, whichever is most reliable. If I could find a way to > make the measurement time window small, then it is more likely to be a > singular value, or a mean that I will take. But, I can't narrow the > measurement time window down without an aggressive filter that is really > smoothing out the data. > > So, as you suggested, I have not been able to converge on a measurement > method (slope vs average) nor a time window that is reliable. In terms > of constraints, in the future I can upgrade my hardware to accomodate > any filter that will do the job. The relatively pitiful ARM MCU I am > now using is rather limited in the discrete time filters that I can > implement in it. > > Your idea of a moving average filter is intriguing, and indeed I have > tried something similar in a spreadsheet simulation. When I started the > simulation with an expected value or a seed as you suggested, it was > quite responsive. But, in terms of error I could not get my model to > improve on the LP filter that I'm already using. Perhaps I need to > reconsider a moving average filter. > > Thank you again for your thoughtful reply, and any other thoughts you > might be able to offer.I assume that you're measuring something steady? If so, perhaps you need to fix the underlying hardware rather than trying to "software" yourself around it. -- Tim Wescott Control system and signal processing consulting www.wescottdesign.com
Recommend a faster responding Low Pass Filter than a 2nd Order Butterworth?
Started by ●March 15, 2012
Reply by ●March 16, 20122012-03-16
Reply by ●March 16, 20122012-03-16
On 15.3.12 10:36 , andrew66 wrote:> I have a signal that generally lasts about 6 seconds and has a lot of > Low-Frequency,< 10 Hz, noise in it. I would like to use a filter to > obtain a reliable answer for the mean value of the signal, as soon as > possible following the start of the data stream. > > I use a 2nd order discrete-time Low Pass Butterworth Filter with a 10 Hz > sample rate, and I can change the cutoff frequency on the fly, via a lookup > table that has the filter coefficients in it. > > The output of the 0.2Hz cutoff version of this filter is smooth and I can > readily calculate a slope or average value from it. But, it takes too long > to respond to the input signal. > > A lowpass filter with a higher cutoff frequency, on the other hand, such as > a 1 Hz cutoff version, responds more quickly of course, but it is more > difficult to pick an average slope or value from it. > > My question is, are there DSP filtering techniques that would be more > effective for this application? By “more effective”, I simply mean it > can filter out the low noise but do it more quickly. For example, do you > know what they use for a rumble filter in real-time audio applications?Just to be sure, and as nobody has asked about it yet: How do you clean up the signal before sampling? Your description of the signal and the sample rate do not match, or you will get problems from Nyquist. -- Tauno Voipio
Reply by ●March 16, 20122012-03-16
On Mar 15, 5:58�pm, "andrew66" <andrew.meier@n_o_s_p_a_m.air- weigh.com> wrote:> >On Thu, 15 Mar 2012 15:36:07 -0500, andrew66 wrote: > > >> I have a signal that generally lasts about 6 seconds and has a lot of > >> Low-Frequency, < 10 Hz, noise in it. �I would like to use a filter to > >> obtain a reliable answer for the mean value of the signal, as soon as > >> possible following the start of the data stream. > > >> I use a 2nd order discrete-time Low Pass Butterworth Filter with a 10 > Hz > >> sample rate, and I can change the cutoff frequency on the fly, via a > >> lookup table that has the filter coefficients in it. > > >> The output of the 0.2Hz cutoff version of this filter is smooth and I > >> can readily calculate a slope or average value from it. �But, it takes > >> too long to respond to the input signal. > > >> A lowpass filter with a higher cutoff frequency, on the other hand, > such > >> as a 1 Hz cutoff version, responds more quickly of course, but it is > >> more difficult to pick an average slope or value from it. > > >> My question is, are there DSP filtering techniques that would be more > >> effective for this application? �By �more effective�, I simply mean > it > >> can filter out the low noise but do it more quickly. �For example, do > >> you know what they use for a rumble filter in real-time audio > >> applications? > > >Yes maybe, and no. > > >Yes there are DSP filtering techniques that might be more effective for > >this, but you haven't said what your constraints are. �And no, I don't > >know what 'they' use for a rumble filter in real-time audio applications. > > >You talk about getting a mean value, but you also talk about getting > >average and slope. �And, you haven't said anything about the expected > >characteristics of the signal, or the noise. > > >To answer your question _as stated_, without those troublesome > >implications of more depth, and _assuming_ that the noise is purely > >additive and of relatively high frequency, the best estimate of the mean > >value is a moving average filter: add up all the samples so far, divide > >by the number of samples, and there you are. > > >You can speed this up by using a recursive formula: > > >x_mean = (x - x_mean)/n > > >where x is your signal, x_mean is your estimate of the signal mean, and n > >is the number of samples so far, starting at 1. > > >But you probably don't actually want the mean of the signal, because if > >you did then you wouldn't be asking about slope, and you're probably > >smart enough to know how to do an average. > > >So what do you really want? �The mean "for a while"? �How long is "a > >while"? �What made you decide that "a while" is long enough? > > >And do you really want the slope? �Why? > > >Normally, the way that you go about solving this in a structured manner > >is to decide what the structure of your noise and your signal are (i.e., > >is it a constant, is it a ramp, if it is moving what is the > >_mathematical_ description of its motion, etc.), then you use that > >knowledge to design a filter that has the best chance of getting the > >right answer. > > >Anything else is just tossing rocks in the dark, and hoping you'll hit > >something tasty. > > >-- > >My liberal friends think I'm a conservative kook. > >My conservative friends think I'm a liberal kook. > >Why am I not happy that they have found common ground? > > >Tim Wescott, Communications, Control, Circuits & Software > >http://www.wescottdesign.com > > Thank you for your reply and excellent questions. �I would like to begin my > response with a more thorough definition of the signal. �I am sampling the > voltage output of a ratiometric bridge sensor. �Over the 6 second sample > window, the base signal starts at about 500mV for the first 1 - 2 seconds, > when it jumps to 1 - 2 V, and then steadily ramps toward 2 - 4 V for the > remainder of the window. �The ramp slope varies from acquisition to > acquisition. �During this time, there is significant LF ringing noise > present: �1/2 to 3 Hz noise bandwidth, and the p-p noise amplitude can be > as much as 500 mV. > > The reason I mentioned slope, AND mean, AND average, is that I would take > any of these that would yield a reliable measurement. �If the measurement > time window is large, say, half the sample window, then I would take either > the voltage slope between the beginning and end times, or the average, > whichever is most reliable. �If I could find a way to make the measurement > time window small, then it is more likely to be a singular value, or a mean > that I will take. �But, I can't narrow the measurement time window down > without an aggressive filter that is really smoothing out the data. > > So, as you suggested, I have not been able to converge on a measurement > method (slope vs average) nor a time window that is reliable. �In terms of > constraints, in the future I can upgrade my hardware to accomodate any > filter that will do the job. �The relatively pitiful ARM MCU I am now using > is rather limited in the discrete time filters that I can implement in it. > > Your idea of a moving average filter is intriguing, and indeed I have tried > something similar in a spreadsheet simulation. �When I started the > simulation with an expected value or a seed as you suggested, it was quite > responsive. �But, in terms of error I could not get my model to improve on > the LP filter that I'm already using. �Perhaps I need to reconsider a > moving average filter. > > Thank you again for your thoughtful reply, and any other thoughts you might > be able to offer.I'm surprised Tim hasn't recommended a Kalman design. You seem to have a fairly restricted signal that you're measuring - you know the form, but need to determine a few parameters that specify the exact trajectory. You haven't said much about the nature of the interfering noise {is the ringing part of the signal or part of the "noise"?). You might want to consider a Weiner-like analysis - i.e. designing a linear filter based on the relative spectral strengths of the noise vs. the signal. Once through the filter, you may need to reformulate the parameters depending on how the filter mangles the time-domain characteristics of the signal. The implementation details of the filter are (hopefully) less important than getting the spectral response right. I don't know what kind of ARM chip you have, but at these speeds I'd think it was plenty fast and capable, unless perhaps you try implementing some monstrous Kalman in floating point ;)
Reply by ●March 16, 20122012-03-16
On Fri, 16 Mar 2012 10:18:19 -0700, cassiope wrote:> On Mar 15, 5:58 pm, "andrew66" <andrew.meier@n_o_s_p_a_m.air- weigh.com> > wrote: >> >On Thu, 15 Mar 2012 15:36:07 -0500, andrew66 wrote: >> >> >> I have a signal that generally lasts about 6 seconds and has a lot >> >> of Low-Frequency, < 10 Hz, noise in it. I would like to use a >> >> filter to obtain a reliable answer for the mean value of the signal, >> >> as soon as possible following the start of the data stream. >> >> >> I use a 2nd order discrete-time Low Pass Butterworth Filter with a >> >> 10 >> Hz >> >> sample rate, and I can change the cutoff frequency on the fly, via a >> >> lookup table that has the filter coefficients in it. >> >> >> The output of the 0.2Hz cutoff version of this filter is smooth and >> >> I can readily calculate a slope or average value from it. But, it >> >> takes too long to respond to the input signal. >> >> >> A lowpass filter with a higher cutoff frequency, on the other hand, >> such >> >> as a 1 Hz cutoff version, responds more quickly of course, but it is >> >> more difficult to pick an average slope or value from it. >> >> >> My question is, are there DSP filtering techniques that would be >> >> more effective for this application? By “more effective”, I simply >> >> mean >> it >> >> can filter out the low noise but do it more quickly. For example, >> >> do you know what they use for a rumble filter in real-time audio >> >> applications? >> >> >Yes maybe, and no. >> >> >Yes there are DSP filtering techniques that might be more effective >> >for this, but you haven't said what your constraints are. And no, I >> >don't know what 'they' use for a rumble filter in real-time audio >> >applications. >> >> >You talk about getting a mean value, but you also talk about getting >> >average and slope. And, you haven't said anything about the expected >> >characteristics of the signal, or the noise. >> >> >To answer your question _as stated_, without those troublesome >> >implications of more depth, and _assuming_ that the noise is purely >> >additive and of relatively high frequency, the best estimate of the >> >mean value is a moving average filter: add up all the samples so far, >> >divide by the number of samples, and there you are. >> >> >You can speed this up by using a recursive formula: >> >> >x_mean = (x - x_mean)/n >> >> >where x is your signal, x_mean is your estimate of the signal mean, >> >and n is the number of samples so far, starting at 1. >> >> >But you probably don't actually want the mean of the signal, because >> >if you did then you wouldn't be asking about slope, and you're >> >probably smart enough to know how to do an average. >> >> >So what do you really want? The mean "for a while"? How long is "a >> >while"? What made you decide that "a while" is long enough? >> >> >And do you really want the slope? Why? >> >> >Normally, the way that you go about solving this in a structured >> >manner is to decide what the structure of your noise and your signal >> >are (i.e., is it a constant, is it a ramp, if it is moving what is the >> >_mathematical_ description of its motion, etc.), then you use that >> >knowledge to design a filter that has the best chance of getting the >> >right answer. >> >> >Anything else is just tossing rocks in the dark, and hoping you'll hit >> >something tasty. >> >> >-- >> >My liberal friends think I'm a conservative kook. My conservative >> >friends think I'm a liberal kook. Why am I not happy that they have >> >found common ground? >> >> >Tim Wescott, Communications, Control, Circuits & Software >> >http://www.wescottdesign.com >> >> Thank you for your reply and excellent questions. I would like to >> begin my response with a more thorough definition of the signal. I am >> sampling the voltage output of a ratiometric bridge sensor. Over the 6 >> second sample window, the base signal starts at about 500mV for the >> first 1 - 2 seconds, when it jumps to 1 - 2 V, and then steadily ramps >> toward 2 - 4 V for the remainder of the window. The ramp slope varies >> from acquisition to acquisition. During this time, there is >> significant LF ringing noise present: 1/2 to 3 Hz noise bandwidth, and >> the p-p noise amplitude can be as much as 500 mV. >> >> The reason I mentioned slope, AND mean, AND average, is that I would >> take any of these that would yield a reliable measurement. If the >> measurement time window is large, say, half the sample window, then I >> would take either the voltage slope between the beginning and end >> times, or the average, whichever is most reliable. If I could find a >> way to make the measurement time window small, then it is more likely >> to be a singular value, or a mean that I will take. But, I can't >> narrow the measurement time window down without an aggressive filter >> that is really smoothing out the data. >> >> So, as you suggested, I have not been able to converge on a measurement >> method (slope vs average) nor a time window that is reliable. In terms >> of constraints, in the future I can upgrade my hardware to accomodate >> any filter that will do the job. The relatively pitiful ARM MCU I am >> now using is rather limited in the discrete time filters that I can >> implement in it. >> >> Your idea of a moving average filter is intriguing, and indeed I have >> tried something similar in a spreadsheet simulation. When I started >> the simulation with an expected value or a seed as you suggested, it >> was quite responsive. But, in terms of error I could not get my model >> to improve on the LP filter that I'm already using. Perhaps I need to >> reconsider a moving average filter. >> >> Thank you again for your thoughtful reply, and any other thoughts you >> might be able to offer. > > I'm surprised Tim hasn't recommended a Kalman design.I was working up to it, if it looked like it was really necessary. If his measurement were steady and the noise white, then the successive averaging filter that I suggested would be the Kalman solution. I'm quite concerned about how Andrew describes his signal, because somehow I'm getting the impression that what's connected to his sensor is a steady physical value. If it isn't, then he needs to tell us more about what's going on physically. If it is, then he's got something wrong before the ADC that he needs to correct. -- My liberal friends think I'm a conservative kook. My conservative friends think I'm a liberal kook. Why am I not happy that they have found common ground? Tim Wescott, Communications, Control, Circuits & Software http://www.wescottdesign.com
