I have a signal that generally lasts about 6 seconds and has a lot of Low-Frequency, < 10 Hz, noise in it. I would like to use a filter to obtain a reliable answer for the mean value of the signal, as soon as possible following the start of the data stream. I use a 2nd order discrete-time Low Pass Butterworth Filter with a 10 Hz sample rate, and I can change the cutoff frequency on the fly, via a lookup table that has the filter coefficients in it. The output of the 0.2Hz cutoff version of this filter is smooth and I can readily calculate a slope or average value from it. But, it takes too long to respond to the input signal. A lowpass filter with a higher cutoff frequency, on the other hand, such as a 1 Hz cutoff version, responds more quickly of course, but it is more difficult to pick an average slope or value from it. My question is, are there DSP filtering techniques that would be more effective for this application? By “more effective”, I simply mean it can filter out the low noise but do it more quickly. For example, do you know what they use for a rumble filter in real-time audio applications? Thanks!
Recommend a faster responding Low Pass Filter than a 2nd Order Butterworth?
Started by ●March 15, 2012
Reply by ●March 15, 20122012-03-15
On Thu, 15 Mar 2012 15:36:07 -0500, andrew66 wrote:> I have a signal that generally lasts about 6 seconds and has a lot of > Low-Frequency, < 10 Hz, noise in it. I would like to use a filter to > obtain a reliable answer for the mean value of the signal, as soon as > possible following the start of the data stream. > > I use a 2nd order discrete-time Low Pass Butterworth Filter with a 10 Hz > sample rate, and I can change the cutoff frequency on the fly, via a > lookup table that has the filter coefficients in it. > > The output of the 0.2Hz cutoff version of this filter is smooth and I > can readily calculate a slope or average value from it. But, it takes > too long to respond to the input signal. > > A lowpass filter with a higher cutoff frequency, on the other hand, such > as a 1 Hz cutoff version, responds more quickly of course, but it is > more difficult to pick an average slope or value from it. > > My question is, are there DSP filtering techniques that would be more > effective for this application? By “more effective”, I simply mean it > can filter out the low noise but do it more quickly. For example, do > you know what they use for a rumble filter in real-time audio > applications?Yes maybe, and no. Yes there are DSP filtering techniques that might be more effective for this, but you haven't said what your constraints are. And no, I don't know what 'they' use for a rumble filter in real-time audio applications. You talk about getting a mean value, but you also talk about getting average and slope. And, you haven't said anything about the expected characteristics of the signal, or the noise. To answer your question _as stated_, without those troublesome implications of more depth, and _assuming_ that the noise is purely additive and of relatively high frequency, the best estimate of the mean value is a moving average filter: add up all the samples so far, divide by the number of samples, and there you are. You can speed this up by using a recursive formula: x_mean = (x - x_mean)/n where x is your signal, x_mean is your estimate of the signal mean, and n is the number of samples so far, starting at 1. But you probably don't actually want the mean of the signal, because if you did then you wouldn't be asking about slope, and you're probably smart enough to know how to do an average. So what do you really want? The mean "for a while"? How long is "a while"? What made you decide that "a while" is long enough? And do you really want the slope? Why? Normally, the way that you go about solving this in a structured manner is to decide what the structure of your noise and your signal are (i.e., is it a constant, is it a ramp, if it is moving what is the _mathematical_ description of its motion, etc.), then you use that knowledge to design a filter that has the best chance of getting the right answer. Anything else is just tossing rocks in the dark, and hoping you'll hit something tasty. -- My liberal friends think I'm a conservative kook. My conservative friends think I'm a liberal kook. Why am I not happy that they have found common ground? Tim Wescott, Communications, Control, Circuits & Software http://www.wescottdesign.com
Reply by ●March 15, 20122012-03-15
what Tim said. (faster response with higher cutoff frequency, whatever the Q is.) i would say this, given a fixed cutoff frequency, if you want an averaging filter and you don't want the filter to ring or overshoot in the step response, it will be *slower* than a Butterworth. it will have two coincident real poles, what we call "critically damped". i think the Butterworth has a Q of sqrt(1/2) and the fastest unringing 2nd-order has a Q of 1/2. i'm thinking of an all-pole (in the s-plane) 2nd-order filter. you could make it a little faster than a critically damped LPF and a tiny amount of overshoot in the step response might be negligible. so, cutoff frequency and Q. them's the parameters you get to play with. -- r b-j rbj@audioimagination.com "Imagination is more important than knowledge."
Reply by ●March 15, 20122012-03-15
On Thu, 15 Mar 2012 19:31:23 -0400, robert bristow-johnson wrote:> what Tim said. (faster response with higher cutoff frequency, whatever > the Q is.) > > i would say this, given a fixed cutoff frequency, if you want an > averaging filter and you don't want the filter to ring or overshoot in > the step response, it will be *slower* than a Butterworth. it will have > two coincident real poles, what we call "critically damped". i think > the Butterworth has a Q of sqrt(1/2) and the fastest unringing 2nd-order > has a Q of 1/2. i'm thinking of an all-pole (in the s-plane) 2nd-order > filter. > > you could make it a little faster than a critically damped LPF and a > tiny amount of overshoot in the step response might be negligible. > > so, cutoff frequency and Q. them's the parameters you get to play with.Actually, I would dis-recommend any IIR low-pass, unless the signal and noise just happen to demand it (i.e., the signal spectrum is 2nd-order low-pass, the noise is white, and you don't have any particular need to be prompt at startup). -- My liberal friends think I'm a conservative kook. My conservative friends think I'm a liberal kook. Why am I not happy that they have found common ground? Tim Wescott, Communications, Control, Circuits & Software http://www.wescottdesign.com
Reply by ●March 15, 20122012-03-15
>On Thu, 15 Mar 2012 15:36:07 -0500, andrew66 wrote: > >> I have a signal that generally lasts about 6 seconds and has a lot of >> Low-Frequency, < 10 Hz, noise in it. I would like to use a filter to >> obtain a reliable answer for the mean value of the signal, as soon as >> possible following the start of the data stream. >> >> I use a 2nd order discrete-time Low Pass Butterworth Filter with a 10Hz>> sample rate, and I can change the cutoff frequency on the fly, via a >> lookup table that has the filter coefficients in it. >> >> The output of the 0.2Hz cutoff version of this filter is smooth and I >> can readily calculate a slope or average value from it. But, it takes >> too long to respond to the input signal. >> >> A lowpass filter with a higher cutoff frequency, on the other hand,such>> as a 1 Hz cutoff version, responds more quickly of course, but it is >> more difficult to pick an average slope or value from it. >> >> My question is, are there DSP filtering techniques that would be more >> effective for this application? By “more effective”, I simply meanit>> can filter out the low noise but do it more quickly. For example, do >> you know what they use for a rumble filter in real-time audio >> applications? > >Yes maybe, and no. > >Yes there are DSP filtering techniques that might be more effective for >this, but you haven't said what your constraints are. And no, I don't >know what 'they' use for a rumble filter in real-time audio applications. > >You talk about getting a mean value, but you also talk about getting >average and slope. And, you haven't said anything about the expected >characteristics of the signal, or the noise. > >To answer your question _as stated_, without those troublesome >implications of more depth, and _assuming_ that the noise is purely >additive and of relatively high frequency, the best estimate of the mean >value is a moving average filter: add up all the samples so far, divide >by the number of samples, and there you are. > >You can speed this up by using a recursive formula: > >x_mean = (x - x_mean)/n > >where x is your signal, x_mean is your estimate of the signal mean, and n>is the number of samples so far, starting at 1. > >But you probably don't actually want the mean of the signal, because if >you did then you wouldn't be asking about slope, and you're probably >smart enough to know how to do an average. > >So what do you really want? The mean "for a while"? How long is "a >while"? What made you decide that "a while" is long enough? > >And do you really want the slope? Why? > >Normally, the way that you go about solving this in a structured manner >is to decide what the structure of your noise and your signal are (i.e., >is it a constant, is it a ramp, if it is moving what is the >_mathematical_ description of its motion, etc.), then you use that >knowledge to design a filter that has the best chance of getting the >right answer. > >Anything else is just tossing rocks in the dark, and hoping you'll hit >something tasty. > >-- >My liberal friends think I'm a conservative kook. >My conservative friends think I'm a liberal kook. >Why am I not happy that they have found common ground? > >Tim Wescott, Communications, Control, Circuits & Software >http://www.wescottdesign.com >Thank you for your reply and excellent questions. I would like to begin my response with a more thorough definition of the signal. I am sampling the voltage output of a ratiometric bridge sensor. Over the 6 second sample window, the base signal starts at about 500mV for the first 1 - 2 seconds, when it jumps to 1 - 2 V, and then steadily ramps toward 2 - 4 V for the remainder of the window. The ramp slope varies from acquisition to acquisition. During this time, there is significant LF ringing noise present: 1/2 to 3 Hz noise bandwidth, and the p-p noise amplitude can be as much as 500 mV. The reason I mentioned slope, AND mean, AND average, is that I would take any of these that would yield a reliable measurement. If the measurement time window is large, say, half the sample window, then I would take either the voltage slope between the beginning and end times, or the average, whichever is most reliable. If I could find a way to make the measurement time window small, then it is more likely to be a singular value, or a mean that I will take. But, I can't narrow the measurement time window down without an aggressive filter that is really smoothing out the data. So, as you suggested, I have not been able to converge on a measurement method (slope vs average) nor a time window that is reliable. In terms of constraints, in the future I can upgrade my hardware to accomodate any filter that will do the job. The relatively pitiful ARM MCU I am now using is rather limited in the discrete time filters that I can implement in it. Your idea of a moving average filter is intriguing, and indeed I have tried something similar in a spreadsheet simulation. When I started the simulation with an expected value or a seed as you suggested, it was quite responsive. But, in terms of error I could not get my model to improve on the LP filter that I'm already using. Perhaps I need to reconsider a moving average filter. Thank you again for your thoughtful reply, and any other thoughts you might be able to offer.
Reply by ●March 15, 20122012-03-15
> >what Tim said. (faster response with higher cutoff frequency, whatever >the Q is.) > >i would say this, given a fixed cutoff frequency, if you want an >averaging filter and you don't want the filter to ring or overshoot in >the step response, it will be *slower* than a Butterworth. it will have >two coincident real poles, what we call "critically damped". i think >the Butterworth has a Q of sqrt(1/2) and the fastest unringing 2nd-order >has a Q of 1/2. i'm thinking of an all-pole (in the s-plane) 2nd-order >filter. > >you could make it a little faster than a critically damped LPF and a >tiny amount of overshoot in the step response might be negligible. > >so, cutoff frequency and Q. them's the parameters you get to play with. > >-- > >r b-j rbj@audioimagination.com > >"Imagination is more important than knowledge." > > >Thank you, I will study up and refresh my memory on all-pole filters.
Reply by ●March 15, 20122012-03-15
On 3/15/12 9:00 PM, andrew66 wrote:> > Thank you, I will study up and refresh my memory on all-pole filters.in the s-plane, it's simple. the impulse response is the sum of two decaying exponentials. they could be either two complex-conjugate exponentials (with negative real part in the pole) or two with arbitrary real poles. starting with Q low, as Q increases this pair of real poles move together on the negative real axis to a point dependent on the resonant frequency. that happens when Q = 1/2 and is critically damped. then, as Q increases beyond 1/2, the two poles move out from the real axis as complex conjugate poles and the step response will have a little overshoot, but it will settle on a step response converging to 1. -- r b-j rbj@audioimagination.com "Imagination is more important than knowledge."
Reply by ●March 15, 20122012-03-15
On 3/15/12 4:36 PM, andrew66 wrote:> do you > know what they use for a rumble filter in real-time audio applications? >do you mean a 1/f noise filter? a filter that turns white noise generated by a random number generator into pink noise or "flicker noise" which has power spectrum that is proportional to 1/f. of course, it can't be 1/f at DC. so you approximate 1/f real good for a range of frequencies and then let the gain at low frequencies level off to some finite value for f=0. that gain might not be 1 if you want the filter to preserve power level for other broadband signals (like a white input). it will be greater than 1 for low frequencies and less than 1 at high frequencies so, at some frequency f0, it will have a unity gain. anyway, if it *is* about the pinking filter with |H(f)|^2 = f0/f there is http://www.firstpr.com.au/dsp/pink-noise/ but right now it doesn't seem to work. or is somekinda anti-rumble high pass filter? then forget about everything i just said. -- r b-j rbj@audioimagination.com "Imagination is more important than knowledge."
Reply by ●March 16, 20122012-03-16
On 3/15/12 10:21 PM, robert bristow-johnson wrote:> On 3/15/12 4:36 PM, andrew66 wrote: > > if it *is* about the pinking filter with > > |H(f)|^2 = f0/f > > there is http://www.firstpr.com.au/dsp/pink-noise/ but right now it > doesn't seem to work.got wayback machine to work: http://web.archive.org/web/20110723091646/http://www.firstpr.com.au/dsp/pink-noise/ -- r b-j rbj@audioimagination.com "Imagination is more important than knowledge."
Reply by ●March 16, 20122012-03-16
andrew66 <andrew.meier@n_o_s_p_a_m.air-weigh.com> wrote: (snip)> My question is, are there DSP filtering techniques that would be more > effective for this application? By �??more effective�??, I simply mean it > can filter out the low noise but do it more quickly. For example, do you > know what they use for a rumble filter in real-time audio applications?I don't know about real-time audio, but my audio amplifier has a switchable 2nd order 18Hz filter. That might be more for warped records than rumble, though. -- glen
