63Hz bandpass filter to remove dithering and other noise

I have very little DSP experience and knowledge. If you are irritated by
newbies and possible dumb questions, please skip the rest of my post.

I have a 63Hz sine wave signal with dithering noise on it that I would like
to remove. I need some guidance where to look for the best type of filter
to remove this dithering noise as well as some slow changing DC offsets.
The 63Hz sine wave is modulated on top of a square wave. I am only
interested in the amplitude of the sine wave modulation.

I tried using a brick wall FFT filter in Excel. When I selected a few
frequency bins either side of my main frequency, I got a lot of distortion
that I assume was caused by the Gibbs effect if I understand some of the
other posts correctly. When I only chose the main frequency bin, the signal
I got out was a pure sine wave as expected but it had a much smaller
amplitude than the signal with noise.

I then tried using a Hamming low pass filter as well as a Chebyshef low
pass filter in Matlab to try to first remove the dithering noise. In both
cases the outputs had very distorted sine waves that looks like what I
understand ringing to be. I believe I must have used wrong settings for the
control variables of the filters.

Any guidance to a website that would explain the process to create the
right digital filter, in terminology that a beginner can understand, would
be very welcome.

	 

_____________________________		
Posted through www.DSPRelated.com

On 10.11.14 15:37, Vicsafric wrote:
> I have very little DSP experience and knowledge. If you are irritated by
> newbies and possible dumb questions, please skip the rest of my post.
>
> I have a 63Hz sine wave signal with dithering noise on it that I would like
> to remove. I need some guidance where to look for the best type of filter
> to remove this dithering noise as well as some slow changing DC offsets.
> The 63Hz sine wave is modulated on top of a square wave. I am only
> interested in the amplitude of the sine wave modulation.
>
> I tried using a brick wall FFT filter in Excel. When I selected a few
> frequency bins either side of my main frequency, I got a lot of distortion
> that I assume was caused by the Gibbs effect if I understand some of the
> other posts correctly. When I only chose the main frequency bin, the signal
> I got out was a pure sine wave as expected but it had a much smaller
> amplitude than the signal with noise.
>
> I then tried using a Hamming low pass filter as well as a Chebyshef low
> pass filter in Matlab to try to first remove the dithering noise. In both
> cases the outputs had very distorted sine waves that looks like what I
> understand ringing to be. I believe I must have used wrong settings for the
> control variables of the filters.
>
> Any guidance to a website that would explain the process to create the
> right digital filter, in terminology that a beginner can understand, would
> be very welcome.


There is not enough information to make a sensible suggestion.

What is the frequency range of the dithering noise?

What is your modulation: amplitude, frequency/phase or both?

What is the frequency range of interest in the modulating signal.

What is minimum / maximum modulation depth (or index in FM/PM)?

----

Please understand that modulation brings new frequency components
on sides of the carrier signal. To filter successfully, enough of
these components must be passed and enough of the noise rejected.

-- 

Tauno Voipio

Hi,

I'm not disagreeing with Tauno, but you could try this:
- calculate the frequency f for each fft bin
- calculate c(f) = exp(abs(f-63 Hz)^2 / alpha)
- take fft; multiply bin-by-bin with c(f); take ifft
- experiment with alpha until the result looks right

This isn't optimal and I'm not advertising exp() as a practical bandpass
response in general, but Excel can do this.

Alternatively, zeroing all FFT bins that are smaller than (for example)
0.1*peak is crude but might do the job with even less math.	 

_____________________________		
Posted through www.DSPRelated.com

On Mon, 10 Nov 2014 07:37:05 -0600, Vicsafric wrote:

> I have very little DSP experience and knowledge. If you are irritated by
> newbies and possible dumb questions, please skip the rest of my post.

We try to welcome newbies, although you have to accept that sometimes you 
need to slog through hard math.

Answer Tauno's questions, and we should be able to offer suggestions.

-- 

Tim Wescott
Wescott Design Services
http://www.wescottdesign.com

On 10.11.14 20:45, Tim Wescott wrote:
> On Mon, 10 Nov 2014 07:37:05 -0600, Vicsafric wrote:
>
>> I have very little DSP experience and knowledge. If you are irritated by
>> newbies and possible dumb questions, please skip the rest of my post.
>
> We try to welcome newbies, although you have to accept that sometimes you
> need to slog through hard math.
>
> Answer Tauno's questions, and we should be able to offer suggestions.
>

And I forgot one question: Sample rate?

-- 

-TV

On 11/10/2014 8:37 AM, Vicsafric wrote:
> I have very little DSP experience and knowledge. If you are irritated by
> newbies and possible dumb questions, please skip the rest of my post.
>
> I have a 63Hz sine wave signal with dithering noise on it that I would like
> to remove. I need some guidance where to look for the best type of filter
> to remove this dithering noise as well as some slow changing DC offsets.
> The 63Hz sine wave is modulated on top of a square wave. I am only
> interested in the amplitude of the sine wave modulation.
>
> I tried using a brick wall FFT filter in Excel. When I selected a few
> frequency bins either side of my main frequency, I got a lot of distortion
> that I assume was caused by the Gibbs effect if I understand some of the
> other posts correctly. When I only chose the main frequency bin, the signal
> I got out was a pure sine wave as expected but it had a much smaller
> amplitude than the signal with noise.
>
> I then tried using a Hamming low pass filter as well as a Chebyshef low
> pass filter in Matlab to try to first remove the dithering noise. In both
> cases the outputs had very distorted sine waves that looks like what I
> understand ringing to be. I believe I must have used wrong settings for the
> control variables of the filters.
>
> Any guidance to a website that would explain the process to create the
> right digital filter, in terminology that a beginner can understand, would
> be very welcome.

I'm not as expert as some of these other posters, but that might be an 
advantage in some respects for your problem.  I think the issue is more 
of an unreasonable expectation.  But I'm not sure what you are looking 
for so I may be mistaken.

The context of dithering I have seen is either when samples have random 
noise added, such as a delta-sigma converter, or you may be referring to 
a dithering of the frequency of your signal.

If you have dithering noise added to your signal, then that can be 
filtered out easily with a bandpass filter.  If the frequency of your 
signal is being dithered, then I think a filter will be much less 
effective.

Random sample dithering spreads the noise across the full bandwidth of 
your sample rate.  A bandpass filter can remove much of that noise. 
Dithering the frequency of the signal spreads the signal over a 
relatively narrow range of the available spectrum making it much harder 
to remove.

Is any of this on target with your application?

BTW, if you had ringing on a sine wave signal, what did that look like?

-- 

Rick

"Vicsafric" <102359@dsprelated> writes:
> [...]
> I then tried using a Hamming low pass filter as well as a Chebyshef low
> pass filter in Matlab to try to first remove the dithering noise. In both
> cases the outputs had very distorted sine waves that looks like what I
> understand ringing to be. I believe I must have used wrong settings for the
> control variables of the filters.

What duration is your sine wave? Generally, the narrower the filter you
use, the longer you have to wait for startup transients to die out. 

Otherwise, linear system theory predicts there should be nothing but
static amplitude and phase changes to your sine wave.
-- 
Randy Yates
Digital Signal Labs
http://www.digitalsignallabs.com

Thank you all very much for all the replies. It has guided me to look again
at what I actually need as an output and what I am putting into the
filters.

It is possible that I have an unreasonable expectation of what is
achievable with a digital filter. The dithering noise I want to remove is
random noise added to my input signal by a delta-sigma analogue to digital
converter. This definitely has a much higher frequency than my modulation
frequency. I find lots of places where it is stated that this dithering
noise is easily removed by a low pass or band pass filter. I had to
conclude that I was making some basic mistake when I tried to implement the
low pass filters in Matlab which then caused the distorted output from the
filters. That was why I asked for help here.

I had another carefull look at my Matlab plots and filter output data. I
can see from my Matlab pwelch (power spectral density estimate) plot that
the low pass filters definitely removes more than 30dB of the higher
frequency power. This seems to prove that these filters are working.

The output signal that I believed to be "distorted" is a sine wave (without
the dithering noise) which I now realise has a much lower frequency
modulated on its peak to peak amplitude. I realise now that this is still
there because I did not use a band pass filter.

I suspect that as soon as I figure out how to implement a bandpass filter
in Matlab the final output should be usable data.

I appreciate all the suggestions and help. I apologise if I wasted your
time. I just did not have any idea what to look for to get a solution.

Now I will add the information I was asked for just in case somebody wants
to know.

I do not see any frequency range on the MCP3903 datasheet regarding the
dithering noise of the delta-sigma analogue to digital converter.

The modulation frequency is 63Hz with an amplitude that makes out 66% of
the maximum level. The actual voltages received change but when the peak to
peak modulation is 2V, the maximum voltage of the modulation is at 3V and
the minimum at 1V.

The modulation frequency does not change. I suspect I might be using wrong
terminology when I talk about the modulation frequency. I switch on a blue
LED and I change the light intensity according to a calculated sine wave
ampliude at a frequency of 63 Hz.

I sample my voltages at 3.33 KHz. The duration of the light pulse is 75ms
during which time I take 250 measurements.




>On 11/10/2014 8:37 AM, Vicsafric wrote:
>> I have very little DSP experience and knowledge. If you are irritated
by
>> newbies and possible dumb questions, please skip the rest of my post.
>>
>> I have a 63Hz sine wave signal with dithering noise on it that I would
like
>> to remove. I need some guidance where to look for the best type of
filter
>> to remove this dithering noise as well as some slow changing DC
offsets.
>> The 63Hz sine wave is modulated on top of a square wave. I am only
>> interested in the amplitude of the sine wave modulation.
>>
>> I tried using a brick wall FFT filter in Excel. When I selected a few
>> frequency bins either side of my main frequency, I got a lot of
distortion
>> that I assume was caused by the Gibbs effect if I understand some of
the
>> other posts correctly. When I only chose the main frequency bin, the
signal
>> I got out was a pure sine wave as expected but it had a much smaller
>> amplitude than the signal with noise.
>>
>> I then tried using a Hamming low pass filter as well as a Chebyshef low
>> pass filter in Matlab to try to first remove the dithering noise. In
both
>> cases the outputs had very distorted sine waves that looks like what I
>> understand ringing to be. I believe I must have used wrong settings for
the
>> control variables of the filters.
>>
>> Any guidance to a website that would explain the process to create the
>> right digital filter, in terminology that a beginner can understand,
would
>> be very welcome.
>
>I'm not as expert as some of these other posters, but that might be an 
>advantage in some respects for your problem.  I think the issue is more 
>of an unreasonable expectation.  But I'm not sure what you are looking 
>for so I may be mistaken.
>
>The context of dithering I have seen is either when samples have random 
>noise added, such as a delta-sigma converter, or you may be referring to 
>a dithering of the frequency of your signal.
>
>If you have dithering noise added to your signal, then that can be 
>filtered out easily with a bandpass filter.  If the frequency of your 
>signal is being dithered, then I think a filter will be much less 
>effective.
>
>Random sample dithering spreads the noise across the full bandwidth of 
>your sample rate.  A bandpass filter can remove much of that noise. 
>Dithering the frequency of the signal spreads the signal over a 
>relatively narrow range of the available spectrum making it much harder 
>to remove.
>
>Is any of this on target with your application?
>
>BTW, if you had ringing on a sine wave signal, what did that look like?
>
>-- 
>
>Rick
>	 

_____________________________		
Posted through www.DSPRelated.com

On 11/11/2014 9:19 AM, Vicsafric wrote:
> Thank you all very much for all the replies. It has guided me to look again
> at what I actually need as an output and what I am putting into the
> filters.
>
> It is possible that I have an unreasonable expectation of what is
> achievable with a digital filter. The dithering noise I want to remove is
> random noise added to my input signal by a delta-sigma analogue to digital
> converter. This definitely has a much higher frequency than my modulation
> frequency. I find lots of places where it is stated that this dithering
> noise is easily removed by a low pass or band pass filter. I had to
> conclude that I was making some basic mistake when I tried to implement the
> low pass filters in Matlab which then caused the distorted output from the
> filters. That was why I asked for help here.

Have you looked at the signal from a real part?  Normally this noise is 
filtered by the ADC.  In fact the whole point of a D-S ADC is to "shape" 
the noise to the high end of the spectrum where it is more easily 
filtered... and then they do that in the ADC.


> I had another carefull look at my Matlab plots and filter output data. I
> can see from my Matlab pwelch (power spectral density estimate) plot that
> the low pass filters definitely removes more than 30dB of the higher
> frequency power. This seems to prove that these filters are working.
>
> The output signal that I believed to be "distorted" is a sine wave (without
> the dithering noise) which I now realise has a much lower frequency
> modulated on its peak to peak amplitude. I realise now that this is still
> there because I did not use a band pass filter.

I guess my question is what are you using as an input signal?  Did you 
capture some data from your ADC?  How much noise do you have relative to 
the signal?


> I suspect that as soon as I figure out how to implement a bandpass filter
> in Matlab the final output should be usable data.
>
> I appreciate all the suggestions and help. I apologise if I wasted your
> time. I just did not have any idea what to look for to get a solution.
>
> Now I will add the information I was asked for just in case somebody wants
> to know.
>
> I do not see any frequency range on the MCP3903 datasheet regarding the
> dithering noise of the delta-sigma analogue to digital converter.

That's because they should remove enough that you won't care anymore. 
There are specs on

Look at the block diagram in the data sheet on page 2.  See the part 
labeled "Δ-Σ modulator"?  Right after that is a low pass filter that 
should be removing the dithering noise along with the quantization 
noise.  On page 4 they list SINAD as 80 dB minimum and SNR as 80 dB 
minimum.  That tells you how much noise you will have with a close to 
full scale signal.  Is that good enough?


> The modulation frequency is 63Hz with an amplitude that makes out 66% of
> the maximum level. The actual voltages received change but when the peak to
> peak modulation is 2V, the maximum voltage of the modulation is at 3V and
> the minimum at 1V.

I'm not following this at all.  Perhaps you can explain what your 63 Hz 
signal is, I don't think you have done that.  What is being modulated on 
what and with what type of modulation?


> The modulation frequency does not change. I suspect I might be using wrong
> terminology when I talk about the modulation frequency. I switch on a blue
> LED and I change the light intensity according to a calculated sine wave
> ampliude at a frequency of 63 Hz.

Are you saying the LED intensity varies at 63 Hz?


> I sample my voltages at 3.33 KHz. The duration of the light pulse is 75ms
> during which time I take 250 measurements.

So you are sampling the instantaneous voltage at each point in time, 
right?  Or are you trying to determine the amplitude of the envelope of 
the 63 Hz sine wave, as if the 63 Hz sine wave were an AM carrier?

-- 

Rick

>
>Have you looked at the signal from a real part?  Normally this noise is 
>filtered by the ADC.  In fact the whole point of a D-S ADC is to "shape" 
>the noise to the high end of the spectrum where it is more easily 
>filtered... and then they do that in the ADC.
>
I am using real data from measurements with the MCP3903 AFE. I have a
functional system. I am only using Matlab to test the filter designs before
trying to implement the best one in software in the system. The dithering
noise is present on the final data that comes out of the ADC. On my latest
Matlab filter tests I do not see the dithering noise any more after it has
gone through the low pass filters.

>
>I guess my question is what are you using as an input signal?  Did you 
>capture some data from your ADC?  How much noise do you have relative to 
>the signal?
>
I am developing a fluorescence sensor. I shine blue light on algae and
measure the amount of red light it emits. The red light is converted to
voltages by 3 light-to-voltage sensors. These voltages are then amplified
in the AFE before being converted by the internal ADC. At very low algae
concentrations there is very little red light and consequently very small
voltages. At these concentrations the noise makes up about 15% of the
signal amplitude. 

>
>That's because they should remove enough that you won't care anymore. 
>There are specs on
>
>Look at the block diagram in the data sheet on page 2.  See the part 
>labeled "Δ-Σ modulator"?  Right after that is a low pass filter that 
>should be removing the dithering noise along with the quantization 
>noise.  On page 4 they list SINAD as 80 dB minimum and SNR as 80 dB 
>minimum.  That tells you how much noise you will have with a close to 
>full scale signal.  Is that good enough?
>

I have no idea how to use these SINAD and SNR values to calculate the
outputs in my system. The measurement data I have does however show that it
all works well enough except at the very low light levels where there is a
lot of overlap between signals due to the noise.

>
>> The modulation frequency is 63Hz with an amplitude that makes out 66%
of
>> the maximum level. The actual voltages received change but when the peak
to
>> peak modulation is 2V, the maximum voltage of the modulation is at 3V
and
>> the minimum at 1V.
>
>I'm not following this at all.  Perhaps you can explain what your 63 Hz 
>signal is, I don't think you have done that.  What is being modulated on 
>what and with what type of modulation?
>

If I understand the Wikipedia page correctly I am not using "modulation"
since I do not have a carrier frequency. I switch on a LED at 66% of its
maximum intensity for 75ms. While it is on I increase and decrease the
intensity from the 66% level by 33% to form a 63Hz sine wave with a DC
offset. The output looks similar to the (4 sin 7 theta/ 7 pi) animation on
the Wikipedia page on Fourier series, except that the sine wave amplitude
is much bigger.
  http://en.wikipedia.org/wiki/Fourier_series

>
>> The modulation frequency does not change. I suspect I might be using
wrong
>> terminology when I talk about the modulation frequency. I switch on a
blue
>> LED and I change the light intensity according to a calculated sine
wave
>> ampliude at a frequency of 63 Hz.
>
>Are you saying the LED intensity varies at 63 Hz?
Yes

>
>
>> I sample my voltages at 3.33 KHz. The duration of the light pulse is
75ms
>> during which time I take 250 measurements.
>
>So you are sampling the instantaneous voltage at each point in time, 
>right?  Or are you trying to determine the amplitude of the envelope of 
>the 63 Hz sine wave, as if the 63 Hz sine wave were an AM carrier?

Yes, I am sampling the voltage at each moment. 

The amount of red light I measure at each moment gives me an indication of
the amount of algae in the water. I know how much blue light I am emitting.
The algae converts the blue light to a small amount of red light. The
amount of red light then is an indication of how much algae there is in the
water that can emit red light. My final calculation comes down to the
amount of red light at any moment compared to the amount of blue light at
that same moment.

The theory is that if the blue light intensity is changed with a sine wave
it will enable the accurate recovery of the red light intensity (voltage)
by removing the noise.


>
>-- 
>
>Rick
>	 

_____________________________		
Posted through www.DSPRelated.com