I'm facing a problem trying to filter 50Hz and harmonics from a signal s(t).
I've implemented several techniques like Kalman filtering, optimized FIR and IIR filters etc.
The result is not the expected one ( the main problem consist of a big signal delay due to the filter transfer function phase ).
The best solution could be a COMB filter recovering the DC band closed to 0 Hz.
Any suggestion?
Thanks a lot.
I've worked with this problem for many years (40+) in active control of power line current or voltage harmonics. I am not familiar with the "shielded cable algorithm", but I have found very satisfactory results with coherent sampling.
The idea is to phase lock the sampling clock (or interrupt frequency) onto the 50 or 60 Hz signal. A DMA transfer of the signal may provide lower jitter than a direct interrupt handler triggering the signal. There are also other software techniques for keeping the sampling jitter reasonable, depending on the processor and ADC.
A varying average 3kHz sampling rate would, for example, be phase locked to the 50Hz signal at 60x the exact frequency of the line. Always an integer multiple! Frequency locking is not very acceptable - you need to use a true phase locked loop that has low jitter. I typically use a SPLL (software PLL) loop. This avoids zero crossing detectors that have poor frequency response. The sample points should not be bouncing around the harmonics (i.e., you need low phase-jitter). I typically permit the PLL to range between 50Hz +/- 0.5 Hz for tightly regulated utility sources and +/- 5Hz for diesel generators (or worse).
I use a running comb filter achieved by subtracting the oldest sample from a circular buffer and then adding the newest sample to the buffer (running techniques require one memory location for each sample in a single power line cycle). You will get a new result for each sample.
What you do with those samples depends on what you are trying to achieve. For an average DC value, simply keep adding the newest sample to an accumulator and subtracting the oldest sample from the accumulator. All harmonics up through the 30th (for 60x sampling) will be canceled. The result will be an average voltage with a frequency response of DC through the 50Hz signal and then with the traditional single stage comb filter response up through the Nyquist.
To measure harmonics, correlate a sine and cosine wave at the proper harmonic frequency with the running samples. This will produce a real and imaginary rotating frame DC signal for each harmonic. I use these values to real time correct harmonic distortion in power inverter outputs. You can either correct the current to <1% distortion in real time or you can correct the voltage to the same (obviously not both at the same time).
If the noise comes from a single phase source, then the harmonics will start with the 3rd and go through the odd harmonics with harmonics above the 19th typically being negligible. A 3-phase source will not usually have the triplens harmonics, but will have counter-rotating pairs of harmonics. 5th paired with 7th, 11th paired with 13th, 17th paired with 19th, etc. The 5th will rotate in one direction while the 7th will rotate in the opposite direction. For this reason, traditional rotating frame methods such DQ based on Park and Clarke transforms will not be satisfactory as it assumes that all harmonics rotate in the same direction and will result in positive feedback for one harmonic and negative feedback for the paired harmonic.
This method can be used not just for feedback, but when paired with a square root function can calculate very accurate true RMS signals (square root of the sum of the individual squares of each harmonic frequency including the fundamental).
The second algorithm that I am experimenting with is a multi-rate decimation CIC filter. This method uses overlapping comb filters that requires a higher sampling rate. It has no multiplies in it, only adds and subtracts so it is very simple. It also has the advantage that like a sigma-delta ADC, it enhances the resolution of the signal significantly (not the precision - only the resolution). You can model the CIC filter in Matlab very easily with the appropriate add-ins (requires the fixed point math add-in and the filter add-in). The CIC filter is then one of the menu choices. You can add multiple zeros in between the integer harmonics to increase the attenuation significantly over a single zero comb filter.
If by chance you are actually operating a PWM inverter (couldn't tell from your answers, but the numbers are all in the proper range for this), the carrier frequency of the inverter should also be variable at some integer multiple or sub-multiple of the sampling frequency.
If any of this is interesting to you, I can provide a list of papers (and expired freely usable patents including one of mine) that document this technique.
Last month, a subset of the first algorithm was discussed by a blog on the Plexim website in the context of controlling harmonics in power quality inverters. Plex is a power electronics simulator that has advantages over Matlab and SPICE and directly competes with them. While I very briefly introduce the video, I am not associated with Plexim in any way and derive no financial benefit from it.
For those of you who perhaps aren't familiar with power line harmonics, this is not a noise signal floating on top of the desired signal. The distortions are caused by non-linear loads on 50 or 60 Hz generators or other sources including PV inverter systems.
Sometimes the goal is to measure these distortions in the voltage and/or current and report on them while at other times it is to correct one or the other by modulating the inverter itself. This is used for inverter control algorithms and active power filter cancellation algorithms. There is a great deal published on these topics and many researchers have worked on this problem since the 1960's.
The distortions used to be caused by thyristor controlled motor controllers injecting up to 30% current harmonics into the power utility. Also the early PC power supplies generated the same harmonics by diode rectifying the power line and then PWMing the rectified voltage (before the advent of unity power factor control algorithms). This is why various regulations for low distortion and unity power factor controlled power supplies were generated.
As explained before the signal s(t) to be analyzed is a random process with spectrum support
within [0..1 kHz]. Grid harmonics are summed to this signal up to the
13th. Random noise is present but negligible.
The goal is to filter the harmonics cleaning s(t) minimizing the filtering delay. The filter should have an acceptable step response ( no overshoot or sawtooth wave).
I've read this paper and used for some tests :
IEEE JOURNAL OF EMERGING AND SELECTED TOPICS IN POWER ELECTRONICS, VOL. 2, NO. 4, DECEMBER 2014
Improving Dynamic Response of Active Harmonic Compensator Using Digital Comb Filter
Muhammad Tahir, Member, IEEE, and Sudip K. Mazumder, Senior Member, IEEE
The filter seems to work properly but the step response is the following one.
That's not acceptable due to sawtooth wave.
Best regards
Are you collecting a cycle of data and then generating a response once per cycle (i.e., decimation), or are you using running algorithms that generate a new result for every new sample?
The sawtooth shape is usually seen when the sampling rate is too low or if the sampling rate is not coherent (or not harmonic as djmaguire refers to it - same thing, as far as I know). Your sampling rate (60x the fundamental) is more than fast enough so I begin to wonder if you are essentially downsampling by collecting a block of data from one powerline cycle, calculating an FFT on it to extract the harmonics, and then generating a single new response that covers a full powerline cycle. If so, the sawtooth shape may be fixed by using "running" algorithms that do not decimate from the 60x sampling.
A comb filter can easily be operated as running comb, but requires a memory storage word for each sample in the power line cycle. An FFT is not easily converted although I have an IEEE published paper on this topic - simply never understood it :-(. In lieu of a running FFT, I use a correlator (produces identical results to a single frequency DFT) to extract each harmonic of interest. If I want to control the 3rd, 5th, 7th, 11th, and 13th harmonics, then I calculate 5 DFT's or correlations and operate 5 PID (although typically only PI) controllers, one for each harmonic, to correct the output. The output of each PID loop is summed in quadrature to generate a single new PWM pulse. The reference variable for the PID controller is typically 0 so that all distortions are cancelled.
I also wonder a bit (just a little bit) if there is something wrong with your graph. The sawtooth shape is exactly the phase response shape of a comb filter...
The sawtooth shape (not true sawtooth in graph above) is also nearly identical to the unfiltered current waveform shape of a PWM inverter measured between the switch and the output filter. The slope of the falling edge is controlled by the load while the slope of the rising edge is controlled by the output filter inductance, thus they are asymmetrical.
If you are measuring a sensor voltage that represents the current and the sensor is located in between the inverter switches and the output filter, then you will have to add an anti-aliasing filter prior to sampling because a triangular shaped current (such as a PWM inverter produces) has high frequencies present that theoretically approach infinity simply as a property of triangles. When measuring the voltage signal of such an inverter, the inverter output filter inherently provides the necessary anti-aliasing filter. In practice, the waveform from a current sensor that sees the triangular shaped current from an inverter acts as a small low pass filter because most types have very limited bandwidth and are already supplying a parasitic low pass filter. All previous discussions regarding single and multi-phase harmonics of the fundamental are off, if measuring the actual switching currents prior to the output filter. That requires a different technique.
By the way, if you are operating a variable frequency motor directly from an inverter, the motor itself is a 1 pole inductor low pass filter and thus the current will have a sinusoid shape and can be easily measured, but the voltage will be a switched square or rectangular wave and will have near infinite frequencies present so will require an anti-aliasing filter (or simulated output filter) for measurement.
If you tell us a little bit more about the actual application space, we may be able to offer more concrete solutions.
This is a response to Donald's post above. I edited it to add some more information and - inexplicably - the post is jumping around! Please mentally insert it into the post marked deleted above. :)
_____________
This and your earlier explanation are quite helpful. As you surmised, I interpreted the original post as being a stock-and-standard cross-talk issue. It is interesting to learn the true nature of the problem and to learn more about this field of study.
...and I love dynamic sample rates when dealing with harmonic/tonal systems. That approach is very accepted for measurement but underutilized in control.
In my earlier response I had mentioned PLLs tied to a bank of switched capacitor (bandpass) filters. That is the (pseudo-)analog approach to a dynamically sampled comb filter. I used it many years back for the feedback control of cabin noise in small prop planes. It yielded 6dBA of noise reduction at the cruise RPM with no DSP. No processor at all, in fact.
I considered the solution more nifty than groundbreaking. ...but it serves as a good example of dynamic sampling for control.
You can try not to filter the hum from the signal, but to cancel it. To do so, you have to create an synthesized 50Hz-with-harmonics signal, like a clipped 50Hz sine wave or so and run it through an adaptive filter, then you adapt this filter using NLMS.
You have to pay attention, however, not to adapt while other signals are present, that will cause the adaption to stray...
I tried it once and it worked quite well
Hi Dimaios,
What is your sample rate?
Which harmonics are you trying to attenuate? Odd? Even? Odd and even?
How high in frequency do the harmonics extend? 5th harm, 7th harm, 15th harm?
How much delay is acceptable?
Neil
Also, what is the band of the desired signal?
The sampling rate is about 3kHz and we are trying to attenuate both odd and even harmonics.
The harmonics can extend a lot into the spectrum ( up to 700-800 Hz ).
The desired band of the signal is about 1 kHz.
The main goal is to improve the bandwidth if compared to a simple average FIR filter ( 20 ms ).
Another requirement is to attenuate more the undesired harmonics ( compared of course to the average FIR filter ).
The problem is not so easy to solve because adaptive techniques need a lot of time to converge.
I've tried to implement this algorithm:
IEEE JOURNAL OF EMERGING AND SELECTED TOPICS IN POWER ELECTRONICS, VOL. 2, NO. 4, DECEMBER 2014
Improving Dynamic Response of Active Harmonic Compensator Using Digital Comb Filter
Muhammad Tahir, Member, IEEE, and Sudip K. Mazumder, Senior Member, IEEE
... but the step response is orrible.
You can find an algorithm that has good step response in
IEEE Transactions on Industrial Electronics, Vol. 43, No. 1, 2/96,
"DSP Control of High-Power UPS Systems Feeding Nonlinear Loads". It achieves complete correction to better than 1% within a single power line cycle and starts achieving steadily improving correction after the first sample following the step change. It uses the algorithm discussed elsewhere in this thread that I wrote. I am one of the authors of this paper. We included a video of real time performance at the American Controls Conference in Chicago in 1996 and won "Best of Show". I developed this algorithm in 1989 and co-authored it with Texas A&M. The patents have expired and the algorithm is freely usable by others and is presently in production in several parts of the world to correct power line harmonics in single and three phase inverter systems.
A few questions.
Is the 50 Hz a power main signal?
Are the harmonics large compared to signal?
What is sample rate?
How many harmonics (1, 2,... all the way out to fs/2)?
What is the signal to which the 50 Hz has been added (speech, )?
What is BW of that signal?
fred
Is the 50 Hz a power main signal? Yes, but consider that frequency is not fixed but time variable. Alarm condition are signaled outside limits [49.9 - 50.1] Hz. In Italy main islands the range could be up to [49.5 - 50.5] Hz so the harmonic notch filter should be wide enough to be sure that sinusoidal noise is always filtered ( an alternative could be that the filter is adaptive but is more complex )
Are the harmonics large compared to signal?
Not always.
What is sample rate? 3 kHz
How many harmonics (1, 2,... all the way out to fs/2)? Up to 15th.
What is the signal to which the 50 Hz has been added (speech, )? Control signals for industrial control.
What is BW of that signal? Maximum 1 kHz
The best algorithm for achieving what you want (of which I am aware) is called the "shielded cable algorithm." :)
But to your question...
At first... I wasn't sure if this was offline processing... real time in software... real time in hardware... I'm guessing now that it is real time. The hardware/software question is still open.
I had a lot of questions in addition to that. The answers to fred's questions helped more than anything. I'm interested in the low end frequency of your signal of interest. ...but I am guessing we're going down to 0.
Short of properly shielding the signal wire, using differential amplification, or filtering the signal with switch-capacitor notch filters whose clock signal is driven by a PLL connected to the mains, I think that adaptive filtering is your best bet. I have a hard time believing that the rate of change of the mains is so fast as to give a properly composed adaptive filter much trouble.
Post a data set. I'd run it through an adaptive filter for you. It'll either work or it won't.
Cables in the plant are shielded. :)
At first... I wasn't sure if this was offline processing... real time in
software... real time in hardware... I'm guessing now that it is real
time. The hardware/software question is still open.
The application is real-time and elaboration is performed by a Cortex-M7 microcontroller ( firmware implements the digital filter ).
I'm interested in the low end frequency of your signal of interest. ...but I am guessing we're going down to 0.
Yes. It's important to preserve the DC value.
I have a hard time believing that the rate of change of the mains is so
fast as to give a properly composed adaptive filter much trouble.
Yes. The rate of change of the grid has a very limited dynamics.
Post a data set. I'd run it through an adaptive filter for you. It'll either work or it won't.
Now I'm not in the plant. As soon as possible I will provide the real signal for offline elaboration.
Thanks a lot for your support.
Best regards
What I see as the big issue is the same one I see all the time: insufficient defining data. It's hard for anyone to guess at a solution when the problem is unspecified.
I also find it problematic to employ a control signal that needs to maintain "accurate" DC and AC, yet reject specific AC and still be "real time" -- specify the latency requirements. DC is a hard signal to capture accurately, as the best estimate is a plain average. Does the desired AC signal consist of single or multiple frequencies? What is the tolerance? Do you need to determine the phase too? Are you sure that your signal media does not suffer from issues like noise, group delay, etc., in addition to AC hum? Could this be fixed with shielding and/or going differential? Switching from Voltage to current loop? Is the sampling harmonic with the desired signals? If not, then there may be periodic residues that affect the DC. All of these either influence the solution or the expected performance of any given solution. After all, the desired solution may not be possible with the current expectations. That does happen. A lot!
My immediate recommendation is to capture hours or days worth of your signal until you are certain you have good representative sample. Then crunch it with all your solutions, weighing the pro's and con's of each, to decide which is best. This is not a philosophical process, it's an engineering process.
Back in 1992, I had to filter out disturbances in ILS signals monitored by ground equipment far field monitor (FFM) stations. The FFM equipment would get periodic alarms for various incursions into the monitored signal's path by ground vehicles, but mostly by aircraft landing either on the runway in question or and adjacent runway, or taxiing aircraft, etc. The distortion "signature" varied by aircraft - prop planes, jet planes, large and small planes, helicopters, etc. Some of this was influenced by terrain too. What to do? Well, it started by collecting data from two particularly problematic airport's runways. An "expert" logged events by the type of traffic and aircraft while a special program captured raw signals at a rate of about 4 measurements per second in parallel. This ended up with hundreds of runway events and about a half million samples of data (about 35 hours!). I took that data to my"lab" (my desk). I first visually examined all signals in "real time" (I wrote a PC graphing program for that), and analyzed the characteristics of these events. After quickly dismissing analog filtering, I applied various filtering techniques to these. This resulted in a nonlinear filter that "looked" at the effects on the signals consistent with all runway events. The final filter had 4 "tuning" parameters to tweak it in the field. I used the collected data to simulate the filtered monitoring until no alarms occurred in the portions where none should occur, but alarms in the sections where we knew they should occur due to aircraft or vehicles parked near the signal for too long. I then first presented my filter to management then to the FAA. This took about 3 months of work, including the data collection.
I try to answer to your questions:
Does the desired AC signal consist of single or multiple frequencies?
The signal s(t) to be analyzed is a random process with spectrum support within [0..1 kHz]. Grid harmonics are summed to this signal up to the 13th. Random noise is present but negligible.
What is the tolerance?
If you are speaking about harmonics to be filtered, the fundamental frequency variation is within [ -0.1 .. 0.1 ] Hz.
The disturbance amplitude can be up to the signal one or sometimes more.
Do you need to determine the phase too?
No. Harmonics have to be filtered, phase identification is not strictly necessary.
Are you
sure that your signal media does not suffer from issues like
noise, group delay, etc., in addition to AC hum?
AC hum is the main problem.
Could this be fixed with shielding and/or going differential?
Cables are already shielded and the signal is sampled by using a differential I/O module input channel.
Switching from Voltage to current loop?
Not possible. The voltage input is mandatory due to field devices.
Is the sampling harmonic with the desired signals?
I don't understand what you mean, could you rephrase the question?
( I'm a non native speaker and something can "slip"! )
Thanks.
By "Is the sampling harmonic with the desired signals?" I meant what is often called synchronous sampling. For example, for a 30 Hz signal, a sampling rate of 1200 Hz is perfectly harmonic. The benefit is that it always captures a complete set of target frequencies. For this example, 1200/30 = 40, so any multiple of 40 samples is harmonic and always captures an integral number of cycles and therefore is always centered on one of the spectral "bins", which have sinc response. Conversely, if nonharmonic sampled, the resulting spectrum of the incomplete waveforms results in smearing with lots of harmonics. Once an artifact of any source of noise affects any desired bins, that cannot be undone and becomes part of the error (i.e. noise).
Something missing in these discussions seems to be real time datawindowing, which is meant to reduce the smearing effects of nonharmonic data on spectral bins. All windowing widens bins - e.g. if the bin width before windowing is 30 Hz, then afterward it approaches 15 Hz, so if you really need the resolution, you then need to double the number of samples in your window.
djmaguire, assuming that you are not talking about using a shielded cable, can you provide a link to this topic? I did not find any reference in a quick Google search...
I am always interested in adding a new tool to my toolbox ;-)
Thanks,
Don
I apologize. The "shielded cable algorithm" was a (bad) joke about trying to keep the signal from being contaminated in the first place by hardware methods.
I am a firm believer that the easiest signal processing is the signal processing that you don't have to do!
:-)
what you want is a periodic notch... this is a comb filter with the periodic zeros accompanied by periodic poles slightly offset from the zeros.
Same as zero packing a notch filter which is accomplished by replacing delay with multiple delays. convert (Z-1)/(Z-a) to (Z^60 -1)/(Z^60-aa). see attached
ooops, I just noticed that there is a DC component in the signal you want preserved... I'll have to modify my notch distribution... will get to it shortly...
fred
Hi dimaios.
Would a filter whose freq magnitude response looks like the following:
interest you? With an Fs = 1600 Hz sample, the filter has E = 8 notches in the positive freq range with the 1st notch located at +50 Hz. The other positive freq notches are at 150, 250, 350, etc, Hz (odd harmonics of 50 Hz). The filter structure is a tapped-delay line and it has 25 symmetrical coefficients. Also, the filter is linear phase.
hello all,
the multiple notch filters do leave a residue as the signal plus 50 Hz harmonics go through multiple notch filter. We can do some house cleaning with a median filter.
fred harris
see attached
Hi Fred,
thanks for your reply.
The approch is good but the problem f the sawtooth wave step response.
As explained before I've realized the filter proposed in the paper
IEEE JOURNAL OF EMERGING AND SELECTED TOPICS IN POWER ELECTRONICS, VOL. 2, NO. 4, DECEMBER 2014
Improving Dynamic Response of Active Harmonic Compensator Using Digital Comb Filter
Comparing your step response with the one presented in the paper you can see that are quite similar.
I think that there is an implicit requirements conflict when asking for the maximum possible frequency bandwidth and at the same time no sawtooth wave step response.
Perhaps this is a limit using the linear system elaboration. The next step will involve neural networks or sliding modes ( there are few papers treating this techniques ).
For now I've discarded the solution that use EKF and UKF because the convergence is not always garanteed.
Thanks for your support.
Best regards
Hi Rick,
that was one of the first attempts, unfortunately I've to filter both odd and even multiple of the fondamental.
Best regards
