In a previous post, I asked about removing a DC spike from a spectrum generated by a quadrature receiver.  I have been able to remove the DC spike but the spike is actually broad going over a range of about 80Hz on either side of DC.  So basically about 7 or 8 bins on either side of the "0" bin.

Now, if I remove the antenna input to the receiver, this part of the spectrum appears always the same at a given gain setting and frequency. In addition, adding an antenna will show pretty well exactly the same peaks.  They appear normally distributed around 0. Fortunately, in my application (a panadapter) the frequency and gain are always the same.

I have an idea that I can work directly with the spectrum data itself, not needing any filters in the time domain or anything like that.  I was wondering what the community thinks of my concept.  Here goes....

I provide a user interactive utility that allows one to establish the average height and width of this "normally" distributed "peak".  The values would be obtained by a set of cursors that one can move on the screen.  Once set,  a correction array would be established that would use the average of the peak values by bin minus the average spectrum value.  This would then be subtracted from the signal itself at these bins in real time.

So does this sound reasonable?

Thanks, Tom

DC leakage is often the price for the simplicity of quadrature receiver versus IF. In the real world the leakage would affect not only DC carrier itself, but neighbour bins as well, and that's exactly what you observe. Removing just DC with sort of bandstop filter might be hard, as that might require very narrow and very steep notch. On the other hand, that is fixing hardware imperfection. Since you know that, you may try to improve your hardware a little. Most of chips in you receive path such as IQ demodulators and ADCs have some digital interface for calibration. Applying zero signal to receiver input, you may observe some DC is still there. Then you adjust receiver chain in a way, that reduces DC leakage to acceptable level and proceed with normal operation. From my experience, such a calibration might be required every time receiver changes frequency and/or gain.

It'll probably work better to subtract the power (energy, take your pick) of each bin.

In a broad sense, it doesn't matter if you filter in time or in frequency.  Only the details matter in implementation.

If all you did was zero out the zero-frequency component then what you describe should be expected.  Choose a different frequency domain filter and see how that works.

That would probably work, but I can think of easier ways.

My SDR front end is the AMSAT UK Funcube Dongle Pro+, a unit sold into the ham radio market. It costs around $150 and samples at 192 kHz with 16 bits/sample.

I remove DC with a simple exponential high pass filter that works like this.

For each sample in a 4096-sample block, subtract the current estimates of the I and Q DC offsets. Average the now-DC-adjusted samples over the block (sum them up and divide by the sample count) and add some small fraction of the average to the current DC offset estimates. Move to the next block and repeat.

I'm currently using 0.05 as my "small fraction", which is a totally empirical value.

If the DC offset estimates are correct, there will be no DC in the samples after adjustment so they will average to zero and no change will be made to the estimates. If an estimate is low, the average (DC) in a block will be slightly positive, and that will gradually bring up the estimate. If an estimate is high, it will gradually decrease.

This produces a very sharp notch around DC so there's little effect on low frequencies. But it does not remove the noise "hump" that you report; I see it too. I found that it can usually be covered up by suitably adjusting the analog gains ahead of the A/D converter but I also try to keep my (usually narrowband) signals away from 0 Hz whenever I have a choice. When tuning the radio to a new frequency, I'll generally set the first IF at 48 kHz (Fs/4), i.e., by tuning the (analog) LO in the tuner 48 kHz below the desired signal. Fine tuning adjusts only the second (digital) LO, which means the first IF will vary, but is very quick and phase-continuous. Only if I try to tune outside the passband will I retune the first (analog) LO to shift it to a new frequency range.

Hi this is interesting but I don't think I quite follow. So, for my first block, I start with .05, subtract this from each sample,  and then average the block. How do I determine "some small fraction of the average" to add to each sample again? 

Thanks.

Sorry, I guess my explanation was confusing. I'll try to express it in simplified C. Hopefully I haven't made a mistake in the process.

complex float samples[4096]; // Input sample block

complex float DC_correction;

complex float average;

int i;

average = 0;

for(i=0;i<4096;i++){

samples[i] -= DC_correction;

average += samples[i];

}

average /= 4096;

DC_correction += 0.05 * average;

(repeat for next block using the now-updated value of DC_correction)

Does this help?

If you want the filter to respond more quickly to changes in DC offset at the expense of removing more signal at very low frequencies, increase that constant from 0.05. That was just an empirically determined value for my particular system.

Oh, another thought. You might experiment to see if changes to the analog configuration ahead of the A/D, especially gain changes, cause transients in the DC signal. If so, you might want to modify this code to react quickly when you expect them, and then average over a longer period after things have settled down.

Hi,

This is perfect.  Only question is if you receive the I and Q samples independently which is the case for me. My limited DSP knowledge suggests that you could apply this independently to I and Q.  Am I correct? 

This is great.Thanks

Hi,

In my code I have access to the I and Q samples independently at this point. So, I have done what you suggested with independent I and Q corrections for the DC.

It makes almost no difference at all(!) to the DC spike. I am beginning to think that there is something else going on here.  I tried factors from 0.001 to 2.5 and the only thing I see is that the DC Spike getting larger when the empirical correction factor gets over 1.8. When the value is at .05 or even much less, I see a very small reduction in the DC spike.

Now, something else must be happening.  Is a typical DC spike constant in amplitude?  Mine varies quite a bit with time.

Something is not right

Yeah, something is not right. Did I make a sign error in my pseudo-code? Hmm...no, I don't think so. Can't think of anything else other than to check that you're computing your averages over blocks. I do it in two steps, first by taking the average (DC) of the current block and then averaging that into a smoothed average but that's not the only way to do that. But you could do it in a sample-by-sample basis if you use a very small scale factor. Since the DC offset doesn't change very fast (otherwise it wouldn't be DC!) you definitely want to average it over a fairly long period.

BTW, one advantage of this method over what you're doing with a FFT is that you can make the notch much narrower this way. As you know, the FFT frequency bin size depends inversely on the length of the FFT, so to get a very narrow notch you'd have to use a very long FFT that would also introduce a lot of delay. With a small gain factor this simple exponential smoother can operate over quite a few FFT blocks and therefore produce a much narrower notch.

But remember it's always best to keep your signal away from DC if you can.

Hi

I think I may have found where I am having an issue. I compared my spectrum to that of HDSDR, and I found that there was no spike at DC as in my setup even if I turned off the automatic DC correction. So I looked at the gain settings in use and saw that they were much higher than mine, by about 36dB. So I raised the gain and found that at higher gains the DC spike did indeed disappear although I started getting many stationary spikes in the spectrum. There are a few things to configure there but it looks like this might be where I am having an issue. Even the hump is gone.

So in summary I think the methodology you gave me works, but I am just not allowing enough real signal to be available to overcome the offset. Does this make sense?

Thank you!

How big an offset were you seeing? I was seeing typically 120-130 counts in 16-bit samples on the AMSAT UK Funcube Dongle Pro+. Be careful you don't overdrive the A/D; I'm not sure of the ideal level but I typically keep the RMS signal level at about -15 dBFS or less.

Ok, this is probably where my issue is. The offset was about 15dB above the baseline and the baseline was - 90dBm or even more.  But the scales are all relative.  I am comparing mine to a commercial panadapter.  Now if I raise the overall signal by 40dBm the offset and hump is gone.  But as I said there are other stationary spikes that appear sometimes.  But the SDRPlay has a LNA that is automatically enabled it seems at some values of the gain. If I do it exactly as HDSDR it seems good sometimes.  I just can't as yet do it the right way.  I need to study the API a little more.

None the less, your help lead me in the right direction!  Not only that, the algorithm you posted will surely help others as well.  In fact I can use it for a KX3 that has a small DC offset which is probably a true offset as opposed to that above.

Many thanks for your help!

Well I investigated all possibilities and HDSDR also shows the spike at -70dB.  In HDSDR, if I then enable DC correction it disappears as it should.  However in the case of the code above it makes no difference at all.

About the only thing I can see, is that all samples are integers between between -2047 and plus 2047.  I am subtracting the average which is not an integer.  But I wouldn't think this would matter.

Here is what the peak looks like in HDSDR and what happens to it when the gain is changed.

http://va2fsq.com/wp-content/uploads/spike.wmv

I'm at a loss here.

Success.  While I do not know why the above example didn't work this one, based on an IIR filter does!!

double wt_i = 0;
double prev_wt_i = 0;
double alpha = 0.988;  //scale factor. 
double wt_q = 0;
if (correctDC)
{
for (int i = 0; i < 16384; i++)
   {
       wt_i = iData[i] + alpha * prev_wt_i;
       iData[i] = wt_i - prev_wt_i;
       prev_wt_i = wt_i;
       wt_q = qData[i] + alpha * prev_wt_q;
       qData[i] = wt_q - prev_wt_q;
       prev_wt_q = wt_q;
    }
}

iData and qData are the samples and alpha is a scale factor.  If 1 everything at DC goes through and 0, nothing goes through.

73

8 years later, this high-pass filter method works well, if I drop alpha to 0.4 the hump disappears as the rest of the freq bin magnitudes increase to the same level. Of course there is a big hole around DC.  Is there any way to solve this without reducing frequencies around zero?  An experienced RF design engineer said the hump is caused by LO leakage, must be a way to filter that out?

I took the liberty to edit your post to highlight your code snippet, I hope you don't mind.

Is the 73 number at the end of your post a typo?

Hi Thanks!

No the 73 in ham radio lingo means Best Regards.

How did you do that for the future?

Thanks

Check this out:

https://www.dsprelated.com/thread/1114/forums-upda...

it shows how to add a code snippet to your posts.

Let me know if you have any question.

73 ;)

Note that it is also important to ensure equal gains in the I and Q channels, and that they are exactly in quadrature (90 degrees apart). The main symptom of a gain or phase imbalance is poor image rejection, i.e., if you're tuned to a signal at +48 kHz in the A/D stream, you'll also pick up a little of whatever happens to be at -48 kHz.

I balance I&Q gain by estimating the average amplitude of each channel over a block and tweaking their levels so that they are the same. I balance phase by accumulating the dot product of the I and Q samples, also averaged over a block; if the channels are in quadrature, this dot product should be close to zero because the dot product of two vectors 90 degrees apart is zero.

This breaks down if you happen to have a strong signal component (especially a carrier) very close to DC, so this is something you should try to avoid. It also avoids having it removed by your DC offset correction routine.

I have tried out this idea and it does work, but it is dependent on the average signal level over the whole spectrum. So if there are no signals, it works pretty well but with a lot of signals the average value of the spectrum is high and therefore the hump reduction is not sufficient.

Would a median value work better?  Or in other words, how do you determine the baseline average of the noise in a spectra that has a lot of peaks?

Thanks

Dear Tom, please go ahead and:

1- click on the 'thumbsup' button associated to the post(s) that you find helpful

2- click on the beer button to reward the users who helped you the most (no cost to you)

Thank you!