DSPRelated.com
Forums

CW/Morse keying detection revisited

Started by John Barrett August 27, 2008
I'm working on a CW/Morse demodulator/detector/decoder that takes 8khz 16bit 
audio from any SSB reciever, and attempts to detect the state of the signal 
(key down, key up) at 700hz. The current "circuit" (all software DSP on a PC 
in C#) consists of a sin/cos converter to IQ running with 700hz sin/cos 
signals, followed by a 300 tap 50hz LP FIR on I and Q, with integrated 
decimation to a 500hz sample rate.

The FIR outputs are squared, summed, and the final output square rooted :) 
Everything to this point appears to be working, excepting that out of band 
(<650hz, >750hz) signals have a strong effect on the output, and I am more 
or less unhappy with the size of the "noise window" (difference between the 
peaks and troughs when no signal is present) given that many signals that I 
can hear clearly enough to possibly copy show very little change in the 
outputs from the DSP code, barely raising the signal out of the noise 
window.

This is my 1st cut at doing DSP, and I have no formal training, so some of 
the questions are liable to be pretty fundamental :) The code described 
above is based on the recommendations of the developer of a BPSK 
demodulator.

I need to achieve 2 goals in the next iteration of the code:
1. Fix any problems with the "demodulator" design (IQ conversion, filtering, 
and summing) to clean up the noise issues.
2. convert the design to be frequency agile -- able to detect CW at any 
frequency in the range 200-2500hz

After reading a number of articles on DSP SSB demodulators, I'm a little 
confused because some show a Hilbert Transform after the LPF on the I 
channel and a matching delay on the Q channel. I need to know if the lack of 
this step in the process could be the cause of the excessive (in my opinion) 
noise at the summing output.

I'm also confused about the LPFs after tthe IQ conversion and what exactly 
they are intended to accomplish, as I've seen designs with both LP and BP 
filters in that location. When using LP filters, is that essetially setting 
the width of the window around the center frequency defined by the feed to 
the IQ converter ?? or is it something else ??

Any help to straighten this mess out would be greatly appreciated !!

John AE5ET Barrett
Plano, TX 


"John Barrett" <john.ae5et@verizon.net> wrote in message 
news:6Y-dnXvpgonwQCnVnZ2dnUVZ_hWdnZ2d@giganews.com...
> I'm working on a CW/Morse demodulator/detector/decoder that > takes 8khz 16bit audio from any SSB reciever, and attempts to > detect the state of the signal (key down, key up) at 700hz. > The current "circuit" (all software DSP on a PC in C#) > consists of a sin/cos converter to IQ running with 700hz > sin/cos signals, followed by a 300 tap 50hz LP FIR on I and > Q, with integrated decimation to a 500hz sample rate. > > The FIR outputs are squared, summed, and the final output > square rooted :) Everything to this point appears to be > working, excepting that out of band (<650hz, >750hz) signals > have a strong effect on the output, and I am more or less > unhappy with the size of the "noise window" (difference > between the peaks and troughs when no signal is present) > given that many signals that I can hear clearly enough to > possibly copy show very little change in the outputs from the > DSP code, barely raising the signal out of the noise window.
Your detector is designed for a signal that has almost all of its energy between 650 and 750 Hz. Does it? These "out-of-band signals" that you mentioned--could it be that they are contributing to the generation of non-linear byproducts in your analog front end or overflows in your digital processing?
> Your detector is designed for a signal that has almost all of its energy > between 650 and 750 Hz. Does it?
No answer, just what output of SSB receiver sounds like. Morse signals are one frequency, that one make sound like 700 Hz by tuning. Could as well be 600, 712, whatever. Problem normally is that you do hear (by ear) many simultanious morse signals, as they are narrow spaced (could be 200 Hz apart), ie many simultanious communications inside same 3 KHz LF output from receiver. On a shortwave radio you can insert filters, so you hear less signals at the same time. Maybe a physical "CW filter" centered around 700Hz Along with real signal are all kinds of other noice sources too. Christen Fihl, OZ1AAB
John Barrett wrote:
> I'm working on a CW/Morse demodulator/detector/decoder that takes 8khz 16bit > audio from any SSB reciever, and attempts to detect the state of the signal > (key down, key up) at 700hz. The current "circuit" (all software DSP on a PC > in C#) consists of a sin/cos converter to IQ running with 700hz sin/cos > signals, followed by a 300 tap 50hz LP FIR on I and Q, with integrated > decimation to a 500hz sample rate. > > The FIR outputs are squared, summed, and the final output square rooted :) > Everything to this point appears to be working, excepting that out of band > (<650hz, >750hz) signals have a strong effect on the output, and I am more > or less unhappy with the size of the "noise window" (difference between the > peaks and troughs when no signal is present) given that many signals that I > can hear clearly enough to possibly copy show very little change in the > outputs from the DSP code, barely raising the signal out of the noise > window. > > This is my 1st cut at doing DSP, and I have no formal training, so some of > the questions are liable to be pretty fundamental :) The code described > above is based on the recommendations of the developer of a BPSK > demodulator. > > I need to achieve 2 goals in the next iteration of the code: > 1. Fix any problems with the "demodulator" design (IQ conversion, filtering, > and summing) to clean up the noise issues. > 2. convert the design to be frequency agile -- able to detect CW at any > frequency in the range 200-2500hz > > After reading a number of articles on DSP SSB demodulators, I'm a little > confused because some show a Hilbert Transform after the LPF on the I > channel and a matching delay on the Q channel. I need to know if the lack of > this step in the process could be the cause of the excessive (in my opinion) > noise at the summing output. > > I'm also confused about the LPFs after tthe IQ conversion and what exactly > they are intended to accomplish, as I've seen designs with both LP and BP > filters in that location. When using LP filters, is that essetially setting > the width of the window around the center frequency defined by the feed to > the IQ converter ?? or is it something else ?? > > Any help to straighten this mess out would be greatly appreciated !! > > John AE5ET Barrett > Plano, TX > >
Your basic design sounds right, although you may want to play with your filter bandwidth. The Hilbert transform shouldn't be necessary if you're getting a good clean I/Q down conversion. You would need it if you were trying to discriminate anything but signal right at your audio carrier frequency (700Hz), but since you're sticking to that you don't need the transform. Yes. The combination of the I/Q conversion and the low-pass filters is, essentially, creating a narrow bandpass filter around 700Hz whose width is twice the width of the lowpass filter. With the right kind of bandpass filter instead you could receive signals off of 700Hz with a 700Hz carrier, but you'd need that Hilbert transform to do it right. Here are some suggestions: One, make yourself some cannonical data captures, in .wav files or whatever. Then you can always test the same things. Two, take the output of your lowpass filters, and for a test, before you do all the squaring and summing, upconvert them with the same 700Hz I and Q signals that you used to downconvert. With code in you _should_ hear cleanly filtered, if slightly "ringy" code. Three, If you want to spend the time learning it, get Scilab (it's free, www.scilab.org) and do your algorithm development on that. There's a steep initial learning curve, but for this kind of stuff you can do in three lines of code what it takes thirty in C++ or C#. Four, take FFTs of your incoming data and your IQ data (treat your IQ data as complex, of course). Compare the two -- they should match up, after correcting for the sampling rate differences. You'll want Scilab for this... (in fact, a waterfall display of the incoming audio would be a good way to pick out weak-signal code -- it's what the moonbounce people are doing these days, I understand). -- Tim Wescott Wescott Design Services http://www.wescottdesign.com Do you need to implement control loops in software? "Applied Control Theory for Embedded Systems" gives you just what it says. See details at http://www.wescottdesign.com/actfes/actfes.html
"Tim Wescott" <tim@seemywebsite.com> wrote in message 
news:m-KdnfTysN0awCjVnZ2dnUVZ_qLinZ2d@web-ster.com...
> John Barrett wrote: >> I need to achieve 2 goals in the next iteration of the code: >> 1. Fix any problems with the "demodulator" design (IQ conversion, >> filtering, and summing) to clean up the noise issues. >> 2. convert the design to be frequency agile -- able to detect CW at any >> frequency in the range 200-2500hz >> >> After reading a number of articles on DSP SSB demodulators, I'm a little >> confused because some show a Hilbert Transform after the LPF on the I >> channel and a matching delay on the Q channel. I need to know if the lack >> of this step in the process could be the cause of the excessive (in my >> opinion) noise at the summing output. >> >> I'm also confused about the LPFs after tthe IQ conversion and what >> exactly they are intended to accomplish, as I've seen designs with both >> LP and BP filters in that location. When using LP filters, is that >> essetially setting the width of the window around the center frequency >> defined by the feed to the IQ converter ?? or is it something else ?? >> >> Any help to straighten this mess out would be greatly appreciated !! >> >> John AE5ET Barrett >> Plano, TX > Your basic design sounds right, although you may want to play with your > filter bandwidth. > > The Hilbert transform shouldn't be necessary if you're getting a good > clean I/Q down conversion. You would need it if you were trying to > discriminate anything but signal right at your audio carrier frequency > (700Hz), but since you're sticking to that you don't need the transform. > > Yes. The combination of the I/Q conversion and the low-pass filters is, > essentially, creating a narrow bandpass filter around 700Hz whose width is > twice the width of the lowpass filter. With the right kind of bandpass > filter instead you could receive signals off of 700Hz with a 700Hz > carrier, but you'd need that Hilbert transform to do it right. > > Here are some suggestions: > > One, make yourself some cannonical data captures, in .wav files or > whatever. Then you can always test the same things. > > Two, take the output of your lowpass filters, and for a test, before you > do all the squaring and summing, upconvert them with the same 700Hz I and > Q signals that you used to downconvert. With code in you _should_ hear > cleanly filtered, if slightly "ringy" code. > > Three, If you want to spend the time learning it, get Scilab (it's free, > www.scilab.org) and do your algorithm development on that. There's a > steep initial learning curve, but for this kind of stuff you can do in > three lines of code what it takes thirty in C++ or C#. > > Four, take FFTs of your incoming data and your IQ data (treat your IQ data > as complex, of course). Compare the two -- they should match up, after > correcting for the sampling rate differences. You'll want Scilab for > this... > > (in fact, a waterfall display of the incoming audio would be a good way to > pick out weak-signal code -- it's what the moonbounce people are doing > these days, I understand). >
My test application has multiple FFT and osciloscope displays monitoring the "circuit" at various points, so I can see what is happening at each point in the signal path.. (The problem is understanding what I should be seeing !!) I can get the IQ data at the full sample rate (I can disable the decimator built into the FIR, getting full rate output, and add a seperate decimator after I tap off the full rate signal), but my FFTs are only designed to handle simple input, not complex. Just to confirm -- since the low pass filters are defining the bandpass around the center frequency defined by the IQ conversion, I should already be frequency agile by changing the the IQ conversion frequency ?? And I'm guessing that your comment about my filters means that I should tighten up, perhaps even look at brick wall filters to knock out that adjacent channel interference ?? I'm not horribly worried about tap counts at this point -- the lowpass FIRs only get executed at a 500hz or 1000hz rate so I've got plenty of CPU available to handle big FIRs. I downloaded SciLab last night in the hopes that it would help me design better filters, but my hopes were dashed due to the steep learning curve -- I've got some more time to poke at it today so I might be able to figure it out. I think I got a basic handle on the language syntax, but the documentation is a little light for someone that doesnt understand all the math behind signal processing. Anyone know of a good link or 3 on designing filters with Scilab that could give me a leg up on the process ?? I've found a few for Matlab, but there are functions missing in Scilab -- remezord, for instance..... I'd prefer examples for symetrical linear phase filters -- I'm reading the Scilab signal.PDF now (and jusr regenerated a couple of my key filters with linear phase, so I'm getting some place !!) An example of taking a sound file as input and performing IQ conversion and filtering would also be really helpful !! I'd love to be able to model the entire process in Scilab, but I need to understand how to get my per-sample based post-processing implemented such that scilab can apply it to a given signal.
John Barrett wrote:
> "Tim Wescott" <tim@seemywebsite.com> wrote in message > news:m-KdnfTysN0awCjVnZ2dnUVZ_qLinZ2d@web-ster.com... >> John Barrett wrote: >>> I need to achieve 2 goals in the next iteration of the code: >>> 1. Fix any problems with the "demodulator" design (IQ conversion, >>> filtering, and summing) to clean up the noise issues. >>> 2. convert the design to be frequency agile -- able to detect CW at any >>> frequency in the range 200-2500hz >>> >>> After reading a number of articles on DSP SSB demodulators, I'm a little >>> confused because some show a Hilbert Transform after the LPF on the I >>> channel and a matching delay on the Q channel. I need to know if the lack >>> of this step in the process could be the cause of the excessive (in my >>> opinion) noise at the summing output. >>> >>> I'm also confused about the LPFs after tthe IQ conversion and what >>> exactly they are intended to accomplish, as I've seen designs with both >>> LP and BP filters in that location. When using LP filters, is that >>> essetially setting the width of the window around the center frequency >>> defined by the feed to the IQ converter ?? or is it something else ?? >>> >>> Any help to straighten this mess out would be greatly appreciated !! >>> >>> John AE5ET Barrett >>> Plano, TX >> Your basic design sounds right, although you may want to play with your >> filter bandwidth. >> >> The Hilbert transform shouldn't be necessary if you're getting a good >> clean I/Q down conversion. You would need it if you were trying to >> discriminate anything but signal right at your audio carrier frequency >> (700Hz), but since you're sticking to that you don't need the transform. >> >> Yes. The combination of the I/Q conversion and the low-pass filters is, >> essentially, creating a narrow bandpass filter around 700Hz whose width is >> twice the width of the lowpass filter. With the right kind of bandpass >> filter instead you could receive signals off of 700Hz with a 700Hz >> carrier, but you'd need that Hilbert transform to do it right. >> >> Here are some suggestions: >> >> One, make yourself some cannonical data captures, in .wav files or >> whatever. Then you can always test the same things. >> >> Two, take the output of your lowpass filters, and for a test, before you >> do all the squaring and summing, upconvert them with the same 700Hz I and >> Q signals that you used to downconvert. With code in you _should_ hear >> cleanly filtered, if slightly "ringy" code. >> >> Three, If you want to spend the time learning it, get Scilab (it's free, >> www.scilab.org) and do your algorithm development on that. There's a >> steep initial learning curve, but for this kind of stuff you can do in >> three lines of code what it takes thirty in C++ or C#. >> >> Four, take FFTs of your incoming data and your IQ data (treat your IQ data >> as complex, of course). Compare the two -- they should match up, after >> correcting for the sampling rate differences. You'll want Scilab for >> this... >> >> (in fact, a waterfall display of the incoming audio would be a good way to >> pick out weak-signal code -- it's what the moonbounce people are doing >> these days, I understand). >> > > My test application has multiple FFT and osciloscope displays monitoring the > "circuit" at various points, so I can see what is happening at each point in > the signal path.. (The problem is understanding what I should be seeing !!) > I can get the IQ data at the full sample rate (I can disable the decimator > built into the FIR, getting full rate output, and add a seperate decimator > after I tap off the full rate signal), but my FFTs are only designed to > handle simple input, not complex. > > Just to confirm -- since the low pass filters are defining the bandpass > around the center frequency defined by the IQ conversion, I should already > be frequency agile by changing the the IQ conversion frequency ?? And I'm > guessing that your comment about my filters means that I should tighten up, > perhaps even look at brick wall filters to knock out that adjacent channel > interference ?? I'm not horribly worried about tap counts at this point -- > the lowpass FIRs only get executed at a 500hz or 1000hz rate so I've got > plenty of CPU available to handle big FIRs. > > I downloaded SciLab last night in the hopes that it would help me design > better filters, but my hopes were dashed due to the steep learning curve -- > I've got some more time to poke at it today so I might be able to figure it > out. I think I got a basic handle on the language syntax, but the > documentation is a little light for someone that doesnt understand all the > math behind signal processing. > > Anyone know of a good link or 3 on designing filters with Scilab that could > give me a leg up on the process ?? I've found a few for Matlab, but there > are functions missing in Scilab -- remezord, for instance..... I'd prefer > examples for symetrical linear phase filters -- I'm reading the Scilab > signal.PDF now (and jusr regenerated a couple of my key filters with linear > phase, so I'm getting some place !!) > > An example of taking a sound file as input and performing IQ conversion and > filtering would also be really helpful !! I'd love to be able to model the > entire process in Scilab, but I need to understand how to get my per-sample > based post-processing implemented such that scilab can apply it to a given > signal. > >
'loadwave' and 'savewave' do what they sound like, and 'sound' plays a sound. Your can type 'help wav' to find out things that deal with .wav files, and 'help sound' for other helpful audio things. This is totally un-run code, so take it for what it's worth: x = loadwave('my file name'); // read in file ix = 1:size(x, 1); // make an index t = (ix - 1) * 0.000125; // time assuming 8ksps xc = x .* exp(%i * 2 * %pi * t * 700); // IQ downconvert @ 700Hz xcf = convol(h, xc); // h your filter xd = sqrt(abs(xcf)); plot2d(t, xd); // graph the envelope -- Tim Wescott Wescott Design Services http://www.wescottdesign.com Do you need to implement control loops in software? "Applied Control Theory for Embedded Systems" gives you just what it says. See details at http://www.wescottdesign.com/actfes/actfes.html
Hi John.

I ported a multimode application to PocketPC, see http://pocketdigi.sourceforge.net.
The application contains CW decoder. The algorithm works like this:

1) Convert your CW signal down to DC by multiplying the input signal
by digital controlled IQ oscillator. One gets a complex signal at
baseband, which will have aliases, but the aliases do not interfere
with your CW signal. If you treat your input signal as complex with
imaginatory part set to zero, you will see, that the negative high
frequencies will "roll over" to the high real frequencies. No problem
as they will be filtered out in the next step.

2) Low pass complex filter. Just filter the real and complex parts
separately with the same FIR low pass filter.

3) Calculate amplitude by sqrt(r^2+i^2)

4) Do some low pass filtering of the resulting signal. On short waves
there are often noise peaks, which will distort the CW baseband
waveform.

5) Extract baseband clock

6) Extract morse data

Steps 2) and 4) are dependent on modulation speed.

73, Vojtech OK1IAK, AB2ZA
On Aug 27, 12:51&#4294967295;am, "John Barrett" <john.ae...@verizon.net> wrote:
> I'm working on a CW/Morse demodulator/detector/decoder that takes 8khz 16bit > audio from any SSB reciever, and attempts to detect the state of the signal > (key down, key up) at 700hz. The current "circuit" (all software DSP on a PC > in C#) consists of a sin/cos converter to IQ running with 700hz sin/cos > signals, followed by a 300 tap 50hz LP FIR on I and Q, with integrated > decimation to a 500hz sample rate.
I tested this setup using a synthesized on-off keyed 880 Hz. Morse signal at 15 WPM (dot length approximately 15.75 msec). I tried several variations including 64-tap FIR and 300-tap FIR low-pass filters, corrupting the signal with white noise at -10 dB, and varying the I/Q generator's NCO from 860 to 900 Hz. In all variations, the magnitude of the I/Q signal perfectly reflected the keying with almost zero noise evident in the detected signal. The 64-tap FIR produces a cleaner signal than the 300-tap FIR due to less ringing. No post- detection filtering was indicated. This series of tests vindicates the theory of what I understand you're trying to do. If it's not working, my guess is that you've got one or more implementation errors.
On Aug 27, 12:51&#4294967295;am, "John Barrett" <john.ae...@verizon.net> wrote:
> I'm working on a CW/Morse demodulator/detector/decoder that takes 8khz 16bit > audio from any SSB reciever, and attempts to detect the state of the signal > (key down, key up) at 700hz. The current "circuit" (all software DSP on a PC > in C#) consists of a sin/cos converter to IQ running with 700hz sin/cos > signals, followed by a 300 tap 50hz LP FIR on I and Q, with integrated > decimation to a 500hz sample rate. > > The FIR outputs are squared, summed, and the final output square rooted :) > Everything to this point appears to be working, excepting that out of band > (<650hz, >750hz) signals have a strong effect on the output, and I am more > or less unhappy with the size of the "noise window" (difference between the > peaks and troughs when no signal is present) given that many signals that I > can hear clearly enough to possibly copy show very little change in the > outputs from the DSP code, barely raising the signal out of the noise > window. > > This is my 1st cut at doing DSP, and I have no formal training, so some of > the questions are liable to be pretty fundamental :) The code described > above is based on the recommendations of the developer of a BPSK > demodulator. > > I need to achieve 2 goals in the next iteration of the code: > 1. Fix any problems with the "demodulator" design (IQ conversion, filtering, > and summing) to clean up the noise issues. > 2. convert the design to be frequency agile -- able to detect CW at any > frequency in the range 200-2500hz > > After reading a number of articles on DSP SSB demodulators, I'm a little > confused because some show a Hilbert Transform after the LPF on the I > channel and a matching delay on the Q channel. I need to know if the lack of > this step in the process could be the cause of the excessive (in my opinion) > noise at the summing output. > > I'm also confused about the LPFs after tthe IQ conversion and what exactly > they are intended to accomplish, as I've seen designs with both LP and BP > filters in that location. When using LP filters, is that essetially setting > the width of the window around the center frequency defined by the feed to > the IQ converter ?? or is it something else ?? > > Any help to straighten this mess out would be greatly appreciated !!
Getting an algorithm to do what the brain can do very well is very very hard. The on/off nature of CW puts it at an extreme disadvantage for automated decoding compared to, say, PSK or BPSK, so you will very definitely need tricks that are not in the PSK/BPSK books. Most experienced CW operators, and all EME operators, can easily hear signals that are under the technical Minimum Discernable Signal (MDS) limit you'd guess by simply using S/N and noise in bandwidth. For the human ear, reducing bandwidth is not always the solution. In particular, narrow bandwidth leads to ringing, which not only is extremely fatiguing, but also makes out of band impulses show up as an apparent signal. The 100Hz bandwidth you are trying is way too narrow for optimal human brain/ear copy. One thing to look at, I don't know how they do it inside: CW Skimmer. I believe (but have no evidence) that it is looking not for simple on/ off signal presence/no presence, but for coherent signals that happen to match morse characters or even patterns of entire callsigns, probably using something like PRML techniques, and it does it across a spectrum. I've seen CW Skimmer in action during a contest. It is amazing! I could imagine hooking it up to a mouse and when you see (not hear) someone calling TEST you just click on the waterfall and work them. I'm not sure this is the way I'd enjoy CW, but I'm sure others will be doing it soon. Tim N3QE
"John Hadstate" <jh113355@hotmail.com> wrote in message 
news:469e82c7-4892-4e78-bc2b-f86828849ba7@x35g2000hsb.googlegroups.com...
On Aug 27, 12:51 am, "John Barrett" <john.ae...@verizon.net> wrote:
>> I'm working on a CW/Morse demodulator/detector/decoder that takes 8khz >> 16bit >> audio from any SSB reciever, and attempts to detect the state of the >> signal >> (key down, key up) at 700hz. The current "circuit" (all software DSP on a >> PC >> in C#) consists of a sin/cos converter to IQ running with 700hz sin/cos >> signals, followed by a 300 tap 50hz LP FIR on I and Q, with integrated >> decimation to a 500hz sample rate. > >I tested this setup using a synthesized on-off keyed 880 Hz. Morse >signal at 15 WPM (dot length approximately 15.75 msec). I tried >several variations including 64-tap FIR and 300-tap FIR low-pass >filters, corrupting the signal with white noise at -10 dB, and varying >the I/Q generator's NCO from 860 to 900 Hz. In all variations, the >magnitude of the I/Q signal perfectly reflected the keying with almost >zero noise evident in the detected signal. The 64-tap FIR produces a >cleaner signal than the 300-tap FIR due to less ringing. No post- >detection filtering was indicated. > >This series of tests vindicates the theory of what I understand you're >trying to do. If it's not working, my guess is that you've got one or >more implementation errors.
Did your test include a decimate to 500hz sample rate after the filters ?? Turns out that I was taking too big a bite out of the sample rate.. I split the filtering into 2 stages.. a 40 tap low pass with transition band from 100hz to 800hz with factor 4 decimation, followed by a 120 tap low pass with 20hz to 60hz transition and a decimation of 2 (total decimation of 8, 1000hz sample rate). I was looking at upping the final sample rate before edge detection in any case to get better definition of the edges, and I'm much happier with the output from the split filters than I was with a single filter and high decimation. Dropped my computation load from 300000 multiplies per second to (80000+120000)=200000 multiplies per second. I may still add a 3rd filter and take it back down to 500 samples.. we'll see how it goes.