Hi group, This is probably a stupid question, but here goes... I'm currently working on a software FSK-modem for decoding shortwave data transmissions. ( RTTY, SITOR-A, SITOR-B, etc) ( OS: Win-XP Pro, coding: Borland Delphi) I have connected the receiver sound output to the soundcard, and have come this far, mostly by reading this NG: I have implemented the bandpassfilters, the envelope detectors, and the decision circuit, and have even managed a FFT-transform to visualize the tone spectrum - note: I'm an experienced programmer, but definitely *not* a mathematician :-( Screenshot: <http://users.pandora.be/dirk.claessens2/DSP/screen.jpg> The sound card is configured to sample at 10240Hz, and the datablock size is set to 1024 (8 bit) samples. After filtering etc..., I end up with consecutive chunks of 1024 samples representing the data pattern of ones & zeroes. Now here's my problem: how do I reconstruct/extract the actual data characters/bytes, timing etc...?? I know that RTTY starts with a start bit etc..., but how should this be extracted out of an array that looks like 11111111111100000001111111110000111110001111?? Thanks for any advice... Dirk.
FSK modem: bit/timing recovery
Started by ●January 23, 2007
Reply by ●January 23, 20072007-01-23
DSP-Newbie wrote:> Hi group, > > This is probably a stupid question, but here goes... > I'm currently working on a software FSK-modem for decoding shortwave > data transmissions. ( RTTY, SITOR-A, SITOR-B, etc) > ( OS: Win-XP Pro, coding: Borland Delphi) > > I have connected the receiver sound output to the soundcard, and have > come this far, mostly by reading this NG: > > I have implemented the bandpassfilters, the envelope detectors, and the > decision circuit, and have even managed a FFT-transform to visualize the > tone spectrum - note: I'm an experienced programmer, but definitely > *not* a mathematician :-( > > Screenshot: <http://users.pandora.be/dirk.claessens2/DSP/screen.jpg> > > The sound card is configured to sample at 10240Hz, and the datablock > size is set to 1024 (8 bit) samples. After filtering etc..., I end up > with consecutive chunks of 1024 samples representing the data pattern of > ones & zeroes. > > Now here's my problem: how do I reconstruct/extract the actual data > characters/bytes, timing etc...?? I know that RTTY starts with a start > bit etc..., but how should this be extracted out of an array that looks > like 11111111111100000001111111110000111110001111?? > > Thanks for any advice... > > Dirk.You've come a long way. Congratulations! The basic RTTY demodulated signal is just like any other TTY signal. To make sense of it, you need to know the polarity out of the decoder (inverted or not) and the code used (Probably ASCII, maybe Baudot), and the "framing". The framing may be synchronous (probably not) or asynchronous with stop and start bits in the data stream. (You also need to knw the bit rate, but to get this far, you evidently do.) A piece of hardware called a UART handles the details, and software UARTs are abundant. http://tinyurl.com/2al6av explains a lot. Come back with more questions if you need to. Jerry -- Engineering is the art of making what you want from things you can get. ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
Reply by ●January 23, 20072007-01-23
Jerry Avins wrote:> DSP-Newbie wrote:>> >> Now here's my problem: how do I reconstruct/extract the actual data >> characters/bytes, timing etc...?? I know that RTTY starts with a start bit >> etc..., but how should this be extracted out of an array that looks like >> 11111111111100000001111111110000111110001111?? >> >> Thanks for any advice... >> >> Dirk. > > You've come a long way. Congratulations! The basic RTTY demodulated signal > is just like any other TTY signal. To make sense of it, you need to know the > polarity out of the decoder (inverted or not) and the code used (Probably > ASCII, maybe Baudot), and the "framing". The framing may be synchronous > (probably not) or asynchronous with stop and start bits in the data stream. > (You also need to knw the bit rate, but to get this far, you evidently do.) > > A piece of hardware called a UART handles the details, and software UARTs are > abundant. http://tinyurl.com/2al6av explains a lot. Come back with more > questions if you need to. > > JerryThanks for the reply Jerry, Are you suggesting I should send the bit stream to the soundcard output, and connect it to the PC's serial port? This might work for RTTY, but not for SITOR-A/B signals, since these are synchronous. I know it can be done without using the serial port, many programs exist that do it. Don't ask why then I'm not using one of them :0) I guess I have not stated my problem clearly enough. =The soundcard sends chunks of 1024 samples at a rate of 10 chunks/sec, sampled at 10240Hz. = If we assume a 100 baud signal, 1 bit time = 10msec. = Since the sampling frequence is 10240Hz, I see roughly 100 identical samples at the output of the decision circuit for each 1/0 present in the original data stream. (101 becomes 111111111111111000000000000000000011111111111111 ) Should I "compress" the data by a factor samplefreq./baud or some such?
Reply by ●January 23, 20072007-01-23
On 2007-01-23, DSP-Newbie <No@way.invalid> wrote:> > Screenshot: <http://users.pandora.be/dirk.claessens2/DSP/screen.jpg>Neat!> Now here's my problem: how do I reconstruct/extract the actual data > characters/bytes, timing etc...?? I know that RTTY starts with a start > bit etc..., but how should this be extracted out of an array that looks > like 11111111111100000001111111110000111110001111??Look at some UARTs. You should be able to find some software or Verilog versions with some googling. Most work by oversampling the data by 16x (looks like I trimmed it, but I think you did 10x?). Then it looks for an edge, counts halfway into the bit time and then samples the data. Fancier ones might vote on 3 middle-of-bit samples. Then it just counts off another bit time and resamples. I'm not familiar with RTTY, but if every byte has a 'stop' bit, then you just restart this over and over, so slight rate changes never accumulate and should not affect you over the course of one byte. To demodulate PWM from WWV, where I had sampled 100 times for every bit and needed to find the phase of the bits so I could measure the pulsewidths, first I edge detected my binary stream by subtracting each sample from the next sample. That left me with an array of the positive- going edges. Then I created a histogram of their offsets mod 100 (my oversample factor). The peak (or even the average) gives you the phase offset (mod 100) and then you can count the 1s in each bin. -- Ben Jackson AD7GD <ben@ben.com> http://www.ben.com/
Reply by ●January 23, 20072007-01-23
"DSP-Newbie" <No@way.invalid> wrote in message news:mn.bcf37d719f5c170d.59994@way.invalid...> I guess I have not stated my problem clearly enough. > =The soundcard sends chunks of 1024 samples at a rate of 10 chunks/sec, > sampled at 10240Hz. > = If we assume a 100 baud signal, 1 bit time = 10msec. > = Since the sampling frequence is 10240Hz, I see roughly 100 identical > samples at the output of the decision circuit for each 1/0 present in the > original data stream. > (101 becomes 111111111111111000000000000000000011111111111111 ) > > > Should I "compress" the data by a factor samplefreq./baud or some such?I don't think you should bother feeding it back into the serial port - emulating a UART in software is simpicity itself compared to what you've already achieved. Here's the way UARTs do it. You need to generate a clock, and the really simple way to do this (but certainly not recommended for production strength for low SNR RTTY - see later) is by simply detecting edge changes. You know how many samples per bit there are (100), so you reset a counter when you see either a state change, or expect that the clock should change (100 samples later): note there will of course not always be a state change. Your bit value is taken half a bit time later (counter = 50 samples). Note that the counter is incremented each sample time. For RTTY there will always be a state change within six bits because it's a five bit code plus a start and a stop bit. The Start bit is a space (0) and the stop bit is a mark (1). The LSB is sent first. Your software should look for a combination of Space - x - x - x - x - x - Mark sequence to get in character (or 'frame') sync. With five bits, it's obviously not ASCII - it's called Baudot although back in my five hole paper tape days here in the UK it was known as Murray code. To get ASCII you'll need two lookup tables - one for Figure shift mode and one for Letter shift mode. Two unique codes switch between Letter shift and Figure shift. http://en.wikipedia.org/wiki/Baudot_code To generate a better clock when SNR isn't so good, it is common to use Costas loop. With the scheme above, a bit of noise will quite easily disrupt the clock by resetting it too often. To get a better data sample, it is common to take an aggregate, that can be weighted, of samples over the entire bit period. There are several other ways to demodulate too. Cheers, Howard
Reply by ●January 23, 20072007-01-23
DSP-Newbie wrote:> Jerry Avins wrote: >> DSP-Newbie wrote: > >>> >>> Now here's my problem: how do I reconstruct/extract the actual data >>> characters/bytes, timing etc...?? I know that RTTY starts with a >>> start bit etc..., but how should this be extracted out of an array >>> that looks like 11111111111100000001111111110000111110001111?? >>> >>> Thanks for any advice... >>> >>> Dirk. >> >> You've come a long way. Congratulations! The basic RTTY demodulated >> signal is just like any other TTY signal. To make sense of it, you >> need to know the polarity out of the decoder (inverted or not) and the >> code used (Probably ASCII, maybe Baudot), and the "framing". The >> framing may be synchronous (probably not) or asynchronous with stop >> and start bits in the data stream. (You also need to knw the bit rate, >> but to get this far, you evidently do.) >> >> A piece of hardware called a UART handles the details, and software >> UARTs are abundant. http://tinyurl.com/2al6av explains a lot. Come >> back with more questions if you need to. >> >> Jerry > > Thanks for the reply Jerry, > > Are you suggesting I should send the bit stream to the soundcard output, > and connect it to the PC's serial port? This might work for RTTY, but > not for SITOR-A/B signals, since these are synchronous.Not at all. I wouldn't know how to make that work since tones aren't bits. Synchronous signaling dispenses with framing bits, but you have to know where a character starts. There's a character called "SYN" that is sent repeatedly for long enough so that the receiver can lock up. Nine is usually enough. The receiver starts anywhere and decodes 8 bits to form a character. If that is not SYN, it slips a bit and does it again. When the decoded character is SYN, the receiver adjusts its clock rate to match the transmitted clock and continues doing that throughout the transmission. The transmitter sends SYNs whenever it has no data.> I know it can be done without using the serial port, many programs > exist that do it. Don't ask why then I'm not using one of them :0)Of course. If you understand what the UART (or USART) does, it's not difficult to write software to simulate it.> I guess I have not stated my problem clearly enough. > =The soundcard sends chunks of 1024 samples at a rate of 10 chunks/sec, > sampled at 10240Hz. > = If we assume a 100 baud signal, 1 bit time = 10msec.That leaves no time for framing bits with asynchronous transmission. A Model 33 KSR runs at 110 bits.second, there being 11 bits per character. Since the receiver restarts its clock at the beginning of every character and the stop bits can be mistimed without problem, the receiver clock must match the transmitter clock within half a bit out of nine. Five percent accuracy would be adequate in the absence of jitter. Usually, one percent is specifies for each clock, so the timing mismatch doesn't exceed two percent.> = Since the sampling frequence is 10240Hz, I see roughly 100 identical > samples at the output of the decision circuit for each 1/0 present in > the original data stream. > (101 becomes 111111111111111000000000000000000011111111111111 ) > > > Should I "compress" the data by a factor samplefreq./baud or some such?You should extract the actual bits, yielding 101, and work with them. 111111111111111000000000000000000011111111111111 is an ideal case. you can tolerate 111011111101111000000000001110000011111101111111 and still extract the bit pattern if you have a rough idea where the edges of the bits really are. Jerry -- Engineering is the art of making what you want from things you can get. ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
Reply by ●January 24, 20072007-01-24
DSP-Newbie wrote:> Jerry Avins wrote: >> DSP-Newbie wrote: > >>> >>> Now here's my problem: how do I reconstruct/extract the actual data >>> characters/bytes, timing etc...?? I know that RTTY starts with a start bit >>> etc..., but how should this be extracted out of an array that looks like >>> 11111111111100000001111111110000111110001111??Jerry, Howard, Ben, Thanks for the tips, it looks like I'll have to do it the hard way... Edge detection + sample counting & some bookkeeping between samples should do it. Back to the drawing board :0) Thanks - Dirk
Reply by ●January 24, 20072007-01-24
Ben Jackson wrote:> > Look at some UARTs. You should be able to find some software or Verilog > versions with some googling. Most work by oversampling the data by 16x > (looks like I trimmed it, but I think you did 10x?). Then it looks for > an edge, counts halfway into the bit time and then samples the data. > Fancier ones might vote on 3 middle-of-bit samples. Then it just counts > off another bit time and resamples.I get the picture...> I'm not familiar with RTTY, but if > every byte has a 'stop' bit, then you just restart this over and over, > so slight rate changes never accumulate and should not affect you over > the course of one byte. >RTTY mostly uses Baudot 5 bit code, + 1.5 start & 1 stop bit. SITOR-B ( and many others ) use a 7-bit FEC code but are synchronous, so there I must jump on the bit stream and look for special sync characters to find the frame sync.> To demodulate PWM from WWV, where I had sampled 100 times for every bit > and needed to find the phase of the bits so I could measure the > pulsewidths, first I edge detected my binary stream by subtracting each > sample from the next sample. That left me with an array of the positive- > going edges. Then I created a histogram of their offsets mod 100 (my > oversample factor). The peak (or even the average) gives you the > phase offset (mod 100) and then you can count the 1s in each bin.Interesting tip! Thanks - Dirk
Reply by ●January 25, 20072007-01-25
"DSP-Newbie" <No@way.invalid> wrote in message news:mn.c2ab7d715664af75.59994@way.invalid...> > RTTY mostly uses Baudot 5 bit code, + 1.5 start & 1 stop bit.Now you've got my memory working, I believe it's 1 start and 1.5 stop. I must admit I'd forgotten about using 1.5 bit times for a stop for RTTY. The baud rate for RTTY by the way is 45.45 baud. This will almost certainly mean you'll have a non-integer number of samples which very slightly complicates things. If you're using a PC to decode, an easy option is to use floating point for your clock and when you reset without a state change, subtract the floating point number of samples per bit from the counter rather than zeroing it. Howard
Reply by ●January 25, 20072007-01-25
Howard Long wrote:> "DSP-Newbie" <No@way.invalid> wrote in message > news:mn.c2ab7d715664af75.59994@way.invalid... >> RTTY mostly uses Baudot 5 bit code, + 1.5 start & 1 stop bit. > > Now you've got my memory working, I believe it's 1 start and 1.5 stop. I > must admit I'd forgotten about using 1.5 bit times for a stop for RTTY.Never count stop bits except to open a gate. When you get to the last data bit, wait some reasonable time and then start looking for a start bit. A stop bit is MARK and the signaling code is MARK idle. the stop-bit time is an approximate minimum -- approximate because receive and send clocks might be a few percent apart. There is no maximum time. Stuttering is allowed.> The baud rate for RTTY by the way is 45.45 baud. This will almost certainly > mean you'll have a non-integer number of samples which very slightly > complicates things. If you're using a PC to decode, an easy option is to use > floating point for your clock and when you reset without a state change, > subtract the floating point number of samples per bit from the counter > rather than zeroing it.I take it you've been there! :-) Jerry -- Engineering is the art of making what you want from things you can get. ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
