Sign in

Not a member? | Forgot your Password?

Search compdsp

Search tips


Discussion Groups

IIR Filter Design Software

Free Online Books

See Also

Embedded SystemsFPGA

Discussion Groups | Comp.DSP | FSK modem: bit/timing recovery

There are 21 messages in this thread.

You are currently looking at messages 1 to .


Is this discussion worth a thumbs up?

0

FSK modem: bit/timing recovery - DSP-Newbie - 2007-01-23 08:15:00

Hi group,

This is probably a stupid question, but here goes...
I'm currently working on a software FSK-modem for decoding shortwave 
data transmissions. ( RTTY, SITOR-A, SITOR-B, etc)
( OS: Win-XP Pro, coding: Borland Delphi)

I have connected the receiver sound output to the soundcard, and have 
come this far, mostly by reading this NG:

I have implemented the bandpassfilters, the envelope detectors, and the 
decision circuit, and have even managed a FFT-transform to visualize 
the tone spectrum - note: I'm an experienced programmer, but definitely 
*not* a mathematician :-(

Screenshot: <http://users.pandora.be/dirk.claessens2/DSP/screen.jpg>;

The sound card is configured to sample at 10240Hz, and the datablock 
size is set to 1024 (8 bit) samples. After filtering etc..., I end up 
with consecutive chunks of 1024 samples representing the data pattern 
of ones & zeroes.

Now here's my problem: how do I reconstruct/extract the actual data 
characters/bytes, timing etc...?? I know that RTTY starts with a start 
bit etc..., but how should this be extracted out of an array that looks 
like    11111111111100000001111111110000111110001111??

Thanks for any advice...

Dirk.

_____________________________
 Free pdf download: Digital Signal Processing Maths.


Re: FSK modem: bit/timing recovery - Jerry Avins - 2007-01-23 11:12:00

DSP-Newbie wrote:
> Hi group,
> 
> This is probably a stupid question, but here goes...
> I'm currently working on a software FSK-modem for decoding shortwave 
> data transmissions. ( RTTY, SITOR-A, SITOR-B, etc)
> ( OS: Win-XP Pro, coding: Borland Delphi)
> 
> I have connected the receiver sound output to the soundcard, and have 
> come this far, mostly by reading this NG:
> 
> I have implemented the bandpassfilters, the envelope detectors, and the 
> decision circuit, and have even managed a FFT-transform to visualize the 
> tone spectrum - note: I'm an experienced programmer, but definitely 
> *not* a mathematician :-(
> 
> Screenshot: <http://users.pandora.be/dirk.claessens2/DSP/screen.jpg>;
> 
> The sound card is configured to sample at 10240Hz, and the datablock 
> size is set to 1024 (8 bit) samples. After filtering etc..., I end up 
> with consecutive chunks of 1024 samples representing the data pattern of 
> ones & zeroes.
> 
> Now here's my problem: how do I reconstruct/extract the actual data 
> characters/bytes, timing etc...?? I know that RTTY starts with a start 
> bit etc..., but how should this be extracted out of an array that looks 
> like    11111111111100000001111111110000111110001111??
> 
> Thanks for any advice...
> 
> Dirk.

You've come a long way. Congratulations! The basic RTTY  demodulated 
signal is just like any other TTY signal. To make sense of it, you need 
to know the polarity out of the decoder (inverted or not) and the code 
used (Probably ASCII, maybe Baudot), and the "framing". The framing
may 
be synchronous (probably not) or asynchronous with stop and start bits 
in the data stream. (You also need to knw the bit rate, but to get this 
far, you evidently do.)

A piece of hardware called a UART handles the details, and software 
UARTs are abundant. http://tinyurl.com/2al6av explains a lot. Come back 
with more questions if you need to.

Jerry
-- 
Engineering is the art of making what you want from things you can get.
¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯

_____________________________
 Free pdf download: Digital Signal Processor Fundamentals and System Design.


Re: FSK modem: bit/timing recovery - DSP-Newbie - 2007-01-23 15:07:00

Jerry Avins wrote:
> DSP-Newbie wrote:

>> 
>> Now here's my problem: how do I reconstruct/extract the actual data 
>> characters/bytes, timing etc...?? I know that RTTY starts with a start
bit 
>> etc..., but how should this be extracted out of an array that looks
like    
>> 11111111111100000001111111110000111110001111??
>> 
>> Thanks for any advice...
>> 
>> Dirk.
>
> You've come a long way. Congratulations! The basic RTTY  demodulated signal

> is just like any other TTY signal. To make sense of it, you need to know
the 
> polarity out of the decoder (inverted or not) and the code used (Probably 
> ASCII, maybe Baudot), and the "framing". The framing may be
synchronous 
> (probably not) or asynchronous with stop and start bits in the data stream.

> (You also need to knw the bit rate, but to get this far, you evidently
do.)
>
> A piece of hardware called a UART handles the details, and software UARTs
are 
> abundant. http://tinyurl.com/2al6av explains a lot. Come back with more 
> questions if you need to.
>
> Jerry

Thanks for the reply Jerry,

Are you suggesting I should send the bit stream to the soundcard 
output, and connect it to the PC's serial port? This might work for 
RTTY, but not for SITOR-A/B signals, since these are synchronous.

I know it can be done without using the serial port,  many programs 
exist that do it. Don't ask why then I'm not using one of them :0)

I guess I have not stated my problem clearly enough.
=The soundcard sends chunks of 1024 samples at a rate of 10 chunks/sec, 
sampled at  10240Hz.
= If we assume a 100 baud signal, 1 bit time = 10msec.
= Since the sampling frequence is 10240Hz, I see roughly 100 identical 
samples at the output of the decision circuit for each 1/0 present in 
the original data stream.
(101 becomes 111111111111111000000000000000000011111111111111 )


Should I "compress" the data by a factor samplefreq./baud or some
such?

_____________________________
 Free pdf download: Digital Signal Processing Maths.


Re: FSK modem: bit/timing recovery - Ben Jackson - 2007-01-23 18:06:00

On 2007-01-23, DSP-Newbie <N...@way.invalid> wrote:
>
> Screenshot: <http://users.pandora.be/dirk.claessens2/DSP/screen.jpg>;

Neat!

> Now here's my problem: how do I reconstruct/extract the actual data 
> characters/bytes, timing etc...?? I know that RTTY starts with a start 
> bit etc..., but how should this be extracted out of an array that looks 
> like    11111111111100000001111111110000111110001111??

Look at some UARTs.  You should be able to find some software or Verilog
versions with some googling.  Most work by oversampling the data by 16x
(looks like I trimmed it, but I think you did 10x?).  Then it looks for
an edge, counts halfway into the bit time and then samples the data.
Fancier ones might vote on 3 middle-of-bit samples.  Then it just counts
off another bit time and resamples.  I'm not familiar with RTTY, but if
every byte has a 'stop' bit, then you just restart this over and over,
so slight rate changes never accumulate and should not affect you over
the course of one byte.

To demodulate PWM from WWV, where I had sampled 100 times for every bit
and needed to find the phase of the bits so I could measure the
pulsewidths, first I edge detected my binary stream by subtracting each
sample from the next sample.  That left me with an array of the positive-
going edges.  Then I created a histogram of their offsets mod 100 (my
oversample factor).  The peak (or even the average) gives you the
phase offset (mod 100) and then you can count the 1s in each bin.

-- 
Ben Jackson AD7GD
<b...@ben.com>
http://www.ben.com/

_____________________________
 Free pdf download: Digital Signal Processing Maths.


Re: FSK modem: bit/timing recovery - Howard Long - 2007-01-23 18:09:00

"DSP-Newbie" <N...@way.invalid> wrote in message 
news:m...@way.invalid...
> I guess I have not stated my problem clearly enough.
> =The soundcard sends chunks of 1024 samples at a rate of 10 chunks/sec, 
> sampled at  10240Hz.
> = If we assume a 100 baud signal, 1 bit time = 10msec.
> = Since the sampling frequence is 10240Hz, I see roughly 100 identical 
> samples at the output of the decision circuit for each 1/0 present in the 
> original data stream.
> (101 becomes 111111111111111000000000000000000011111111111111 )
>
>
> Should I "compress" the data by a factor samplefreq./baud or some
such?

I don't think you should bother feeding it back into the serial port - 
emulating a UART in software is simpicity itself compared to what you've 
already achieved.

Here's the way UARTs do it.

You need to generate a clock, and the really simple way to do this (but 
certainly not recommended for production strength for low SNR RTTY - see 
later) is by simply detecting edge changes. You know how many samples per 
bit there are (100), so you reset a counter when you see either a state 
change, or expect that the clock should change (100 samples later): note 
there will of course not always be a state change. Your bit value is taken 
half a bit time later (counter = 50 samples). Note that the counter is 
incremented each sample time.

For RTTY there will always be a state change within six bits because it's a 
five bit code plus a start and a stop bit. The Start bit is a space (0) and 
the stop bit is a mark (1). The LSB is sent first.

Your software should look for a combination of Space - x - x - x - x - x - 
Mark sequence to get in character (or 'frame') sync.

With five bits, it's obviously not ASCII - it's called Baudot although back 
in my five hole paper tape days here in the UK it was known as Murray code. 
To get ASCII you'll need two lookup tables - one for Figure shift mode and 
one for Letter shift mode. Two unique codes switch between Letter shift and 
Figure shift. http://en.wikipedia.org/wiki/Baudot_code

To generate a better clock when SNR isn't so good, it is common to use 
Costas loop. With the scheme above, a bit of noise will quite easily disrupt 
the clock by resetting it too often.

To get a better data sample, it is common to take an aggregate, that can be 
weighted, of samples over the entire bit period.

There are several other ways to demodulate too.

Cheers, Howard

_____________________________
 Free pdf download: Complex Digital Signal Processing in Telecommunications.


Re: FSK modem: bit/timing recovery - Jerry Avins - 2007-01-23 19:00:00

DSP-Newbie wrote:
> Jerry Avins wrote:
>> DSP-Newbie wrote:
> 
>>>
>>> Now here's my problem: how do I reconstruct/extract the actual data

>>> characters/bytes, timing etc...?? I know that RTTY starts with a 
>>> start bit etc..., but how should this be extracted out of an array

>>> that looks like    11111111111100000001111111110000111110001111??
>>>
>>> Thanks for any advice...
>>>
>>> Dirk.
>>
>> You've come a long way. Congratulations! The basic RTTY  demodulated 
>> signal is just like any other TTY signal. To make sense of it, you 
>> need to know the polarity out of the decoder (inverted or not) and the

>> code used (Probably ASCII, maybe Baudot), and the "framing".
The 
>> framing may be synchronous (probably not) or asynchronous with stop 
>> and start bits in the data stream. (You also need to knw the bit rate,

>> but to get this far, you evidently do.)
>>
>> A piece of hardware called a UART handles the details, and software 
>> UARTs are abundant. http://tinyurl.com/2al6av explains a lot. Come 
>> back with more questions if you need to.
>>
>> Jerry
> 
> Thanks for the reply Jerry,
> 
> Are you suggesting I should send the bit stream to the soundcard output, 
> and connect it to the PC's serial port? This might work for RTTY, but 
> not for SITOR-A/B signals, since these are synchronous.

Not at all. I wouldn't know how to make that work since tones aren't 
bits. Synchronous signaling dispenses with framing bits, but you have to 
know where a character starts. There's a character called "SYN" that
is 
sent repeatedly for long enough so that the receiver can lock up. Nine 
is usually enough. The receiver starts anywhere and decodes 8 bits to 
form a character. If that is not SYN, it slips a bit and does it again. 
When the decoded character is SYN, the receiver adjusts its clock rate 
to match the transmitted clock and continues doing that throughout the 
transmission. The transmitter sends SYNs whenever it has no data.

> I know it can be done without using the serial port,  many programs 
> exist that do it. Don't ask why then I'm not using one of them :0)

Of course. If you understand what the UART (or USART) does, it's not 
difficult to write software to simulate it.

> I guess I have not stated my problem clearly enough.
> =The soundcard sends chunks of 1024 samples at a rate of 10 chunks/sec, 
> sampled at  10240Hz.
> = If we assume a 100 baud signal, 1 bit time = 10msec.

That leaves no time for framing bits with asynchronous transmission. A 
Model 33 KSR runs at 110 bits.second, there being 11 bits per character. 
Since the receiver restarts its clock at the beginning of every 
character and the stop bits can be mistimed without problem, the 
receiver clock must match the transmitter clock within half a bit out of 
nine. Five percent accuracy would be adequate in the absence of jitter. 
Usually, one percent is specifies for each clock, so the timing mismatch 
doesn't exceed two percent.

> = Since the sampling frequence is 10240Hz, I see roughly 100 identical 
> samples at the output of the decision circuit for each 1/0 present in 
> the original data stream.
> (101 becomes 111111111111111000000000000000000011111111111111 )
> 
> 
> Should I "compress" the data by a factor samplefreq./baud or some
such?

You should extract the actual bits, yielding 101, and work with them. 
111111111111111000000000000000000011111111111111 is an ideal case. you 
can tolerate 111011111101111000000000001110000011111101111111 and still 
extract the bit pattern if you have a rough idea where the edges of the 
bits really are.

Jerry
-- 
Engineering is the art of making what you want from things you can get.
¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯

_____________________________
 Free pdf download: Complex Digital Signal Processing in Telecommunications.


Re: FSK modem: bit/timing recovery - DSP-Newbie - 2007-01-24 05:16:00

DSP-Newbie wrote:
> Jerry Avins wrote:
>> DSP-Newbie wrote:
>
>>> 
>>> Now here's my problem: how do I reconstruct/extract the actual data

>>> characters/bytes, timing etc...?? I know that RTTY starts with a
start bit 
>>> etc..., but how should this be extracted out of an array that looks
like   
>>>  11111111111100000001111111110000111110001111??


Jerry, Howard, Ben,

Thanks for the tips, it looks like I'll have to do it the hard way...
Edge detection + sample counting & some bookkeeping between samples 
should do it.

Back to the drawing board :0)

Thanks - Dirk

_____________________________
 Free pdf download: Complex Digital Signal Processing in Telecommunications.


Re: FSK modem: bit/timing recovery - DSP-Newbie - 2007-01-24 05:23:00

Ben Jackson wrote:

>
> Look at some UARTs.  You should be able to find some software or Verilog
> versions with some googling.  Most work by oversampling the data by 16x
> (looks like I trimmed it, but I think you did 10x?).  Then it looks for
> an edge, counts halfway into the bit time and then samples the data.
> Fancier ones might vote on 3 middle-of-bit samples.  Then it just counts
> off another bit time and resamples.

I get the picture...


>  I'm not familiar with RTTY, but if
> every byte has a 'stop' bit, then you just restart this over and over,
> so slight rate changes never accumulate and should not affect you over
> the course of one byte.
>

RTTY mostly uses Baudot 5 bit code, + 1.5 start & 1 stop bit.
SITOR-B ( and many others ) use a 7-bit FEC code but are synchronous, 
so there I must jump on the bit stream and look for special sync 
characters to find the frame sync.

> To demodulate PWM from WWV, where I had sampled 100 times for every bit
> and needed to find the phase of the bits so I could measure the
> pulsewidths, first I edge detected my binary stream by subtracting each
> sample from the next sample.  That left me with an array of the positive-
> going edges.  Then I created a histogram of their offsets mod 100 (my
> oversample factor).  The peak (or even the average) gives you the
> phase offset (mod 100) and then you can count the 1s in each bin.

Interesting tip!

Thanks - Dirk

_____________________________
 Free pdf download: Digital Signal Processing Maths.


Re: FSK modem: bit/timing recovery - Howard Long - 2007-01-25 01:07:00

"DSP-Newbie" <N...@way.invalid> wrote in message 
news:m...@way.invalid...
>
> RTTY mostly uses Baudot 5 bit code, + 1.5 start & 1 stop bit.

Now you've got my memory working, I believe it's 1 start and 1.5 stop. I 
must admit I'd forgotten about using 1.5 bit times for a stop for RTTY.

The baud rate for RTTY by the way is 45.45 baud. This will almost certainly 
mean you'll have a non-integer number of samples which very slightly 
complicates things. If you're using a PC to decode, an easy option is to use 
floating point for your clock and when you reset without a state change, 
subtract the floating point number of samples per bit from the counter 
rather than zeroing it.

Howard

_____________________________
 Free pdf download: Digital Signal Processor Fundamentals and System Design.


Re: FSK modem: bit/timing recovery - Jerry Avins - 2007-01-25 09:46:00

Howard Long wrote:
> "DSP-Newbie" <N...@way.invalid> wrote in message 
> news:m...@way.invalid...
>> RTTY mostly uses Baudot 5 bit code, + 1.5 start & 1 stop bit.
> 
> Now you've got my memory working, I believe it's 1 start and 1.5 stop. I 
> must admit I'd forgotten about using 1.5 bit times for a stop for RTTY.

Never count stop bits except to open a gate. When you get to the last 
data bit, wait some reasonable time and then start looking for a start 
bit. A stop bit is MARK and the signaling code is MARK idle. the 
stop-bit time is an approximate minimum -- approximate because receive 
and send clocks might be a few percent apart. There is no maximum time. 
Stuttering is allowed.

> The baud rate for RTTY by the way is 45.45 baud. This will almost certainly

> mean you'll have a non-integer number of samples which very slightly 
> complicates things. If you're using a PC to decode, an easy option is to
use 
> floating point for your clock and when you reset without a state change, 
> subtract the floating point number of samples per bit from the counter 
> rather than zeroing it.

I take it you've been there! :-)

Jerry
-- 
Engineering is the art of making what you want from things you can get.
¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯

_____________________________
 Free pdf download: Digital Signal Processor Fundamentals and System Design.


| 1 | | 3 |