Data transfer for real time analysis

hi,

We are planning a data acquisition system with 4-12 sub units. Each sub unit
produce data of 25MBit/sec. The distance between the boxes and computer is
about 2 meters.
I'm willing to transfer and analyze the data in real-time. I'm willing to
close a control loop in 1-5msec (I. wish 100usec ......).
What is the recommended serial protocol?
The sub units will be in-house made the host computer for data analysis and
storage will be a PC (probably with raid HD).
What communication HW should I use in my sub units?
What communication HW should I use in the PC?

Regards
    Pini

In article <c6h98p$3jm$1@news.iucc.ac.il>, Pini wrote:
> 
> We are planning a data acquisition system with 4-12 sub units. Each sub unit
> produce data of 25MBit/sec. The distance between the boxes and computer is
> about 2 meters.
> I'm willing to transfer and analyze the data in real-time. I'm willing to
> close a control loop in 1-5msec (I. wish 100usec ......).
> What is the recommended serial protocol?
> The sub units will be in-house made the host computer for data analysis and
> storage will be a PC (probably with raid HD).
> What communication HW should I use in my sub units?
> What communication HW should I use in the PC?

You're kidding yourself if you think a _reliable_ control loop can
be closed in 1msec or less using general purpose networks and computers.
A dedicated (OS-free) CPU and hard-wired serial links (like you see
on DSP chips) can do the job.  So can an FPGA.  But to get an answer
to your question, you need to
  1) specify your problem better (I/O, volume, budget, compute horsepower)
  2) move it to a better set of groups, that's more hardware-oriented.
     comp.arch.embedded might still be OK, but drop windowsce and Linux,
     and maybe add comp.arch.fpga.

    - Larry

On Mon, 26 Apr 2004 00:07:11 +0200, "Pini" <pinit@ee.bgu.ac.il> wrote:

>hi,
>
>We are planning a data acquisition system with 4-12 sub units. Each sub unit
>produce data of 25MBit/sec. The distance between the boxes and computer is
>about 2 meters.
>I'm willing to transfer and analyze the data in real-time. I'm willing to
>close a control loop in 1-5msec (I. wish 100usec ......).
>What is the recommended serial protocol?
>The sub units will be in-house made the host computer for data analysis and
>storage will be a PC (probably with raid HD).

Unless you have complex interconnections between the units (input from
unit 1 controls outputs on units 3, 7 and 12) or are planning to sell
several hundred systems each year, I would suggest using a three level
hierarchy. 

Use one high level PC system for user interface, data representation
and permanent storage, use 1-3 "PC" computers to run the control loop
and connect 4 sub-units to each control loop unit using standard 100
Mbit/s Ethernet (requires a 4 line PCI card on each control system).
All kinds of Ethernet hardware is available off the shelf and the
prices are low.

If you want 100 us control loop cycles, you definitively are going to
need some real time kernels (or real time extensions to Windows or
Linux) in those control units. 

Assuming 25 MBit/s and 100 us cycle time, that would be 2.5
kbit/cycle, that will produce a medium sized Ethernet frame, thus one
frame from the sub-unit to the controller and one the other way should
be more than sufficient. 

With control loop cycle times in the few millisecond range, you might
even be able to use standard Windows NT/CE or Linux if you can
tolerate some data losses now and then, but of course, you should not
touch (or even gaze at :-) these  control units when they are running
the control loop or you may loose a sample. In this case it might also
be advantageous to run non-standard Jumbo Ethernet frames (several
kilobytes/frame). 

In any case, use Ethernet addressing and a simple master slave
relationship and you do not have to worry about collisions or IP
overhead (although UDP/IP overhead is quite low). 

Do some data reduction in the controlling system (filtering, FFT)
before passing on the data to the high level PC system for data
analysis, presentation and storage.

Paul

Pini wrote:

> hi,
> 
> We are planning a data acquisition system with 4-12 sub units. Each sub unit
> produce data of 25MBit/sec. The distance between the boxes and computer is
> about 2 meters.
> I'm willing to transfer and analyze the data in real-time. I'm willing to
> close a control loop in 1-5msec (I. wish 100usec ......).
> What is the recommended serial protocol?
> The sub units will be in-house made the host computer for data analysis and
> storage will be a PC (probably with raid HD).
> What communication HW should I use in my sub units?
> What communication HW should I use in the PC?
> 
> Regards
>     Pini
> 
> 

Look at the CAN bus, and at the TTI (time-triggered-interval) bus for 
automotive applications.  TI, Motorola and Analog Devices all put CAN 
controllers on some of their chips these days, and with a good real-time 
OS (_not_ a windows-like OS!) you can certainly meet the 5ms figure, and 
probably the 1ms figure.  100us will be out of reach unless you're doing 
very simple control calculations and you stick it in the CAN 
controller's ISR.

-- 

Tim Wescott
Wescott Design Services
http://www.wescottdesign.com

On Mon, 26 Apr 2004 08:30:51 -0700, Tim Wescott
<tim@wescottnospamdesign.com> wrote:

>
>Look at the CAN bus, and at the TTI (time-triggered-interval) bus for 
>automotive applications.  TI, Motorola and Analog Devices all put CAN 
>controllers on some of their chips these days, and with a good real-time 
>OS (_not_ a windows-like OS!) you can certainly meet the 5ms figure, and 
>probably the 1ms figure.  100us will be out of reach unless you're doing 
>very simple control calculations and you stick it in the CAN 
>controller's ISR.

Please tell more about the TTI bus. 

I have had some problems running just 0.5 Mbit/s on average over
standard CAN bus. I would be very interested to hear how to _reliably_
run 25 Mbit/s over TTI bus. 

Apparently the TTI bus is also sufficiently fast so that you can run
both the measurement and control frames within 100 us.

Paul

















:-) :-).

Paul Keinanen wrote:
> On Mon, 26 Apr 2004 08:30:51 -0700, Tim Wescott
> <tim@wescottnospamdesign.com> wrote:
> 
> 
>>Look at the CAN bus, and at the TTI (time-triggered-interval) bus for 
>>automotive applications.  TI, Motorola and Analog Devices all put CAN 
>>controllers on some of their chips these days, and with a good real-time 
>>OS (_not_ a windows-like OS!) you can certainly meet the 5ms figure, and 
>>probably the 1ms figure.  100us will be out of reach unless you're doing 
>>very simple control calculations and you stick it in the CAN 
>>controller's ISR.
> 
> 
> Please tell more about the TTI bus. 
> 
> I have had some problems running just 0.5 Mbit/s on average over
> standard CAN bus. I would be very interested to hear how to _reliably_
> run 25 Mbit/s over TTI bus. 
> 
> Apparently the TTI bus is also sufficiently fast so that you can run
> both the measurement and control frames within 100 us.
> 
> Paul
>
> :-) :-).
> 

Dang!  I missed the data rate -- I'm hanging my head in shame here. 
I've had good success with 1Mb/s on CAN, but that's over short runs in a 
well-controlled environment.

I don't know much about the TTI bus, other than (a) it's proponents 
think very highly of it if you need a hard-real-time data communications 
bus for control systems, and (b) at the time that I was really studying 
it hard there was no silicon available, and floods of CAN silicon.  I 
suspect that it won't support 25Mb/s either.

I know you could meet 25Mb/s with isochronous packets on IEEE-1394b, but 
you'd be hard pressed to meet 250us; 375us starts to get doable, and 
500us may even be practical.  I've heard that there's an isochronous 
protocol that can be used with Ethernet, but "I've heard" is about the 
most that I can say about it.

-- 

Tim Wescott
Wescott Design Services
http://www.wescottdesign.com

 
Pini-

More data is needed before you design. Windows CE can run quite well
"Inside " the one millisecond level, but consider.


1) how many loops and other calculations? 25 Megabit per second could
mean A tremendous number of variables.  Just slamming these onto a hard
drive could give you a load that might make real time PID loop
processing impossible .

2) whatever you do think twice about closing the <1ms loop through a
stack of any kind .  Then after you think twice , don't do it .

3) You might take a look at embedded xp with the RTX extensions from
venturecom.  this is real time about as hard as you're ever going to
see , and you can buy as much processor as you want .

Larry "RT is Me" Ricci (eMvp) 

  
> hi,
> 
> We are planning a data acquisition system with 4-12 sub units. Each
> sub unit produce data of 25MBit/sec. The distance between the boxes
> and computer is about 2 meters.
> I'm willing to transfer and analyze the data in real-time. I'm
> willing to close a control loop in 1-5msec (I. wish 100usec ......).
> What is the recommended serial protocol?
> The sub units will be in-house made the host computer for data
> analysis and storage will be a PC (probably with raid HD).
> What communication HW should I use in my sub units?
> What communication HW should I use in the PC?
> 
> Regards
>     Pini



-- 
Lawrence 'Larry' Ricci  MVP windows CE 

On Mon, 26 Apr 2004 15:24:44 -0700, Tim Wescott
<tim@wescottnospamdesign.com> wrote:

>>>100us will be out of reach unless you're doing 
>>>very simple control calculations and you stick it in the CAN 
>>>controller's ISR.

>> Apparently the TTI bus is also sufficiently fast so that you can run
>> both the measurement and control frames within 100 us.
>> :-):-)

>I've had good success with 1Mb/s on CAN, but that's over short runs in a 
>well-controlled environment.

I guess you must code all the data you want to transfer into the 11
bit frame IDs and send no actual data bytes to reach the 100 us
measurement+control cycle. This can be practical if there are only two
stations on the line.

A typical CAN frame with 8 data bytes is in the order of 100 bits, so
at 1 Mbit/s the frame time is typically 100 us. With worst case bit
stuffing the measurement and control frame pair can take about 250-300
us total. If you also want to transfer some low priority frames
between these real time frames, the control cycle time should be at
least 400 us or about 2.5 kHz sampling frequency, assuming no protocol
stack overhead. 

Paul
   

Seems like a task for ATM. 

Maybe 1000 MBit Ethernet can do this _quite_ reliably but you can't be
sure.

There are Networks done with firewile (e.g. by Yamaha for musical
equipment). Perhaps this is worth a look, too.

-Michael

Paul Keinanen wrote:

> On Mon, 26 Apr 2004 15:24:44 -0700, Tim Wescott
> <tim@wescottnospamdesign.com> wrote:
> 
> 
>>>>100us will be out of reach unless you're doing 
>>>>very simple control calculations and you stick it in the CAN 
>>>>controller's ISR.
> 
> 
>>>Apparently the TTI bus is also sufficiently fast so that you can run
>>>both the measurement and control frames within 100 us.
>>>:-):-)
> 
> 
>>I've had good success with 1Mb/s on CAN, but that's over short runs in a 
>>well-controlled environment.
> 
> 
> I guess you must code all the data you want to transfer into the 11
> bit frame IDs and send no actual data bytes to reach the 100 us
> measurement+control cycle. This can be practical if there are only two
> stations on the line.
> 
> A typical CAN frame with 8 data bytes is in the order of 100 bits, so
> at 1 Mbit/s the frame time is typically 100 us. With worst case bit
> stuffing the measurement and control frame pair can take about 250-300
> us total. If you also want to transfer some low priority frames
> between these real time frames, the control cycle time should be at
> least 400 us or about 2.5 kHz sampling frequency, assuming no protocol
> stack overhead. 
> 
> Paul
>    

I was focusing on the upper time limit of 5ms.  If you want 100us you 
certainly need some other technology.

-- 

Tim Wescott
Wescott Design Services
http://www.wescottdesign.com