SynDEx Communications Under Linux

Pierre POMIERS <>
Tony NOEL <>
This document aims to present how to program SynDEx CAN (or others) communications sequence under Linux (RTLinux or RTAI). It will collect precise information about system calls under Linuses, obtained from various programming guides. By this way, this will help us to design a good communication sequence functioning implementation according to the SynDEx concept.

1 SynDEx communication sequence [1]

1.1 Principles of operation

Each communication medium, i.e. here each CAN bus, is a sequential resource where interprocessor data transfers must be ordered. Rather than a dynamic ordering, done at run time by the CAN arbitration on transfers contents, and which requires identification information to be transferred and decoded, a static ordering is chosen at compile time, where each transfer is uniquely identified simply by its position in the communication sequence. As the CAN bus has no centralized communication sequencer, the communication sequence must be distributed and synchronized on all the processors sharing the bus: at each communication step, all processors know the transfer type and size, the transmitting processor knows its source buffer address, each receiving processor knows its destination buffer address, and each remaining processor is synchronizing, i.e. waits for data as if it were receiving, but does not store the data. Moreover, to avoid dynamic buffer allocation and overruns, the transmitting processor does not start transmitting data before it has received, from each receiving processor, an empty synchronizing frame meaning that the destination buffer is available for reception.

1.2 Present implementation

Each communication step (see CAN_send_, CAN_recv_ and CAN_sync_ macros) takes as arguments the data buffer name, the transmitting processor type and name, and the receiving processors names. Each communication step is processed in several stages under the control of an automaton. Initially, and after each communication step completes, a synch-frame counter is reset, and then incremented by the interrupt triggered by each received synch-frame (automaton state: CAN_init_state). When a communication step is initiated (see CAN_send_shared, CAN_recv_shared, and CAN_sync_shared), the base address and the byte size of the data buffer (computed from the data buffer name), as well as the address of the next instruction in the communication sequence, are stored in a "transfer context" for later use, and the synch-frame counter is decremented by the number of synch-frames to be received. Moreover, in the case of a receive step, a synch-frame is transmitted, regardless of the number of synch-frames already received, but with a CAN message identifier different for each processor; as only the number of synch-frames matters, it is easier to let the CAN bus arbitrate any contention between synch-frames, than to statically order synch-frames (which would require each receiving processor to transmit its synch-frame only after it has received the number of synch-frames which were to be transmitted before it). The automaton then waits for synch-frames in the CAN_send/recv_waitingSynch_state, counting receive interrupts. When the synch-frame counter reaches zero, all the receiving processors have their destination buffer ready for reception, then data transfers may begin using and updating the "transfer context" which holds the number of data bytes remaining to transfer and the address of the next byte to transfer. In the case of the transmitting processor (CAN_sending_state), CAN frame buffers are loaded with data and submitted for transmission; for a receiving processor (CAN_recving_state), frame buffers are (stored and re-)submitted for reception; for a synchronizing processor (CAN_sync_state), frame buffers are only submitted for reception (their contents is not stored to memory). After each frame buffer(s) submission, the automaton waits for the end-of-transfer interrupt, letting the processor execute the computation sequence. When the number of bytes remaining to transfer reaches zero, the communication sequence is resumed (CAN_resume_state) at the address stored when the communication step was initiated.

1.3 What happens from now?

The communication sequence described before is the one implemented both for 555 and 386/DJGPP machines. Its functioning reliability has been proofed through numbers of tests. Porting SynDEx communication kernel under Linux will consist in re-programming the set of CAN macros using Linux system programming possibilities. This way is a bit different from what has been done before. Indeed, when we were programming whether under 386/DJGPP or 555, we had a total control of the machine resources (IRQs, interrupt vector table, system calls...). Using Linux, we do not have direct access to low level programming anymore. As a secured Operating System, Linux kernel forbid direct hardware access by handling all the machine resources.

2 How do we access BIOS functions from Linux? [2]

No way. This is protected mode, use OS services instead. Again, you can't use int 0x10, int 0x13, etc. Fortunately almost everything can be implemented through system calls or library functions. In the worst case you may go through direct port access, or make a kernel patch to implement needed functionality.

2.1 Kernel modules [4]

Getting access to the Linux kernel "World" is done using what we call `modules'. Modules can be seen as pieces of programs adding functionality to an actual kernel (in a way like kernel patches). Before we go further, it is worth underlying some fundamental difference between a kernel module and a classical application.

While an application performs a single task from beginning to end, a kernel module registers itself in order to serve future requests. There are two main parts in a modules. First, the module's "main" function init_module(), which is in fact the module entry point, aims to prepare module's functions for later invocations. The second entry point of a module, cleanup_module, gets invoked just before the module is unloaded and aims to terminate the module's run.

Moreover, it is important to precise that, unlike application, is not linked through libraries (or any other external references) but directly linked to the kernel. Hence, the only functions a module can call, are the one exported by the kernel. This is a very important point. It very constrain our way to program SynDEx applications.

2.2 Influence on the SynDEx code generation

Before, all code needed by a SynDEx generated application, was included in a same assembler file. This was possible because no OS was restricting the access to hardware programming. Now, under Linux, as modules things are different. In order to interface our applications with the hardware (always through kernel function calls), we will have to make use of module(s).

For the moment, documentation is not very clean on this point: it is not sure it is possible to handler several IRQs to a same module (as we did in a classical 386/DJGPP SynDEx application). Some tests has been done to test if multiple IRQs assignment was possible or not. A rapid analysis of the /proc/interrupts Linux system file shows us that our assumption was correct. Nevertheless, we are not sure about the OS reactions to this way of programming yet? If this become to be impossible, we have to imagine the possibility to split our SynDEx application into several parts: one CAN communication module, eventually one Ethernet communication module and one "main" application process (or task whether we will use Linux, RTLinux or RTAI). All these pieces of program running together will have to communicate in order to exchange information. This additional mechanism will surely be realized making use of shared memory or fifos or others.

2.3 Preliminary information about Interrupt Handlers [3]

There are two types of interaction between the CPU and the rest of the computer's hardware. The first type is when the CPU gives orders to the hardware, the other is when the hardware needs to tell the CPU something. The second, called interrupts, is much harder to implement because it has to be dealt with when convenient for the hardware, not the CPU. Hardware devices typically have a very small amount of ram, and if you don't read their information when available, it is lost.

Under Linux, there are two types of IRQs, short and long. A short IRQ is one which is expected to take a very short period of time, during which the rest of the machine will be blocked and no other interrupts will be handled. A long IRQ is one which can take longer, and during which other interrupts may occur (but not interrupts from the same device). If at all possible, it's better to declare an interrupt handler to be long.

When the CPU receives an interrupt, it stops whatever it's doing (unless it's processing a more important interrupt, in which case it will deal with this one only when the more important one is done), saves certain parameters on the stack and calls the interrupt handler. This means that certain things are not allowed in the interrupt handler itself, because the system is in an unknown state. The solution to this problem is for the interrupt handler to do what needs to be done immediately, usually read something from the hardware or send something to the hardware, and then schedule the handling of the new information at a later time (this is called the `bottom half') and return. The kernel is then guaranteed to call the bottom half as soon as possible -- and when it does, everything allowed in kernel modules will be allowed.

The way to implement this is to call request_irq to get your interrupt handler called when the relevant IRQ is received (in our case there are 16 of them as Linux will run on an Intel platforms). This function receives the IRQ number, the name of the function, flags, a name for /proc/interrupts and a parameter to pass to the interrupt handler. The flags can include SA_SHIRQ to indicate you're willing to share the IRQ with other interrupt handlers (usually because a number of hardware devices sit on the same IRQ) and SA_INTERRUPT to indicate this is a fast interrupt. This function will only succeed if there isn't already a handler on this IRQ, or if you're both willing to share.

Then, from within the interrupt handler, we communicate with the hardware and then use queue_task_irq with tq_immediate and mark_bh(BH_IMMEDIATE) to schedule the bottom half.

2.4 Implementation differences between Linux and (RTLinux/RTAI) [5][6]

At that step, we do not know yet what type of platform OS we will use: Linux, RTLinux or RTAI? Let us not focus on this decision for the moment, but let us just check what difference we would encounter while implementing the communication macros. Before going on, let us remember that RTLinux and RTAI are based on the same principle and are considerate to be real-time Linuses.

For writing modules under "classical" Linux, we have to use functions provided by module dedicated library such as "module.h". They allow to manipulate hardware interrupts, interrupt handling. Normally, they should also allow you to deal with scheduling. Nevertheless, Linux was originally designed as a time-sharing system, that is to say that it strive to optimize average performance. Hence, the execution of a module running under Linux kernel control, will be constrained by other running process: in other words, there is really no way to guaranty a precise execution rate or time schedule.

Writing modules under (RTLinux/RTAI) is also possible. An other set of functions is available to this aim. Here, the main advantage of using (RTLinux/RTAI) is that it will allow modules execution to respect hard real-time constraints. In other words, it will provide low latency and high predictability.

Finally, it seems to be not so different implementing a module under Linux or (RTLinux/RTAI). This only important thing we have to keep in mind is that, "classical Linux" modules can only be interfaced with other "classical Linux" process. On the opposite, "(RTLinux/RTAI)" modules can be used both by other "(RTLinux/RTAI)" processes or by "classical Linux" processes. So, for the moment, just considering future users application, it seems to be less restrictive to implement "(RTLinux/RTAI)" modules: hence, users already using (RTLinux/RTAI) functionality will continue to have the possibility to use them into their SynDEx application, where for the other users, it will be totally transparent. Nevertheless, if we decide to implement SynDEx macros using (RTLinux/RTAI) modules, future users wanting to design SynDEx application for heterogeneous architecture including Linux machine, will be obliged to install (RTLinux/RTAI) on their Linux box. Is it really a constraint? I am not sure (in fact I am sure it is not), but some persons could be reluctant to do so.


Considering what we described above, there are not many way to program SynDEx communication macros. We have to use modules programming to access hardware resources: no other way is possible (unless we become kernel wizards and re-program our own SynDEx oriented Linux kernel... but I think it is not the purpose).

This way of programming macros will change a bit the way of macroprocessing applications code. Unlike before, we will not have just a single file to compile and execute but several communicating modules. In fact, we will have to deal with two programming layers: one concerning the modules code proper and a second one, relying on Unix mechanisms for modules "management".

The question of Linux or (RTLinux/RTAI) modules implementation is also an important point. Implementing modules under "classical Linux" is something very possible, but the "time-sharing" schedule method used by this OS is not compatible with our needs. It would be possible to program our own "SynDEx" scheduler and our own synchronizing methods but it would surely be very time consuming. On the other side, we have (RTLinux/RTAI). One could feel there is a paradox designing SynDEx using an other real-time OS. But, from a programming point of view, (RTLinux/RTAI) have to be seen as a toolbox providing many reliable methods (shared memory, semaphores, real-time scheduler...). Using (RTLinux/RTAI) would allow us to focus on the real SynDEx software architecture, without spending time programming already existing tools.

Now, before ending this paper, let us just have a purely technical word about (RTLinux and RTAI). RTAI is an Italian variant of RTLinux that has gone off on its own path. GPL encourages such divergence. Many differences are purely cosmetic, however the fundamental difference between the two systems is attitude towards features. The main thing that could make us choose RTAI rather that RTLinux is its aptitude to handle partially-linked modules through standard or math library for example (what RTLinux does not seem to propose). Moreover, unlike RTLinux, RTAI does not obliged user to patch and recompile its working Linux kernel. RTAI features (like fifos, scheduler and so on) can be inserted into Linux kernel using insmod from the shell command line.


[1] CAN.m4x
[3] Linux Kernel Module Programming Guide (LKMPG).
[4] Alessandro RUBINI, "Linux Device Drivers", O'REILLY, February 1998.
[5] Michael Barabanov, "A Linux-based Real-Time Operating System", June 1997.
[6] Dipartimento di Ingegneria Aerospaziale - Politecnico di Milano
Real Time Application Interface web site