Overview — eCos Port of a Subset of the TMC Library
The Tilera Multicore Components or TMC Library provides a variety of
primitives for building parallel programs. It is documented in UG527,
“The Applications Libraries Reference Manual”. The eCos
CYGPKG_HAL_TILEGX_TMC implements a subset
of this library. The subset supports communication between Linux
applications running on some of the tiles on a TILE-Gx chip and eCos
applications running on other tiles. The package also contains a
number of example applications demonstrating the communication
CYGPKG_HAL_TILEGX_TMC is automatically
included in any configuration for a TILE-Gx target. It does not have
to be added to the configuration. The package does not add any
overhead to eCos applications which do not use any of its
functionality, so there is no reason for ever removing the package
from the configuration.
The Tilera TMC library consists of the following components:
- UDN helper routines for communication over the UDN bus. Most of these have been ported to eCos.
Performance tuning. The routine
- Spinning shared memory synchronization primitives. These have all been ported to eCos.
- Scheduler shared memory synchronization primitives. These have not been ported. The primitives interact with the Linux kernel, and there is no way for an eCos application running on a BME tile to do that.
- Specified-attribute memory page allocation. These have not been ported. Allocating memory pages involves calling into the hypervisor and the hypervisor is no longer present on BME tiles running eCos. However it is possible for a Linux application to allocate one or more pages, pass the details on to an eCos application, and have the latter map the pages into its address space.
- Common memory, allowing pages to be mapped at the same virtual address in different processes. These have not been ported. Linux applications run in 64-bit mode with a 64-bit address space, whereas eCos applications run in 32-bit mode with 32-bit addresses. This makes it difficult to use the same virtual addresses.
- CPU sets and affinitization. These have not been ported. The primitives are intended to allow threads to be bound to specific tiles. Since each eCos instance runs on only one tile there is no point in attempting to support such affinitization.
- User space interrupt installation routines. These have not been ported. eCos has its own model of how interrupts should be handled and its own routines for managing interrupts. Trying to support the TMC's model of interrupt handling as well would complicate things for little or no gain.
- Interprocessor interrupt event-handling. These have not been ported. It is not clear that they are actually useful since there is no primitive for generating an IPI interrupt. Instead UDN communications provides a way of sending data asynchronously to another tile, and if desired that UDN communication can be processed by an interrupt handler.
Using mspaces for standard malloc/free. These have not been ported.
Under Linux they provide an alternative implementation of the C
free()routines, offering some control over home caching and memory page sizes. For eCos applications it makes more sense to use the standard eCos heap using memory provided by the hypervisor during startup.
Cache control. These functions have not been ported, and the Tilera
documentation recommends against using these low-level shared memory
primitives. The main functionality provided, memory fences to
guarantee visibility of stores to cache coherent memory, is instead
provided by the eCos
- Multiple heap allocation. These routines allow for the allocation of separate mspaces with control over cache homing and other functionality. They involve interaction with the hypervisor's memory page support which is not possible for an eCos application.
- Task management and cleanup. These primitives relate to support for multiple processes and interaction with the Tilera shepherd process. Since eCos does not support multiple processes, only multiple threads, the primitives are not applicable.
The package only provides implements of the header
<tmc/perf.h>. The other
header files do exist but will generate a compile-time warning if they
are included. Providing these dummy headers prevents the compiler from
accidentally including the Linux TMC headers.
The eCos TMC support is subject to a number of important restrictions which application developers must be aware of.
When a Linux process uses
tmc_alloc_map() or a
similar routine to allocate a block of memory, that memory is owned by
the Linux process. Details of the block including the physical address
can be passed on to eCos applications which can then map it into their
address space. If the Linux process exits or gets killed off the Linux
kernel and hypervisor will reclaim the allocated block, which may then
get reused for some other Linux process or for the kernel or
hypervisor itself. Meanwhile the eCos application may still have a
mapping to the underlying physical memory and may still write to it,
corrupting memory that now belongs to some random other part of the
system. Neither the hypervisor nor the Linux kernel have any way of
keeping track of what memory has been mapped into an eCos
application's address space, so they cannot do anything to avoid this
The only solution is to make sure that the eCos application is always informed when the Linux process exits, so that it can unmap any shared memory pages. That is not always easy to achieve, especially in a debug environment, but it is the application developer's responsibility.
Separately, UDN communication is subject to a major restriction. By
default a Linux application does not have access to the UDN bus, and
must explicitly obtain such access from the Linux kernel by a call to
tmc_udn_init(). The kernel only grants UDN access
to one task per tile. In a debug environment LittleBoPeep runs on one
of the tiles providing gdb debug functionality for eCos tiles, and
LittleBoPeep needs to use the UDN bus for this. Therefore Linux
applications requiring UDN access cannot run on the same tile as
LittleBoPeep, usually the highest-numbered tile not used for BME.
The following UDN routines are supported:
The functions related to UDN channel
tmc_udn3_available_count() are not supported.
UDN channel 3 is used for communication between LittleBoPeep and
gdbstubs, and if application code tried to use this channel as well
then things would get very confusing with UDN traffic going to the
There are four other UDN functions in the Tilera TMC library which are
not supported under eCos:
tmc_udn_persist_after_exec(). Under Linux
tmc_udn_init() is needed to request access to the
UDN bus from the Linux kernel. That is not necessary under eCos since
eCos applications run at protection level 2, which is sufficient for
UDN access. The three other functions are also related to access
rights and are equally unnecessary.
Spinning Shared Memory Synchronization
The following are supported:
These data types and functions have the same semantics as the Tilera TMC versions, so the Tilera documentation can be consulted for more details.
An important point about these routines is that the various unlock functions and the barrier wait function involve a memory barrier, guaranteeing that all memory writes are visible to other tiles. Therefore code that only manipulates shared data while owning a lock automatically avoids many memory consistency problems.
The package comes with a number of example applications in
- This example illustrates UDN communication between a TILE-Gx Linux application and one or more instances of an eCos application.
This example is derived
udn0. It adds direct communication between the instances of the eCos application.
- This example sets up a block of shared memory between a Linux application and one or more instances of an eCos application. The shared memory is used to hold large amounts of data. The Linux application sends UDN messages to control what each eCos instance does with that shared data.
- This example also sets up a block of shared memory. However UDN communication is used only during initialization, to set up the shared memory. All subsequent communication between Linux and eCos goes via the shared memory.
- This is a testcase for the various spinning shared memory synchronization primitives.
- This is an example involving a Linux host application, a TILE-GX Linux application, and one or more instances of an eCos worker application. It combines ethernet traffic using the gxio/mpipe library routines running in the TILE-Gx Linux application, and communication between that and the eCos workers over shared memory and the UDN bus. It was written to a specific customer's requirements and may be of limited interest to other users.
|2020-12-09||eCosPro Non-Commercial Public License|