Chapter 155. Basic concepts

155.1. Structure

lwIP has been incorporated into eCos as a single package (CYGPKG_NET_LWIP) which contains all the core lwIP code and the bulk of the eCos port. The remaining elements that constitute the eCos port can be found in the generic Ethernet driver package (CYGPKG_IO_ETH_DRIVERS) and so is only relevant when using an Ethernet-based network card rather than SLIP or PPP. Support for SLIP and PPPoS (PPP-over-Serial) is layered over the standard eCos serial driver API.

The port to eCos has been constructed using the sys_arch porting abstraction within lwIP, and this allows the eCos port to be cleanly separated from the core lwIP code, although it still remains in the lwIP eCos package.

155.2. Application Programming Interfaces (APIs)

There are three different APIs which may be used by applications to interface with the stack: the raw API, the sequential API, or the BSD sockets compatibility API. Each one in turn builds on the functionality provided by the previous API. This allows users the flexibility of choosing a fairly bare implementation to squeeze the maximum out of the available resources; or to use a more powerful API to simplify application coding and reduce time-to-market. Note that despite the presence of the BSD sockets compatibility API, the lwIP stack implementation is not in any way related to the other BSD-derived TCP/IP stacks present in eCos.

The raw API provides an event-based interface with callbacks directly into the application in order to handle incoming/outgoing data and events. There is no inter-thread protection and can only operate with a single thread of execution.

The sequential API is a more traditional style of network interface API which provides functions that may be called synchronously to perform network operations, and where those operations can be considered complete (or will complete asynchronously with no further application interaction) when those functions return to the application.

When using the sequential API (or the BSD sockets API which is layered on top of it), lwIP maintains its own internal thread for network data processing and event management. This is usually referred to as lwIP's TCP/IP thread (even though that is a slight misnomer). This thread uses mailboxes to communicate with application threads, and semaphores to provide mutual exclusion protection.

The BSD sockets compatibility API included in lwIP provides a subset of the Berkeley sockets interface introduced in the BSD 4.2 operating system. The Berkeley sockets interface, recently standardised by ISO/IEC in POSIX 2003.1, will be familiar to those who have developed network applications on Linux, POSIX, UNIX or to a limited extent Windows with Winsock.

As the BSD sockets API provided as part of lwIP is only a subset of the full sockets, it should be considered only as an aid to development or for when porting existing code. It should not be considered as a drop-in replacement for applications written for a complete BSD network stack implementation which supports a wealth of features that do not exist in, and in many cases would be inappropriate for, a low footprint implementation such as lwIP.

155.3. Protocol implementations

lwIP implements a variety of protocols. Support for each protocol can be individually included in or excluded from the configuration, subject to dependency constraints. The protocol implementations are mostly compartmentalised into separate source modules. Support exists for TCP, UDP, UDP-Lite, IP (IPv4 and IPv6), ICMP, ICMP6, ARP when using Ethernet, IGMP, DHCP, AutoIP and Stateless AutoConfiguration, DNS, MLD, ND, SNMP, SNTP, TFTP, SLIP and PPP.

In most cases functionality has been intentionally restricted to avoid "bloat" (unnecessary features increasing resource use), or in some cases completely omitted. This is covered in slightly more detail in Section 155.6, “Limitations”.

155.4. Packet data buffers

lwIP does not only possess features allowing it by itself to maintain a small footprint, but also has design aspects which allow it to work with the application to reduce footprint. One important case of this is lwIP packet data buffers.

Packet data buffers in lwIP are termed pbufs. Pbufs can be chained together in fairly arbitrary ways to create a pbuf chain. The idea is that the application can pass the stack a pbuf of data to transmit, and the stack can prepend and possibly append other pbufs to encapsulate the data in protocol headers/footers without having to copy the data elsewhere, thus saving resources. In some cases, depending on precisely how the pbuf was allocated, the stack may even be able to fit protocol headers inside the pbuf passed to it. It also means that the application can itself provide data allocated in differing ways and from different locations, but assembled together as a pbuf chain. This will ensure that the data is treated as if it were all allocated contiguously. When using the sequential API, the underlying pbufs are wrapped in a netbuf construct in order to provide a simpler API to manipulate data in buffers; but the underlying functionality remains based on pbufs.

When a pbuf is created, it must be one of a variety of types:

PBUF_RAM
This is a conventional buffer, which points to data allocated from a pool in RAM managed by lwIP. On creation the buffer size must be given.
PBUF_ROM

This is a buffer pointing to immutable read-only data. This allows fixed literal data to be stored in ROM/Flash rather than using up precious RAM. Note that data pointed at by a PBUF_ROM pbuf does not literally have to point at read-only memory. All it means is that the data must not change, even if control has returned to the application. The pbuf data may still be being referenced as part of a packet waiting in a queue to be transmitted, or more often, waiting in a queue in case retransmission is necessary.

[Caution]Caution

Not all architectures will allow ethernet transfers direct from ROM, so the underlying hardware device driver may need to perform copying of data as required.

PBUF_REF
This is a buffer pointing to mutable data, passed in by reference. This means data provided by the application allocated from its own resources, and which could change in the future. This differs from PBUF_RAM packets in that the data is allocated by the application, and not from lwIP's PBUF_RAM buffer memory pool. As the application could change the data after control is returned to it, if lwIP finds it must enqueue the pbuf, it will internally copy the data to a new PBUF_RAM. The benefits of this type of packet occur when the packet does not need to be enqueued, and so no PBUF_RAM pbuf needs to be allocated.
PBUF_POOL
The buffer is allocated as a chain of fixed size pbufs from a fixed size pool.

155.5. Configurability

lwIP was designed from the outset to have a low resource footprint. One of the techniques it uses to achieve this goal is its high level of configurability.

lwIP allows both coarse- and fine-grained control of functionality. Large sections of potentially unused functionality can be selected to be removed by the user, including entire protocol stacks. Such examples of removable coarse-grained functionality include UDP, TCP, SLIP, PPP stacks, ethernet/ARP support, IP fragmentation and/or reassembly, or the sequential API.

It has a somewhat modular and layered design to assist with this. It is intentionally only somewhat modular: other TCP/IP stacks have strictly enforced interfaces and abstractions between protocol layers. These abstractions are frequently cumbersome and can result in unnecessary resource implications. lwIP deliberately violates some of these protocol interface layering abstractions where doing so could improve resource utilization. An example is reserving an estimated appropriate amount of space for protocol headers when constructing packets, where the choice of protocol dictates the amount of space.

Where lwIP really stands out is in its fine-grained control over the various pools of resources. Most resources are compartmentalised into fixed size memory pools to allow sizes to be constrained deterministically. The application designer will know, or can choose, the maximum number of network connections which are to be supported depending on application requirements. They also know the level of data throughput required for transmission or reception and can control the levels of the necessary resources appropriately, such as numbers of buffers (separately for incoming or outgoing packet data) and their sizes, numbers of protocol control blocks, TCP window sizes and more.

In this way, application designers can choose a configuration that maximises performance within the limitations of available memory. Clearly, the more constrained the memory, the greater the potential for adverse consequences for performance, or the number of supported connections. However, it should be realised that even with copious quantities of memory resources available to lwIP, it cannot be expected that a stack intentionally designed from the outset to be sparing with memory will perform as well as a stack intentionally designed from the outset for high performance. Nevertheless careful tuning of lwIP almost always results in significant performance improvements.

A simple real-world test application, from a target platform with only a total of 128K of RAM, would when performing a simple test transmitting multiple 1400-byte packets obtain a throughput of 1000k/s. However, the same platform configuration sending multiple 8192-byte packets would see the throughput drop to 100k/s since fragmentation and buffer availability now impact the lwIP performance. System designers need to consider how the application makes use of the available lwIP APIs in conjunction with the resources available to maximise application network throughput.

155.6. Limitations

As already mentioned, lwIP does not seek to provide a complete implementation of a TCP/IP stack providing the same level of functionality provided in large OSes such as Linux, Windows, *BSD, etc. While some aspects are controlled by configuration, in other cases functionality is intentionally limited to fit the design requirements of a compact footprint.

While a complete list of the limitations would be too numerous to enumerate, here are some of the most relevant ones to be aware of:

  • Retransmission and windowing algorithms are implemented simply, at the expense of some performance.
  • Routing is simplified - one gateway per interface. IP forwarding follows the same rules as the host itself.
  • No support for NAT, nor packet filtering.
  • The TCP, DHCP and IP protocols can contain options in their packets. Relatively few of these options are supported by lwIP.
  • IPv4 Path MTU discovery (from RFC1191) is not supported. Ordinarily it is used to avoid fragmentation of packets resulting from the maximum MTU of an intermediate link between source and destination being smaller than the packet sizes actually transmitted. lwIP does however allow the TCP Maximum Segment Size (MSS) to be configured.
  • No complex data structures, caches and search trees to optimise speed. Generally simple lists are used.
  • Thread safety (for the sequential and BSD compatibility API) is implemented in a very simple form. Individual connections should not be operated on by multiple threads simultaneously. The mutual exclusion that is provided is at a very coarse grain - the network processing operations themselves are not multi-threaded.
  • Selective Acknowledgements (SACKs) (from RFC2018) are not provided in the TCP implementation. SACKs are a commonly implemented approach to increasing performance on links subject to packet loss, packet errors or congestion.
  • Most ICMP packet types are ignored.
  • If IP fragmentation and reassembly support is enabled then a limited sequence of IP fragments can be reassembled at one time (controlled by lwIP configuration options). If the number of active sequences supported is exceeded then packet fragments for new sequences are simply dropped, with the hope that a subsequent retransmission may be successful. Received IP fragments are allowed to be reassembled out of order however.
  • The BSD sockets compatibility API does not implement all socket options, API functions, nor API semantics.
  • Error handling for application errors is frequently only handled with asserts - used only during debug builds during development, allowing for smaller production code in release builds.
  • The TCP persist timer is not implemented. If a remote peer has filled its receive window and as a result lwIP stops sending, then when the remote peer processes more data it sends an ACK to update the window. However if that ACK is lost, then if data is entirely unidirectional (lwIP to remote host), the connection could stall. In practice, this has not been something people have experienced really.
  • TCP data is not split in the unsent queue, resulting in somewhat inefficient use of receiver windows.
  • The DNS client support only returns IPv4 addresses.
  • The SNMP agent only provides traps to IPv4 addresses.
  • The SNTP client provides a minimal SNTPv4 implementation.
  • The lwIP IPv6 implementation does not currently track router advertisement routeinfo information. The IPv6 routing simply uses the normal /64 prefix for matching destination addresses against acquired (source) addresses for each indivdual lwIP network interface. If a destination address cannot be matched against an acquired source /64 address then the routingerror (ERR_RTE) code is returned. This is not normally a limitation when using link-local or global addresses, but if an organisation is using unique-local addressing the lwIP stack by default will limit addressing to destinations on the same subnet (i.e. the matching /64 prefix). However, an eCos specific extension exists for supporting unique-local addresses, where the CDL option CYGNUM_LWIP_IPV6_UNIQUELOCAL_MASK can define the number of global prefix bits which are matched (from /48 to /64). This option can be configured to allow destination addresses for other unique-local subnets to be matched against the specific /64 interface unique-local address.

There are many more examples.

If a lwIP Direct device driver is being used then see Section 160.1.4, “GDB/RedBoot” regarding limitation of remote network GDB/RedBoot debugging.

155.7. Quick Start

Incorporating lwIP into your application is straightforward. The essential starting point is to incorporate the lwIP eCos package (CYGPKG_NET_LWIP) into your configuration.

This may be achieved directly using ecosconfig add on the command line, or the Build->Packages… menu item within the eCos Configuration Tool. If you wish to support Ethernet devices, you will also need to include the Common Ethernet Support (CYGPKG_IO_ETH_DRIVERS) eCos package. For SLIP/PPPoS support, you will need to enable the Hardware serial device drivers (CYGPKG_IO_SERIAL_DEVICES) configuration option within the Serial device drivers (CYGPKG_IO_SERIAL) eCos package.

[Note]Note

When using serial devices it is important to ensure the I/O driver configuration provides the necessary buffers and/or hardware flow-control to avoid the possibility of PPPoS/SLIP RX data loss.

Alternatively, as a convenience, configuration templates have been provided to permit an easy starting point for creating a configuration incorporating lwIP. Two templates are provided: lwip_eth for those intending to use lwIP with Ethernet; and lwip_ppp for those intending to use lwIP with PPP. These may be used either by providing the template name as an extra argument on the command line to ecosconfig new; or with the Build->Templates… menu item within the eCos Configuration Tool. Both these templates are basic, incorporating only those packages which are essential for lwIP operation.

At this stage it would be appropriate to tailor the lwIP package configuration to the application requirements. At a minimum it would be appropriate to consider whether a static IP address, or a dynamic IP address served from a DHCP server, is required. Note that if RedBoot is used on the target and incorporates network support, then you must not give lwIP and RedBoot the same IP address. For the same reason, you must not configure both lwIP and RedBoot to obtain an IP address via DHCP.

Prior to coding your application to perform lwIP stack operations using its APIs, the stack must be initialised. This does not happen automatically, and instead a C function must be called:

int cyg_lwip_init (void);

The function declaration can be obtained by including the network.h header file:

#include network.h

cyg_lwip_init returns 0 on success and non-zero on failure. Note that 0 may be returned even if no network interfaces were successfully initialised. This is because in some cases interfaces are brought up asynchronously in any case, devaluing such an error indication; and because an interface not coming up may be expected. If the application needs to determine the status of interfaces, it should query the stack using the netif_* functions using the lwip/netif.h header file.

The cyg_lwip_init function must be called from a thread context. Raw API users need not call this function, although they instead will be required to perform their own stack initialization. Consult the raw API documentation for more information.

[Note]Note

The cyg_lwip_init function, depending on the configuration, may block for some time waiting for interfaces to acquire network addresses. Alternatively the:

int cyg_lwip_init_nowait (void);

function can be called to perform just the necessary low-level initialisation, without the extra addresswait functionality. The cyg_lwip_init function itself uses the cyg_lwip_init_nowait routine prior to waiting for network addresses to be assigned.

Alternatively, the (weak) helper function:

void init_all_network_interfaces (void);

is provided. By default it just calls the lwIP specific cyg_lwip_init initialisation, but it may be overridden by drivers or run-time support if alternative initialisation strategies are required.

[Note]Note

The init_all_network_interfaces name is the same as used by the alternative CYGPKG_NET BSD networking world.

If obtaining an address via DHCP it can be convenient to enable the network interface debugging configuration option within lwIP (CYGDBG_LWIP_DEBUG_NETIF). This will allow the IP address which was set to be viewed on the diagnostic output console. Similarly the helper function:

int cyg_lwip_netif_print_info (struct netif *netif , cyg_lwip_printf *pf );

can be called after cyg_lwip_init to output specific interface address information via the supplied printf-alike routine. For example using diag_printf for the pf parameter will display the address information on the diagnostic output console without having to enable the network interface debugging feature.