Chapter 159. Writing Ethernet Device Drivers

159.1. Generic Ethernet API

This section provides a simple description of how to write a low-level, hardware dependent ethernet driver. In eCos this is known as a “standard” driver.

There is a high-level driver (which is only code — with no state of its own) that is part of the stack. There will be one or more low-level drivers tied to the actual network hardware. Each of these drivers contains one or more driver instances. The intent is that the low-level drivers know nothing of the details of the stack that will be using them. Thus, the same driver can be used by the eCos supported TCP/IP stack, RedBoot, or any other, with no changes.

A driver instance is contained within a struct eth_drv_sc:

struct eth_hwr_funs {
    // Initialize hardware (including startup)
    void (*start)(struct eth_drv_sc *sc,
                  unsigned char *enaddr,
                  int flags);
    // Shut down hardware
    void (*stop)(struct eth_drv_sc *sc);
    // Device control (ioctl pass-thru)
    int  (*control)(struct eth_drv_sc *sc,
                    unsigned long key,
                    void *data,
                    int   data_length);
    // Query - can a packet be sent?
    int  (*can_send)(struct eth_drv_sc *sc);
    // Send a packet of data
    void (*send)(struct eth_drv_sc *sc,
                 struct eth_drv_sg *sg_list,
                 int sg_len,
                 int total_len,
                 unsigned long key);
    // Receive [unload] a packet of data
    void (*recv)(struct eth_drv_sc *sc,
                 struct eth_drv_sg *sg_list,
                 int sg_len);
    // Deliver data to/from device from/to stack memory space
    // (moves lots of memcpy()s out of DSRs into thread)
    void (*deliver)(struct eth_drv_sc *sc);
    // Poll for interrupts/device service
    void (*poll)(struct eth_drv_sc *sc);
    // Get interrupt information from hardware driver
    int (*int_vector)(struct eth_drv_sc *sc);
    // Logical driver interface
    struct eth_drv_funs *eth_drv, *eth_drv_old;
};

struct eth_drv_sc {
    struct eth_hwr_funs *funs;
    void                *driver_private;
    const char          *dev_name;
    int                  state;
    struct arpcom        sc_arpcom; /* ethernet common */
};
[Note]Note

If you have two instances of the same hardware, you only need one struct eth_hwr_funs shared between them.

There is another structure which is used to communicate with the rest of the stack:

struct eth_drv_funs {
    // Logical driver - initialization
    void (*init)(struct eth_drv_sc *sc,
                 unsigned char *enaddr);
    // Logical driver - incoming packet notifier
    void (*recv)(struct eth_drv_sc *sc,
                 int total_len);
    // Logical driver - outgoing packet notifier
    void (*tx_done)(struct eth_drv_sc *sc,
                    CYG_ADDRESS key,
                    int status);
};

Your driver does not create an instance of this structure. It is provided for driver code to use in the eth_drv member of the function record. Its usage is described below in Section 159.3, “Upper Layer Functions”

One more function completes the API with which your driver communicates with the rest of the stack:

extern void eth_drv_dsr(cyg_vector_t vector,
                        cyg_ucount32 count,
                        cyg_addrword_t data);

This function is designed so that it can be registered as the DSR for your interrupt handler. It will awaken the “Network Delivery Thread” to call your deliver routine. See Section 159.2.7, “Deliver function”.

You create an instance of struct eth_drv_sc using the ETH_DRV_SC() macro which sets up the structure, including the prototypes for the functions, etc. By doing things this way, if the internal design of the ethernet drivers changes (e.g. we need to add a new low-level implementation function), existing drivers will no longer compile until updated. This is much better than to have all of the definitions in the low-level drivers themselves and have them be (quietly) broken if the interfaces change.

The “magic” which gets the drivers started (and indeed, linked) is similar to what is used for the I/O subsystem. This is done using the NETDEVTAB_ENTRY() macro, which defines an initialization function and the basic data structures for the low-level driver.

  typedef struct cyg_netdevtab_entry {
      const char        *name;
      bool             (*init)(struct cyg_netdevtab_entry *tab);
      void              *device_instance;
      unsigned long     status;
  } cyg_netdevtab_entry_t;

The device_instance entry here would point to the struct eth_drv_sc entry previously defined. This allows the network driver setup to work with any class of driver, not just ethernet drivers. In the future, there will surely be serial PPP drivers, etc. These will use the NETDEVTAB_ENTRY() setup to create the basic driver, but they will most likely be built on top of other high-level device driver layers.

To instantiate itself, and connect it to the system, a hardware driver will have a template (boilerplate) which looks something like this:

#include <cyg/infra/cyg_type.h>
#include <cyg/hal/hal_arch.h>
#include <cyg/infra/diag.h>
#include <cyg/hal/drv_api.h>
#include <cyg/io/eth/netdev.h>
#include <cyg/io/eth/eth_drv.h>

ETH_DRV_SC(DRV_sc,
           0,             // No driver specific data needed
           "eth0",        // Name for this interface
           HRDWR_start,
           HRDWR_stop,
           HRDWR_control,
           HRDWR_can_send
           HRDWR_send,
           HRDWR_recv,
           HRDWR_deliver,
           HRDWR_poll,
           HRDWR_int_vector
);

NETDEVTAB_ENTRY(DRV_netdev,
                "DRV",
                DRV_HRDWR_init,
                &DRV_sc);

This, along with the referenced functions, completely define the driver.

[Note]Note

If one needed the same low-level driver to handle multiple similar hardware interfaces, you would need multiple invocations of the ETH_DRV_SC()/NETDEVTAB_ENTRY() macros. You would add a pointer to some instance specific data, e.g. containing base addresses, interrupt numbers, etc, where the

        0, // No driver specific data

is currently.

159.2. Review of the functions

Now a brief review of the functions. This discussion will use generic names for the functions — your driver should use hardware-specific names to maintain uniqueness against any other drivers.

159.2.1. Init function

static bool DRV_HDWR_init(struct cyg_netdevtab_entry *tab)

This function is called as part of system initialization. Its primary function is to decide if the hardware (as indicated via tab->device_instance) is working and if the interface needs to be made available in the system. If this is the case, this function needs to finish with a call to the ethernet driver function:

    struct eth_drv_sc *sc = (struct eth_drv_sc *)tab->device_instance;
    ….initialization code….
    // Initialize upper level driver
    (sc->funs->eth_drv->init)( sc, unsigned char *enaddr );

where enaddr is a pointer to the ethernet station address for this unit, to inform the stack of this device's readiness and availability.

[Note]Note

The ethernet station address (ESA) is supposed to be a world-unique, 48 bit address for this particular ethernet interface. Typically it is provided by the board/hardware manufacturer in ROM.

In many packages it is possible for the ESA to be set from RedBoot, (perhaps from 'fconfig' data), hard-coded from CDL, or from an EPROM. A driver should choose a run-time specified ESA (e.g. from RedBoot) preferentially, otherwise (in order) it should use a CDL specified ESA if one has been set, otherwise an EPROM set ESA, or otherwise fail. See the cl/cs8900a ethernet driver for an example.

159.2.2. Start function

static void
HRDWR_start(struct eth_drv_sc *sc, unsigned char *enaddr, int flags)

This function is called, perhaps much later than system initialization time, when the system (an application) is ready for the interface to become active. The purpose of this function is to set up the hardware interface to start accepting packets from the network and be able to send packets out. The receiver hardware should not be enabled prior to this call.

[Note]Notes:
  • This function will be called whenever the up/down state of the logical interface changes, e.g. when the IP address changes, or when promiscuous mode is selected by means of an ioctl() call in the application. This may occur more than once, so this function needs to be prepared for that case.
  • In future, the flags field (currently unused) may be used to tell the function how to start up, e.g. whether interrupts will be used, alternate means of selecting promiscuous mode etc.

159.2.3. Stop function

static void HRDWR_stop(struct eth_drv_sc *sc)

This function is the inverse of “start.” It should shut down the hardware, disable the receiver, and keep it from interacting with the physical network.

159.2.4. Control function

static int
HRDWR_control(
        struct eth_drv_sc *sc, unsigned long key,
        void *data, int len)

This function is used to perform low-level “control” operations on the interface. These operations would typically be initiated via ioctl() calls in the BSD stack, and would be anything that might require the hardware setup to change (i.e. cannot be performed totally by the platform-independent layers).

The key parameter selects the operation, and the data and len params point describe, as required, some data for the operation in question.

Available Operations:

ETH_DRV_SET_MAC_ADDRESS
This operation sets the ethernet station address (ESA or MAC) for the device. Normally this address is kept in non-volatile memory and is unique in the world. This function must at least set the interface to use the new address. It may also update the NVM as appropriate.
ETH_DRV_GET_IF_STATS_UD, ETH_DRV_GET_IF_STATS

These acquire a set of statistical counters from the interface, and write the information into the memory pointed to by data. The “UD” variant explicitly instructs the driver to acquire up-to-date values. This is a separate option because doing so may take some time, depending on the hardware.

The definition of the data structure is in cyg/io/eth/eth_drv_stats.h.

This call is typically made by SNMP.

ETH_DRV_SET_MC_LIST

This entry instructs the device to set up multicast packet filtering to receive only packets addressed to the multicast ESAs in the list pointed to by data.

The format of the data is a 32-bit count of the ESAs in the list, followed by packed bytes which are the ESAs themselves, thus:

#define ETH_DRV_MAX_MC 8
struct eth_drv_mc_list {
    int len;
    unsigned char addrs[ETH_DRV_MAX_MC][ETHER_ADDR_LEN];
};
ETH_DRV_SET_MC_ALL
This entry instructs the device to receive all multicast packets, and delete any explicit filtering which had been set up.

This function should return zero if the specified operation was completed successfully. It should return non-zero if the operation could not be performed, for any reason.

159.2.5. Can-send function

static int HRDWR_can_send(struct eth_drv_sc *sc)

This function is called to determine if it is possible to start the transmission of a packet on the interface. Some interfaces will allow multiple packets to be "queued" and this function allows for the highest possible utilization of that mode.

Return the number of packets which could be accepted at this time, zero implies that the interface is saturated/busy.

159.2.6. Send function

struct eth_drv_sg {
    CYG_ADDRESS  buf;
    CYG_ADDRWORD len;
};

static void
HRDWR_send(
        struct eth_drv_sc *sc,
        struct eth_drv_sg *sg_list, int sg_len,
        int total_len, unsigned long key)

This function is used to send a packet of data to the network. It is the responsibility of this function to somehow hand the data over to the hardware interface. This will most likely require copying, but just the address/length values could be used by smart hardware.

[Note]Note

All data in/out of the driver is specified via a “scatter-gather” list. This is just an array of address/length pairs which describe sections of data to move (in the order given by the array), as in the struct eth_drv_sg defined above and pointed to by sg_list.

Once the data has been successfully sent by the interface (or if an error occurs), the driver should call (sc->funs->eth_drv->tx_done)() (see Section 159.3.2, “Callback Tx-Done function”) using the specified key. Only then will the upper layers release the resources for that packet and start another transmission.

[Note]Note

In future, this function may be extended so that the data need not be copied by having the function return a “disposition” code (done, send pending, etc). At this point, you should move the data to some “safe” location before returning.

159.2.7. Deliver function

static void
HRDWR_deliver(struct eth_drv_sc *sc)

This function is called from the “Network Delivery Thread” in order to let the device driver do the time-consuming work associated with receiving a packet — usually copying the entire packet from the hardware or a special memory location into the network stack's memory.

After handling any outstanding incoming packets or pending transmission status, it can unmask the device's interrupts, and free any relevant resources so it can process further packets.

It will be called when the interrupt handler for the network device has called

    eth_drv_dsr( vector, count, (cyg_addrword_t)sc );

to alert the system that “something requires attention.” This eth_drv_dsr() call must occur from within the interrupt handler's DSR (not the ISR) or actually be the DSR, whenever it is determined that the device needs attention from the foreground. The third parameter (data in the prototype of eth_drv_dsr() must be a valid struct eth_drv_sc pointer sc.

The reason for this slightly convoluted train of events is to keep the DSR (and ISR) execution time as short as possible, so that other activities of higher priority than network servicing are not denied the CPU by network traffic.

To deliver a newly-received packet into the network stack, the deliver routine must call the following which will in turn call the receive function, which we talk about next.

(sc->funs->eth_drv->recv)(sc, len);

See also Section 159.3.3, “Callback Receive function” below.

159.2.8. Receive function

static void
HRDWR_recv(
        struct eth_drv_sc *sc,
        struct eth_drv_sg *sg_list, int sg_len)

This function is a call back, only invoked after the upper-level function

(sc->funs->eth_drv->recv)(struct eth_drv_sc *sc, int total_len)

has been called itself from your deliver function when it knows that a packet of data is available on the interface. The (sc->funs->eth_drv->recv)() function then arranges network buffers and structures for the data and then calls HRDWR_recv() to actually move the data from the interface.

A scatter-gather list (struct eth_drv_sg) is used once more, just like in the send case.

159.2.9. Poll function

static void
HRDWR_poll(struct eth_drv_sc *sc)

This function is used when in a non-interrupt driven system, e.g. when interrupts are completely disabled. This allows the driver time to check whether anything needs doing either for transmission, or to check if anything has been received, or if any other processing needs doing.

It is perfectly correct and acceptable for the poll function to look like this:

static void
HRDWR_poll(struct eth_drv_sc *sc)
{
   my_interrupt_ISR(sc);
   HRDWR_deliver(struct eth_drv_sc *sc);
}

provided that both the ISR and the deliver functions are idempotent and harmless if called when there is no attention needed by the hardware. Some devices might not need a call to the ISR here if the deliver function contains all the “intelligence.”

159.2.10. Interrupt-vector function

static int
HRDWR_int_vector(struct eth_drv_sc *sc)

This function returns the interrupt vector number used for receive interrupts. This is so that the common GDB stubs can detect when to check for incoming “CTRL-C” packets (used to asynchronously halt the application) when debugging over ethernet. The GDB stubs need to know which interrupt the ethernet device uses so that they can mask or unmask that interrupt as required.

159.3. Upper Layer Functions

Upper layer functions are called by drivers to deliver received packets or transmission completion status back up into the network stack.

These functions are defined by the hardware independent upper layers of the networking driver support. They are present to hide the interfaces to the actual networking stack so that the hardware drivers may be used by different network stack implementations without change.

These functions require a pointer to a struct eth_drv_sc which describes the interface at a logical level. It is assumed that the low level hardware driver will keep track of this pointer so it may be passed “up” as appropriate.

159.3.1. Callback Init function

void (sc->funs->eth_drv->init)(
                struct eth_drv_sc *sc, unsigned char *enaddr)

This function establishes the device at initialization time. It should be called once per device instance only, from the initialization function, if all is well (see Section 159.2.1, “Init function”). The hardware should be totally initialized (not “started”) when this function is called.

159.3.2. Callback Tx-Done function

void (sc->funs->eth_drv->tx_done)(
                struct eth_drv_sc *sc,
                unsigned long key, int status)

This function is called when a packet completes transmission on the interface. The key value must be one of the keys provided to HRDWR_send() above. The value status should be non-zero (details currently undefined) to indicate that an error occurred during the transmission, and zero if all was well.

It should be called from the deliver function (see Section 159.2.7, “Deliver function”) or poll function (see Section 159.2.9, “Poll function”).

159.3.3. Callback Receive function

void (sc->funs->eth_drv->recv)(struct eth_drv_sc *sc, int len)

This function is called to indicate that a packet of length len has arrived at the interface. The callback HRDWR_recv() function described above will be used to actually unload the data from the interface into buffers used by the device independent layers.

It should be called from the deliver function (see Section 159.2.7, “Deliver function”) or poll function (see Section 159.2.9, “Poll function”).

159.4. Calling graph for Transmission and Reception

It may be worth clarifying further the flow of control in the transmit and receive cases, where the hardware driver does use interrupts and so DSRs to tell the “foreground” when something asynchronous has occurred.

159.4.1. Transmission

  1. Some foreground task such as the application, SNMP “daemon”, DHCP management thread or whatever, calls into network stack to send a packet, or the stack decides to send a packet in response to incoming traffic such as a “ping” or ARP request.
  2. The driver calls the HRDWR_can_send() function in the hardware driver.
  3. HRDWR_can_send() returns the number of available "slots" in which it can store a pending transmit packet. If it cannot send at this time, the packet is queued outside the hardware driver for later; in this case, the hardware is already busy transmitting, so expect an interrupt as described below for completion of the packet currently outgoing.
  4. If it can send right now, HRDWR_send() is called. HRDWR_send() copies the data into special hardware buffers, or instructs the hardware to “send that.” It also remembers the key that is associated with this tx request.
  5. These calls return … time passes …
  6. Asynchronously, the hardware makes an interrupt to say “transmit is done.” The ISR quietens the interrupt source in the hardware and requests that the associated DSR be run.
  7. The DSR calls (or is) the eth_drv_dsr() function in the generic driver.
  8. eth_drv_dsr() in the generic driver awakens the “Network Delivery Thread” which calls the deliver function HRDWR_deliver() in the driver.
  9. The deliver function realizes that a transmit request has completed, and calls the callback tx-done function (sc->funs->eth_drv->tx_done)() with the same key that it remembered for this tx.
  10. The callback tx-done function uses the key to find the resources associated with this transmit request; thus the stack knows that the transmit has completed and its resources can be freed.
  11. The callback tx-done function also enquires whether HRDWR_can_send() now says “yes, we can send” and if so, dequeues a further transmit request which may have been queued as described above. If so, then HRDWR_send() copies the data into the hardware buffers, or instructs the hardware to "send that" and remembers the new key, as above. These calls then all return to the “Network Delivery Thread” which then sleeps, awaiting the next asynchronous event.
  12. All done …

159.4.2. Receive

  1. Asynchronously, the hardware makes an interrupt to say “there is ready data in a receive buffer.” The ISR quietens the interrupt source in the hardware and requests that the associated DSR be run.
  2. The DSR calls (or is) the eth_drv_dsr() function in the generic driver.
  3. eth_drv_dsr() in the generic driver awakens the “Network Delivery Thread” which calls the deliver function HRDWR_deliver() in the driver.
  4. The deliver function realizes that there is data ready and calls the callback receive function (sc->funs->eth_drv->recv)() to tell it how many bytes to prepare for.
  5. The callback receive function allocates memory within the stack (eg. MBUFs in BSD/Unix style stacks) and prepares a set of scatter-gather buffers that can accommodate the packet.
  6. It then calls back into the hardware driver routine HRDWR_recv(). HRDWR_recv() must copy the data from the hardware's buffers into the scatter-gather buffers provided, and return.
  7. The network stack now has the data in-hand, and does with it what it will. This might include recursive calls to transmit a response packet. When this all is done, these calls return, and the “Network Delivery Thread” sleeps once more, awaiting the next asynchronous event.