Interrupt Handling - Using Onload

A thread accelerated by Onload will either be interrupt driven or it will be spinning.

When the thread is interrupt driven, a thread which calls into Onload to read from its receive queue and for which there are no received packets to be processed, will ‘sleep’ until an interrupt(s) from the kernel informs it that there is more work to do.

When a thread is spinning, it is busy waiting on its receive queue until packets are received - in which case the packets are retrieved and the thread returns immediately to the receive queue, or until the spin period expires. If the spin period expires the thread will relinquish the CPU core and ‘sleep’ until an interrupt from the kernel informs it that further packets have been received. If the spin period is set greater than the packet inter-arrival rate, the spinning thread can continue to spin and retrieve packets without interrupts occurring. Even when spinning, an application might experience a few interrupts.

As a general rule, when spinning, only a few interrupts will be expected so performance is typically insensitive as to which CPU core processes the interrupts.

However, when Onload is interrupt driven performance can be sensitive to where the interrupts are handled and will typically benefit to be on the same CPU socket as the application thread handling the socket I/O. The method required depends on the setting of the EF_PACKET_BUFFER_MODE environment variable:

• If EF_PACKET_BUFFER_MODE=0 or 2, an Onload stack will use one or more of the interrupts assigned to the NET driver receive queues. The CPU core handling the interrupts is defined by the RSS mapping of receive queues to CPU cores:
  • If sfcaffinity_config has been used to affinitize RSS channel interrupts, the interrupt handling core for the stack can be set using the EF_IRQ_CORE environment variable.

    It is only possible for interrupts to be handled on the requested core if a NET driver interrupt is assigned to the selected core.

    See Affinitizing RSS Channels to CPUs.

  • Otherwise, the interrupt handling core for the stack can be set using the EF_IRQ_CHANNEL environment variable. Onload interrupts are handled by the same core assigned to the NET driver receive channel.
• If EF_PACKET_BUFFER_MODE=1 or 3, the onload stack creates dedicated interrupts. The interrupt handling core for the stack can be set using the EF_IRQ_CORE environment variable.

For more information about these environment variables, see:

• EF_IRQ_CHANNEL
• EF_IRQ_CORE
• EF_PACKET_BUFFER_MODE.

When Onload is using a NET driver RSS channel for its source of interrupts, it can be useful to dedicate this channel to Onload and prevent the driver from using this channel for RSS traffic. See Restrict RSS to Local NUMA Node and Restrict RSS Receive Queues for methods of how to achieve this.