Programmed I/O

PIO (programmed input/output) describes the process whereby data is directly transferred by the CPU to or from an I/O device. It is an alternative to bus master DMA techniques where data are transferred without CPU involvement.

Solarflare SFN8000 and X2 series adapters support TX PIO, where packets on the transmit path can be “pushed” to the adapter directly by the CPU. This improves the latency of transmitted packets but can cause a very small increase in CPU utilization. TX PIO is therefore especially useful for smaller packets.

The Onload TX PIO feature is enabled by default but can be disabled via the environment variable EF_PIO. An additional environment variable, EF_PIO_THRESHOLD specifies the size of the largest packet size that can use TX PIO.

The number of PIO buffers available depend on the adapter type being used and the number of PCIe Physical Functions (PF) exposed per port.

For optimum performance, PIO buffers should be reserved for critical processes and other processes should set EF_PIO to 0 (zero).

The Onload stackdump utility provides additional counters to indicate the level of PIO use - see TX PIO Counters for details.

The Solarflare net driver will also use PIO buffers for non-accelerated sockets and this will reduce the number of PIO buffers available to Onload stacks. To prevent this set the driver module option piobuf_size=0. Driver module options can be set in a user-created file (sfc.conf) in the /etc/modprobe.d directory:

options sfc piobuf_size=0

An Onload stack requires one PIO buffer for each VI it creates. An Onload stack will create one VI for each physical interface that it uses.

When both accelerated and non-accelerated sockets are using PIO, the number of PIO buffers available to Onload stacks can be calculated from the available PIO regions:

Using the above example values applied to a SFN7x22 adapter, each PF on the adapter requires:

piobuf_size * rss_cpus * num_PFs/ region size = 0.5 regions - (round up - so each port needs 1 region).

This leaves 16-2 = 14 regions for Onload stacks which also require one region per port, per stack. Therefore from our example we can have 7 onload stacks using PIO buffers.

PIO buffers are allocated on a first-come, first-served basis. The following warning might be observed when stacks cannot be allocated any more PIO buffers:

WARNING: all PIO bufs allocated to other stacks. Continuing without PIO.  Use EF_PIO to control this

To ensure more buffers are available for Onload, it is possible to prevent the net driver from using PIO buffers. This can be done by setting the sfc driver module option in a user-created file in the /etc/modprobe.d directory:

options sfc piobuf_size=0

Reload the Onload kernel drivers for the changes to be effective:

# onload_tool reload

The per-stack EF_PIO variable can also be unset for stacks where PIO buffers are not required. If there is contention for PIO buffers, consider disabling PIO for any stacks that primarily receive, so the buffers are available for stacks that perform latency-critical sends.