Wire Order Delivery

Onload User Guide (UG1586)
Document ID
UG1586
Release Date
2023-07-31
Revision
1.2 English

When a TCP or UDP application is working with multiple network sockets simultaneously it is difficult to ensure data is delivered to the application in the strict order it was received from the wire across these sockets.

The onload_ordered_epoll_wait() API is an Onload alternative implementation of epoll_wait() providing additional data allowing a receiving application to recover in-order timestamped data from multiple sockets. To maintain wire order delivery, only a specific number of bytes, as identified by the onload_ordered_epoll_event, should be recovered from a ready socket.

  • All sockets must be in the same Onload stack. This can be confirmed using the onload_fd_stat() function.
  • Ordering is done on a per-stack basis - for TCP and UDP sockets.
  • Only data received from an Onload stack with a hardware timestamp will be ordered.
  • The environment variable EF_RX_TIMESTAMPING must be enabled: 
    EF_RX_TIMESTAMPING=1
  • File descriptors where timestamping information is not available can be included in the epoll set, but received data will be returned from these unordered.
  • The application must use the epoll API and the onload_ordered_epoll_wait() function.
  • The application must set the per-process environment variable EF_UL_EPOLL=1 or EF_UL_EPOLL=3.
  • EPOLLET and ONESHOT flags should NOT be used.
  • Concurrent use of the ordering data is not safe, and so onload_ordered_epoll_wait() must not be called from multiple threads.
  • See onload_ordered_epoll_wait for further details.

To prevent packet coalescing in the receive queue, resulting in multiple packets received with the same hardware timestamp, the EF_TCP_RCVBUF_STRICT variable should be disabled (default setting). The following figure demonstrates the Wire Order Delivery feature.

Figure 1. Wire Order Delivery

onload_ordered_epoll_wait() returning at point X would allow the following data to be recovered:

  • Socket A: timestamp of packet 1, bytes in packet 1.
  • Socket B: timestamp of packet 2, bytes in packets 2 and 3.
  • onload_ordered_epoll_wait() returning again would recover timestamp of packet 4 and bytes in packet 4.

The Wire Order Delivery feature is only available on Solarflare Flareon or XtremeScale™ adapters having a PTP/HW timestamping activation key. When receiving across multiple adapters, Solarflare sfptpd (PTP) can ensure that adapters are closely synchronized with each other and, if required, with an external PTP clock source.