Onload supports different implementations of epoll controlled by the EF_UL_EPOLL environment variable - see Multiplexed I/O for configuration details.
There are various limitations and differences in Onload vs. kernel behavior - refer to Multiplexed I/O for details.
- When using EF_UL_EPOLL
=1or 3, it has been identified that the behavior of
epoll_wait()differs from the kernel when the
EPOLLONESHOTevent is requested, resulting in two ‘wakeups’ being observed, one from the kernel and one from Onload. This behavior is apparent on
SOCK_STREAMsockets for all combinations of
EPOLLOUTevents. This applies for all types of accelerated sockets. EF_EPOLL_CTL_FAST is enabled by default and this modifies the semantics of epoll. In particular, it buffers up calls to
epoll_ctl()and only applies them when
epoll_wait()is called. This can break applications that do
epoll_wait()in one thread and
epoll_ctl()in another thread. The issue only affects EF_UL_EPOLL
=2and the solution is to set EF_EPOLL_CTL_FAST
=0if this is a problem. The described condition does not occur if EF_UL_EPOLL
- When EF_EPOLL_CTL_FAST is enabled and an application is testing the readiness of
an epoll file descriptor without actually calling
epoll_wait(), for example by doing epoll within epoll() or epoll within
select(), if one thread is calling
epoll_wait()and another thread is doing
epoll_ctl(), then EF_EPOLL_CTL_FAST should be disabled. This applies when using EF_UL_EPOLL 1, 2 or 3.
If the application is monitoring the state of the epoll file descriptor indirectly, for example by monitoring the epoll fd with poll, then EF_EPOLL_CTL_FAST can cause issues and should be set to zero.
To force Onload to follow the kernel behavior when using the
epoll_wait()call, the following variables should be set:
- A socket should be removed from an epoll set only when all references to the socket are closed.
=1(default) or EF_UL_EPOLL
=3, a socket is removed from the epoll set if the file descriptor is closed, even if other references to the socket exist. This can cause problems if file descriptors are duplicated using
dup(), dup2() or fork(). For example:
Workaround is set EF_UL_EPOLL
s = socket(); s2 = dup(s); epoll_ctl(epoll_fd, EPOLL_CTL_ADD, s, ...); close(s); /* socket referenced by s is removed from epoll set when using onload */
- When Onload is unable to accelerate a
connected socket, for example because no route to the destination exists which uses
a Solarflare interface, the socket will be handed off to the kernel and is removed
from the epoll set. Because the socket is no longer in the epoll set, attempts to
modify the socket with
epoll_ctl()will fail with the
ENOENT(descriptor not present) error. The described condition does not occur if EF_UL_EPOLL
- If an epoll file descriptor is passed to the
write()functions these will return a different errorcode than that reported by the kernel stack. This issue exists for all implementations of epoll.
EPOLLETis used and the event is ready,
epoll_wait()is triggered by ANY event on the socket instead of the requested event. This issue should not affect application correctness.
- Users should be aware that if a server is overclocked the
epoll_wait()timeout value will increase as CPU MHz increases resulting in unexpected timeout values. This has been observed on Intel based systems and when the Onload epoll implementation is EF_UL_EPOLL
=1or 3. Using EF_UL_EPOLL
=2this behavior is not observed.
- On a spinning thread, if epoll acceleration is disabled by setting EF_UL_EPOLL
=0, sockets on this thread will be handed off to the kernel, but latency will be worse than expected kernel socket latency.
- To ensure that non-accelerated file descriptors are checked in poll and select functions, the following options should be disabled (set to zero):
- EF_SELECT_FAST and EF_POLL_FAST
- When using
select()calls, to ensure that non-accelerated file descriptors are checked when there are no events on any accelerated descriptors, set the following options:
- EF_POLL_FAST_USEC and EF_SELECT_FAST_USEC, setting both to zero.