Applications which rapidly open and close a large number of connections to other machines might experience occasional connection failures due to the rapid reuse of TCP sequence numbers being detected as retransmits in the TIME-WAIT state. This is most commonly a problem with Windows and FreeBSD TCP stacks.
The standard RFC-derived algorithm for avoiding this problem relies on a clock ticking at a rate which is faster than bytes are transmitted. A link running at 100 Mb can theoretically transmit faster than the clock can tick, however, and 10 Gb+ links can practically do this.
Onload has added the EF_TCP_ISN_MODE option to provide a solution. The default “clocked” setting uses the standard best-effort algorithm. The “clocked+cache” setting will store the last sequence number used for every remote endpoint to guarantee that the problem is avoided. This mode is recommended for applications such as proxies which rapidly open and close connections to a variety of unknown, third-party servers.
The following settings can be used to fine-tune the clocked+cache mode:
- EF_TCP_ISN_CACHE_SIZE
Number of entries to allocate in the cache of remote endpoints. The default value of 0 selects a size automatically.
For more information, see EF_TCP_ISN_CACHE_SIZE. - EF_TCP_ISN_INCLUDE_PASSIVE
Store data for closed passively-opened connections in the cache. This data would only be needed by an application which closed its listening socket and continued to run, so the option is disabled by default
For more information, see EF_TCP_ISN_INCLUDE_PASSIVE. - EF_TCP_ISN_OFFSET
Distance by which to step the initial sequence number of new connections relative to the previous connection. Only extremely specialized applications would consider changing the default.
For more information, see EF_TCP_ISN_OFFSET. - EF_TCP_ISN_2MSL
Maximum amount of time that any remote TCP stack's implementation will leave a socket in the TIME-WAIT state. This is configurable in many systems, however the default value of 240 seconds is a maximum common value across a variety of operating systems.
For more information, see EF_TCP_ISN_2MSL.