ONLOAD_MSG_WARM

Applications that send data infrequently may see increased send latency compared to an application that is making frequent sends. This is due to the send path and associated data structures not being cache and TLB resident (which can occur even if the CPU has been otherwise idle since the previous send call).

Onload allows applications to repeatedly call send() to keep the TCP fast send path ‘warm’ in the cache without actually sending data. This is particularly useful for applications that only send infrequently and helps maintain low latency performance for those TCP connections that do not send often. These “fake” sends are performed by setting the ONLOAD_MSG_WARM flag when calling the TCP send calls. The message warm feature does not transmit any packets.

char buf[10];
send(fd, buf, 10, ONLOAD_MSG_WARM);

Onload stackdump supports counters to indicate the level of message warm use:

• warm_aborted is a count of the number of times a message warm send function was called, but the sendpath was not exercised due to Onload locking constraints.
• warm is a count of the number of times a message warm send function was called when the send path was exercised.
• A send() ONLOAD_MSG_WARM can return 0 (length of data sent) if the send() was unsuccessful. This will be due to normal stack and TCP networking conditions such as cannot get stack lock, insufficient send window available, other packets in the send queue or retransmit queue etc.