Onload uses certain physical resources on the network adapter. If these resources are exhausted, it is not possible to create new Onload stacks and not possible to accelerate new sockets or applications. The onload_stackdump utility should be used to monitor hardware resources. Physical resources include:
Virtual NICs
Virtual NICs provide the interface by which a user level application sends and receives network traffic. When these are exhausted it is not possible to create new Onload stacks, meaning new applications cannot be accelerated. However, Solarflare network adapters support large numbers of Virtual NICs, and this resource is not typically the first to become unavailable.
Endpoints
Onload represents sockets and pipes as structures called endpoints. The maximum number of accelerated endpoints permitted by each Onload stack is set with the EF_MAX_ENDPOINTS variable. The stack limit can be reached sooner than expected when syn-receive states (the number of half-open connections) also consume endpoint buffers. Four syn-receive states consume one endpoint. The maximum number of syn-receive states can be limited using the EF_TCP_SYNRECV_MAX variable.
Filters
Filters are used to deliver packets received from the wire to the appropriate application. When filters are exhausted it is not possible to create new accelerated sockets. The general recommendation is that:
- On X3-series adapters, applications do not allocate more than 256 filters or create more than 256 outgoing connections.
- On X2-series adapters, applications do not allocate more than 16384 filters or create more than 16384 outgoing connections.
- On earlier adapters, applications do not allocate more than 4096 filters or create more than 4096 outgoing connections.
The limit does not apply to inbound connections to a listening socket.
Buffer Table
The buffer table provides address protection and translation for DMA buffers. When all buffer resources are exhausted it is not possible to create new Onload stacks, and existing stacks are not able to allocate more DMA buffers.
When hardware resources are exhausted, normal operation of the system should continue, but it will not be possible to accelerate new sockets or applications.
TX, RX Ring Buffer Size
Onload does not obey RX, TX ring sizes set in the kernel, but instead uses the values specified by EF_RXQ_SIZE and EF_TXQ_SIZE. Both default to 512.