The Onload environment variable EF_PREFAULT_PACKETS will cause the user process to ‘touch’ the specified number of packet buffers when an Onload stack is created. This means that memory for these packet buffers is pre-allocated and memory-mapped into the user-process address space.
Pre-allocation is advised to prevent latency jitter caused by the allocation and memory-mapping overheads.
When deciding how many packets to prefault, the user should look at the alloc value when the onload_stackdump packets command is run. The alloc value is a high water mark identifying the maximum the number of packets being used by the stack at any singular point. Setting EF_PREFAULT_PACKETS to at least this value is recommended.
onload_stackdump packets$ onload_stackdump packets
ci_netif_pkt_dump_all: id=0
pkt_sets: pkt_size=2048 set_size=1024 max=32 alloc=2
pkt_set[0]: free=544
pkt_set[1]: free=446 current
pkt_bufs: max=32768 alloc=2048 free=990 async=0
pkt_bufs: rx=1058 rx_ring=992 rx_queued=2 pressure_pool=64
pkt_bufs: tx=0 tx_ring=0 tx_oflow=0
pkt_bufs: in_loopback=0 in_sock=0
994: 0x200 Rx
n_zero_refs=1054 n_freepkts=1 estimated_free_nonb=1053
free_nonb=0 nonb_pkt_pool=ffffffffffffffffWhen deciding how many packets to prefault the user should consider that Onload must allocate from the EF_MAX_PACKET pool, a number of packet buffers per receive ring per interface. Once these have been allocated, any remainder can be prefaulted.
Users who require to prefault the maximum possible number of available packets can set EF_PREFAULT_PACKETS and EF_MAX_PACKETS to the same value:
EF_PREFAULT_PACKETS=64000 EF_MAX_PACKETS=64000 onload <myapplication>...Users can alternatively set EF_PREALLOC_PACKETS to allocate EF_MAX_PACKETS packet buffers during stack creation:
EF_PREALLOC_PACKETS=1 EF_MAX_PACKETS=64000 onload <myapplication>...Refer to Parameter Reference for details of these variables.