TCP Stacks

Onload User Guide (UG1586)

Document ID

UG1586

Release Date

2023-07-31

Revision

1.2 English

This section describes typical output for a TCP stack.

Note: Depending on the state of the stack, some of the output shown in this section might be omitted. The sample output in this section is from several different stacks, to illustrate these different states, and so might not be self-consistent.

The output starts with basic information about the stack, shown in the following table:

Table 1. Stackdump Output: TCP Stack
Sample Output	Description
`ci_netif_dump_to_logger:`	Function dumping the stack.
`stack=7`	Stack id.
`name=`	Stack name as set by EF_NAME.
`cplane_pid=813`	Process id of Onload control plane server.
`namespace=net:[4026531956]`	Namespace id.
`ver=201811`	Onload version.
`uid=0`	User id.
`pid=1930`	Process id of creator process.
`ns_flags=0`	Flags as a hexadecimal value, followed by names of flags including: ONLOAD_UNSUPPORTED SOCKCACHE_FORKED.
`creation_time=2019-01-25 15:16:14`	Creation time of stack.
`(delta=19secs)`	Age of stack, as a delta between stack creation and the stackdump.
`lock=20000000 LOCKED`	Internal stack lock status.
`nics=3`	Hexadecimal bitfield identifying adapters used by this stack. For example, 0x3 = 0b11, so the stack is using adapters 1 and 2.
`primed=1`	1 if the event queue will generate an interrupt when the next event arrives, otherwise 0.
`ref=`	Count of references to this stack.
`trusted_lock=`	Kernel side stack lock: `0`: unlocked `2`: awaiting free Otherwise: locked, bitmask gives more information.
`k_ref=`	Count of kernel references to this stack.
`n_ep_closing=`	Count of kernel references to this stack for closing endpoints.
`sock_bufs:`	Sockets buffers which can be allocated:
`max=8192`	Maximum number.
`n_allocated=4`	Number currently allocated.
`aux_bufs:`	Aux buffers, used by partially opened TCP connections (incoming connections) before they are established and promoted to use socket buffers. The number of aux buffers is limited to EF_TCP_SYNRECV_MAX `* 2`
`free=6`	Number of free aux buffers.
`aux_bufs[syn-recv state]:`	Aux buffers for the syn-recv state:
`n=0`	Number currently allocated
`max=2048`	Maximum number.
`aux_bufs[syn-recv bucket]:`	Aux buffers for the syn-recv bucket:
`n=0`	Number currently allocated
`max=8192`	Maximum number.
`aux_bufs[epoll3 state]:`	Aux buffers for the epoll3 state:
`n=0`	Number currently allocated
`max=8192`	Maximum number.
`pkt_sets:`	Packet sets:
`pkt_size=2048`	Size of a packet buffer, in bytes.
`set_size=1024`	Number of packet buffers in each packet set.
`max=32`	Maximum number of packet sets available to this stack.
`alloc=2`	Number of packet sets currently allocated.
`pkt_set[0]:`	Packet set 0:
`free=112`	Number of free packet buffers in the set, each of size `pkt_sets -> pkt_size`.
`pkt_set[1]:`	Packet set 1:
`free=880`	Number of free packet buffers in the set, each of size `pkt_sets -> pkt_size`.
`current`	This is the packet set currently being used.
`pkt_bufs:`	Packet buffers:
`max=32768`	Maximum number of packet buffers this stack can allocate, each of size `pkt_sets -> pkt_size`.
`alloc=576`	Number of packet buffers that have been allocated.
`free=57`	Number of packet buffers that are free, and can be reused by either receive or transmit rings.
`async=0`	Number of packet buffers used by Onload in one of its asynchronous queues.
`pkt_bufs:`	Receive packet buffers:
`rx=1056`	Number of receive packet buffers that are currently in use.
`rx_ring=992`	Number of packet buffers that have been pushed to the receive ring.
`rx_queued=0`	Number of packet buffers that are in the application’s receive queue.
`pressure_pool=64`	Number of packet buffers in the pressure pool. This is a pool of packet buffers used when the stack is under memory pressure. Its size is `rx - (rx_ring + rx_queued)`. This might be followed by flags indicating a memory pressure condition: `CRITICAL`: the number of packets in the receive socket buffers is approaching the EF_MAX_RX_PACKETS value. `LOW`: there are not enough packet buffers available to refill the RX descriptor ring.
`pkt_bufs:`	Transmit packet buffers:
`tx=2`	Number of transmit packet buffers that are currently in use.
`tx_ring=1`	Number of packet buffers that remain in the transmit ring.
`tx_oflow=0`	The number of extra packets that are ready to send to the transmit queue, but that the transmit queue does not have space to accept.
`pkt_bufs:`	Other packet buffer totals:
`in_loopback=0`	Number of packet buffers currently used in TCP loopback connection.
`in_sock=991`	Number of packet buffers currently used by a TCP socket.
`pkt_bufs:`	Other packet buffer totals:
`rx_reserved=`	Total number of receive packet buffers that are reserved by `ESTABLISHED` sockets.
`signal_q=[%d,%d]`	Asynchronous signal queue head and tail (Windows only).
`completion_q=%d`	Asynchronous completion queue (Windows only).
`time:`	Internal timer values. To convert ticks to milliseconds, multiply by `ci_ip_time_tick2ms`:
`netif=5eb5c61`	Current cached time, in ticks.
`poll=5eb5c61`	Scheduler’s view of time, in ticks.
`now=5eb5c61 (diff=0.000sec)`	Time now from cache of real ticks, and difference between this time and the `netif` time. If the difference is more than 5 seconds, it is followed by: `!! STUCK !!`
`ERRORS:`	Errors, if any, including: `PPL` `LOOP` `ASS` `SYNRECV`.
`active cache:`	TCP socket caching:
`hit=0`	Number of cache hits (were cached).
`avail=0`	Number of sockets available for caching.
`cache=EMPTY`	Current cache state, either “EMPTY” or “yes”.
`pending=EMPTY`	Current pending state, either “EMPTY” or “yes”.
`passive scalable cache:`	TCP socket caching:
`cache=EMPTY`	Current cache state, either “EMPTY” or “yes”.
`pending=EMPTY`	Current pending state, either “EMPTY” or “yes”.
`readylist:`	Ready list (one line per list):
`id=%d`	Ready list id
`pid=%d`	Process id of process managing ready list
`ready=%s`	Current ready list state, either “EMPTY” or “yes”.
`unready=%s`	Current unready list state, either “EMPTY” or “yes”.
`flags=%x`	Ready list flags, as a hexadecimal value.

There is then a section that is repeated for each virtual interface associated with the stack, describing the virtual interface to the NIC. This is shown in the following table:

Table 2. Stackdump Output: Virtual Interface for a TCP Stack
Sample Output	Description
`ci_netif_dump_vi:`	Function dumping the stack’s virtual interface to the NIC
`stack=7`	Stack id.
`intf=0`	Interface (port) number.
`dev=(pci address)`	PCI address of NIC.
`hw=0C0`	Hardware version, given as an architecture / variant / revision tuple.
`vi=240`	Identifies the VI in use by the stack.
`pd_owner=1`	Will be zero when using physical addressing mode.
`channel=0`	Identifies the receive queue being used on this interface.
`tcpdump=off`	One of the following: `all` `nomatch` `off`.
`vi_flags=3800000`	VI flags, as a hexadecimal value.
`oo_vi_flags=3`	Hexadecimal bitfield identifying features requested on this VI. For details, see src/include/ci/internal/oo_vi_flags.h.
`evq:`	Event queue data:
`cap=2048`	Maximum number of events the queue can hold, set by EF_RXQ_SIZE, EF_TXQ_SIZE.
`current=16de30`	The current event queue location.
`is_32_evs=0`	Is 1 if there are 32 or more events pending.
`is_ev=0`	Is 1 if there are any events pending.
`evq:`	Further event queue data:
`sync_major=ffffffff`	Major part of the timestamp (seconds).
`sync_minor=0`	Minor part of the timestamp (upper part of ns).
`sync_min=0`	Smallest possible seconds value for timestamp.
`evq:`	Further event queue data:
`sync_synced=0`	Timestamp synchronized with adapter
`sync_flags=0`	Time synchronization flags
`rxq:`	Receive queue data:
`cap=511`	Total capacity.
`lim=511`	Maximum fill level for receive descriptor ring, specified by EF_RXQ_LIMIT.
`spc=1`	Amount of empty buffers ready to be used.
`level=510`	How full the receive queue currently is.
`total_desc=93666`	Total number of descriptors that have been pushed to the receive queue.
`txq:`	Transmit queue data:
`cap=511`	Total capacity.
`lim=511`	Maximum fill level for transmit descriptor ring.
`spc=511`	Amount of empty buffers ready to be used.
`level=0`	How full the transmit queue currently is.
`pkts=0`	How many packets are represented by the descriptors in the transmit queue.
`oflow_pkts=0`	How many packets are in the overflow transmit queue (that is, waiting for space in the NIC's transmit queue).
`txq:`	Further transmit queue data:
`pio_buf_size=2048`	PIO buffer size.
`tot_pkts=93669`	Total number of packet buffers used.
`bytes=0`	Number of packet bytes currently in the queue.
`txq:`	Further transmit queue data:
`ts_nsec=40000000`	Nanoseconds from timestamp in tx queue state.
`clk:`	Flags from last synchronization: `SET` `SYNC`.
`last_rx_stamp:`	Last receive timestamp:
`0:0`	Given as seconds:nanoseconds
`ctpio:`	Cut-through PIO
`max_frame_len=500`	Maximum frame length for the CTPIO low-latency transmit mechanism.
`frame_len_check=500`	Frame length check, CTPIO is disabled if this is zero.
`ct_thresh=65535`	Cut-through threshold for CTPIO transmits.
`ERRORS:`	Errors, if any, including: `REMAP`.
`vi=240`	Identifies the VI in use by the stack when there is a separate receive queue for UDP.
`evq:`	Event queue data for when there is a separate receive queue for UDP:
`cap=2048`	Maximum number of events the queue can hold, set by EF_RXQ_SIZE, EF_TXQ_SIZE.
`current=16de30`	The current event queue location.
`is_32_evs=0`	Is 1 if there are 32 or more events pending, otherwise 0.
`is_ev=0`	Is 1 if there are any events pending, otherwise 0.
`rxq:`	Receive queue data for when there is a separate receive queue for UDP:
`cap=511`	Total capacity.
`lim=511`	Maximum fill level for receive descriptor ring, specified by EF_RXQ_LIMIT.
`spc=1`	Amount of empty buffers ready to be used.
`level=510`	How full the receive queue currently is.
`total_desc=93666`	Total number of descriptors that have been pushed to the receive queue.

There is then a section giving extra information about the stack, shown in the following table:

Table 3. Stackdump Output: Extra Information for a TCP Stack
Sample Output	Description
`ci_netif_dump_extra:`	Function dumping the extra information
`stack=7`	Stack id.
`in_poll=0`	Is 1 if the process is currently polling, otherwise 0.
`post_poll_list_empty=1`	Is 1 if there are tasks to be done once polling is complete, otherwise 0.
`poll_did_wake=0`	Is 1 if while polling, the process identified a socket which needs to be woken following the poll, otherwise 0.
`rx_defrag_head=-1`	Reassembly sequence number. -1 means no re-assembly has occurred.
`rx_defrag_tail=-1`	Reassembly sequence number. -1 means no re-assembly has occurred.
`tx_may_alloc=1`	The number of packet buffers TCP could use.
`can=1`	The number of packet buffers TCP can use now.
`nonb_pool=1`	The number of packet buffers available to TCP process without holding the lock.
`send_may_poll=0`	Is 1 if using EF_POLL_ON_DEMAND, otherwise 0.
`is_spinner=0,0`	First value is 1 if a thread is spinning, otherwise 0. Second value is the number of spinning threads.
`hwport_to_intf_i=0,-1,-1,-1,-1,-1`	Internal mapping of hardware ports to internal interfaces.
`intf_i_to_hwport=0,0,0,0,0,0`	Internal mapping of internal interfaces to hardware ports.
`uk_intf_ver=03e89aa26d20b98fd08793e771f2cdd9`	md5 user/kernel interface checksum computed by both kernel and user application to verify internal data structures.
`deferred count 0/32`	NUMA node parameters - refer to Onload Deployment on NUMA Systems.
`numa nodes:`	Further NUMA node parameters - refer to Onload Deployment on NUMA Systems.
`creation=0`
`load=0`
`numa node masks:`	Further NUMA node parameters - refer to Onload Deployment on NUMA Systems.
`packet alloc=1`
`sock alloc=1`
`interrupt=1`

Finally, there is a list of process ids shown in the following table:

Table 4. Stackdump Output: Process Ids for a TCP Stack
Sample Output	Description
`pids:14025`	List of processes being accelerated by Onload on this stack.