Minimizing Lock Contention

Onload User Guide (UG1586)

Document ID

UG1586

Release Date

2023-07-31

Revision

1.2 English

Lock contention can greatly affect performance. When threads share a stack, a thread holding the stack lock will prevent another thread from doing useful work. Applications with fewer threads might be able to create a stack per thread (see EF_STACK_PER_THREAD and Stacks API).

Use onload_stackdump to identify instances of lock contention:

# onload_stackdump lots | egrep "(lock_)|(sleep)"

Table 1. Onload_stackdump Lock and Sleep Counters
Counter	Description
`periodic_lock_contends`	Number of times periodic timer could not get the stack lock.
`interrupt_lock_contends`	Number of times the interrupt handler could not get the stack lock because it is already held by user level or other context.
`timeout_interrupt_lock_contends`	Number of times timeout interrupts could not lock the stack.
`sock_sleeps`	Number of times a thread has blocked on a single socket.
`sock_sleep_primes`	Number of times select/poll/epoll enabled interrupts.
`unlock_slow`	Number of times the slow path was taken to unlock the stack lock.
`unlock_slow_pkt_waiter`	Number of times packet memory shortage provoked the unlock slow path.
`unlock_slow_socket_list`	Number of times the deferred socket list provoked the unlock slow path.
`unlock_slow_need_prime`	Number of times interrupt priming provoked the unlock slow path.
`unlock_slow_wake`	Number of times the unlock slow path was taken to wake threads.
`unlock_slow_swf_update`	Number of times the unlock slow path was taken to update sw filters.
`unlock_slow_close`	Number of times the unlock slow path was taken to close sockets/pipes.
`unlock_slow_syscall`	Number of times a syscall was needed on the unlock slow path.
`lock_wakes`	Number of times a thread is woken when blocked on the stack lock.
`stack_lock_buzz`	Number of times a thread has spun waiting for the stack lock.
`sock_lock_sleeps`	Number of times a thread has slept waiting for a sock lock.
`sock_lock_buzz`	Number of times a thread has spun waiting for a sock lock.
`tcp_send_ni_lock_contends`	Number of times TCP `sendmsg()` contended the stack lock
`udp_send_ni_lock_contends`	Number of times UDP `sendmsg()` contended the stack lock
`getsockopt_ni_lock_contends`	Number of times `getsockopt()` contended the stack lock.
`setsockopt_ni_lock_contends`	Number of times `setsockopt()` contended the stack lock.
`lock_dropped_icmps`	Number of dropped ICMP messages not processed due to contention.

Solution

Performance will be improved when stack contention is kept to a minimum. When threads share a stack it is preferable for a thread to spin rather than sleep when waiting for a stack lock. The EF_BUZZ_USEC value can be increased to reduce ‘sleeps’. Where possible use stacks per thread.