IRQ Balancing - 2020.2 English

H.264/H.265 Video Codec Unit v1.2 Solutions LogiCORE IP Product Guide (PG252)

Document ID
PG252
Release Date
2020-11-24
Version
2020.2 English

Various multimedia use-cases involving video codecs such as audio/video conferencing, video-on-demand, playback, and record use-cases also involve multiple other peripherals such as ethernet, video capture pipeline related IPs icluding image sensor and image signal processing engines, DMA engines, and display pipeline related IP like video mixers and HDMI transmitters, which in turn use unique interrupt lines for communicating with the CPU.

In these scenarios, it becomes important to distribute the interrupt processing load across multiple CPU cores instead of utilizing the same core for all the peripherals/IP. Distributing the IRQ across CPU cores optimizes the latency and performance of the running use-case as the IRQ context switching and ISR handling load gets distributed across multiple CPU cores.

Each peripheral/IP is assigned a unique interrupt number by the Linux kernel. Whenever a peripheral or IP needs to signal something to the CPU (like it has completed a task or detected something), it sends a hardware signal to the CPU and the kernel retrieves the associated IRQ number and then calls the associated interrupt service routine. The IRQ numbers can be retrieved using the following command. This command also lists the number of interrupts processed by each core, the interrupt type, and comma-delimited list of drivers registered to receive that interrupt.

$cat /proc/interrupts

TheZynq UltraScale+ MPSoC has four CPU cores available. If running a plain PetaLinux image without any irqbalance daemon, then by default all IRQ requests are processed by CPU 0 by the Linux scheduler. To assign a different CPU core to process a particular IRQ number, the IRQ affinity for that particular interrupt needs to be changed. The IRQ affinity value defines which CPU cores are allowed to process that particular IRQ. For more information, see https://www.kernel.org/doc/Documentation/IRQ-affinity.txt.

By default, the IRQ affinity value for each peripheral is set to 0xf, which means that all four CPU cores are allowed to process interrupt as shown in the following example using the IRQ number 42.

$cat /proc/irq/42/smp_affinity
output: f

To restrict this IRQ to a CPU core n, you have to set a mask for only the nth bit. For example, if you want to route to only CPU core 2, then set the mask for the second bit using the value 0x4.

echo 4 > /proc/irq/42/smp_affinity

The following section shows how IRQ balancing can be performed before running a multistream video conferencing use-case that involves multiple peripherals and video IP.

As explained in Encoder Features > DMA_PROXY module usage, various DMA channels are used for constructing encoded output, which in turn also utilize different interrupt lines as depicted by the zynqmp-dma blocks in the following figure.

Figure 1. Default IRQ Assignment to CPU Core 0

As seen in the previous figure, all interrupt requests from different peripherals goes to CPU 0 by default.

To distribute the interrupt requests across different CPU cores as show in the following figure, follow these steps:

Figure 2. Distributed Interrupt Layout
  1. Find the IRQ numbers for each of the above peripherals.
    root@zcu106-zynqmp:~/llp2_2020.2_scripts/Serial/1080p/nv12# cat /proc/interrupts | grep al5                                                                                                                  
     49:    1250127      47679          0          0     GICv2 127 Level     a0120000.al5d, a0100000.al5e
    
    root@zcu106-zynqmp:~/llp2_2020.2_scripts/Serial/1080p/nv12# cat /proc/interrupts | grep xilinx_frame
     52:      18662          0        822          0     GICv2 122 Level     xilinx_framebuffer
     53:      19170          0          0          0  interrupt-controller@a0055000   3 Level   -level     xilinx_framebuffer
     54:      18825          0          0          0  interrupt-controller@a0055000   0 Level   -level     xilinx_framebuffer
     55:      18463          0          0          0  interrupt-controller@a0055000   1 Level   -level     xilinx_framebuffer
     57:          0          0          0          0     GICv2 121 Level     xilinx_framebuffer
    
    root@zcu106-zynqmp:~/llp2_2020.2_scripts/Serial/1080p/nv12# cat /proc/interrupts | grep xilinx-hdmi 
     56:     544834          0    4911768          0     GICv2 123 Level     xilinx-hdmi-rx
     58:      86730          0          0     813482     GICv2 125 Level     xilinx-hdmitxss
    
    root@zcu106-zynqmp:~/llp2_2020.2_scripts/Serial/1080p/nv12# cat /proc/interrupts | grep mixer      
     59:      86752          0          0     814292     GICv2 128 Level     xlnx-mixer
    
    root@zcu106-zynqmp:~/llp2_2020.2_scripts/Serial/1080p/nv12# cat /proct/interrupts | grep zynqmp-dma
    12:   42151036          0          0          0     GICv2 156 Level         zynqmp-dma
    13:   31494805   10644207          0          0     GICv2 157 Level     zynqmp-dma
    14:   31483922          0   10643127          0     GICv2 158 Level     zynqmp-dma
    15:   31518024          0          0   10595920     GICv2 159 Level     zynqmp-dma
    NOTE: Here there are multiple zynqmp-dma interrupt lines so to check which ones are getting utilized,
    you first need to run the usecase and then check which interrupt lines are getting utilized.
    

    The numbers on the left are the IRQ numbers for the respective peripherals.

  2. Assign CPU 1 to VCU IRQ with number 49.
    echo 2 > /proc/irq/49/smp_affinity #VCU
  3. Assign CPU 2 to HDMI RX and the framebuffer write IP to CPU 2
    echo 4 > /proc/irq/52/smp_affinity #Primary HDMI Rx
    
    echo 4 > /proc/irq/56/smp_affinity #Primary HDMI Tx
  4. Assign CPU 3 to HDMI TX and Video mixer IP
    echo 8 > /proc/irq/58/smp_affinity #Tx
    
    echo 8 > /proc/irq/59/smp_affinity #Mixer

    By default, the interrupts for video1 xilinx_framebuffer DMA engine and various other peripherals are already being processed by CPU 0 so there is no need to modify the smp_affinity for the same. Using the previous commands, the IRQ is distributed as per the scheme mentioned in the previous figure, which can also be seen by running the following command when the use-case is running and observing whether interrupts for the peripherals are going to respective CPU cores as intended or not. Likewise, similar scheme of distributing interrupts can be followed for other use-cases too depending upon the peripherals being used, system load, and intended performance.

    $ cat /proc/interrupts
  5. Assign a unique CPU to each zynqmp-dma channel if possible
    echo 0 > /proc/irq/12/smp_affinity #zynqmp-dma1 
    echo 2 > /proc/irq/13/smp_affinity #zynqmp-dma2 
    echo 4 > /proc/irq/14/smp_affinity #zynqmp-dma3 
    echo 8 > /proc/irq/15/smp_affinity #zynqmp-dma4				

By default the interrupts for other peripherals will be processed by cpu 0 so there is no need to modify the smp_affinity for the same. Using the preceding commands, the IRQ will get distributed as per the scheme mentioned in gex1601891035556.html#gex1601891035556__image_lvr_zfs_fnb, which can also be seen by running the following command when the use-case is running:

 cat /proc/interrupts 
 12:       42151036          0                 0               0     GICv2 156 Level     zynqmp-dma
 13:      31494805      10644207          0               0     GICv2 157 Level     zynqmp-dma
 14:      31483922          0             10643127    0     GICv2 158 Level     zynqmp-dma
 15:      31518024          0                  0         10595920  GICv2 159 Level     zynqmp-dma

 49:    1250127      47679          0          0     GICv2 127 Level     a0120000.al5d, a0100000.al5e
 52:      18662          0        822          0     GICv2 122 Level     xilinx_framebuffer
 53:      19170          0          0          0  interrupt-controller@a0055000   3 Level   -level     xilinx_framebuffer
 54:      18825          0          0          0  interrupt-controller@a0055000   0 Level   -level     xilinx_framebuffer
 55:      18463          0          0          0  interrupt-controller@a0055000   1 Level   -level     xilinx_framebuffer
 56:     544834          0    5528886          0     GICv2 123 Level     xilinx-hdmi-rx
 57:          0          0          0          0     GICv2 121 Level     xilinx_framebuffer
 58:      86730          0          0     915677     GICv2 125 Level     xilinx-hdmitxss
 59:      86752          0          0     915711     GICv2 128 Level     xlnx-mixer 
 60:      42021          0          0          0     GICv2 138 Level     a00c0000.sync_ip