Memory Stall Analysis - 2021.2 English

1. Select Trace view.
2. Choose Stalls view in the bottom and select Memory Stalls from the drop-down list.
   Figure 1. Memory Stall in Trace View
   
   

   The stall is named as MS_<NUM>. The number is increased by time. Each stall has the following associated information.

   Stall ID

   The memory stall id. The earlier the stall happens, the smaller the number. The number is unique across all types of stall.

   Stalled Tile

   The AI Engine tile where the stalled kernel is located.

   Stalled Kernel

   The kernel that is stalled. It is named <Kernel_function_name>.<Schedule_ID>.<Graph_instance_name>. Sometimes it is shown as _main and then cross-probe is required to find the real kernel function.

   Start (ns)

   The start time that the stall happens

   Duration (ns)

   The duration of the stall.

   Bank Conflict

   The memory bank where the stall happens on.

   Buffer 1, Buffer 2, Buffer 3

   The buffers that cause the memory stall. It can be one buffer or multiple buffers.

3. When you click each line of the stalls in the Stalls view, it goes to the start of the memory stall in Trace view. Zoom in and out of the Trace view to observe how frequently the memory stalls occur and the position of the stall in kernel running.
   Note: If large number of memory stalls occur repeatedly in the running kernel, it indicates that the stalls can happen inside loop. It is best to investigate and resolve. If memory stalls only happen once at the start of the kernel running, or a very small number of stalls happen in kernel running, it can usually be neglected. From the name of the buffers that cause the stall, it can be identified whether it is window buffer or system buffer or something else. If it is a window buffer or RTP buffer that can be controlled in the graph, one way is to place it manually using constraints if better placement can be identified. If it is system memory (named system<NUM>+<NUM>), it is required to identify the variables that are involved in the stall.
4. Click the row of the specific stall and switch to Events view.
   Figure 2. Events View of Memory Stall
   
   
5. The Events view shows the events that happen in the device. The cycle where the memory stall happens is highlighted. You can see the tile on which the DM_BANK_CONFLICT event has occurred and the variables that are being read or written. In the previous figure, the variables delay_line and eq_coef1 are being read at the same cycle.
6. Try to explore some cycles before or after the stall cycle to find more hints. For example, the following figure shows the events happened after the previous stall cycle. It is seen that delay_line and eq_coef1 are also being read at the same cycle but different part (128 bits). By examining the source code and assembly code, it can be found that delay_line and eq_coef1 are both issued 256 bits at the same cycle, and that causes the memory stall. The two 256 bits memory access are split into two cycles due to the memory stall.
   Figure 3. Events View of Memory Stall
   
   

   One way to resolve it is documented in the Load and Store with Virtual Resource Annotations section in AI Engine Kernel Coding Best Practices Guide (UG1079). For example, redefine points to eq_coef0 and delay_line and annotate them with __aie_dm_resource_a. Use the following new points instead.

   const v8cint16 __aie_dm_resource_a* __restrict coeff = (v8cint16 __aie_dm_resource_a*) eq_coef0; const v8cint16 coe = *coeff;

   v16cint16 __aie_dm_resource_a* __restrict p_buff = (v16cint16 __aie_dm_resource_a*) &delay_line; v16cint16 buff=*p_buff;

The following table lists some possible scenarios that cause memory stalls and possible solutions.