Troubleshooting Event Trace in Hardware - 2022.2 English

AI Engine Tools and Flows User Guide (UG1076)
Document ID
UG1076
Release Date
2022-10-19
Version
2022.2 English
Table 1. Troubleshooting Event Trace in Hardware
Issue Resolution
Trace packets are being dropped on certain streams. (This can be determined by observing black bars on event trace streams in the Vitis Analyzer GUI.) Option 1: Increase the number of trace streams coming out of AI Engine using the option below in build time.
aiecompiler --num-trace-streams=<N>

Option 2: If using PLIO event trace streams, use theaiecompiler flag --trace-plio-width=128 which uses the128-bit PLIO stream width. This ensures the PLIO stream width matches the AI Engine data rates.

Option 3: Reduce the number of tiles connected per trace stream. You can configure the AI Engine event trace metric to be applied on specific graphs or tiles. More details on these options can be found in Table 1 and Table 1.

Option 4: aiecompiler allocates trace streams for every tile in a horizontal fashion. To ensure optimal trace nets routing it is recommended to use location constraints to ensure AI Engines are placed vertically wherever possible so that all active AI Engines equally distributed to trace streams.
Kernel function names are not visible from trace. This could be because the kernel is inlined either by the compiler or via an attribute. Apply --xlopt=0 to compile the design or specify kernel functions with __attribute__((noinline)) attribute to disable kernel in-lining.
Kernels start time are off by more than one hundred cycles. Apply --broadcast-enable-core=true option to the compiler to ensure the design starts all kernels in a few clock cycles range.
Tool issues a warning message indicating trace buffer is full. Increase event trace buffer size.
For the XSDB flow, specify larger value with the -depth option in the AI Engine trace start command.
%xsdb aietrace start -graphs dut -config-level functions_all_stalls -work-dir ./Work -link-summary ./bf_hw.xsa.xclbin.link_summary -base-address 0x900000000 -depth 0x8000000
For the XRT flow, update xrt.ini file and aie_trace_buffer_size line.
[Debug]
aie_trace=true
aie_trace_buffer_size=100M
aie_trace_metrics = functions_all_stalls
Note: The following warning message is issued by both XSDB and XRT when the trace buffer is full.

Warning message:

AI Engine Trace Buffer size is full, Device trace could be incomplete.
Need to know if the PL is sending or receiving data to/from the AI Engine as expected.

Add monitors on PL kernels and their memory-mapped AXI4 masters.

v++ -l --profile_kernel <data:[kernel_name|all]:[compute_unit_name
|all]:[interface_name|all](:[counters|all])><[stall|exec]:[kernel_name|all]:[compute_u
nit_name|all](:[counters|all])>

For example, to monitor every kernel master, add the following:

v++ -l --profile_kernel data:all:all:all

Running on hardware, add the following lines in the xrt.ini file.

[Debug]
profile=true
native_xrt_trace=true
data_transfer_trace=coarse

After running the application on hardware, device_trace_0.csv, native_trace.csv, summary.csv, and xrt.run_summary files are generated on sd_card. Copy those files to where your project is and at same level as the project's Work directory. Issue the command vitis_analyzer xrt.run_summary and select the timeline Trace view to inspect PL trace.