Troubleshooting Event Trace in Hardware - 2021.2 English

Versal ACAP AI Engine Programming Environment User Guide (UG1076)
Document ID
UG1076
Release Date
2021-12-17
Version
2021.2 English
Table 1. Troubleshooting Event Trace in Hardware
Issue- Resolution
Trace packets are being dropped on certain streams. (This can be determined by observing black bars on event trace streams in the Vitis Analyzer GUI.) Increase the number of trace streams coming out of AI Engine using the option below. 
aiecompiler --num-trace-streams=<N>
Trace packets are being dropped on certain streams and the number of trace stream is 16 in a PLIO event trace flow. (This can be determined by observing black bars on event trace streams in the Vitis Analyzer GUI.) Use advanced options in v++ system.cfg file to set the aie trace clock frequency.
[advanced]param=compiler.aieTraceClockSelect=fastest
where compiler.aieTraceClockSelect is the trace clock frequency on the PL trace kernel. The exact frequency setting is dependent on the platform used. Using the "fastest" option could impact PL timing results.
Kernel function names are not visible from trace. This could be because the kernel is inlined either by the compiler or via an attribute. Apply --xlopt=0 to compile the design or specify kernel functions with __attribute__((noinline)) attribute to disable kernel in-lining.
Kernels start time are off by more than one hundred cycles. Apply --broadcast-enable-core=true option to the compiler to ensure the design starts all kernels in a few clock cycles range.
Tool issues a warning message indicating trace buffer is full. Increase event trace buffer size.
For the XSDB flow, specify larger value with the -depth option in the AI Engine trace start command.
%xsdb aietrace start -graphs dut -config-level functions_all_stalls -work-dir ./Work -link-summary ./bf_hw.xclbin.xclbin.link_summary -base-address 0x900000000 -depth 0x8000000
For the XRT flow, update xrt.ini file and aie_trace_buffer_size line.
[Debug] aie_trace=true
aie_trace_buffer_size=100M
aie_trace_metrics = functions_all_stalls
Note: The following warning message is issued by both XSDB and XRT when the trace buffer is full.

Warning message: AI Engine Trace Buffer size is full, Device trace could be incomplete.
Need to know if the PL is sending or receiving data to/from the AI Engine as expected.

Add monitors on PL kernels and their memory-mapped AXI4 masters.

v++ -l --profile_kernel <data:[kernel_name|all]:[compute_unit_name
|all]:[interface_name|all](:[counters|all])><[stall|exec]:[kernel_name|all]:[compute_u
nit_name|all](:[counters|all])>

For example, to monitor every kernel master, add the following:

v++ -l --profile_kernel data:all:all:all

Running on hardware, add the following lines in the xrt.ini file.

[Debug]
profile=true
native_xrt_trace=true
data_transfer_trace=coarse

After running the application on hardware, device_trace_0.csv, native_trace.csv, summary.csv, and xrt.run_summary files are generated on sd_card. Copy those files to where your project is and at same level as the project's Work directory. Issue the command vitis_analyzer xrt.run_summary and select the timeline Trace view to inspect PL trace.