Dense Non-Pyramidal LK Optical Flow - 2023.2 English

Optical flow is the pattern of apparent motion of image objects between two consecutive frames, caused by the movement of object or camera. It is a 2D vector field, where each vector is a displacement vector showing the movement of points from first frame to second.

Optical Flow works on the following assumptions:

• Pixel intensities of an object do not have too many variations in consecutive frames
• Neighboring pixels have similar motion

Consider a pixel I(x, y, t) in first frame. (Note that a new dimension, time, is added here. When working with images only, there is no need of time). The pixel moves by distance (dx, dy) in the next frame taken after time dt. Thus, since those pixels are the same and the intensity does not change, the following is true:

Taking the Taylor series approximation on the right-hand side, removing common terms, and dividing by dt gives the following equation:

Where , , and .

The above equation is called the Optical Flow equation, where, fx and fy are the image gradientsand ft is the gradient along time. However, (u, v) is unknown. It is not possible to solve this equation with two unknown variables. Thus, several methods are provided to solve this problem. One method is Lucas-Kanade. Previously it was assumed that all neighboring pixels have similar motion. The Lucas-Kanade method takes a patch around the point, whose size can be defined through the ‘WINDOW_SIZE’ template parameter. Thus, all the points in that patch have the same motion. It is possible to find (fx, fy, ft ) for these points. Thus, the problem now becomes solving ‘WINDOW_SIZE * WINDOW_SIZE’ equations with two unknown variables,which is over-determined. A better solution is obtained with the “least square fit” method. Below is the final solution, which is a problem with two equations and two unknowns:

API Syntax

template< int WINDOW_SIZE, int TYPE, int ROWS, int COLS, int NPC, bool USE_URAM=false, int XFCVDEDPTH_IN_0 = _XFCVDEPTH_DEFAULT, int XFCVDEDPTH_IN_1 = _XFCVDEPTH_DEFAULT, int XFCVDEDPTH_IN_2 = _XFCVDEPTH_DEFAULT, int XFCVDEDPTH_IN_3 = _XFCVDEPTH_DEFAULT>
void DenseNonPyrLKOpticalFlow (xf::cv::Mat<TYPE, ROWS, COLS, NPC, XFCVDEDPTH_IN_0> & frame0, xf::cv::Mat<TYPE, ROWS, COLS, NPC, XFCVDEDPTH_IN_1> & frame1, xf::cv::Mat<XF_32FC1, ROWS, COLS, NPC, XFCVDEDPTH_IN_2> & flowx, xf::cv::Mat<XF_32FC1, ROWS, COLS, NPC, XFCVDEDPTH_IN_3> & flowy)

Parameter Descriptions

The following table describes the template and the function parameters.

Resource Utilization

The following table summarizes the resource utilization of DenseNonPyrLKOpticalFlow for a 4K image, as generated in the Vivado HLS 2019.1 version tool for the Xilinx Xczu9eg-ffvb1156-1-i-es1 FPGA at 300 MHz.

The following table summarizes the resource utilization of DenseNonPyrLKOpticalFlow for a 4K image, as generated in the Vivado HLS version tool for the Xilinx Xczu7eg-ffvb1156-1 FPGA at 300 MHz with UltraRAM enabled.

Performance Estimate

The following table summarizes performance estimates of the DenseNonPyrLKOpticalFlow function for a 4K image, generated using Vivado HLS 2019.1 version tool for the Xilinx Xczu9eg-ffvb1156-1-i-es1 FPGA.