In Scaler Only mode, at the start of the first frame, the Scaler will copy the coefficients from the AXI4-Lite interface into a local RAM. This will take about 500 cycles. After this, the scaler should be able to start filling its internal line buffer. Scaler starts generating output when the line buffer is filled about half full. If the supports 4:2:0, the buffer for chroma resampling will also need to be filled. As a safe measure, Scaler uses the number of vertical taps for the number of lines of latency. After every line, the Scaler will have to do some bookkeeping, and therefore, for about 20 cycles the Scaler will not be able to accept data. You can reduce the initial latency by reducing the number of vertical taps.