Usage - 2023.1 English

H.264/H.265 Video Codec Unit v1.2 Solutions LogiCORE IP Product Guide (PG252)

Document ID
PG252
Release Date
2023-05-16
Version
2023.1 English

Normal Latency

This is standard latency and can be used with any of the rate control mode.

Gstreamer Encoder Pipeline
gst-launch-1.0 v4l2src io-mode=4 device=/dev/video0 ! video/x-raw, width=1920, height=1080, framerate=60/1, format=NV12 ! omxh264enc ! video/x-h264, alignment=au! Fakesink
Gstreamer Decoder Pipeline
gst-launch-1.0 filesrc location="input-file.mp4" ! qtdemux name=demux demux.video_0 ! h264parse ! video/x-h264, alignment=au ! omxh264dec low-latency=0 ! queue max-size-bytes=0 ! fakevideosink
Ctrl-SW Command
The v4l2 capture control software encoder application demonstrates Xilinx’s Low-Latency feature using the VCU ctrlsw APIs. It is an enhanced version of normal VCU ctrlsw app (ctrlsw_encoder).

Refer to this wiki page section for commands and additional details.

No Reordering (Reduced Latency)

For encoder, set rate control mode to low-latency with I only, IPPP, or low-delay-P mode. For decoder, set low-latency parameter with alignment to AU through caps for sink pad.

Gstreamer Encoder Pipeline
gst-launch-1.0 v4l2src io-mode=4 device=/dev/video0 ! video/x-raw, width=1920, height=1080, framerate=60/1, format=NV12 ! omxh264enc gop-mode= (basic or low-delay-p), b-frames=0 control-rate=low-latency ! fakesink
Gstreamer Decoder Pipeline
gst-launch-1.0 filesrc location="input-file.mp4" ! qtdemux name=demux demux.video_0 ! h264parse ! video/x-h264, alignment=au ! omxh264dec low-latency=1 ! queue max-size-bytes=0 ! fakevideosink
Ctrl-SW command
At the control software level, it can be specified using command line arguments.
ctrlsw_decoder -avc -in input-avc-file.h264 -out ouput.yuv --no-reordering
Note: There is no argument at encoder side for no-reordering.

Low Latency

This mode supports sub-frame latency. For encoder, recommended to set rate control mode to low-latency rate for best performance and pass alignment = NAL through caps at encoder source pad. For decoder, set low-latency parameter with alignment set to NAL through caps for sink pad. Low latency can be enabled at the encoder and decoder sides as follows:

omxh264enc ! video/x-h264, alignment=nal ! ... ! omxh264dec low-latency=1 ! ...
Gstreamer Encoder Pipeline
gst-launch-1.0 v4l2src io-mode=4 device=/dev/video0 ! video/x-raw, width=1920, height=1080, framerate=60/1, format=NV12 ! omxh264enc gop-mode= (default or low-delay-p), b-frames=0 control-rate=low-latency ! video/x-h264, alignment=nal ! fakesink
Gstreamer Decoder Pipeline
gst-launch-1.0 filesrc location="input-file.mp4" ! qtdemux name=demux demux.video_0 ! h264parse! video/x-h264, alignment=nal ! omxh264dec low-latency=1 ! queue max-size-bytes=0 ! fakevideosink
Ctrl-SW command
At the control software level, it can be specified through command line arguments.
ctrlsw_encoder –cfg encode_simple.cfg –-slicelat
ctrlsw_decoder -avc -in input-avc-file.h264 -out ouput.yuv -slicelat

Xilinx Low Latency

Xilinx low latency can be enabled by adding an extra caps video/x-raw(memory:XLNXLL) at v4l2src and decoder side as follows

v4l2src ! video/x-raw(memory:XLNXLL), width=3840, height=2160, ... ! omxh264enc ! video/x-h264, alignment=nal ! ... ! omxh264dec low-latency=1 ! video/x-raw(memory:XLNXLL) ...
4x 1080p60 AVC NV12 Xilinx Low-Latency Pipeline
gst-launch-1.0 -v v4l2src io-mode=dmabuf device=/dev/video0 ! video/x-raw/(memory:XLNXLL/), width=1920, height=1080, format=NV12, framerate=60/1 ! queue ! omxh264enc control-rate=low-latency target-bitrate=6000 prefetch-buffer=TRUE num-slices=8 periodicity-idr=240 cpb-size=500 initial-delay=250 gdr-mode=horizontal gop-mode=low-delay-p ! video/x-h264, alignment=nal ! queue max-size-buffers=0 ! omxh264dec low-latency=1 ! video/x-raw/(memory:XLNXLL/) ! queue max-size-bytes=0 ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=kmssink bus-id=a00c0000.v_mix plane-id=30 hold-extra-sample=1 show-preroll=false sync=true' sync=true -v
gst-launch-1.0 -v v4l2src io-mode=dmabuf device=/dev/video1 ! video/x-raw/(memory:XLNXLL/), width=1920, height=1080, format=NV12, framerate=60/1 ! queue ! omxh264enc control-rate=low-latency target-bitrate=6000 prefetch-buffer=TRUE num-slices=8 periodicity-idr=240 cpb-size=500 initial-delay=250 gdr-mode=horizontal gop-mode=low-delay-p ! video/x-h264, alignment=nal ! queue max-size-buffers=0 ! omxh264dec low-latency=1 ! video/x-raw/(memory:XLNXLL/) ! queue max-size-bytes=0 ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=kmssink bus-id=a00c0000.v_mix plane-id=31 hold-extra-sample=1 show-preroll=false sync=true' sync=true -v
gst-launch-1.0 -v v4l2src io-mode=dmabuf device=/dev/video2 ! video/x-raw/(memory:XLNXLL/), width=1920, height=1080, format=NV12, framerate=60/1 ! queue ! omxh264enc control-rate=low-latency target-bitrate=6000 prefetch-buffer=TRUE num-slices=8 periodicity-idr=240 cpb-size=500 initial-delay=250 gdr-mode=horizontal gop-mode=low-delay-p ! video/x-h264, alignment=nal ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=fakesink sync=true' sync=true -v
gst-launch-1.0 -v v4l2src io-mode=dmabuf device=/dev/video3 ! video/x-raw/(memory:XLNXLL/), width=1920, height=1080, format=NV12, framerate=60/1 ! queue ! omxh264enc control-rate=low-latency target-bitrate=6000 prefetch-buffer=TRUE num-slices=8 periodicity-idr=240 cpb-size=500 initial-delay=250 gdr-mode=horizontal gop-mode=low-delay-p ! video/x-h264, alignment=nal ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=fakesink sync=true' sync=true -v
4kp60 AVC NV12 Xilinx Low-Latency Serial Pipeline
gst-launch-1.0 -v v4l2src io-mode=dmabuf device=/dev/video0 ! video/x-raw\(memory:XLNXLL\), width=3840, height=2160, format=NV12, framerate=60/1 ! omxh264enc num-slices=8 control-rate=low-latency gop-mode=low-delay-p target-bitrate=25000 cpb-size=500 gdr-mode=horizontal initial-delay=250 periodicity-idr=240 filler-data=0 prefetch-buffer=true ! video/x-h264, alignment=nal ! queue max-size-buffers=0 ! omxh264dec low-latency=1 ! video/x-raw\(memory:XLNXLL\) ! queue max-size-bytes=0 ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=kmssink bus-id=a0070000.v_mix plane-id=33  hold-extra-sample=1 show-preroll=false sync=true' sync=true -v
2x 4kp30 HEVC NV12 Xilinx Low-Latency Serial Pipeline
ENC_EXTRA_OP_BUFFERS=10 gst-launch-1.0 -v v4l2src io-mode=dmabuf device=/dev/video0 ! video/x-raw\(memory:XLNXLL\), width=3840, height=2160, format=NV12, framerate=30/1 ! omxh265enc control-rate=low-latency target-bitrate=12500 filler-data=0 prefetch-buffer=true num-slices=8 periodicity-idr=240 cpb-size=500 gdr-mode=horizontal initial-delay=250 gop-mode=low-delay-p ! video/x-h265, alignment=nal ! queue max-size-buffers=0 ! omxh265dec low-latency=1 ! video/x-raw\(memory:XLNXLL\) ! queue max-size-bytes=0 ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=kmssink bus-id=a0070000.v_mix plane-id=33 hold-extra-sample=1 show-preroll=false render-rectangle=<0,0,1920,2160> max-lateness=10000000 sync=true' sync=true -v > /run/pipeline0.txt 2>&1 &
ENC_EXTRA_OP_BUFFERS=10 gst-launch-1.0 -v v4l2src io-mode=dmabuf device=/dev/video1 ! video/x-raw\(memory:XLNXLL\), width=3840, height=2160, format=NV12, framerate=30/1 ! omxh265enc control-rate=low-latency target-bitrate=12500 filler-data=0 prefetch-buffer=true num-slices=8 periodicity-idr=240 cpb-size=500 gdr-mode=horizontal initial-delay=250 gop-mode=low-delay-p ! video/x-h265, alignment=nal ! queue max-size-buffers=0 ! omxh265dec low-latency=1 ! video/x-raw\(memory:XLNXLL\) ! queue max-size-bytes=0 ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=kmssink bus-id=a0070000.v_mix plane-id=34  hold-extra-sample=1 show-preroll=false render-rectangle=<1920,0,3840,2160> max-lateness=10000000 sync=true' sync=true -v > /run/pipeline1_$filename.txt 2>&1 &
4kp60 AVC NV12 Xilinx Low-Latency Streaming Pipeline
Client
gst-launch-1.0 udpsrc port=5004 buffer-size=60000000 caps="application/x-rtp, media=video, clock-rate=90000, payload=96, encoding-name=H264" ! rtpjitterbuffer latency=7 ! rtph264depay ! h264parse ! video/x-h264, alignment=nal ! omxh264dec low-latency=1 ! video/x-raw\(memory:XLNXLL\) ! queue max-size-bytes=0 ! fpsdisplaysink name=fpssink text-overlay=false video-sink="kmssink bus-id=a0070000.v_mix plane-id=33" sync=true -v > /run/client0_pipeline.log 2>&1 &
Server

gst-launch-1.0 -v v4l2src device=/dev/video0 io-mode=4 ! video/x-raw\(memory:XLNXLL\),format=NV12,width=3840,height=2160,framerate=60/1 ! omxh264enc num-slices=8 periodicity-idr=240 cpb-size=500 gdr-mode=horizontal initial-delay=250 control-rate=low-latency prefetch-buffer=true target-bitrate=25000 gop-mode=low-delay-p ! video/x-h264, alignment=nal ! rtph264pay ! udpsink buffer-size=60000000 host=192.168.0.2 port=5004 max-lateness=-1 qos-dscp=60 async=false max-bitrate=120000000 -v > /run/server0_pipeline.log 2>&1 &
Ctrl-SW command
The Xilinx Low Latency mode is not supported in the example control software encoder application ctrlsw_encoder, that comes with petalinux as this latency mode is used mainly for live use-cases that the ctrlsw_encoder does not support. However, there is a separate v4l2 capture control software encoder application implemented which also supports Xilinx’s Low-Latency feature using the ctrlsw APIs. It is an extension to the normal VCU ctrlsw app (ctrlsw_encoder) to support live record and streamout use-cases. Please refer to this wiki page for commands and additional details.