To model and control video capture pipelines such as the ones used in this TRD on Linux systems, multiple kernel frameworks and APIs must work in conjunction. For simplicity, the overall solution is referred to as Video4Linux (V4L2), although the framework only provides part of the required functionality. Individual components are discussed in the following sections.
Driver Architecture
The following figure shows the VL42 driver stack (a generic V4L2 driver model of a video pipeline). The video pipeline driver loads the necessary sub-device drivers and registers the device nodes it needs, based on the video pipeline configuration specified in the device tree.
The framework exposes the following device node types to user space to control certain aspects of the pipeline:
- Media device node:
/dev/media* - Video device node:
/dev/video* - V4L subdevice node:
/dev/v4l-subdev*
/dev/media1,
/dev/media2, and so on.These steps describe the software data flow:
- The V4L2 source driver allocates a frame buffers for the capture device.
- The V4L2 framework imports/exports the DMA_BUF file descriptor (FD) to the GStreamer element.
- The frame buffer is shared with the DRM display device using the DMA_BUF framework.
Media Framework
The main goal of the media framework is to discover the device topology of a video pipeline and to configure it at run time. To achieve this, pipelines are modeled as an oriented graph of building blocks called entities and are connected through pads.
An entity is a basic media hardware building block. It can correspond to a large variety of blocks such as physical hardware devices (image sensors), logical hardware devices (soft IP cores inside the PL), DMA channels, or physical connectors. Physical or logical devices are modeled as sub-device nodes, and DMA channels as video nodes.
A pad is a connection endpoint through which an entity can interact with other entities. Data produced by an entity flows from the entity's output to one or more entity inputs. A link is a point-to-point oriented connection between two pads, either on the same entity or on different entities. Data flows from a source pad to a sink pad.
A media device node is created that allows the user space application to configure the video pipeline and its sub-devices through the libmediactl and libv4l2subdev libraries. The media controller API provides this functionality:
- Enumerates entities, pads, and links
- Configures pads
- Sets media bus format
- Sets dimensions (width/height)
- Configures links
- Enable/disable
- Validates formats
The following figure shows the media graph for the CSI RX video capture pipelines as generated by the media-ctl utility. The numbers on the edges are pads and the solid arrows represent active links. The gray boxes are video nodes that correspond to frame buffer write channels (outputs).
