Olivier Crête
April 09, 2025
Share this post:
Reading time:
The GStreamer project has recently released version 1.26, and once again, Collabora has made numerous contributions and we're proud to have had more contributors for this release than any other company. We focused our attention on three key areas: efficient use of hardware, the MPEG-5 LCEVC codec, and improving the support for analytics and machine learning.
For optimal hardware usage, the most important improvement in the last two releases of GStreamer has been the introduction of DRM modifiers. This is especially important when dealing with high-resolution video such as 4K or even 8K or when working with HDR10 formats. The Linux kernel's DRM subsystem describes video buffer formats using two values. The first one is the traditional fourcc describing the color representation. But many hardware platforms offer more options than fourcc can appropriately describe. The video content of the buffer could be compressed, secure, or stored in tiles. The hardware has often been designed to only support its highest resolution or color format when using compression or tiles, making 4K or HDR10 possible on those platforms. To represent these options, a second value called a "modifier" is used. These modifiers can be hardware-specific, and they generally indicate that it may not be possible to access or interpret the content of the memory from application software. Therefore, image buffers described with modifiers can generally only be used by hardware drivers. Since GStreamer 1.24, GStreamer supports describing video formats using those modifiers directly. To make things more complicated, the Video4Linux2 subsystem of the Linux kernel, used for capture devices such as cameras and for video codecs, does not use the DRM modifiers, but instead has its own list of pixel formats. The V4L2 formats don't have modifiers but instead describe each combination with a new identifier. To bridge these two worlds, Robert Mader has added code to GStreamer to convert between these two formats. This makes it possible to capture in the most optimal format available.
Another important feature is the GPU's ability to write into memory allocated by other subsystems, as the GPU is generally the most flexible part of the pipeline, having memory layout requirements that are less strict than codecs or display controllers. To achieve this, we've enabled various elements that render using a GPU to import a buffer pool from a downstream element. Nicolas Dufresne has added this to the qmlgl6src element, which renders a Qt6 QML scene into a video stream. Jakub Adam has added the same functionality to the gldownload element, which allows other GStreamer OpenGL elements to render into a downstream buffer pool. This allows features such as PCIe Peer-to-peer (P2P) communication to work seamlessly when supported by the underlying hardware and drivers. Nicolas has also made the waylandsink element prefer DMABuf support when this has been negotiated, avoiding inappropriate fallback to system memory when sending buffers directly to a Wayland compositor. Nicolas has also improved the qmlgl6src element to reduce its rendering latency, making it more suitable for remote UI use cases.
The team has also been busy making other improvements to GStreamer's Video4Linux2 plugins. These elements can now better support frame rate negotiation. The way in which GStreamer and V4L2 each negotiate frame rates is slightly different, and the algorithms should now work better together. The stateless decoders can now handle changing the resolution at random points in AV1 and VP9 streams, as this is permitted by their respective formats. The stateful encoders can also now support an "unknown" frame rate, no longer requiring a frame rate to be arbitrarily set when the stream doesn't have the frame rate information. We've also added support for a number of buffer formats, such as P010, and formats specific to MediaTek and Rockchip SoCs.
Julian Bouzas has been hard at work adding support for the new MPEG-5 Low Complexity Enhancement Video Coding (LCEVC) standard to GStreamer. LCEVC is a new type of video compression standard as It enhances the quality of an image previously encoded with another codec. Julian has integrated it into GStreamer by adding a second enhancement decoder in addition to the base layer decoder. The LCEVC decoder is based on the open source LCEVCDec library. Julian has also contributed an encoder plugin based on V-Nova's SDK. This work was the result of a close collaboration with V-Nova, inventors of LCEVC.
The other area in which our team has been especially active is the integration of Machine Learning and Analytics into GStreamer. GStreamer has already been adopted by most major chips vendors as the base for their video analytics SDK, but sadly, each vendor has built its own environment. We're taking the best ideas from all of them and bringing them together upstream.
The first part of our effort was to create a new GstAnalyticsRelationMeta, which can carry multiple elementary bits of information and relate them to each other. Daniel Morin has enhanced this in the 1.26 release by adding a new type of information for segmentation masks. To make this more useful, he's added an N-to-N relationship, so that a segmentation mask with multiple different types can be related to an array of classes for classification.
Benjamin Gaignard has contributed a set of Rust-based elements to convert between a GstAnalyticsRelationMeta and the XML format described by the ONVIF Analytics specification.
The second part of our analytics effort is to cleanly separate a machine learning inference engine, such as ONNX Runtime or LiteRT, from a model-specific tensor decoder. In GStreamer 1.26, we've separated the tensor decoders from the ONNX Runtime inference element. We've also created a new GstTensorMeta, making it possible to attach arbitrary tensors to buffers so that downstream pipeline elements can interpret them.
We've also improved the Python bindings, making it easier to write analytics elements in Python, as this is the machine learning community's preferred language.
For my part, I've also been writing new elements in Rust; I've created a pair of elements: originalbuffersave and originalbufferrestore. The first element stores a meta on a buffer referencing itself; this reference is kept intact even if the buffer is resized or cropped. The second element restores the original buffer, dropping the modified version, but keeping all metadata appended to the buffer. This makes it possible to resize or crop a video buffer before running analytics steps on it and then restore the original full-resolution buffer.
In the next release we will contribute an element to the LiteRT inference framework (formerly known as Tensorflow Lite or TFLite). We will also contribute tensor decoders for more models, in particular for the Yolo v8 and later segmentation models and the Ultra-Light-Fast-Generic-Face-Detector model. Daniel Morin has also been preparing the ground to add a new tensordecodebin element that can automatically plug in the right tensor decoder based on the model loaded into the inference element. He's been improving the caps negotiation code to make this possible.
Robert Mader has also been busy adding support for the rotation tag to the QuickTime (MP4) and Matroska muxer. This makes it possible to store video that has been described as rotated without actually modifying the encoded image.
We've also been fixing bugs and making minor changes all over the place. Nicolas has been the maintainer for Video4Linux2 support in GStreamer for many years, and he's continued his work in making these elements even better. Jordan Yelloz has improved the mpegtsmux element to work correctly when configured in Constant Bitrate (CBR) mode even when the application changes the bitrate to another one at runtime.
We're looking forward to many more improvements across all those fronts in the next GStreamer release, so expect many new and exciting merge requests to land soon. If you want help with GStreamer, please reach out!
10/04/2025
Three demos. One stand. From end-to-end HDR and a brand-new SoC running PanVK, to NVK and WebGPU out of the box — discover how Collabora…
09/04/2025
Collabora once again played a key role in the latest release of GStreamer, contributing enhancements such as improved hardware efficiency,…
26/03/2025
The kernel release emphasizes the continuing growth and maturity of the Linux ecosystem and delivers new features, improvements, and optimizations…
Whether writing a line of code or shaping a longer-term strategic software development plan, we'll help you navigate the ever-evolving world of Open Source.
한국어 버전의 Collabora.com 보기
Acesse Collabora.com em Português
Comments (0)
Add a Comment