The GStreamer community recently announced the release of version 1.22 of the open source multimedia framework, and once again Collabora has made a number of key contributions. In this cycle, 17 Collabora engineers contributed 227 patches to the core GStreamer repository!

As spotlighted in the previous release, our contributions focused on making GStreamer the best framework for multimedia features on embedded platforms. We're particularly proud of our work on the support of hardware codec accelerators, also known as "stateless codecs".

Hardware accelerated video decoders

Collabora has been doing a lot of work on the kernel side of stateless codecs, but when we say they are stateless, we really mean that the hardware or firmware doesn't keep a state. The decoding process is still a stateful process and this state is mostly managed by the userspace application. GStreamer has one of the implementations of this state tracking. In the 1.22 release, following the merging of a HEVC stateless API into the Linux kernel, Benjamin Gaignard, Sebastian Fricke, and Nicolas Dufresne merged the GStreamer code to enable HEVC decoding using stateless hardware.

We've also made a number of smaller improvements to support various hardware specificities. Benjamin added the P010 10-bit format to the stateless elements. Nicolas added a new format he called "NV12_16L32S" which MediaTek calls MM21: it's a tiled format specific to some MediaTek codecs, by adding support for it to both the GStreamer v4l2 elements, but also to the OpenGL elements, we can now display the output of the decoder directly. As part of this work, Nicolas improved GStreamer helpers allowing arbitrary tile dimensions. While he was working on the OpenGL elements, he also added support for other tiled formats. Nicolas also made a compressed header option in GStreamer in order to support MediaTek decoders.

We've also been preparing for the arrival of stateless AV1 decoders, Nicolas contributed some fixes to the AV1 base decoder and improved the signalling in the parser to report the right alignment. And going back to some older codecs, Nicolas finally fixed the support for interlaced H.264 decoding with stateless decoders.

Straight to Wayland

On many embedded systems, the GPU is just too slow to process full-resolution video. But they've been designed so that the buffers coming out of the hardware decoders can be fed directly into the display controller. On Linux, this is best supported by giving the buffer to the Wayland compositor and letting it figure out how to deal with the display controller. In GStreamer, we already had waylandsink to enable this use-case, but it was difficult to use because it required the application developer to use the low-level Wayland APIs directly. Robert Mader and George Kiagiadakis have now created a gtkwaylandsink which provides a GTK+ 3 widget, making it easy to integrate it into a GTK+ application. It uses the same model that we already had for gtkglsink which combines a tightly integrated pair of a GStreamer element and a GTK+ widget. Robert also added support for rotation to both versions of the waylandsink, enabling them to tell the display controller that the content should be rotated before displaying it on screen - or displaying pre-rotated content on rotated displays more efficiently.

Making GStreamer shine on tiny devices

On the build side, Xavier Claessens is still on his quest to make Meson awesome and have GSteamer use it to its full potential. Building on his contribution of a development environment setup module to Meson, he improved its use in GStreamer. He also moved the harness enabling GStreamer to be used in Android applications from the Cerbero build system into the core of GStreamer, making it possible for Android applications with Meson only. Also continuing on our previous work to make it possible to build GStreamer and its dependencies as a single large library called gstreamer-full, he added the last missing piece, support for GIO plugins. This is in particular essential to enable the use of TLS libraries using the GLib abstraction. To help with memory management in C, GLib can use GCC/clang extensions to automatically free the memory when a variable goes out of scope. Xavier uses this mechanism to unmap buffers with g_auto(GstBufferMapInfo).

And much more

Jakub Adam had a completely different focus; he's been working on improving screen capture. On the Windows side, he made it possible for the d3d11screencapturesrc element to grab only a portion of the screen. On Linux, he improved ximagesrc so that it can follow changes in the screen's resolution. Colin Kinloch added a little bit of code to enable the serialization and deserialization of the GDateTime format, making it possible to use it on the command line to configure element properties. He also made all elements use the common code to parse the rotation tags, ensuring that all elements have consistent behaviour.

Daniel Morin made his first contribution to GStreamer, porting the ONNX-Runtime based elements to the latest version.

And finally, I once again worked on GStreamer WebRTC support, fixing various issues discovered when creating a WebRTC bridge where GStreamer receives using WebRTC and then sends out again using WebRTC. After adding support for asynchronous DNS resolution to libnice, I remove it from GStreamer's webrtc element, making the code more simple, but also make it possible to start the connection attempt in parallel with the DNS resolution of the relay servers. I also looked at the wpesrc element, adding support key modifiers, so it can receive keyboard events such as "ctrl-v", not only receive it as the letter "v".

We're already working on some exciting changes for the next version of GStreamer. If you want to use GStreamer in your project or want to see it improved, please contact us!

Related Posts

Adding VP9 and MPEG2 stateless support in v4l2codecs for GStreamer

GStreamer 1.20: Embedded & WebRTC lead the way

Generate a minimal GStreamer build, tailored to your needs

Related Posts

Adding VP9 and MPEG2 stateless support in v4l2codecs for GStreamer

GStreamer 1.20: Embedded & WebRTC lead the way

Generate a minimal GStreamer build, tailored to your needs