Daniel Morin
March 19, 2024
Share this post:
Reading time:
The highly anticipated release of GStreamer version 1.24 is finally here! It is the culmination of 13 months of meticulous and dedicated effort by the entire community. A total of 231 developers contributed to this release, resulting in over 3141 patches. Thanks to sponsors like STMicroelectronics, Netflix, Living Optics, and ChromeOS, Collabora once again came in first place with the most contributors for this release, with 16 developers taking part.
GStreamer remains the pinnacle open source multimedia framework, and with this release rises to new heights in both embedded and network streaming. Building on our innovative legacy, we also took a bold step forward by introducing a machine learning analytics foundation in GStreamer 1.24. This enables GStreamer to position itself as not only a preeminent multimedia framework, but as the go-to choice for building robust analytics pipelines.
For a more complete list of enhancements from the community, please see the project's release notes.
A remarkable stride forward has been made by Jordan Yelloz by creating the GStreamer W3C Media Source Extensions (MSE) library. This library introduces a GStreamer API aligned with the W3C Media Source Extensions specification, empowering applications to seamlessly integrate existing code for media processing with GStreamer without the necessity of a web browser engine. Built upon WebKit's robust implementation of the Media Source specification, this library offers a versatile foundation. It offers a C API similar to the JavaScript API available to in browser code.
Having had the privilege of collaborating with this library to develop an application, I can attest to its efficacy in simplifying the implementation of adaptive streaming, slicing, and time-shifting. It streamlines these processes, making them much more straightforward and efficient.
In another collaborative effort, Olivier Crête and Jordan Yelloz have combined their expertise to introduce a new signaller, implemented in Rust, for sending streams to LiveKit SFU. This noteworthy addition significantly simplifies the implementation of WebRTC use-cases, making it possible to easily send streams to the LiveKit SFU without having to write code. Concurrently, Olivier has enhanced adaptivedemux2 to parse the cookies from headers correctly and pass back the cookies to further requests, as required by some web services.
In my own contributions, I have enhanced the functionality of the RTSP Connection by adding support for redirect answers [301, 302, 307, 308] when operated over an HTTP tunnel. With these impactful changes in place, rtspsrc can now seamlessly connect to an RTSP server positioned behind proxies. This capability proves especially valuable in environments where controlled access to a video source, such as IP cameras, is a common use-case.
Say "bonjour" to efficiency with Nicolas' work on DRM (Direct Rendering Manager) Dumb allocators; we can now allocate buffers from DRM drivers. The DRM Dumb allocator is no longer exclusive to kmssink but also available to waylandsink. This can make software rendering more efficient on cost optimized processors that don't include a GPU, as "Dumb" buffers are especially allocated for software rendering. We'd like to thank STMicroelectronics for sponsoring this work and making it possible. In this domain, both kmssink and gtkwaylandsink have seen substantial improvements. These include forcing redraw on resolution changes, implementing a mechanism to report dropped frames for more accurate render/drop rate calculations by fpsdisplaysink, enhanced pool handling for increased resilience, and addressing memory leaks.
Daniel Stone and Robert Mader contributed to the addition of support for wp_single_pixel_buffer_v1, eliminating the need for a 1x1 wl_shm buffer. This allows the Wayland compositor to use a more efficient code path when clearing the surface.
Depending on your use case, shared memory might be more suitable. In such scenarios, consider GstShmAllocator by Xavier Claessens. For an even better solution, if available on your platform, explore unixfd – a new plugin providing sink and source elements that facilitate inter-process pipelines. These elements communicate over a Unix socket, enabling zero-copy transfer of fd backed buffers across pipelines in different processes. Xavier's unixfd is complemented by its new generic metas GstMeta serialization API, allowing the transmission and storage of metas. Presently, gdppay and unixfdsin leverage this generic serdes API, with the expectation that other payloaders, muxers, and sinks will soon make use of it. This work was sponsored by Netflix.
Robert Mader contributed some improvements to the Linux OpenGL stack, making it prefer OpenGL ES over OpenGL by default so OES_EGL_image_external can be used, which is useful when using YUV formats. He also added DMABuf passthrough support to a couple of OpenGL elements to make them easier to use in combination with the new GTK4 graphics offloading feature.
In the realm of Video4Linux (v4l2), the Linux kernel API for video sources and accelerators, Nicolas introduced new pixel formats, including 10bit 4:2:0, and has added support for VeriSilicon and MediaTek specific formats. Continuing on this stride, Nicolas also added arbitrary tile size required for MediaTek tiling in v4l2 and OpenGL, enabling native support for accelerated playback on MediaTek System-on-Chips. I also contributed by incorporating bayer multibytes formats (10, 12, 14, 16 bits) into v4l2src. A special thanks is owed to Living Optics for underwriting this work. In a project sponsored by ChromeOS, utilizing the new stateless decoder API in v4l2, Daniel Almeida expanded the v4l2codecs plugin to support the new AV1 decoder.
On the codec front, Nicolas extended support for 8-bits monochrome to x264enc. Those familiar with codec development understand the significance of having the right tools, and Benjamin Gaignard's new plugin fits this description perfectly. The elements of this plugin, namely av12json, h2642json, and h2652json, parse frame/NAL headers and generate easily readable JSON streams. Thank you, Benjamin; this tool is precisely what I needed to enhance the h264parse element for better compliance with AU splitting. The AU splitting in h264parse has undergone a refactoring based on a look-ahead algorithm, now supporting less common but valid bitstream NAL sequences often encountered in low-latency scenarios.
Collabora's developers are at the forefront of GStreamer analytics. While GStreamer has long been utilized in analytics pipelines, its native support was previously rudimentary. The latest release marks a significant milestone in terms of analytics support. Drawing upon my years of experience in video analytics to Aaron's groundbreaking initiative with analytics support from the previous release and Olivier's profound expertise in GStreamer we collectively achieved the desired outcome to bring generic, efficient, and extensible analytics support to GStreamer.
A notable improvement lies in the overhaul of OnnxInference, previously known as onnxobjectdetector, which has undergone major refactoring and is now integrated into a collection of generic machine learning elements. Built upon the widely adopted open-source framework, ONNX Runtime, with multiple backend support for enhanced performance, onnxinference has been made generic and compatible with any ONNX model. Notably, the separation of inference and tensor-decoding has facilitated the generality of onnxinference, making it adaptable to diverse models. Tensor-decoding, on the other hand, remains model-specific and is handled by specialized tensor-decoders like ssdobjectdetector that are independent from inference engine. This major improvement is the result of a collaboration between Aaron Boxer, Olivier Crête, and myself.
However, generating metadata is just the beginning of a comprehensive analytics pipeline. Real-world analytics pipelines are characterized by their multi-level, multi-modal, and heterogeneous nature. To address this complexity, the introduction of Analytics-Meta serves as a transport mechanism for analytics results. Beyond its memory efficiency, Analytics-Meta is specifically designed to describe relationships between analytics results. It facilitates heterogeneous analytics pipelines (combining computer vision and machine learning), multi-level scenarios (involving multiple ML models), and generic analytics result consumption, such as analytics overlay and Onvif serialization elements. Analytics-Meta is deep rooted in GStreamer with Olivier Crête's core change to allow metadata re-use. This substantial progress significantly enhances GStreamer's analytics support. For more in-depth information on Analytics-Meta, please visit Analytics-Meta. This major advancement is spearheaded by Olivier Crête and myself, Daniel Morin.
Detlev Casanova enhanced our CI system using the VIrtual StateLess (visl) decoder driver as well as Fluster to eliminate the need for hardware decoders and make it possible to test the v4l2 codec stack in GStreamer's cloud-based CI system. The comprehensive v4l2-stateless-decoders test suite now includes tests for VP8, VP9, H.264, and H.265, executed within a dedicated environment leveraging QEMU and VirtMe. These efforts were made possible thanks to ChromeOS's support.
After community consultations, Olivier Crête has assumed the stewardship role and decided to retire gst-omx due to its prolonged state of disuse over the years.
Our next plans for GStreamer will be to add AV1 support to visl and expand analytics support with multi-modal analysis, tfliteinference, and AnalyticsMeta2Onfiv.
Whether you are eager to delve into the capabilities of GStreamer 1.24 or have inquiries about harnessing its innovative features for optimal hardware performance, Collabora's multimedia team is here to support you, providing expertise and assistance to help you make the most of the latest feature releases within the GStreamer framework. Don't hesitate to contact us; we are ready to assist you in leveraging and implementing these cutting-edge enhancements.
20/12/2024
The Rockchip RK3588 upstream support has progressed a lot over the last few years. As 2024 comes to a close, it is a great time to have…
09/12/2024
Collabora will be at NeurIPs this week to dive into the latest academic findings in machine learning and research advancements that are…
05/12/2024
Now based on Debian Bookworm, Apertis is a collaborative OS platform that includes an operating system, but also tools and cloud services…
Whether writing a line of code or shaping a longer-term strategic software development plan, we'll help you navigate the ever-evolving world of Open Source.
한국어 버전의 Collabora.com 보기
Acesse Collabora.com em Português
Comments (0)
Add a Comment