Generating MPEG-DASH streams for Open Source adaptive streaming with GStreamer

Generating MPEG-DASH streams for Open Source adaptive streaming with GStreamer

Stéphane Cerveau
June 12, 2020

Share this post:

Reading time:

Adaptive streaming is a technique to provide flexibility and scalability by offering variable bit-rate streams to the client. Designed to work over HTTP, it provides media content as separate streams with media type and various bit-rates, the client will be able to select according to its network bandwidth or its CPU power.

The most popular adaptive streaming systems are:

HLS (Apple HTTP Live Streaming)
MSS (Microsoft Smooth Streaming)
ADS (Adobe HTTP Dynamic Streaming)
MPEG-DASH (MPEG Dynamic Adaptive Streaming over HTTP)

MPEG-DASH is the most complete adaptive streaming technique. This format is based on an XML description file called an MPD (Media Presentation Description). This format describes a set of representations which has a media type (audio, video or subtitles) and provides various bit-rate or media format.

This solution is an open standard and is widely supported by the industry. For more information about it, you can visit the DASH-IF website.

In the example below, the MPD describes a static content with three media content type (adaptation sets). Each adaptations sets contains representations. The video has 5 different representations, which allows to switch to 5 different bit rates according to the playback constraints.

<MPD mediaPresentationDuration="PT634.566S" minBufferTime="PT2.00S" profiles="urn:hbbtv:dash:profile:isoff-live:2012,urn:mpeg:dash:profile:isoff-live:2011" type="static" xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:mpeg:DASH:schema:MPD:2011 DASH-MPD.xsd">
 <BaseURL>./</BaseURL>
 <Period>
  <AdaptationSet id="1" mimeType="video/mp4" contentType="video" subsegmentAlignment="true" subsegmentStartsWithSAP="1" par="16:9">
   <SegmentTemplate duration="120" timescale="30" media="$RepresentationID$/$RepresentationID$_$Number$.m4v" startNumber="1" initialization="$RepresentationID$/$RepresentationID$_0.m4v"/>
   <Representation id="bbb_30fps_1280x720_4000k" codecs="avc1.64001f" bandwidth="4952892" width="1280" height="720" frameRate="30" sar="1:1" scanType="progressive"/>
   <Representation id="bbb_30fps_320x180_200k" codecs="avc1.64000d" bandwidth="254320" width="320" height="180" frameRate="30" sar="1:1" scanType="progressive"/>
   <Representation id="bbb_30fps_480x270_600k" codecs="avc1.640015" bandwidth="759798" width="480" height="270" frameRate="30" sar="1:1" scanType="progressive"/>
   <Representation id="bbb_30fps_640x360_800k" codecs="avc1.64001e" bandwidth="1013310" width="640" height="360" frameRate="30" sar="1:1" scanType="progressive"/>
   <Representation id="bbb_30fps_3840x2160_12000k" codecs="avc1.640033" bandwidth="14931538" width="3840" height="2160" frameRate="30" sar="1:1" scanType="progressive"/>
  </AdaptationSet>

  <AdaptationSet id="2" mimeType="audio/mp4" contentType="audio" subsegmentAlignment="true" subsegmentStartsWithSAP="1">
   <Accessibility schemeIdUri="urn:tva:metadata:cs:AudioPurposeCS:2007" value="6"/>
   <Role schemeIdUri="urn:mpeg:dash:role:2011" value="main"/>
   <SegmentTemplate duration="192512" timescale="48000" media="$RepresentationID$/$RepresentationID$_$Number$.m4a" startNumber="1" initialization="$RepresentationID$/$RepresentationID$_0.m4a"/>
   <Representation id="bbb_a64k" codecs="mp4a.40.5" bandwidth="67071" audioSamplingRate="48000">
    <AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/>
   </Representation>
  </AdaptationSet>

  <AdaptationSet id="3" mimeType="image/jpeg" contentType="image">
    <SegmentTemplate media="$RepresentationID$/tile_$Number$.jpg" duration="100" startNumber="1"/>
    <Representation bandwidth="12288" id="thumbnails_320x180" width="3200" height="180">
      <EssentialProperty schemeIdUri="http://dashif.org/thumbnail_tile" value="10x1"/>
    </Representation>
  </AdaptationSet>

 </Period>
</MPD>

DASH in GStreamer

Since 2012, GStreamer includes only a DASH client called dashdemux whereas, for HLS, it provides the both elements, demuxer and sink.

DASH Demuxer

This element landed in the GStreamer repository in 2012 and had evolved a lot since that time to support the various use-cases DASH offers in its specifications and its applications. Indeed it is able to support multiple streams (video/audio/subtitles) and allows the user to select from the available streams or automatically select the best representation according to the network.

DASH Sink

A first attempt to propose a DASH sink was also in 2012. The design was not mature enough to be landed and the author never had the time to complete it.

In 2014, a new element called splitmuxsink was introduced. It handles the most complex part of creating a fragmented stream, it cuts files with synchronized audio and video. Based on this element, a new HLS sink called hlssink2 was created in 2017. I decided to finally create a DASH sink based on this approach to fill the gap in GStreamer.

MPD Parser

In order to unify the MPD support, a first task has been to relocate and redesign the base classes to read, and write, an MPD file. Based on XML, the Media Presentation Description scheme is based on multiple nodes owning children and properties as described above. A first important work item was to split the code in 'objects' each identifying an XML node from the MPD schema, including the root node, periods, adaptation sets, etc. An object oriented approach has been selected to unify the work regarding the parsing, object property manipulation and the XML format generation.

The sink

Inspired from the work on hlssink2, the dash sink is a "super bin" and includes a splitmuxsink to provide the multiple media segments. Most of the challenge, here, was to write the MPD compliant with the DASHIF conformance test here with usable and suitable media segments.

This plugin is now capable of:

Multiple input audio/video streams
Multiple periods
TS segment supported (MP4 support is not yet complete), need additional work for the segment transition in short segment scheme.
Fragment segment with given duration
Static/Dynamic MPD (Passing the DASH-IF conformance test)

An example is worth a thousand words

In the following pipeline a static MPD file is created in /tmp along with one single segment long for a single video stream during a period of 60s. The segment will be encoded as H.264 and encapsulated in MPEG transpor stream files.

$ gst-launch-1.0 -m dashsink name=dashsink mpd-root-path=/tmp target-duration=60 dynamic=false period-duration=60000 muxer=ts  v4l2src ! video/x-raw,framerate=30/1,width=320,height=240 ! videoconvert ! queue ! x264enc bitrate=400 ! dashsink.video_0

Future work includes implementing support for creating MP4 fragment files following the CMAF (MPEG-A Part 19) specification.

If you would like to learn more about dashsink or any other parts of GStreamer, please contact us!

Cross-compiling with gst-build and GStreamer

Reducing the size of a Rust GStreamer plugin

Getting started with GStreamer's gst-build

Cross-compiling with gst-build and GStreamer

Reducing the size of a Rust GStreamer plugin

Getting started with GStreamer's gst-build

Comments (15)

wallywall:
Jun 13, 2020 at 03:37 AM

Excellent work!

Reply to this comment

Reply to this comment
cb:
Jan 04, 2021 at 11:57 AM

I found this today, this is awesome! Im using hlssink2 for a while and always was looking for something similar for dash.

Is there a reason why just ts and mp4 muxers are supported right now. I wanted to give it a try with VP9 ( The main reason why I would like to switch from HLS to DASH) but it does not seem to work. So it would be great if webm would be supported as well.

Reply to this comment

Reply to this comment
1. Stéphane Cerveau:
  Jan 11, 2021 at 04:09 PM
  
  Thank you for your interest.
  
  Indeed webm is not supported yet by the dash sink. On the first version of the element, the target was to support the most common supported formats for DASH which are TS and MP4.
  
  If you need this support, feel free to contribute to the code on the GStreamer gitlab codebase. You can participate or follow this entry https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/issues/1496#note_753320. Or you can contact us and we will be pleased to add this support for you.
  
  Reply to this comment
  
  Reply to this comment
vandana:
Feb 16, 2021 at 02:29 PM

Thanks for the pipeline example for dashsink.

However I get the below error when i have to read from dvbsrc instead of v4l2src:

could not link dvbsrc0 to videoconvert0, dvbsrc0 can't handle caps video/x-raw, framerate=(fraction)30/1, width=(int)320, height=(int)240
WARNING: erroneous pipeline: could not link dvbsrc0 to videoconvert0, dvbsrc0 can't handle caps video/x-raw, framerate=(fraction)30/1, width=(int)320, height=(int)240

My pipeline:
gst-launch-1.0 -m dashsink name=dashsink mpd-root-path=/tmp target-duration=60 dynamic=false period-duration=60000 muxer=ts dvbsrc modulation=5 adapter=0 frequency=165000000 delsys=dvb-c-b ! video/x-raw,framerate=30/1,width=320,height=240 ! videoconvert ! queue ! x264enc bitrate=400 ! dashsink.video_0

Reply to this comment

Reply to this comment
1. Stéphane Cerveau:
  Feb 16, 2021 at 02:44 PM
  
  Hello,
  
  According to my understanding, the problem is related to the accepted caps of your "dvbsrc" and not really related to the "dashsink" itself.
  
  Could you please confirm that this pipeline works first:
  
  gst-launch-1.0 dvbsrc modulation=5 adapter=0 frequency=165000000 delsys=dvb-c-b ! video/x-raw,framerate=30/1,width=320,height=240 ! videoconvert ! fakesink
  
  Reply to this comment
  
  Reply to this comment
  1. vandana:
    Feb 17, 2021 at 11:24 AM
    
    You were correct on the capabilities. The fakesink pipeline also gives error:
    0:00:00.014350347 27012 0x55e7e4a02a30 ERROR GST_PIPELINE subprojects/gstreamer/gst/parse/grammar.y:755:gst_parse_perform_link: could not link dvbsrc0 to videoconvert0, dvbsrc0 can't handle caps video/x-raw, framerate=(fraction)30/1, width=(int)320, height=(int)240
    WARNING: erroneous pipeline: could not link dvbsrc0 to videoconvert0, dvbsrc0 can't handle caps video/x-raw, framerate=(fraction)30/1, width=(int)320, height=(int)240
    
    When i see the dvbsrc capacilities using gst-inspect-1.0, I get:
    
    Pad Templates:
    SRC template: 'src'
    Availability: Always
    Capabilities:
    video/mpegts
    mpegversion: 2
    systemstream: true
    
    How do I resolve this capability incompatibility? Without this, I get error:
    gst-launch-1.0 -m dashsink name=dashsink mpd-root-path=/tmp target-duration=60 dynamic=false period-duration=60000 muxer=ts dvbsrc modulation=5 adapter=0 frequency=165000000 delsys=dvb-c-b ! queue ! x264enc bitrate=400 ! dashsink.video_0
    0:00:00.031501501 27077 0x55bcda470e30 ERROR GST_PIPELINE subprojects/gstreamer/gst/parse/grammar.y:774:gst_parse_perform_link: could not link queue0 to x264enc0
    WARNING: erroneous pipeline: could not link queue0 to x264enc0
    
    gst-launch-1.0 -m dashsink name=dashsink mpd-root-path=/tmp target-duration=60 dynamic=false period-duration=60000 muxer=ts dvbsrc modulation=5 adapter=0 frequency=165000000 delsys=dvb-c-b ! dashsink.video_0
    0:00:00.017987747 27082 0x556785554700 ERROR GST_PIPELINE subprojects/gstreamer/gst/parse/grammar.y:774:gst_parse_perform_link: could not link dvbsrc0 to dashsink
    WARNING: erroneous pipeline: could not link dvbsrc0 to dashsink
    
    gst-launch-1.0 -m dashsink name=dashsink mpd-root-path=/tmp target-duration=60 dynamic=false period-duration=60000 muxer=ts dvbsrc modulation=5 adapter=0 frequency=165000000 delsys=dvb-c-b ! queue ! dashsink.video_0
    0:00:00.042886015 27085 0x55eb6d745b30 ERROR GST_PIPELINE subprojects/gstreamer/gst/parse/grammar.y:774:gst_parse_perform_link: could not link queue0 to dashsink
    WARNING: erroneous pipeline: could not link queue0 to dashsink
    
    Reply to this comment
    
    Reply to this comment
    1. Stéphane Cerveau:
      Feb 17, 2021 at 02:19 PM
      
      Hello,
      
      Indeed dvbsrc ouputs mpeg2 ts stream which needs to be demuxed and parsed to be passed to the dashsink.
      
      Can you give a try to this pipeline where dvbsrc stream is demuxed by tsdemux and parsed by mpegvideoparse.
      
```
gst-launch-1.0 dashsink name=dashsink mpd-root-path=/tmp target-duration=60 dynamic=false period-duration=60000 muxer=ts dvbsrc modulation=5 adapter=0 frequency=165000000 delsys=dvb-c-b ! queue ! tsdemux ! mpegvideoparse ! dashsink.video_0
```
      I assume you only have one video stream to feed the dashsink.
      
      Regards.
      
      Reply to this comment
      
      Reply to this comment
      1. vandana:
        Feb 18, 2021 at 10:39 AM
        
        Not the same error, but got this: Pls note that the gstreamer was built using gst-build. Could that be a problem? I am running hlssink and udpsink with this compiled version, and all work ok, I think.
        
        gst-launch-1.0 dashsink name=dashsink mpd-root-path=/tmp target-duration=60 dynamic=false period-duration=60000 muxer=ts dvbsrc modulation=5 adapter=0 frequency=165000000 delsys=dvb-c-b ! queue ! tsdemux ! mpegvideoparse ! dashsink.video_0
        
        Setting pipeline to PAUSED ...
        Pipeline is live and does not need PREROLL ...
        Pipeline is PREROLLED ...
        Setting pipeline to PLAYING ...
        New clock: GstSystemClock
        
        (gst-launch-1.0:13326): GStreamer-CRITICAL **: 10:35:50.452: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
        
        (gst-launch-1.0:13326): GStreamer-CRITICAL **: 10:35:50.452: gst_structure_get_name: assertion 'structure != NULL' failed
        
        (gst-launch-1.0:13326): GStreamer-CRITICAL **: 10:35:50.452: gst_structure_get_value: assertion 'structure != NULL' failed
        
        (gst-launch-1.0:13326): GStreamer-CRITICAL **: 10:35:50.452: gst_structure_get_string: assertion 'structure != NULL' failed
        Caught SIGSEGV
        exec gdb failed: No such file or directory
        Spinning. Please run 'gdb gst-launch-1.0 13326' to continue debugging, Ctrl-C to quit, or Ctrl-\ to dump core.
        
        Reply to this comment
        
        Reply to this comment
        
        Stéphane Cerveau:
        Feb 18, 2021 at 03:21 PM
        
        Hello,
        
        Interesting as my test was using a filesrc and not the dvbsrc.
        
        Could you please open an issue on https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/ where we could track the problem and try to address it (you can put @dabrain34 in cc) ?
        
        Could you please provide the full backtrace in the issue in order to see where it could fail ?
        
        All the critical logs can also be debugged using the environment variable G_DEBUG=fatal-warnings
        
        Regards.
        
        Stéphane
        
        Reply to this comment
        
        Reply to this comment
vandana:
Feb 22, 2021 at 09:29 AM

Have posted my issue;
https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/issues/1534
Didnt find any way to do a cc, hence giving the ticket link here.
Thanks for all your guidance.

Reply to this comment

Reply to this comment
vandana:
Mar 05, 2021 at 05:59 AM

Stephane, am back again. I just cannot arrive at the correct pipeline which would read from dvbsrc, transcode using x264enc and avenc_aac and output both audio and video streams in the dashsink. Could you pls suggest, whats wrong with this one:
gst-launch-1.0 dashsink name=dashsink mpd-baseurl=http://localhost/media/kllywq84 mpd-root-path=/var/www/localhost/media/kllywq84 mpd-filename=live.mpd target-duration=5 min-buffer-time=10 minimum-update-period=10 dynamic=true muxer=ts dvbsrc modulation=5 adapter=0 frequency=147000000 delsys=dvb-c-b ! tsdemux name=demux demux. ! queue ! audioresample ! audioconvert ! avenc_aac bitrate = 128000 demux. ! queue ! x264enc bitrate=1200 key-int-max=60 ! video/x-h264,stream-format=byte-stream,profile=main ! dashsink.

The last one which worked for me, and which you suggested, doesnt have audio, and no transcoding.
gst-launch-1.0 --gst-debug=3 dashsink name=dashsink mpd-baseurl=http://50.239.254.201/media/kllywq84/ mpd-root-path=/var/www/vividcoredash/media/kllywq84/ mpd-filename=live.mpd target-duration=60 dynamic=false period-duration=60000 muxer=ts dvbsrc modulation=5 adapter=0 frequency=165000000 delsys=dvb-c-b ! queue ! tsdemux ! mpegvideoparse ! dashsink.video_0

Reply to this comment

Reply to this comment
1. Stéphane Cerveau:
  Mar 05, 2021 at 02:18 PM
  
  Hello Vandana,
  
  As far as I understand you issue, you should first decode your branches to then encode to another format. Moreover, as I recommend you on the GStreamer mailing list, you have to give a name to your dashsink pad (dashsink.video_0 or dashsink_audio_0) according to the media type branch your are passing.
  
  I am sorry to hear you are having other issues with the dashsink.
  Please continue sharing feedback directly on the GStreamer mailing list or on Gitlab for issues where others can help as well.
  
  Otherwise, you can also get in touch with us to look at setting up a proper support contract between your company and Collabora.
  
  Best regards.
  
  Stéphane.
  
  Reply to this comment
  
  Reply to this comment
vandana:
Mar 09, 2021 at 07:04 AM

Thanks for the tip Stephane, added a decodebin and the transcding pipeline for video worked.

Reply to this comment

Reply to this comment
Gregoire Gentil:
Aug 21, 2022 at 04:46 AM

I have a transfer of a live video stream from an embedded device to the javascript function of a client browser:

enbedded device (gstreamer x264enc-hardware ! whatever-I-want ! appsink)

=== transfer of data stream with a proprietary protocol ===>

HTML5 browser (javascript function receiving data sent by the appsink)

In other words, I'm trying to display a h264 live stream created on an embedded device with a proprietary transfer protocol, the data re-appearing in a javascript function inside an HTML5 browser.

I was thinking of using MediaSource in the browser to decode h264 and display the image.

The video stream settings (video only, resolution, bandwidth) are fixed and known on both sides. So, everything can be hard-coded.

What could I use on the embedded device (replacement of the "whatever-I-want" gstreamer plugin) so that the work in the HTML5 browser is not too complicated.

One solution would be to use the broadway.js library that can decode h264 in javascript and do nothing on the embedded device side but it obviously doesn't leverage MediaSource of .

Could I use Gstreamer avmux_dash and hope that MediaSource can input the data?

Reply to this comment

Reply to this comment
1. Olivier Crête:
  Aug 26, 2022 at 08:06 PM
  
  The easiest way is probably to add a "cmafmux" from the Rust plugins to make CMAF fragments, and you should then be able to feed those to the MediaSource API. I'd stay away from soemthign like broadway.js as this will be much much slower than the decoder already present in the browser.
  
  If you care about latency, you may want to look into WebRTC. There is the webrtcsink element that makes it easy.
  
  Reply to this comment
  
  Reply to this comment

Add a Comment

Search the newsroom

Latest Blog Posts

Faster inference: torch.compile vs TensorRT

19/12/2024

In the world of deep learning optimization, two powerful tools stand out: torch.compile, PyTorch’s just-in-time (JIT) compiler, and NVIDIA’s…

Mesa CI and the power of pre-merge testing

08/10/2024

Having multiple developers work on pre-merge testing distributes the process and ensures that every contribution is rigorously tested before…

A shifty tale about unit testing with Maxwell, NVK's backend compiler

15/08/2024

After rigorous debugging, a new unit testing framework was added to the backend compiler for NVK. This is a walkthrough of the steps taken…

A journey towards reliable testing in the Linux Kernel

01/08/2024

We're reflecting on the steps taken as we continually seek to improve Linux kernel integration. This will include more detail about the…

Building a Board Farm for Embedded World

27/06/2024

With each board running a mainline-first Linux software stack and tested in a CI loop with the LAVA test framework, the Farm showcased Collabora's…

Smart audio filters with WirePlumber 0.5

26/06/2024

WirePlumber 0.5 arrived recently with many new and essential features including the Smart Filter Policy, enabling audio filters to automatically…

About Collabora

Whether writing a line of code or shaping a longer-term strategic software development plan, we'll help you navigate the ever-evolving world of Open Source.

한국의 국기 한국어 버전의 Collabora.com 보기