Implementing fallback in media frameworks

When our preferred datatype is not supported, we need to make sure by way of some fallback logic that audio and video are still playing .

Sometimes, a Dolby mediatype is not supported on the playback system. To provide ubiquitous service coverage across devices, we need to have alternative formats, and this lesson teaches you how to make sure that a fallback is always available and played back as a last resort.

One approach to this might be hard coding the fallback into the player using the methods we learned in the last example. However, it is more flexible and scalable to make the player code generic, and codify the fallback into a generic media asset description.

MPEG DASH has this functionality, and we will be using this particular format to demonstrate the general idea.

MPEG DASH provides a standardized description of media presentation, that is, the experience available from continuous media content, in a so-called manifest file. The format of this file is formally specified in ISO/IEC 23009-1, but this web page provides a more gentle introduction.

The manifest file specifies the location of (segmented) media assets, and is able to provide alternative assets (different languages, different bit rates, or different codecs) with multiple so-called Adaptation Sets. (Within each AdaptationSet, one or more Representations are present to allow each client to adapt to its respective network condition. We will not make use of this feature in this tutorial.)

For our example, each AdaptationSet is targeted at a specific codec.

Figure 1. Composition of a manifest file with fallback

In Querying mediatype support we learned how the player can find out whether the mediatype is supported (and therefore the AdaptationSet can be played). So, to play back MPEG DASH from a manifest file, the player interprets the manifest, filters out all AdaptationSets that use unsupported mediatypes, and picks one of the remaining AdaptationSets.

The manifest file is XML, and interpreting the file is not straightforward. There are a number of libraries that deal with this task. We will take a look at a particular example, dash.js.

dash.js is a full fledged media player – it does everything we talked about so far, plus it also takes care of selecting the right mediatype, downloading assets in real-time, adapting the bit rate to the connection speed and feeding the downloaded media segements to the Media Source Extension (MSE) API. If we use dash.js, the task becomes one of designing the manifest file such that dash.js can do its job.

Below is a manifest file that has four different AdaptationSets. The first is an adaptation set containing video; the last three are all using different codecs (MPEG AAC for the first, Dolby Digital Plus for the second and Dolby AC-4 for the last).

Figure 2. A manifest file with three audio adaptation sets
<?xml version="1.0" ?>
<MPD xmlns="urn:mpeg:dash:schema:mpd:2011" profiles="urn:mpeg:dash:profile:isoff-on-demand:2011" minBufferTime="PT5.00S" mediaPresentationDuration="PT30.040S" type="static">
  <!-- Created with Bento4 mp4-dash.py, VERSION=2.0.0-637 -->
  <Period>
    <!-- Video -->
    <AdaptationSet id="1" mimeType="video/mp4" segmentAlignment="true" startWithSAP="1" maxWidth="1920" maxHeight="1080">
      <Label lang="en">Video</Label>
      <Role schemeIdUri="urn:mpeg:dash:role:2011" value="main"/>
      <SegmentTemplate timescale="1000" duration="5000" initialization="$RepresentationID$/init.mp4" media="$RepresentationID$/seg-$Number$.m4s" startNumber="1"/>
      <Representation id="video/avc1" codecs="avc1.4D4029" width="1920" height="1080" scanType="progressive" frameRate="25" bandwidth="638625"/>
    </AdaptationSet>
    <!-- Audio -->
    <AdaptationSet id="2" mimeType="audio/mp4" startWithSAP="1" segmentAlignment="true" lang="en" selectionPriority="97">
      <Label lang="en">AAC - Stereo - English</Label>
      <Role schemeIdUri="urn:mpeg:dash:role:2011" value="main"/>
      <SegmentTemplate timescale="1000" duration="5000" initialization="$RepresentationID$/init.mp4" media="$RepresentationID$/seg-$Number$.m4s" startNumber="1"/>
      <Representation id="audio/en/mp4a/1" codecs="mp4a.40.2" bandwidth="63929" audioSamplingRate="48000">
        <AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/>
      </Representation>
    </AdaptationSet>
    <AdaptationSet id="3" mimeType="audio/mp4" startWithSAP="1" segmentAlignment="true" lang="en" selectionPriority="98">
      <Label lang="en">Dolby Digital Plus - 5.1 - English</Label>
      <Role schemeIdUri="urn:mpeg:dash:role:2011" value="main"/>
      <SegmentTemplate timescale="1000" duration="5000" initialization="$RepresentationID$/init.mp4" media="$RepresentationID$/seg-$Number$.m4s" startNumber="1"/>
      <Representation id="audio/en/ec-3/1" codecs="ec-3" bandwidth="385179" audioSamplingRate="48000">
        <AudioChannelConfiguration schemeIdUri="urn:mpeg:mpegB:cicp:ChannelConfiguration" value="6"/>
      </Representation>
    </AdaptationSet>
    <AdaptationSet id="5" mimeType="audio/mp4" startWithSAP="1" segmentAlignment="true" lang="en" selectionPriority="100">
      <Label lang="en">Dolby AC-4 - 5.1 - English</Label>
      <Role schemeIdUri="urn:mpeg:dash:role:2011" value="main"/>
      <SegmentTemplate timescale="1000" duration="5000" initialization="$RepresentationID$/init.mp4" media="$RepresentationID$/seg-$Number$.m4s" startNumber="1"/>
      <Representation id="audio/und/ac-4/1" codecs="ac-4.02.01.01" bandwidth="193766" audioSamplingRate="48000">
        <AudioChannelConfiguration schemeIdUri="urn:mpeg:mpegB:cicp:ChannelConfiguration" value="6"/>
      </Representation>
    </AdaptationSet>
  </Period>
</MPD>
  1. mimeType indicates that all assets are packaged into ISO base media files. It has the same value "audio/mp4" across all AdaptationSets.
  2. codecs signals the mediatype. It has a different value in each AdaptationSet. The player needs to construct the MIME type mentioned in Querying mediatype support by concatenating the mimeType and codecs attributes, and use this to check with the user agent whether the type is supported. dash.js does this for you, under the hood.

With this, we start a new project with one file in your development environment:

Figure 3. Using dash.js to play back MPEG DASH. When you start the example, you should hear audio no matter what device you are on. On some devices, you may hear the fallback audio, and on others, the Dolby audio. The voice tells you the selected audio format.
Tip: Start the example or view the full source code
<h1>Web App: Implementing fallback in DASH.js</h1>
<p>Expect to hear audio; the voice will tell you the selected audio format.</p>

<video data-dashjs-player="" autoplay src="media/v01/dash/lesson_4.mpd" controls>This device does not support the &lt;video&gt; element</video>

<script src="http://cdn.dashjs.org/latest/dash.all.debug.js"></script>
  1. We are using the HTML video element, a sibling of the audio element that we used in Playing Dolby Audio.
  2. We are calling dash.js in what is called the factory mode. In factory mode, dash.js finds the video element in your DOM and replaces the functionality provided by the native media player.

In subsequent lessons, we will learn how to extend and customize dash.js, for example to pick immersive audio only for systems that are equipped for it.