HDS & Bootstrap Data

inspect-hds-bootstrap

Working with HDS Bootstrap Data

I’ve always been curious about the bootstrap data for HDS content. Recently, I had the chance to find out more about it and get in some fun development with Node.js. We’ve been kicking around the idea of building a tool set for Adobe Media Server using Node.js and possibly socket.io. Last weekend we got some of the ideas going and one of those was parsing the hds bootstrap data created when content is packages for HDS delivery.

The bootstrap data can live in a couple of places:

  1. In the <metadata> node of an F4M file
  2. In an external .bootstrap file

The .bootstap file contains binary data and the F4M file contains the same binary data that has been Base64 encoded. So, getting to the data is pretty trivial – either read in the .bootstrap file or un-encode the string in that is in the F4M. Getting to the data contained in the bootstrap binary data is the fun part.

Understanding the bootstrap data

To do so, check out the F4V file format specification. This PDF gives you the details for the entire F4V file format. If you read through the PDF, you’ll  see that it is built using what are called “boxes”. These boxes are given identifiers such as “abst”,  “adaf”,  “adkm”,  “aeib”,  “afra”,  & “afrt” to name a few. Each box contains a header, that header identifies the box by its identifier and lets you know how much data is contained in the box. These boxes are also arranged into a hierarchy, so each box has some data that is specific to some part of the data contained in the file.

It is all in the boxes

The boxes that we are concerned with are “abst” or the bootstrap information box, “asrt” or the segement run table box, and “afrt” or the fragment run table box.

The abst box

The bootstrap information box contains information needed to bootstrap playing of HDS content – specifically to construct the URLs necessary to retrieve the fragments for playback. This includes information about the server, media, & segment information.

The asrt box

The segment run table box contains data about the segments for the media item. There can be multiple ASRT boxes – each representing a different quality level. There are some rules that you’ll want to pay attention to for the data in the asrt box:

  • An asrt box can represent fragment runs for several quality levels.
  • Each entry gives the first segment number for a run of segments with the same count of fragments.
    • The count of segments having this same count of fragments can be calculated by subtracting the first segment number in this entry from the first segment number in the next entry.

The afrt box

The fragment run table box is used to find the fragment corresponding to a given time. Similar to the asrt box, there are some rules that you’ll want to pay attention to:

  • Fragments are individually identifiable by the URL scheme based on segment number and fragments number.
  • Fragments may vary both in duration and in number of samples.
  • Duration of the fragments are stored in the this box.
  • A Fragment Run Table may represent fragments for more than one quality level.
  • Each fragment run table entry gives the first fragment number for a run of fragments with the same duration.
    • The count of fragments having this same duration can be calculated by subtracting the first fragment number in this entry from the first fragment number in the next entry.

Parsing the bootstrap data using Node.js

Parsing binary data in Node.js can be done using “Buffer”. For the most part parsing the bootstrap data was pretty straight forward. There is one issue that I ran into with 64bit Integers which was solved easily enough (there are node modules for just about anything) using the node-int64 module to represent the 64Bit Integers. Once that was solved it was just a matter of parsing through the box header to figure out where you are in the dataset, and then creating the appropriate data structures to represent what you want and need in from the bootstrap data.

In our case we want to be able to monitor live events across multiple servers to make sure that they are all on the same segment and fragment. We’re building a services that in the case that something happens to a server and it goes haywire, will notify another service that can then restart or shut down that particular server or let caching servers know that they need to flush or refresh cache. We’re still dreaming up things we can use this type of data for.

Just want to get to that data?

If you have a .bootstrap file you can use the f4fpackager.exe that is part of the Adobe Media Server toolset to inspect the bootstrap data. All you need to do is run the tool with the argument “–inspect-bootstrap”. So the command looks something like the following if you have a bootstrap file named mydata.bootstrap:

[shell].f4fpackager.exe –input-file=mydata.bootstrap –inspect-bootstrap[/shell]

Anyways, if you have any questions or input let me know in the comments.

14 Replies to “HDS & Bootstrap Data”

  1. Hey John, your posts are a great help in understanding the internals of HDS and how it works – many thanks for putting them up. I have just started getting into HDS and learning about the file, metadata and transport formats (I just finished writing a basic ISO BMFF decoder so I can see what the MP4 looks like before and after F4F packaging).

    Sorry if this is somewhat off-topic to this particular post, but one thing that sticks in my mind is that Flash cannot do HTTP byte range requests (without a custom HTTP client implementation using sockets, and even then requiring a socket policy server), and yet it is the default option to create a single segment F4F file (which surely would necessitate byte range requests for streaming). All the bootstrap data in the world can’t help Flash get around this HTTP request limitation, do you know what mechanism it uses?

    1. In short, the bootstrap data isn’t for byte rage requests. The bootstrap data contains information necessary to request fragment for a timecode from the server. Using the data in the bootstrap you can build a table of URLs that map to time codes. Then you can request a fragment using the URL from the server. THe HTTP Origin Module (an Apache module) is responsible for identifying and returning the fragment.

      FYI, you can see the client-side of things at work in the org.osmf.net.httpstreaming.f4f package of the OSMF framework.

      1. Hi John, thanks for your reply back! I did some reading of the OSMF source over the weekend and found the time-period/byte-range/fragment translation code which then needs to be decoded by the Origin Module. It’s a bit sad that Adobe couldn’t build native HDS support into Flash and do away with this custom mechanism to get around Flash’s inability to do byte-range requests, but there you go.

        Now at least I understand how it works. Cheers!

  2. Hi John, useful post, thanks! I’m playing with using the alternate media streams in a manifest at the moment, OSMF specifically excludes anything other than audio or video types, but I’m wondering if there’s a clean way of muxing in alternate streams with a type of ‘text’ – so subtitles can be carried in a separate stream, for instance. I’ve gotten so far with updating OSMF to cope with this, but now I’m stuck at the nitty gritty – I can get the text-stream bootstrap, but then…nothing…! I’m trying to mirror the way LBA works..got any ideas?

  3. Hey John, your posts are a great help in understanding the internals of HDS and how it works – many thanks for putting them up. I have just started getting into HDS and learning about the file, metadata and transport formats (I just finished writing a basic ISO BMFF decoder so I can see what the MP4 looks like before and after F4F packaging).

    Sorry if this is somewhat off-topic to this particular post, but one thing that sticks in my mind is that Flash cannot do HTTP byte range requests (without a custom HTTP client implementation using sockets, and even then requiring a socket policy server), and yet it is the default option to create a single segment F4F file (which surely would necessitate byte range requests for streaming). All the bootstrap data in the world can’t help Flash get around this HTTP request limitation, do you know what mechanism it uses?

    1. In short, the bootstrap data isn’t for byte rage requests. The bootstrap data contains information necessary to request fragment for a timecode from the server. Using the data in the bootstrap you can build a table of URLs that map to time codes. Then you can request a fragment using the URL from the server. THe HTTP Origin Module (an Apache module) is responsible for identifying and returning the fragment.

      FYI, you can see the client-side of things at work in the org.osmf.net.httpstreaming.f4f package of the OSMF framework.

      1. Hi John, thanks for your reply back! I did some reading of the OSMF source over the weekend and found the time-period/byte-range/fragment translation code which then needs to be decoded by the Origin Module. It’s a bit sad that Adobe couldn’t build native HDS support into Flash and do away with this custom mechanism to get around Flash’s inability to do byte-range requests, but there you go.

        Now at least I understand how it works. Cheers!

  4. Hi John, useful post, thanks! I’m playing with using the alternate media streams in a manifest at the moment, OSMF specifically excludes anything other than audio or video types, but I’m wondering if there’s a clean way of muxing in alternate streams with a type of ‘text’ – so subtitles can be carried in a separate stream, for instance. I’ve gotten so far with updating OSMF to cope with this, but now I’m stuck at the nitty gritty – I can get the text-stream bootstrap, but then…nothing…! I’m trying to mirror the way LBA works..got any ideas?

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.