w3c/mse-byte-stream-format-isobmff

ISO BMFF Byte Stream Format for Media Source Extensions (MSE)

ISO BMFF Byte Stream Format

This repository contains the latest Editor's Draft of the ISO BMFF Byte Stream Format specification for Media Source Extensions, developed by the Media Working Group.

Note: The registry was copied over from the main Media Source Extensions repository end of November 2020. Please refer to the main repository for the history of changes made to the specification before that date.

mse-byte-stream-format-isobmff's Issues

Use of `ftyp` box

The BSF spec indicates the following about the "File Type Box" (https://www.w3.org/TR/mse-byte-stream-format-isobmff/#iso-init-segments):

An ISO BMFF initialization segment is defined in this specification as a single File Type Box (ftyp) followed by a single Movie Box (moov).
The user agent must run the append error algorithm if any of the following conditions are met:

1. A File Type Box contains a major_brand or compatible_brand that the user agent does not support.
2. A box or field in the Movie Box is encountered that violates the requirements mandated by the major_brand or one of the compatible_brands in the File Type Box.

In my testing, none of the browsers I tested (Chrome, Firefox, Safari) care about the presence or contents of the ftyp box.
Bullet 2. above would mean that browsers have a mapping (brand, requirements), which I don't think they have.
Also 1) contradicts the design of ISOBMFF which says that you can process a file if you understand at least one brand.

I think the BSF can safely be updated to match what implementations do and simply say that an Initialization Segment is a moov box, optionally preceded by boxes.

Note that the same type of sentence exists also for the styp box, and could be safely removed I think.

Handling of box size fields

Reading the BSF, I think there is a need to clarify the general box parsing behavior, specifically with respect to the size field. Let me recap.

In ISOBMFF a Box starts with:

1. a size field on 4 bytes
2. a type field on 4 bytes
3. possibly a largesize field on 8 bytes
4. possible a UUID on 16 bytes
5. possibly version and flags fields on 1+3 bytes if the type is a FullBox

The size field documents the size of the entire box including the type and size itself.

size is an integer that specifies the number of bytes in this box, including all its fields and contained boxes;

So strictly speaking the size value should not be smaller than 8 (or more if UUID and/or FullBox). However, there are 2 special cases:

• a value of 0:

if size is 0, then this box shall be in a top-level container, and be the last box in that container (typically, a file or data object delivered over a protocol), and its payload extends to the end of that container (normally only used for a MediaDataBox)

I don't think it makes sense to tolerate size-0 in MSE context.

Clarification 1: I would like to clearly document that setting the size field to 0 is an error case, i.e. "MUST trigger the append error algorithm".

• a value of 1:

if size is 1 then the actual size is in the field largesize

The simplest clarification we could do is:
Clarification 2: Setting the size field to a value greater or equal to 2 and smaller than 8 is an error.

We could change the value 8 depending on UUID and/or FullBox status if desired, but that's quite complex, see table below:

size == 1	type == 'uuid'	Is FullBox?	Minimum (large)size value (bytes)
false	false	false	8
true	false	false	16
false	true	false	24
true	true	false	32
false	false	true	12
true	false	true	20
false	true	true	28
true	true	true	36

We could also discuss if largesize should be permitted at all. Does MSE support an mdat with a size greater than 4GB?

More generally, we could simply say:
Clarification 3: If the size field of a box indicates a value less than the size of the fields known to the parser, the parser MUST run the append error algorithm.

Another important aspect is the behavior of a parser when the box contains more data than the parser understands. In other words, the parser knows how to parse X bytes but the size field indicates X+Y (Y>0). Such extensibility mechanism has been used in the past. For example, the avcC box has been extended in the past to include more data at the end of the box when an 'old' field has a known value.

Clarification 4: Parsers should discard unknown trailing bytes of a box and not trigger an error.

Finally, a box can contain other boxes so the size of the containing box is constrained.

Clarification 5: If the size of a containing box is less than the sum of the sizes (or largesizes if used) of its child box, plus its own header, it is an error.

Link to ISO/IEC-14496-12 is broken

The working draft links to those pages

• Inline in the text: https://standards.iso.org/ittf/PubliclyAvailableStandards/c061988_ISO_IEC_14496-12_2012.zip
• In normative references: http://standards.iso.org/ittf/PubliclyAvailableStandards/c068960_ISO_IEC_14496-12_2015.zip

However, both of them lead to 404 on ISO's website.

GitHub.io links returns 404

The link https://w3c.github.io/mse-isobmff-byte-stream-format/ provided in the description of the repo gives a 404. @tidoust should it be linking to the main MSE repo? or should this repo have a gh-pages branch?

Broken references in ISO BMFF Byte Stream Format

While crawling ISO BMFF Byte Stream Format, the following links to other specifications were detected as pointing to non-existing anchors:

• https://html.spec.whatwg.org/multipage/media.html#audiotrack-audiotrack
• https://html.spec.whatwg.org/multipage/media.html#videotrack-videotrack
• https://html.spec.whatwg.org/multipage/media.html#texttrack-texttrack

_{This issue was detected and reported semi-automatically by Strudy based on data collected in webref.}

Support for unknown boxes

The BSF spec says:

Valid top-level boxes such as pdin, free, and sidx are allowed to appear before the moov box. These boxes must be accepted and ignored by the user agent and are not considered part of the initialization segment in this specification.

and

Valid top-level boxes defined in ISO/IEC 14496-12 other than ftyp, moov, styp, moof, and mdat are allowed to appear between the end of an initialization segment or media segment and before the beginning of a new media segment. These boxes must be accepted and ignored by the user agent and are not considered part of the media segment in this specification.

In my testing, only Safari supports boxes like unkn. All browsers support a free box though. It should be clarified what "Valid top-level box" means. Does it mean an actual box defined somewhere, and if so where? or does it mean any box like unkn should be supported as long as it is well-formed.

Support for edit lists

The BSF spec indicates that implementations must support one type of edit list (called "offset edit list" in CMAF) but is silent about other types. Amongst other types we have:

1. use of fractional rate,
2. use of zero rate (i.e. dwell),
3. use of rate = 2.
4. use of 'empty' edits (media time -1)
5. use of multiple entries

In my testing, there does not seem to be interoperability. Most of the time browsers ignore the edit lists they don't support. It'd probably be better to fire an error.

Number and order of boxes

BSF puts constraints on some boxes (e.g. mandatory presence), but in some cases, in my testing, the browsers I tested (Safari, Chrome, Firefox) won't fail if the box is absent (e.g. tfdt). In other cases (e.g. absence of mvex), they will fail but not in the same way (error thrown upon init segment append vs upon media segment append).

Also, the BSF spec does not put any constraints on the order of non-top-level boxes, but some browsers behave incorrectly in some cases (e.g. mvex first, then mvhd, then trak).

It'd be good to be clearer about the number and order of boxes that are permitted, possibly following or referencing the CMAF specification.