EXT-X-DATERANGE metadata synchronisation vs video stream in presence of frequent EXT-X-DISCONTINUITYs

[lnstadrum]

EXT-X-DATERANGE metadata synchronisation vs video stream in presence of frequent EXT-X-DISCONTINUITYs

[lnstadrum]

Hi @robwalch,

Thanks you, the test branch works as expected on our test stream.

We will indeed have to find a solution for Safari. There actually is a silent audio track in our test stream (as we do have audio in our streams in production), and audio and video tracks do not end at the same moment prior to discontinuity, or do not start at the same moment right after, due to the way our compiled stream is built.

I guess this issue can now be closed. Thanks a lot again for your help!

[robwalch]

In HLS.js, DATERANGE tags are mapped to cues on the TextTrack timeline using playlist time (EXTINF durations). The drift occurs when media parsed and the parsed media duration differs from the duration in the playlist without the total program duration matching up once summed up. This is the case in your sample. The demo page "Timeline" tab shows the parsed segment duration slightly over 1.02s each vs the #EXTINF:1.001000 found in the playlist. This is not a result of DISCONTINUITY tags, but the playlist segment durations each being less than the corresponding parsed segment durations. Generally, HLS.js determines parsed segment duration as the difference between the starting video timestamp of a segment to the starting timestamp of the next.

You can access DateRange data directly in HLS.js using hls.levels[hls.currentLevel].details?.dateRanges. This is a map (Object) of all parsed DateRanges by ID. It's a much more complete and up-to-date collection of logical DateRanges - valid tags with the same ID are merged, and all attributes are available on the object. In v1.6 (with #6213) DateRanges will have a "tag anchor" that reference their adjacent fragment in the playlist so that their start time is always mapped to the PDT and discontinuity domain at that segment position on the playback timeline. The LEVEL_PTS_UPDATED event signals that segment times were updated based on parsed media timestamps and can be used to update app logic.

A fix for this metadata TextTrack specific issue still requires cue timing to be updated after media is parsed (on LEVEL_PTS_UPDATED) in the id3-track-controller. We can look into a fix for this in the next release. Even after the update I would recommend using the aforementioned LevelDetails dateRanges. Using cues in necessary is Safari HLS playback, but produces a cue for every attribute, is missing ID, and does not merge DateRange tags with the same ID.

[robwalch]

Marking as enhancement. This is not a regression.

DateRange TextTrack cues are mapped to playlist time not the video track or parsed media, as of v1.5.x. When the playlist times differ this much from the parsed media (on every segment, not just discontinuity) I think it is fair to consider this a stream issue. To support Interstitials in v1.6, precise mapping of DateRanges to the playback timeline is crucial, so we can afford to compensate for these kind of discrepancies, and explore sliding cue start times (if we cannot adjust them after creating then we'll need to remove and add cues) at a usable interval.

[robwalch]

@lnstadrum,

Using cues in necessary in Safari HLS playback, but produces a cue for every attribute, is missing ID, and does not merge DateRange tags with the same ID.

FYI - The DateRanges in your playlist are invalid and while this should result in them being ignored, Apple HLS clients error and will not play the sample provided. ID and date attributes are expected to be provided as quoted-string values (...bonksound.studio/hls/playlist.m3u8 is missing quotes around ID and START-DATE values and errors rather than plays in Safari and Apple HLS clients).

Ex:

#EXT-X-DATERANGE:ID=4.16016,START-DATE=2024-02-05T00:00:16.016000,DURATION=4.004,X-CUE=" "

should be:

#EXT-X-DATERANGE:ID="4.16016",START-DATE="2024-02-05T00:00:16.016000",DURATION=4.004,X-CUE=" "

HLS.js is not strict about quoted-string format attribute values missing quotes. In #6213 I've added some validation logic that logs warning in to the console when missing quotes are encountered.

[lnstadrum]

Hi @robwalch,

Thank you for looking into this.

• In our application, we build a compilation of videos consisting of parts of existing HLS streams. A "part" is a consecutive subset of segments of another stream (which does not necessarily include the last segment of that stream). The original streams are produced by ffmpeg (so are EXTINF values). To build the compilation we simply copy the segments with their corresponding durations without properly adjusting duration of segments which fall before EXT-DISCONTINUITYs in the compiled stream. So I guess it is rather an issue with our stream indeed. On the other hand, it would be very convenient for us to avoid querying and storing the "accurate" duration (not sure if this is a proper term) of every segment which may potentially be the last one in the corresponding part of the compilation, as far as it makes the process of building the compilation playlist and the logistics of storing the original streams in the DB more complex, and we actually do not know in advance which segment are concerned (the compilations are built on the fly).
• Thank you for pointing the quotes issue. I will fix this in the test stream. I haven't had yet the chance to test playback using the native iOS implementation, but will soon do.

[robwalch]

@Instadrum,

I just added 0c5ca21 to #6213 which updates cue start and end times on PTS update. Let me know if this works for you:

https://feature-date-range-parsing.hls-js-4zn.pages.dev/demo/?src=https%3A%2F%2Fbonksound.studio%2Fhls%2Fplaylist.m3u8

Just confirmed that Safari does not shift cues to align with media this way, so you'll want to make sure your Playlist EXTINF durations align with presentation time in Apple HLS clients (or relative to Safari's HTMLMediaElement.getStartDate()). Providing unmuxed ISO BMFF segmented HLS Playlists may help as then your main playlist has only video segments and there is no start offset when segmented based on audio time or whatever track starts or ends first or last.

I don't think it hurts for HLS.js to make these adjustments as it may push the timeline out based on audio priming delays, overlaps not allowed in MSE, and other presentation time oddities the library has picked up on the way. This change will ensure we keep cues aligned with the libraries interface for DateRange and Program-Date-Time mapping and hls.playingDate.