Timed Metadata for HTTP Live Streaming Support

[video-dev]

Timed Metadata for HTTP Live Streaming Support

[dstieglitz]

Is it possible to get PTS from the underlying stream through the API?

[mangui]

@nireshkumar regarding https://developer.apple.com/library/ios/documentation/AudioVideo/Conceptual/HTTP_Live_Streaming_Metadata_Spec/HTTP_Live_Streaming_Metadata_Spec.pdf, extracting ID3 PES is not a big deal. they could eventually be parsed and returned along with their PTS through a specific event.
currently we have FRAG_PARSING_DATA that handles Audio/Video,
we could add a FRAG_PARSING_METADATA event, that would return an array of ID3 PES, along with their PTS

[mangui]

then it would be up to your application to listen to this event and handle those metadatas (parse, and display them )

[dstieglitz]

@mangui is the PTS available on a frame-by-frame basis (for video) even if there no ID3 metadata? For example, is it possible to implement a video.currentTimePTS() call or similar?

[mangui]

PTS reported by hls.js events are always relative to start position, and expressed in ms.
basically you will always have video.currentPTS = 1000*video.currentTime
so that it will be straightforward to sync metadata and video.

[dstieglitz]

Is it possible to obtain the PTS from the decoding (the PTS written by the encoder, for example)?

[mangui]

the real PTS is parsed, but not exposed. just curious, why would you need it ?

[dstieglitz]

One of our use cases involves tagging live-DVR streams. Relative PTS has no meaning in this scenario, however, typically the CDN will cycle the encoding PTS every 24 hours so we'd like to experiment using the frame PTS plus date data to get frame accurate tagging.

[mangui]

I see, indeed it make sense in that case.
anyway the values are there, it is just a matter of exposing them.

[dstieglitz]

Can you advise on how this might be accomplished? I'd like to give it a go.

[mangui]

@dstieglitz, please find guidelines below :

• retrieve ID3 PID from PMT ( stream type : 0x15) : add a case 0x15 here https://github.com/dailymotion/hls.js/blob/master/src/demux/tsdemuxer.js#L179
• parse/retrieve ID3 PES packet : add a else if(pid === id3id) and do similar processing than for audio or video (refer to https://github.com/dailymotion/hls.js/blob/master/src/demux/tsdemuxer.js#L88-L99 for audio processing)
• add a new method this._parseID3PES() that will just push the parsed PES in an array. pes.pts is the absolute PTS value.
• add a new method flushID3Samples() that should be called from end() : https://github.com/dailymotion/hls.js/blob/master/src/demux/tsdemuxer.js#L126-L141
• in this method, fire a new event FRAG_PARSING_METADATA, which data should contain your array of ID3 packets
• add some logic in demuxer.js and tsdemuxerworker.js so that this event could work with and without webworker

[mangui]

also, if you just want to expose absolute PTS, you need to play with
this._initPTSand this._initDTS
https://github.com/dailymotion/hls.js/blob/master/src/demux/tsdemuxer.js#L824-L828

[dstieglitz]

@mangui thanks!

[nireshkumar]

@mangui Thanks for the reply. Guidelines for implementation looks great.

[dstieglitz]

Is there a test manifest for a live-dvr playlist?

[mangui]

http://dailymotion.github.io/hls.js/demo/?src=http%3A%2F%2Fonline.video.rbc.ru%2Fonline%2Frbctv.m3u8

on Chrome with https://chrome.google.com/webstore/detail/allow-control-allow-origi/nlfbmbojpeacfghkpbjhddihlkkiljbi

[nireshkumar]

@mangui, Our live encoder inject ID3 tags into stream every second. We used to listen to flashls's ‘HLSEvent.ID3_UPDATED’ event and consume ID3 tag data, its working fine. When I added 'case 0x15' here https://github.com/dailymotion/hls.js/blob/master/src/demux/tsdemuxer.js#L179 and played the same stream, ID3 tag id detected only once at the beginning stream or on bitrate switch. Am I missing something ? is it possible replicate the same behaviour as in flashhls in hls.js as well?

[mangui]

ID3 tags should be parsed and dispatched through an event; but it is not hls.js responsability to dispatch individual event at the right timings. it is up to the application to do it.
the difference is that flash allows to inject custom metadata in the NetStream. NetStream will trigger an event with each individual tag at the right timings. but MediaSource does not support this.

[nireshkumar]

Got it. Thanks
Any pointers on how I can extract injected ID3 Tags ?

[mangui]

for tags extraction from tsdemuxer, plz refer to my yesterday's comment.
for id3 tag parsing, i guess there should be JS libs available around.

[nireshkumar]

@mangui I followed the implementation guidelines and its working. As well added FRAG_INIT_SEGMENT_TS_CHANGED event which fires when _initPTS and _initDTS gets set, this exposes absolute PTS.
Here is the change set link : nireshkumar@d8617bc

I would be glad to raise a PR if you want to add metadata parsing feature. Let me know if I need to make more changes to make it PR ready.

[mangui]

@nireshkumar nice !
regarding absolute timestamps, do you need them only for ID3 ?
you should already have absolute PTS/DTS in your ID3 PES packets, dispatched by FRAG_PARSING_METADATA

I would suggest to:
either convert all timestamps to relative ones, and add initPTS/initDTS in FRAG_PARSING_INIT_SEGMENT, no need to create a new event

[mgw-sbex]

Hey, could you give me a call to discuss syncing multiple HLS streams together ? 805.499.1994