[Request] Document Playback Flow

[jellyfin-archive]

[Request] Document Playback Flow

[Artiume]

I'm not sure how JF determines the client's network. Someone else might be able to fill in for that. The client's type will be parsed; based on this information. StreamBuilder will determine the type of playback. S.B. will consider certain things such as bitrate limit and so on. I'll see what I can do to make a more complete map

[anthonylavado]

The FFmpeg commands are here: https://github.com/jellyfin/jellyfin/blob/8ff07e17e6f58a1102b9d8c36a995dec4dc2f041/MediaBrowser.Controller/MediaEncoding/EncodingHelper.cs

[anthonylavado]

Local network detection is pretty rudimentary. Look at the section starting at line 110:
https://github.com/jellyfin/jellyfin/blob/master/Emby.Server.Implementations/Networking/NetworkManager.cs#L110-L156

[anthonylavado]

There’s also an endpoint in the server that’s used to generate random data of a requested size, which is used to calculate connection speed, which factors in to the transcode bitrate calculation. If the speed is high enough to support almost anything, then client considerations come in to play (e.g. Chromecast is set to a max of 10 Mbps as part of a first gen limitation)

[anthonylavado]

Web Interface playback support is determined with BrowserDeviceProfile:
jellyfin/jellyfin-web#703

Other clients are expected to know what they support and can use the data endpoint to do a connection speed test. After that, the client is expected to make a request listing its capabilities and desired bitrate. Of course if the media codecs and player support match up, direct play/stream will happen.

[anthonylavado]

DLNA clients have profiles in the server code, which is used to list their capabilities. The server gets the “product name” when a DLNA request comes in, and then matches it with a profile to know what to send back. The server already includes a bunch of popular ones, but it can always be customized by the end user.

[Artiume]

Local network detection is pretty rudimentary. Look at the section starting at line 110:
https://github.com/jellyfin/jellyfin/blob/master/Emby.Server.Implementations/Networking/NetworkManager.cs#L110-L156

Should this go before L110?
https://github.com/jellyfin/jellyfin/blob/a3615dec693ce8f9b0a803d98e9ba0b1f32f62a7/Emby.Server.Implementations/Networking/NetworkManager.cs#L158-L178
Not sure what the difference in between public bool IsInPrivateAddressSpaceAndLocalSubnet(string endpoint) and if (checkSubnets && IsInPrivateAddressSpaceAndLocalSubnet(endpoint)) for the variable endpoint. But to me, it looks like L150 might return false since the value of L158 hasn't run yet. Or does it not matter?

DLNA clients have profiles in the server code, which is used to list their capabilities. The server gets the “product name” when a DLNA request comes in, and then matches it with a profile to know what to send back. The server already includes a bunch of popular ones, but it can always be customized by the end user.

I believe @PrplHaz4 once showed me a url that has a generic list of DLNA profiles. We could do a review and add additional clients based on the list.

I want to say these are the DLNA profiles currently available. I'm not sure how Jellyfin Clients report their playback compatibility though.

https://github.com/jellyfin/jellyfin/tree/master/Emby.Dlna/Profiles

Extra DLNA Notes:
https://github.com/rivarolle/plex-dlna-profiles

uPnP anyone? Haha
https://openconnectivity.org/certification/upnp-certification/

[anthonylavado]

So speed detection happens at this endpoint:

https://github.com/jellyfin/jellyfin/blob/b3811a9498fe06b68693f4a269de902cdd7eb2a2/MediaBrowser.Api/Playback/MediaInfoService.cs#L60-L71

The actual code for it is here:
https://github.com/jellyfin/jellyfin/blob/b3811a9498fe06b68693f4a269de902cdd7eb2a2/MediaBrowser.Api/Playback/MediaInfoService.cs#L106-L132

Recently fixed for a good reason: jellyfin/jellyfin#2167

To see where it is used in clients:
https://github.com/search?q=org%3Ajellyfin+bitratetest&type=Code

[Artiume]

Totally didn't know github supported code search, I've just been using grep -inr this entire time haha.

I'm gonna try and break down StreamBuilder

Server request client information
https://github.com/jellyfin/jellyfin/blob/b3811a9498fe06b68693f4a269de902cdd7eb2a2/MediaBrowser.Api/Playback/MediaInfoService.cs#L192-L253

Server determining Streaming Logic.
https://github.com/jellyfin/jellyfin/blob/b3811a9498fe06b68693f4a269de902cdd7eb2a2/MediaBrowser.Api/Playback/MediaInfoService.cs#L319-L415

First determine Direct Play
https://github.com/jellyfin/jellyfin/blob/b3811a9498fe06b68693f4a269de902cdd7eb2a2/MediaBrowser.Api/Playback/MediaInfoService.cs#L417-L463

Direct Stream
https://github.com/jellyfin/jellyfin/blob/b3811a9498fe06b68693f4a269de902cdd7eb2a2/MediaBrowser.Api/Playback/MediaInfoService.cs#L465-L498

Transcode
https://github.com/jellyfin/jellyfin/blob/b3811a9498fe06b68693f4a269de902cdd7eb2a2/MediaBrowser.Api/Playback/MediaInfoService.cs#L500-L535

Local or Remote Check
https://github.com/jellyfin/jellyfin/blob/b3811a9498fe06b68693f4a269de902cdd7eb2a2/MediaBrowser.Api/Playback/MediaInfoService.cs#L548-L570

Not really sure how to read the rest of it.

So I want to add a wrapper around the streambuilder to do a logic check where if the current type of mode fails for whatever reason, it'll go to the next level.

Direct Play > Direct Stream > Transcode (If HWA was used, then try Software, if Software fails, either exit or play stream without failed codec). I think it would need to go between L415 and end around L535?

[Artiume]

https://github.com/jellyfin/jellyfin-web/blob/b273853f0c160ade001a08d215bc460440cfba98/src/components/htmlvideoplayer/plugin.js#L887-L896

If a container has an invalid video codec, we could provide better error handling

[Artiume]

https://github.com/jellyfin/jellyfin/blob/1ae9ed6e2aefb323f9959f9ed7a0c7950dd630c6/MediaBrowser.Api/Playback/BaseStreamingService.cs#L271-L282

This looks interesting. Looks like this is where FFmpeg beings and exits. We might be able to catch it here to reattempt.

[Artiume]

https://github.com/jellyfin/jellyfin/blob/1ae9ed6e2aefb323f9959f9ed7a0c7950dd630c6/MediaBrowser.Api/Playback/BaseStreamingService.cs#L675-L845
Input of State Builder for StreamBuilder.

specifically, this reads the properties of the video codec.
https://github.com/jellyfin/jellyfin/blob/1ae9ed6e2aefb323f9959f9ed7a0c7950dd630c6/MediaBrowser.Api/Playback/BaseStreamingService.cs#L807-L832

Output of State Builder
https://github.com/jellyfin/jellyfin/blob/1ae9ed6e2aefb323f9959f9ed7a0c7950dd630c6/MediaBrowser.Api/Playback/BaseStreamingService.cs#L847-L930

DLNA stuff for State Builder
https://github.com/jellyfin/jellyfin/blob/1ae9ed6e2aefb323f9959f9ed7a0c7950dd630c6/MediaBrowser.Api/Playback/BaseStreamingService.cs#L939-L1017

[Artiume]

jellyfin/jellyfin#2390 (comment)

This shows how ffmpeg is built in the end with HWA. My bottom link is the holy grail as to how ffmpeg works.

[Artiume]

https://github.com/jellyfin/jellyfin/blob/master/MediaBrowser.Controller/MediaEncoding/EncodingHelper.cs
This is where the various flags are decided for ffmpeg.

[Artiume]

https://github.com/jellyfin/jellyfin/blob/1d0e21d19501e0d34ba33286133e00db70a649a7/MediaBrowser.Controller/MediaEncoding/EncodingHelper.cs#L2367

This is the logic to burn in dvdsubs

[Artiume]

https://github.com/jellyfin/jellyfin/blob/1d0e21d19501e0d34ba33286133e00db70a649a7/MediaBrowser.Controller/MediaEncoding/EncodingHelper.cs#L2279

This is where logic checks to see if it's a video or audio request.

This is also where we can disable video for carplay.

[Artiume]

https://github.com/jellyfin/jellyfin/blob/1d0e21d19501e0d34ba33286133e00db70a649a7/MediaBrowser.Controller/MediaEncoding/EncodingHelper.cs#L2376

This is where hardware decoding begins.

[Artiume]

Hls segment swap
jellyfin/jellyfin#1802

Keyframe sync support for exoplayer
google/ExoPlayer#2882

[Artiume]

https://github.com/jellyfin/jellyfin/pull/2251/files
PR showing how HWA encoding was updated for AMF

[Artiume]

HLS casting to chromecast
google/ExoPlayer#6482

[Artiume]

intel/intel-vaapi-driver#238
VC-1 support in Linux issue ticket

[Artiume]

https://github.com/jellyfin/jellyfin-web/blob/b377878fd355b8b4c42954ccbc95a69aa227f302/src/scripts/browserdeviceprofile.js#L359

Chromecast doesn't support ac3 and currently defaults to ac3. This code says to do aac, what's up?

[Artiume]

HLS segment muxer

https://github.com/jellyfin/jellyfin/blob/adc3ab19913e697ae0bd1fbf1cf8cda12041b71b/MediaBrowser.Api/Playback/Hls/HlsSegmentService.cs#L42