Is TextTrackRenderer fully implemented?

[anotherhale]

Is TextTrackRenderer fully implemented?

[anotherhale]

Is this the best forum to ask questions regarding Exoplayer? They are not specifically bugs or troubleshooting issues.

Here are a few I have so far:

1. Is Exoplayer production ready? Is see that it does have a 1.0+ tag version. I also do not see any disclaimers in the readme stating that it is not.
2. Can Exoplayer support movie rentals/purchases from Youtube? Amazon? others?

Thanks,
Andy

[ojw28]

1. TextTrackRenderer is fully implemented. It's used by SmoothStreamingRendererBuilder in the demo app to render TTML subtitles, although the demo app doesn't contain a sample video that actually has TTML subtitles, because I couldn't find one that was freely available.
2. Production readiness is ultimately something you need to decide for yourself. We're using it in production at Google in our YouTube, Play Movies and Fiber apps for DASH playbacks, so we consider it production ready for this use case. We believe the SmoothStreaming use case to be more or less production ready too.
3. Play Movies is using ExoPlayer for movie and episode playbacks. I'm not really sure what your question was though.

[anotherhale]

I am trying to open a ttml file from an external source (it is not bundled in the source container). I have copied the ttml file to the appropriate asset folder in my android project:

//text Uri is an android asset:
Uri textUri = "file:///android_asset/subtitles.en.ttml";

I have modified DefaultRendererBuilder.java to try to support this (https://gist.github.com/anotherhale/4739df1b6545b5067d76), but am getting a MediaExtractor error "Failed to instantiate extractor".

Do you have any suggestions on how I would open a ttml file not bundled in the source container?

[anotherhale]

Sorry I am new to media development and am still learning. I realized that this is not working because FrameworkSampleSource does not read standard file input, but media files in a MediaFormatHolder.

So if I put my external TTML file in a media format holder would this work?

Like this?

MP4Box -add subtitle.en.ttml output.mp4
(http://gpac.wp.mines-telecom.fr/2014/09/04/subtitling-with-gpac/)

Thanks.

[ojw28]

You probably want to write a very simple SampleSource that will load the whole TTML file into a byte[], and then provide it to a calling TextTrackRenderer (by passing it back in the SampleHolder provided). You don't need to do anything with the MediaFormatHolder for text.

[nahojkap]

Related question - for TTML, there is additional styling information available - would you consider changing the TextTrackRenderer API to reflect this or is this sometihng you'd suggest to roll your own of?

Thanks,

Johan

[anotherhale]

Johan,

I was going to suggest adding this functionality as well. I would also like to add support for other parsers (SSA/ASS, SRT, STL, etc). I would like to volunteer to do this and have some thoughts on implementation.

[anotherhale]

Oliver, thanks for the suggestion. I created a BufferedByteSampleSource that I think should work - I have not had a chance to test it yet. I will close this out if it is successful.

[nahojkap]

As far as samples for streams with TTML, there is http://streams.smooth.vertigo.com/elephantsdream/Elephants_Dream_1024-h264-st-aac.ism/manifest - not sure how free that is, but it is accessible (and has been for quite a while).

However, the player is currently failing in the extractor trying to get the sample presentation time.

Johan

[ojw28]

• We used to use that video internally, but we found multiple issues with it that made it unsuitable for testing.
• Yes, TextTrackRenderer should probably be enhanced to support styling. Some careful thought would be required to keep it lightweight and general enough to support multiple formats though.
• Fyi, we'll be adding WebVTT support very soon.

[Ood-Tsen]

Is WebVTT used by the HLS Streaming or another protocol , jusk like the NuPlayer did ?

[nahojkap]

Naively I would assume it would be for use with MPEG-DASH primarily.

For styling, could it not be possible to have in essence two TextTrackRenderers - one which is simple (for the more basic subtitles) and supporting a richer set of markup on the text returned?

[Ood-Tsen]

I have tested the link "http://streams.smooth.vertigo.com/elephantsdream/Elephants_Dream_1024-h264-st-aac.ism/manifest" .

There is a null object exception due to there is no proper MediaFormat for the subtitle.

It's seems an known issue in SmoothStreamingChunkSource.java.

[ojw28]

• The problem with the missing MediaFormat is fixed already on the dev branch. The problems with the Elephants Dream video were around reliability of whatever is serving it, and I believe the subtitles contained invalid characters too.
• WebVTT is initially for DASH, but it should be fairly easy to wire it up for other use cases.
• There could be two renderers. Whether that's a good design choice depends on how extensive styling support needs to be, and what the overhead is of the logic for subtitles that don't have any etc.

[Ood-Tsen]

If we selected the 'jpn' text track, XmlPullParserException raised an error.

E/EventLogger(12030): Caused by: org.xmlpull.v1.XmlPullParserException: Dangling < (position:TEXT �路��ʿ���� @28:127 in java.io.InputStreamReader@422855b8)
E/EventLogger(12030): at org.kxml2.io.KXmlParser.peekType(KXmlParser.java:995)
E/EventLogger(12030): at org.kxml2.io.KXmlParser.next(KXmlParser.java:442)
E/EventLogger(12030): at org.kxml2.io.KXmlParser.next(KXmlParser.java:310)
E/EventLogger(12030): at com.google.android.exoplayer.text.ttml.TtmlParser.parse(TtmlParser.java:124)

It seems the 'jpn' text track doesn't encode in UTF-8.

We may detect the character set of input stream instead of use UTF-8 directly.

TextTrackRenderer.java
......
subtitle = subtitleParser.parse(subtitleInputStream, C.UTF8_NAME, sampleHolder.timeUs);

[ojw28]

If you inspect the subtitle files for this stream, you'll see they explicitly state that they're UTF8. I'm pretty sure the subtitles are just broken, as per my reply above.

NB - We should let the parser infer the encoding rather than pass UTF-8, but it doesn't fix anything for this stream (since it infers UTF-8 anyway).

[nahojkap]

Maybe I am doing something wrong but when you said the MediaFormat issue has been solved, did you mean that it should work on the dev branch as is? I just built from a fresh clone and it fails with the same issue.

[ojw28]

Are you sure you're pulling the dev branch? I think you're pulling master, which isn't the dev branch.

[nahojkap]

I was pretty sure, but I cloned your repo again and retried - same issue. I have checked out the dev branch, i can see your commits from today on it (latest is c4e1c35....). It is mighty strange - will triple check again just to be sure.

[Ood-Tsen]

Hello johanlindquist,

You may check this update.

Ood-Tsen@5c5f60f

[ojw28]

Sorry Johan; I didn't realize, but we were talking about different issues!

[ojw28]

FYI - I think a better fix is just to check fragmentRun.length in FragmentedMp4Extractor.seekTo. I'll drop a patch to dev early next week.

[nahojkap]

Great, thanks!

Also, i think the reason the parsing is failing is because it is working
with a partial XML document (contained in a sample?) and not the XML from
the entire smooth streaming fragment.

When I debug it, the data passed to the parser is truncated from what i can
see.

Johan
On Sep 21, 2014 10:16 AM, "ojw28" [email protected] wrote:

FYI - I think a better fix is just to check fragmentRun.length in
FragmentedMp4Extractor.seekTo. I'll drop a patch to dev early next week.

—
Reply to this email directly or view it on GitHub
#39 (comment).

[ojw28]

The single sample should contain the full XML document (in theory ;)).

[nahojkap]

Indeed, you are right - and the English subtitles works as they should ... I know we are parsing the other (bad) XML though so need to check why we can & Exoplayer don't do it.

[ojw28]

The question at the top of this issue has been answered, and so I'm unsure what this is tracking at this point. Closing. Please spin out specific issues separately if needed. Thanks!

[triwav]

Thought I would post this here since trying to find documentation on adding an external caption file is very hard:
https://gist.github.com/triwav/ba72e99e5181427f4645

And to use it I'm doing:
try
{
URL url = new URL(video.getClosedCaptionUrl());
HttpSampleSource subtitleSource = new HttpSampleSource(url, "text/vtt");
textRenderer = new TextTrackRenderer(subtitleSource, new WebvttParser(), this, getContext().getApplicationContext().getMainLooper());
player.prepare(videoRenderer, audioRenderer, textRenderer);
} catch (MalformedURLException e)
{
player.prepare(videoRenderer, audioRenderer);
}

Note that I am having to pass in the mime type which is less than ideal but getTrackInfo could get called before the load has completed resulting in my case the webvtt file getting ignored by the subtitle parser. One bug is that when it gets to the end of the video it will just hang there. I'm currently dealing with this with this code in my onText method:
// Bug manually dealing with for now
int playerPosition = player.getCurrentPosition();
if (previousOnTextTime == playerPosition)
{
player.setRendererEnabled(TYPE_TEXT, false);
}

[Ood-Tsen]

Does ExoPlayer support loading vtt from external file ?

[triwav]

That's the purpose of this HttpSampleSource.

[ojw28]

It's also supported directly by the library if you use SingleSampleChunkSource. Basically the setup should be:

TextTrackRenderer->ChunkSampleSource->SingleSampleChunkSource

I don't think this approach has the issues that you describe above, so you may find it beneficial to switch from HttpSampleSource over to this approach. Effectively, you'll be replacing your custom HttpSampleSource with ChunkSampleSource->SingleSampleChunkSource.

Sorry, I'm aware documentation is poor around how to this kind of thing. I'm trying to find time to write some.

[triwav]

Thanks @ojw28. I tried looking into this but it seems pretty complicated to get setup. It seems like the Allocator requires some extra work to keep track of what's been loaded. If you or someone else provides some documentation I'll investigate further but at this point I think I'll stick with my existing solution.

LoadControl loadControl = new LoadControl()
{

        @Override
        public boolean update(Object loader, long playbackPositionUs, long nextLoadPositionUs, boolean loading, boolean failed)
        {
            return true;
        }


        @Override
        public void unregister(Object loader)
        {
            Debug.toString("unregister");

        }


        @Override
        public void trimAllocator()
        {
            Debug.toString("trimAllocator");

        }


        @Override
        public void register(Object loader, int bufferSizeContribution)
        {
            Debug.toString("register");

        }


        @Override
        public Allocator getAllocator()
        {
            Debug.toString("getAllocator");

// return null;
return new Allocator()
{

                @Override
                public void trim(int targetSize)
                {
                    // TODO Auto-generated method stub
                    Debug.toString("trim");

                }


                @Override
                public int getAllocatedSize()
                {
                    // TODO Auto-generated method stub
                    Debug.toString("getAllocatedSize");

                    return 0;
                }


                @Override
                public Allocation allocate(int size)
                {
                    // TODO Auto-generated method stub
                    Debug.toString("allocate", size);

                    return new Allocation()
                    {

                        @Override
                        public void release()
                        {
                            Debug.toString("Allocation", "release");

                        }


                        @Override
                        public int getFragmentOffset(int index)
                        {
                            Debug.toString("Allocation", "getFragmentOffset");
                            return 0;
                        }


                        @Override
                        public int getFragmentLength(int index)
                        {
                            Debug.toString("Allocation", "getFragmentLength");
                            return 0;
                        }


                        @Override
                        public byte[][] getBuffers()
                        {
                            Debug.toString("Allocation", "getBuffers");
                            return null;
                        }


                        @Override
                        public void ensureCapacity(int size)
                        {
                            Debug.toString("Allocation", "ensureCapacity");

                        }


                        @Override
                        public int capacity()
                        {
                            Debug.toString("Allocation", "capacity");
                            return 0;
                        }
                    };
                }
            };
        }
    };

    DataSource dataSource = new DataSource()
    {

        @Override
        public int read(byte[] buffer, int offset, int readLength) throws IOException
        {
            Debug.toString("read");
            return 0;
        }


        @Override
        public long open(DataSpec dataSpec) throws IOException
        {
            Debug.toString("open", dataSpec.uri);
            // TODO Auto-generated method stub
            return 1000;
        }


        @Override
        public void close() throws IOException
        {
            // TODO Auto-generated method stub

        }
    };

    Uri uri = Uri.parse(video.getClosedCaptionUrl());
    DataSpec dataSpec = new DataSpec(uri);

    Format format = new Format("id", MimeTypes.TEXT_VTT, -1, -1, -1, -1, -1);

    MediaFormat mediaFormat = MediaFormat.createFormatForMimeType(MimeTypes.TEXT_VTT);

    SingleSampleChunkSource chunkSource = new SingleSampleChunkSource(dataSource, dataSpec, format, C.UNKNOWN_TIME_US, mediaFormat);
    SampleSource subtitleSource = new ChunkSampleSource(chunkSource, loadControl, 0, false);
    textRenderer = new TextTrackRenderer(subtitleSource, this, getContext().getApplicationContext().getMainLooper(), new WebvttParser());

[ojw28]

Yes, it is complicated to setup currently. I'm working on making it simpler.

[ojw28]

Simplifications (which will apply to all subtitle formats) are tracked by #587.

[andreimarinescu]

I got this working using the gist posted by @triwav (thanks for this, by the way!). I've created a new gist that is compatible with the latest dev version, I hope it's ok to post it here in case anyone else faces this issue until the simplifications in #587 are added. https://gist.github.com/andreimarinescu/9b2e6c1fed6bbaddb91b

[triwav]

Andre, I just updated my gist to what we're actually using in production today. There were a few weird bugs in that version of the code that this latest version should address. If you experience any bugs with your script feel free to take a look at my latest gist to see if that solves it for you.

[andreimarinescu]

Thanks! I'll have a look. We haven't deployed this into production, probably that's why we haven't hit any roadbumps yet.

[triwav]

Yeah the main things to check is turning on and off captions repeatedly, jumping to a different point in the video and behavior when the video finishes playing.

[abhiguru]

Tried the above and then

HttpSampleSource httpSampleSource = new HttpSampleSource(url , "text/vtt" );
TrackRenderer textRenderer = new TextTrackRenderer(httpSampleSource, player,
mainHandler.getLooper(), new WebvttParser());

The textRenderer that gets generated has no sources in it any idea why ? Here is the code I have https://gist.github.com/abhiguru/9fb26a51c4e8b57d3d48

and here is the error about null sources

Internal runtime error.
java.lang.NullPointerException: Attempt to invoke interface method 'boolean com.google.android.exoplayer.SampleSource$SampleSourceReader.prepare(long)' on a null object reference
at com.google.android.exoplayer.SampleSourceTrackRenderer.doPrepare(SampleSourceTrackRenderer.java:54)
at com.google.android.exoplayer.TrackRenderer.prepare(TrackRenderer.java:110)
at com.google.android.exoplayer.ExoPlayerImplInternal.incrementalPrepareInternal(ExoPlayerImplInternal.java:281)
at com.google.android.exoplayer.ExoPlayerImplInternal.prepareInternal(ExoPlayerImplInternal.java:272)
at com.google.android.exoplayer.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:201)
at android.os.Handler.dispatchMessage(Handler.java:98)
at android.os.Looper.loop(Looper.java:135)
at android.os.HandlerThread.run(HandlerThread.java:61)
at com.google.android.exoplayer.util.PriorityHandlerThread.run(PriorityHandlerThread.java:40)

[ojw28]

You're getting the NullPointerException because you've implemented the register method to return null, which is in violation of the contract of that method.

That aside, it's really quite easy to use an external caption file in recent versions of ExoPlayer. You shouldn't need to be writing custom code. You can use SingleSampleSource and you should need about four lines of code, similar to those in the top post in #753.

[abhiguru]

Thanks for the reply, Eventually I found out about the SingleSampleSource and have been using it without a problem! Maybe we can add that to the wiki for others who come looking ? #753 is a helpful thread thanks for pointing it out.