Rendering: performance issue while rendering running rainbow animated subtitles

[NBruderman]

Rendering: performance issue while rendering running rainbow animated subtitles

[arch1t3cht]

While I can reproduce the lag on mpc-hc+libass, this sample drops 0 frames for me on mpv. So this seems to be specific to mpc-hc's usage of libass.

This file is however slightly peculiar because the strings inside the BlockGroups of the muxed subtitle track all contain null terminators, which I at least haven't seen in an mkv file before. The sample's mediainfo claims it was muxed using Lavf, but from testing and skimming the source code it doesn't look like libavformat includes null terminators in these subtitle packets.

I noticed this because these subtitles would fail to render properly on libass when demuxed with Aegisub (which currently does not account for such packets). Now, theoretically this could be seen as a libass/VSFilter performance differential since VSFilter does render a file like this (that is, a file where every Dialogue event ends with a null byte) normally while libass seems to see any null byte as terminating the entire file, but this could also be intentional.

[NBruderman]

Actually, on mpv I got much less video lag, however, I got audio lags, which didn't occur on mpc-hc.
I understand my sample doesn't contain audio, so I'm adding here the same sample, but with audio intact.

Sample with audio

[astiob]

The ASS consists of the exact same event repeated 596 times with the only differences being the primary colour (but not alpha) and rectangular clip, and with the clips animated (plus a few additional easy events). The event text is plain English. We should be caching all of this, but…

I’m not very good with profilers, but when I try Valgrind’s Callgrind on this, these items seem to stand out:

Accumulated/inclusive times:

14,128,295,879 (98.01%)  libass/ass_render.c:ass_render_frame
 5,498,568,436 (38.14%)   > libass/ass_render.c:render_and_combine_glyphs
 1,785,114,093 (12.38%)      > libass/ass_cache.c:ass_cache_get
 1,246,502,932 ( 8.65%)      > libass/ass_render.c:quantize_transform
   545,154,344 ( 3.78%)      > ???:ldexp [/lib64/libm-2.31.so]
   305,904,900 ( 2.12%)      > ???:sincos [/lib64/libm-2.31.so]
   105,776,216 ( 0.73%)      > ???:frexp [/lib64/libm-2.31.so]
    15,938,912 ( 0.11%)      > ???:malloc [/lib64/ld-2.31.so]
    12,204,948 ( 0.08%)      > ???:lrint [/lib64/libm-2.31.so]
 5,065,330,511 (35.14%)   > libass/ass_shaper.c:ass_shaper_shape
 4,219,346,055 (29.27%)      > ???:hb_shape [/usr/lib64/libharfbuzz.so.0.30400.0]

 1,785,114,093 (12.38%)  < libass/ass_render.c:render_and_combine_glyphs
   629,579,053 ( 4.37%)  < libass/ass_shaper.c:get_cached_metrics
   516,155,753 ( 3.58%)  < libass/ass_render.c:ass_render_frame
     7,927,047 ( 0.05%)  < libass/ass_font.c:ass_font_new
 2,938,775,946 (20.39%)  *  libass/ass_cache.c:ass_cache_get
   623,249,172 ( 4.32%)  >   libass/ass_cache_template.h:bitmap_hash
   476,685,636 ( 3.31%)  >   libass/ass_cache.c:outline_hash
   297,747,436 ( 2.07%)  >   libass/wyhash.h:glyph_metrics_hash
   173,484,742 ( 1.20%)  >   libass/ass_cache.c:composite_hash

Exclusive times:

1,467,734,200 (10.18%)  libass/ass_render.c:ass_render_frame
1,429,651,746 ( 9.92%)  libass/ass_render.c:render_and_combine_glyphs
1,392,355,886 ( 9.66%)  ???:0x000000000006c670 [/usr/lib64/libharfbuzz.so.0.30400.0]
1,148,738,356 ( 7.97%)  libass/ass_render.c:quantize_transform
1,098,743,168 ( 7.62%)  ???:0x0000000000050570 [/usr/lib64/libfreetype.so.6.17.4]
  786,622,629 ( 5.46%)  libass/ass_cache.c:ass_cache_get
  611,028,600 ( 4.24%)  libass/wyhash.h:bitmap_hash
  427,808,556 ( 2.97%)  libass/wyhash.h:outline_hash

It seems that even though the caches are fully working, we’re spending too much time building the cache lookups? We may need to add a direct lookup of composite bitmap by text & style, bypassing the text shaping, glyph-by-glyph lookup and run merging.

[astiob]

But also, we’re outputting 601 nontransparent images per frame and an additional 312 transparent ones. (The ASS has \shad2\4a&HFF&. 🤦) Almost all of these images are 5 px wide.

On my Windows machine, mpv with default settings reports in OSD that it’s dropping some frames but it’s barely noticeable, while with -vf sub it very clearly struggles and renders only a handful of frames every second.

For reference, an isolated call to ass_render_frame on my Windows machine takes 25–30 ms. On a separate Linux machine (where I ran Valgrind) with profile run over 62 frames (where everything but the first event should come from the cache), the average is 17–20 ms per frame.

I’m guessing that this amount of small images is somehow causing issues for both mpv and MPC-HC. But frankly, I’m surprised that vf_sub is having so much trouble.