Performance regression results in severe framedrops for typesetted subtitles

Performance regression results in severe framedrops for typesetted subtitles

[TheOneric]

If mpv were to reset style overrides each frame even if they didn't change, yeah this commit could lead to perf regressions.

Interestingly though, I actually observe better perf after 1a599b1. I am however also using an older mpv (0.35.1) instead of git master. At least a simple greps for changes containing ass_set_selective_style_override or mp_ass_configure doesn't change anything betwenn 0.25.0 and current master, but this does ofc not cover all possibilities.

mpv 0.35.1
gcc 12.2.0
./configure --enable-asm --enable-compare --enable-fuzz \
  CFLAGS="-O2 -g -ftree-vectorize -fvect-cost-model=dynamic -flto=auto" \
  LDFLAGS="-flto=auto"

2023.05.11: 1a599b1db748dfc07df98cd37d8603edb87da2fd
    --no-config 0 dropped frames
    user-conf   5

2023.04.04: 32dea3434dbd85f9fb2700b6863c1bd92f00b86f
    --no-config 24-27
    user-conf   27-29

Before we try to dig deeper, could you check different builds of libass with the same mpv build(setup) to confirm this is actually connected to libass and not a change in mpv, some dependency or the build tools?
If you already have a build setup, keeping the mpv binary the same and ust switching the dynamic libass lib would be easiest. If not, MSYS2 and mpv-build is probably helpful. You can select a specific libass commit by running ./use-libass-custom <commit-hash> before rebuilding and after the initial setup. The relevant commit hashes are listed above in my measurements.
If you are not comfortable building mpv, do you know someone else who observed this regression and is willing to build mpv?

[llyyr]

Interestingly though, I actually observe better perf after 1a599b1.

This has not been my experience. I did some testing with libass-master and without that commit using mpv (both master and 0.35.1) with the following profile and no other configs set.

[bench]
audio=no
untimed=yes
video-sync=display-desync
vulkan-swap-mode=immediate
opengl-swapinterval=0
osd-msg1="FPS: ${estimated-display-fps}"

(although I did try both vo=gpu and vo=gpu-next, the performance is within 2-3fps of each other)

I isolated the part of the subtitles that cause the frame drops into a file here, and modified the timestamps for easier benchmarking out.txt (the ext has to be txt due to github limitations, change it to .ass)

Then ran mpv like this: mpv --profile=bench --sub-file=out.ass [h264 video]

To establish a baseline, I get heavily fluctuating 350-500 fps with very basic (srt conversion) subtitles on the same video with libass 32dea34, and 300-450 fps with libass master with the same subtitles.

On mpv master:
With libass master, I get 40 fps on vo=gpu, 38fps on vo=gpu-next
With libass 32dea34 or libass 0.17.1, I get 85-90 fps on both vo=gpu and vo=gpu-next.

And I also get roughly similar results with mpv 0.35.1.

That commit definitely causes huge performance penalty, however it's also worth mentioning that this particular sign/typesetting is particularly evil because libass is particularly slow even without that commit. My assumption would be that libass has always been slow with this particular type of sign, and this commit just made that particular case worse

[Dudemanguy]

If mpv were to reset style overrides each frame even if they didn't change, yeah this commit could lead to perf regressions.

Yeah this is what's currently being done. I'm not too familiar with that area of mpv but that's one pretty obvious optimization we could make. I see the same kind of behavior that llyyr reported so this commit definitely slows down mpv alot, but if we are misusing the API then it's our fault. There's possibly some other things we are doing unconditionally on every frame that we don't have to do.

[TheOneric]

Thanks both you!

With the FPS benchmark, I too can reproduce lower (~5.5×) perf with mpv after 1a599b1. At the same time, the dropped frame numbers from before remain reproducible and are better with this libass commit. (I’m really curious what weird cache/scheduling/... interactions leads to this dropping fewer frames, but am too busy to spend time at this. If anyone gets the same result and knows, please let me know.)

In any case it is pretty clear that the libass part by itself now performs worse in mpv. And with the confirmation mpv does indeed reset overrides each frame the cause is also nailed down.

mpv just not resetting overrides if there wasn't a change would be ideal and the cleanest solution. In theory (if it turns out many relied on cheap resets) we could copy and cache the last override and early exit ass_set_selective_style_override if there wasn't a change. This will add some additional overhead though and to avoid this mpv might still want to skip resetting every frame.

At first glance (checking mplayer, ffmpeg, VLC, Kodi, gstreamer assrender, AviSynth and VapourSynth assrender). Kodi appears to be the only other user of ass_set_selective_style_override and its invocation appears to be already limited to changes.

@rcombs do you think it makes sense to check for changes in ass_set_selective_style_override, or are you aware of any other users?

[rcombs]

I think the caller is basically always in a better position to know whether a change has occurred than we are.

[TheOneric]

Well then, seems like (for now) there’s nothing to do here then and mpv, the only known affected API-user, is already being patched (thanks Dudemanguy!). In any event thanks for the report and help looking into it.

[Dudemanguy]

@TheOneric: Thank you for exactly identifying the problem in literally the first sentence!

Can confirm that I don't experience frame drops anymore with the latest Windows mpv build. Thanks everyone.