add threading support to signalstats filter

[bavc]

add threading support to signalstats filter

[JeromeMartinez]

QCTools is multithread in a manner we parse several files at the same time while keeping arbitrary 2 threads available for real time usage and other software. Better formula to be discussed.
default config of FFmpeg is used for decoding, maybe it is possible to use multithreaded decoding of FFV1, I'll check.

[pjotrek-b]

Thanks for clearing that up.

I've also tried to generate the qctools.xml.gz with ffprobe directly (as written in the "Data Format" help), and the CPU usage is identical to QCTools'. I've tried adding "-threads 8", but no effect.
Sorry, I know this is more an ffprobe question, but maybe you got a tip for speeding this up?

[JeromeMartinez]

I think it depends of the way you encoded the stream. It is possible for FFV1 (as well as AVC/H264, HEVC/H265...) to be thread-friendly (with multiple slices) but it is not mandatory (if 1 slice, impossible to thread).

We could split the file in several parts and do parallel scan of each part, but it is more work. For the moment, we optimize with several files parsed at the same time (if you provide several files).

[pjotrek-b]

I've tried with both: FFV1.1 (no slices) and FFV1.3 (=16 slices).

I know that since FFV1 changes introduced for version 3, version 1 can now be decoded multithreaded.
For example, running "ffmpeg -i video.avi -an -f -framemd5 video.avi.framemd5" is also only decoding, but using all cores in both cases. So I would assume that ffprobe should also be possible to decode that fast?

[pjotrek-b]

Maybe that's something I shouldn't post here, but rather on ffmpeg-user@ mailing list.

[JeromeMartinez]

First, we need to see if a QCTool option is faulty.
e.g. doing ffprobe command line without stats and see if all cores are used

version 1 can now be decoded multithreaded.

Even with files without slice?

[pjotrek-b]

ffprobe doesn't use all the cores in my test. Just sent that to ffmpeg-user@.
And, yes: FFV1 version 1 can be decoded (only DEcoded!) multithreaded since a very long time now. I guess around 2011 or so. It was a by-product of decoder-changes introduced in version 2/3.

[pjotrek-b]

Related thread on ffmpeg-user@ mailing list:
http://www.mail-archive.com/[email protected]/msg01085.html

It's very likely, that one of the filters used for gathering the values might not be implemented multi-threading. Haven't confirmed that yet, but thought you might want to know.

[JeromeMartinez]

this is weird that the filterchain prevents the FFV1 decoding to be multithread.
Too many differences for FFmpeg list (mix of ffmpeg and ffprobe...), so try to have only few differences.

e.g.
ffmpeg_git -i qctools.avi -threads 8 -f null /dev/null
vs
ffprobe_git -i "movie=qctools.avi" -threads 8 > /dev/null
then adding options step by step so we can understand which option prevent multithread.

Would you mind to do such test?

[pjotrek-b]

I'll do.

[pjotrek-b]

Tried on the same video as before (no audio), and when I deactivate the "signalstats=stat=tout+vrep+brng" filter, my whole CPU is used.

I've also tried removing "tout", "vrep" individually: no change. Seems that it's the "signalstats" filter that won't multithread.

[JeromeMartinez]

We have the guilty code:).
I wonder the reason the filter disables the multithread of the decoding.
So I think that now the best thing is to ask FFmpeg team if this is wanted that a filter disables the multithread decoding.
But before, maybe we need to bench the filter without FFV1 (in order to know the speed of the filter).

@dericed any idea about how we could bench every part of FFmpeg?

But maybe this is not the highest priority...

[dericed]

This query shows commits done to enable slice threading into filters: http://git.videolan.org/?p=ffmpeg.git&a=search&h=HEAD&st=commit&s=slice+threading.

[pjotrek-b]

Thanks!
Hm... I can't finde the "signalstats" filter in that list.

[dericed]

@ubitux wrote several optimizations to the signalstats filter here http://github.com/ubitux/FFmpeg/commits/signalstats. These are anticipated to be included within ffmpeg soon and will be used in the QCTools 0.7 update.

[bdwheele]

I ran a sample file through the process:
MPEG-2 720x486 yuv422p 29.97fps
pcm_s24le @ 48 kHz stereo
55:49 runtime

The 20141119 build (pre-threading):
real 26m39.583s
user 30m37.650s
sys 0m16.084s

The 20150105 build (threaded):
real 7m15.335s
user 36m58.948s
sys 1m59.835s

So, it's a wall-clock improvement of ~3.7x and it ended up using around 5.5 cores for the duration of the processing.

[pjotrek-b]

Sounds great!
Which qctools build includes this change, so I could test it too?

[JeromeMartinez]

you can test with FFmpeg 2.5.2 directly (command line)
I'll provide soon a beta version of QCTools 0.7 which includes the new FFmpeg version.

[pjotrek-b]

I just ran the XML-generating commandline, using ffprobe from git-HEAD.
Definitely faster now, and my CPU graph shows it's multithreading.

It's not using my CPU (i7 QuadCore) to the fullest though, but it's still way better than before.
Thank you very much! :)