Evaluation data issues

[GFENGG]

Evaluation data issues

[ziqihuangg]

Hi, thanks for your question! Currently there are several supported formats (like MP4 and GIF). See this function for video loading: https://github.com/Vchitect/VBench/blob/master/vbench/utils.py#L101

VBench has already considered the influence from frame rate and resolution during evaluation.

For example, some dimensions can be sensitive to video frame rate, like "Dynamic Degree". For higher frame rate videos, the dynamics (movement) between two consecutive frames are generally smaller, resulting in a false result of "low dynamic degree". Therefore, we eliminate frame rate's influence at the preprocessing step: https://github.com/Vchitect/VBench/blob/master/vbench/dynamic_degree.py#L86C9-L86C19

So during evaluation, you can directly supply the raw video without any preprocessing to VBench, and VBench will take care of any preprocessing needed for individual evaluation dimensions.