Comments (6)
Thanks for the question and suggestion! We will provide instructions for evaluating a new video soon.
Currently, we enforce evaluation on the standard VBench prompt lists to ensure fair comparison among different video generation models. That's also why we give warnings when a required video is not found. This is done via defining the set of prompts in: https://github.com/Vchitect/VBench/blob/master/VBench_full_info.json
A quick fix is to define your own VBench_full_info.json file: replacing the prompts in "prompt_en" by your video name.
from vbench.
Thank you very much for your great work! looking forward for instructions for evaluating a new video :)
from vbench.
Thanks for the question and suggestion! We will provide instructions for evaluating a new video soon.
Currently, we enforce evaluation on the standard VBench prompt lists to ensure fair comparison among different video generation models. That's also why we give warnings when a required video is not found. This is done via defining the set of prompts in: https://github.com/Vchitect/VBench/blob/master/VBench_full_info.json A quick fix is to define your own VBench_full_info.json file: replacing the prompts in "prompt_en" by your video name.
If I use my video name replace the prompts in "prompt_en" , but I can't supply the "auxiliary_info", will this have any impact?
Looking forward for you replay.
Thanks!!
from vbench.
Thanks for the question and suggestion! We will provide instructions for evaluating a new video soon.
Currently, we enforce evaluation on the standard VBench prompt lists to ensure fair comparison among different video generation models. That's also why we give warnings when a required video is not found. This is done via defining the set of prompts in: https://github.com/Vchitect/VBench/blob/master/VBench_full_info.json A quick fix is to define your own VBench_full_info.json file: replacing the prompts in "prompt_en" by your video name.If I use my video name replace the prompts in "prompt_en" , but I can't supply the "auxiliary_info", will this have any impact? Looking forward for you replay. Thanks!!
@moclimb Thank you for your interest in our work! In ['subject_consistency', 'background_consistency', 'temporal_flickering', 'motion_smoothness', 'dynamic_degree', 'aesthetic_quality', 'imaging_quality', 'temporal_style', 'overall_consistency']
, you don't need to fill in any auxiliary_info
, you just need to fill in the dimension
corresponding to your prompt
In ['object_class', 'multiple_objects', 'human_action', 'color', 'spatial_relationship', 'scene', 'appearance_style']
, you need to fill in auxiliary_info
in accordance with the corresponding rules. You must note that the object class and multiple objects in the tuple must be an object of COCO. The human action must be one action of Kinetics-400.
from vbench.
Hello, i'm trying to run inference. I've installed the package. I've tried the code here.
I have a new directory of videos, called "videos".
(venv) (base) yonatan:~/VBench$ ls videos/ Iron_Man.mp4 birthday.mp4 lavie_human_action_full_info.json skateboarding_dog.mp4
When I try to run
vbench evaluate --videos_path "videos" --dimension "human_action"
I receive an error indicating that no data is found. When I try through the code, I also get a long list of missing videos:
>>> from vbench import VBench >>> my_VBench = VBench('cuda', 'VBench_full_info.json', 'videos') >>> my_VBench.evaluate(videos_path='videos', name='my_test', dimension_list=['human_action']) WARNING!!! This required video is not found! Missing benchmark videos can lead to unfair evaluation result. The missing video is: A person is riding a bike-0.mp4 ... ...I guess that your code only supports your existing evaluated videos? And the instructions do not yet support inference on new videos?
The question is how can I run the evaluation for a new video + description. Thank you very much 🙏
Hi Yonatan, we now support evaluating user-customised videos: https://github.com/Vchitect/VBench?tab=readme-ov-file#new-evaluate-your-own-videos. You can do this by simply turning the custom_input
flag on.
This new feature now supports those "video quality" dimensions, like temporal consistency, and frame-wise quality. For dimensions that heavily rely on the text prompt, sometimes VBench utilize pre-defined metadata, like the color labels for objects associated with our pre-defined prompt list. Therefore, evaluating videos from user-customised prompts is not yet supported for these prompt-dependent dimensions, but we are working on it. We will gradually enable customised evaluation of more "video-text alignment" dimensions.
from vbench.
I'm closing the issue, feel free to re-open it if you still encountered issues when evaluating new videos.
from vbench.
Related Issues (20)
- FileNotFoundError: [Errno 2] No such file or directory: './VBench_full_info.json' HOT 3
- Cannot find and process videos under provided directory HOT 2
- Evaluation data issues HOT 1
- generating videos from images HOT 9
- how to execute the project ? HOT 2
- FileNotFoundError: [Errno 2] No such file or directory: './VBench_full_info.json' HOT 2
- how to use detectron2(which require torch<=1.10) and the torch>=1.12 in requirement HOT 1
- Evaluation of True Videos HOT 4
- provide the pretrained weights for reproducing the results in leaderboard HOT 3
- What does the result means? HOT 2
- Why is the SVD not included? HOT 2
- Where can I find open source video data? HOT 2
- Where is the human preference data? HOT 3
- About the size and number of frames HOT 2
- Some problems HOT 13
- LeaderBoard code HOT 6
- LeaderBoard sampled videos
- run inference on customized video and prompt?
- score range for each dimension? HOT 2
- Request for detailed instructions for submitting evaluation results to the leaderboard
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vbench.