VideoChat2 Mistral Vs Vicuna Configs

[mcrchopra]

VideoChat2 Mistral Vs Vicuna Configs

[Andy1621]

Good questions!

1. The num_frames in the configs are actually placeholders, and I changed them to 4 in the ipynb. Since UMT uses sic-cos position embedding and is pretrained with 4 frames, I set it to 4 for better position interpolation.
2. In the instruction tuning, we use 8 frames, but we find 16 frames are better for different tasks. We do not use 16 frames considering the efficiency and effectiveness.

3. You can find the data details in data.py. We use only WebVid10M and CC3M in the stage2 of Mistral, since we find that more data is even harmful, leading to about 1~2% accuracy drop in MVBench.

[mcrchopra]

Thank you for the quick response! This is really helpful!