Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedback,
Daechul Ahn1,3, Yura Choi1,3, Youngjae Yu1, Dongyeop Kang2, Jonghyun Choi3,โ
1Yonsei University, 2University of Minnesota, 3Seoul National University
โ Corresponding Author
ACL 2024 (To appear)
ย
- [Aug 07, 2024] We update our trained lora checkpoint of reward model & policy model initialization to Hugginface
- [Aug 06, 2024] Our model is available in HuggingFace Spaces!
- [Jul 16, 2024] ๐๏ธ VLM-RLAIF has been selected for โจoral presentationโจ at ACL 2024! See you in Bangkok ๐น๐ญ
- [Jun 16, 2024] ๐ฅ Our next work on aligning large video multimodal model, i-SRT๐, is now available [arXiv, code]
- [May 31, 2024] ๐ฅณ VLM-RLAIF is accepted to ACL 2024 !
ย
Abstract: Recent advancements in large language models have influenced the development of video large multimodal models (VLMMs). Previous approaches for VLMMs involve Supervised Fine-Tuning (SFT) with instruction-tuned datasets, integrating LLM with visual encoders, and additional learnable parameters. Here, aligning video with text, and vice versa, remains a challenge, primarily due to the insufficient quality and quantity of multimodal instruction-tune data compared to that of text-only. This discrepancy often results in alignments that poorly ground the video content. To address this, we present a novel alignment strategy that employs a multimodal AI system equipped with Reinforcement Learning from AI Feedback (RLAIF), providing self-preference feedback to refine itself and facilitating the alignment of video and text modalities. Our approach uniquely integrates detailed video descriptions as context into a multimodal AI system during preference feedback generation to enrich the understanding of video content, a process we call context-aware reward modeling. Empirical evaluations on various video benchmarks demonstrate that our VLM-RLAIF outperforms existing approaches, including the SFT model.
Check PREPARE_DATASET.md to prepare training & validation datasets
Model | Size | Checkpoint | corr. | detail. | context | temp. | const. |
---|---|---|---|---|---|---|---|
RLAIF | 7B | SNUMPR/vlm_rlaif_video_llava_7b | 3.63 | 3.25 | 4.00 | 3.23 | 3.32 |
SFT | 7B | SNUMPR/vlm_sft_video_llava_7b | 2.79 | 2.82 | 3.37 | 2.28 | 2.49 |
Lora Checkpoints (used to train the model w/ PPO)
Model | Size | Lora Checkpoint |
---|---|---|
Policy init | 7B | SNUMPR/vlm_policy_init_7b_lora |
Reward model | 13B | SNUMPR/vlm_rm_13b_lora |
ย
Dataset Usage | Link |
---|---|
SFT (short) | SNUMPR/vlm_rlaif_datasets/SFT_short.json |
SFT (long) | SNUMPR/vlm_rlaif_datasets/SFT_long.json |
Preference dataset (for RM) | SNUMPR/vlm_rlaif_datasets/RM_13b_v1_dataset_39k.json |
PPO init | SNUMPR/vlm_rlaif_datasets/PPO_init.json |
RLAIF | SNUMPR/vlm_rlaif_datasets/RL_data.json |
ย
Check PREPARE_DATASET.md to prepare training & validation datasets
- Zero-shot QA
bash Evaluation/zeroshotqa/scripts/zeroshotqa_pipeline.sh
- Video Generative Benchmark
bash Evaluation/scripts/videochatgpt_pipeline.sh
ย
- Refer to the RLAIF folder to train reward model, policy model, and do PPO
ย
Available Soon
ย
@inproceedings{ahnCYKC24,
author = {Daechul Ahn and Yura Choi and Youngjae Yu and Dongyeop Kang and Jonghyun Choi},
title = {Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedback},
booktitle = {ACL},
year = {2024}
}
ย
- The majority of this project is released under the Apache 2.0 license as found in the LICENSE file.
- The service is a research preview intended for non-commercial use only, subject to the model License of LLaMA ย