How is it related to Video ChatCaptioner?

[opengvlab]

How is it related to Video ChatCaptioner?

[Andy1621]

@junchen14 Thanks for your question and awesome work! The main difference is shown in the name of the two projects:

1. Video ChatCaptioner and ChatCaptioner focus on generating better and fine-grained captions automatically. They use ChatGPT as a smart questioner, while using BLIP2 as a good answerer.
2. Our project aims to build a chatbot to ask anything about video. It's more like your team's other project miniGPT-4. Different from miniGPT-4 and BLIP2, which implicitly encode the image into abstract tokens, we currently adopt explicit encoding (frame caption, dense caption with box, action class...). Without extra training, we find that the explicit encoding is better now. BTW, we also try to simply adapt miniGPT-4 for video. Unfortunately, without instruction fine-tuning, the results are sometimes unsatisfactory now.

[junchen14]

Thanks for the clarification! It looks like an interesting project and glad to see that miniGPT-4 for video has also been working in progress!