Why T5 is used instead of GPT?

[openmotionlab]

Why T5 is used instead of GPT?

[ChenFengYe]

Hi, the first language model that we used to build MotionGPTs is LLaMA-13B. However, it shows insufficient performance and low training efficiency. We assume the reason is the limited dataset size compared to the large parameters and language data of LLaMA.

Then, we thus choose T5-770M, a small but common language model, as our final backbone, because many previous vision-language multimodal works, like Unified-IO and BLIP, have chosen T5, this encoder-decoder architecture. It shows a strong power to address multi-modal tasks. In addition, the decoder-only model has the advantage for self-supervised without pair data while we have paired data which this advance is greatly weakened. We are still working on collecting a large motion dataset for larger motion-language models.

We have evaluated MotionGPT on GPT-2 and are working on LLaMA-2+LORA. Please refer to the below.

[ChangeNext]

Did you only do fine-tuning, or did you also perform pre-training?