This repository contains the code for the paper titled "PerceptionGPT: Effectively Fusing Visual Perception into LLM". [Link to our paper]
conda create -n perceptiongpt python=3.10
conda activate perceptiongpt
pip install -r requirements.txt
Before conducting training, organize the training dataset following Shikra.
The running config located in config/train_configs/shikra3_rec3_mask_box_cls_refcoco_all.py could be modified accordingly
bash script/run.sh
The project is built on top of the amazing multimodal large language model LLaVA and Shikra. Thanks for these great work!
If you find our work useful for your research or applications, please cite using this BibTeX:
@misc{pi2023perceptiongpt,
title={PerceptionGPT: Effectively Fusing Visual Perception into LLM},
author={Renjie Pi and Lewei Yao and Jiahui Gao and Jipeng Zhang and Tong Zhang},
year={2023},
eprint={2311.06612},
archivePrefix={arXiv},
primaryClass={cs.CV}
}