English | 中文
Paper | Demo_chat | Detailed Caption | Model Weight | Model Weight in wisemodel | Demo in wisemodel
Monkey brings a training-efficient approach to effectively improve the input resolution capacity up to 896 x 1344 pixels without pretraining from the start. To bridge the gap between simple text labels and high input resolution, we propose a multi-level description generation method, which automatically provides rich information that can guide the model to learn the contextual association between scenes and objects. With the synergy of these two designs, our model achieved excellent results on multiple benchmarks. By comparing our model with various LMMs, including GPT4V, our model demonstrates promising performance in image captioning by paying attention to textual information and capturing fine details within the images; its improved input resolution also enables remarkable performance in document images with dense text.
2024.4.5
🚀 Monkey is nominated as CVPR 2024 Highlight paper.2024.3.8
🚀 We introduce TextMonkey, using only public document data, with a (Demo) available and code forthcoming.2024.2.27
🚀 Monkey is accepted by CVPR 2024. The paper has been carefully updated according to the valuable comments.2024.1.3
🚀 Release the basic data generation pipeline. Data Generation2023.12.21
🚀 The JSON file used for Monkey training is provided.2023.12.16
🚀 Monkey can be trained using 8 NVIDIA 3090 GPUs. See subsection train for details.2023.11.25
🚀 Monkey-chat demo is released.2023.11.06
🚀 Monkey paper is released.
- Contextual associations. We introduce a multilevel description generation method that improves the model’s ability to grasp the relationships among multiple targets and more effectively utilize common knowledge in generating text descriptions.
- Support resolution up to 1344 x 896. Surpassing the standard 448 x 448 resolution typically employed for LMMs, this significant increase in resolution augments the ability to discern and understand unnoticeable or tightly clustered objects and dense text.
- Enhanced general performance. We carried out testing across 18 diverse datasets, leading to a very competitive performance by our Monkey model in tasks such as Image Captioning, General Visual Question Answering, Scene Text-centric Visual Question Answering, and Document-oriented Visual Question Answering. In particular, during qualitative evaluations centered on dense text question answering, Monkey has shown promising results, comparing with GPT4V
conda create -n monkey python=3.9
conda activate monkey
git clone https://github.com/Yuliang-Liu/Monkey.git
cd ./Monkey
pip install -r requirements.txt
Demo is fast and easy to use. Simply uploading an image from your desktop or phone, or capture one directly. Demo_chat is also launched as an upgraded version of the original demo to deliver an enhanced interactive experience.
Before 14/11/2023, we have observed that for some random pictures Monkey can achieve more accurate results than GPT4V.
Before 31/1/2024, Monkey-chat achieved the fifth rank in the Multimodal Model category on OpenCompass.
We also provide the source code and the model weight for the original demo, allowing you to customize certain parameters for a more unique experience. The specific operations are as follows:
- Make sure you have configured the environment.
- You can choose to use the demo offline or online:
- Offline:
- Download the Model Weight.
- Modify
DEFAULT_CKPT_PATH="pathto/Monkey"
in thedemo.py
file to your model weight path. - Run the demo using the following command:
python demo.py
- Online:
- Run the demo and download model weights online with the following command:
python demo.py -c echo840/Monkey
The json file used for Monkey training can be downloaded at Link.
The data from our multi-level description generation method is now open-sourced and available for download at Link. Examples:
You can download train images from Train. Extraction code: 4hdh
You can download test images and jsonls from Test. Extraction code: 5h71
The images are from CC3M, COCO Caption, TextCaps, VQAV2, OKVQA, GQA, ScienceQA, VizWiz, TextVQA, OCRVQA, ESTVQA, STVQA, AI2D and DUE_Benchmark. When using the data, it is necessary to comply with the protocols of the original dataset.
We offer evaluation code for 14 Visual Question Answering (VQA) datasets in the evaluate_vqa.py
file, facilitating a quick verification of results. The specific operations are as follows:
- Make sure you have configured the environment.
- Modify
sys.path.append("pathto/Monkey")
to the project path. - Prepare the datasets required for evaluation.
- Run the evaluation code.
Take ESTVQA as an example:
- Prepare data according to the following directory structure:
├── data
| ├── estvqa
| ├── test_image
| ├── {image_path0}
| ├── {image_path1}
| ·
| ·
| ├── estvqa.jsonl
- Example of the format of each line of the annotated
.jsonl
file:
{"image": "data/estvqa/test_image/011364.jpg", "question": "What is this store?", "answer": "pizzeria", "question_id": 0}
- Modify the dictionary
ds_collections
:
ds_collections = {
'estvqa_test': {
'test': 'data/estvqa/estvqa.jsonl',
'metric': 'anls',
'max_new_tokens': 100,
},
...
}
- Run the following command:
bash eval/eval.sh 'EVAL_PTH' 'SAVE_NAME'
We also offer Monkey's model definition and training code, which you can explore above. You can execute the training code through executing finetune_ds_debug.sh
.
The json file used for Monkey training can be downloaded at Link.
ATTENTION: Specify the path to your training data, which should be a json file consisting of a list of conversations.
Inspired by Qwen-VL, we freeze the Large Language Model (LLM) and introduce LoRA into four linear layers "c_attn", "attn.c_proj", "w1", "w2"
for training. This step makes it possible to train Monkey using 8 NVIDIA 3090 GPUs. The specific implementation code is in modeling_qwen_nvdia3090.py
.
- Add LoRA: You need to replace the contents of
modeling_qwen.py
with the contents ofmodeling_qwen_nvdia3090.py
. - Freeze LLM: You need to freeze other modules except LoRA and Resampler modules in
finetune_multitask.py
.
from transformers import AutoModelForCausalLM, AutoTokenizer
checkpoint = "echo840/Monkey"
model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map='cuda', trust_remote_code=True).eval()
tokenizer = AutoTokenizer.from_pretrained(checkpoint, trust_remote_code=True)
tokenizer.padding_side = 'left'
tokenizer.pad_token_id = tokenizer.eod_id
img_path = ""
question = ""
query = f'<img>{img_path}</img> {question} Answer: ' #VQA
# query = f'<img>{img_path}</img> Generate the detailed caption in English: ' #detailed caption
input_ids = tokenizer(query, return_tensors='pt', padding='longest')
attention_mask = input_ids.attention_mask
input_ids = input_ids.input_ids
pred = model.generate(
input_ids=input_ids.cuda(),
attention_mask=attention_mask.cuda(),
do_sample=False,
num_beams=1,
max_new_tokens=512,
min_new_tokens=1,
length_penalty=1,
num_return_sequences=1,
output_hidden_states=True,
use_cache=True,
pad_token_id=tokenizer.eod_id,
eos_token_id=tokenizer.eod_id,
)
response = tokenizer.decode(pred[0][input_ids.size(1):].cpu(), skip_special_tokens=True).strip()
print(response)
Our model can accurately describe the details in the image.
Our model performs particularly well in dense text question answering tasks. For example, in the dense text of item labels, Monkey can accurately answer various information about the item, and its performance is very impressive compared to other LMMs including GPT4V.
Monkey also performs equally well in daily life scenes. It can complete various Q&A and caption tasks and describe various details in the image in detail, even the inconspicuous watermark.
We qualitatively compare with existing LMMs including GPT4V, Qwen-vl, etc, which shows inspiring results. One can have a try using the provided demo.
If you wish to refer to the baseline results published here, please use the following BibTeX entries:
@article{li2023monkey,
title={Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models},
author={Li, Zhang and Yang, Biao and Liu, Qiang and Ma, Zhiyin and Zhang, Shuo and Yang, Jingxu and Sun, Yabo and Liu, Yuliang and Bai, Xiang},
journal={arXiv preprint arXiv:2311.06607},
year={2023}
}
Qwen-VL: the codebase we built upon. Thanks for the authors of Qwen for providing the framework.
We welcome suggestions to help us improve the Monkey. For any query, please contact Dr. Yuliang Liu: [email protected]. If you find something interesting, please also feel free to share with us through email or open an issue. Thanks!