tiger-ai-lab / mantis Goto Github PK
View Code? Open in Web Editor NEWOfficial code for Paper "Mantis: Multi-Image Instruction Tuning"
Home Page: https://tiger-ai-lab.github.io/Mantis/
License: Apache License 2.0
Official code for Paper "Mantis: Multi-Image Instruction Tuning"
Home Page: https://tiger-ai-lab.github.io/Mantis/
License: Apache License 2.0
Thanks for your nice work.
I am training mantis-8b-siglip-llama3 with lora. However, it appears that there is a non-negligible performance drop when using lora finetuning. Two evaluation benchmark results are given here,
- | Mantis-Eval | NLVR2 |
---|---|---|
report in paper | 59.45 | 87.43 |
lora results | 51.61 | 82.06 |
For pretraining and fine-tuning, I did not modify any training-specific hyperparameter.
Perhaps it is caused by some errors when loading lora model during inference. I notice that you did lora finetuning in ablations. Can authors help on this?
Thanks.
Hi,
Thank you for your work on this library. I'd like to know if there's any planned support for Idefics3, this model seems to be better than Idefics2, for visual reasoning https://huggingface.co/HuggingFaceM4/Idefics3-8B-Llama3
Regards,
The FUYU model implementation currently lacks support for multi-GPU setups. This issue has already been addressed and fixed in Huggingface's transformers repository.
Here's the link to the PR in the Hugging Face repository: huggingface/transformers#29880
Why not use q-bench2-a1-pair-test.json for q-bench2?
I am trying to add Mantis to the supported model list in VLLM or Sglang
e7497f3c073c:6956:9753 [1] NCCL INFO comm 0x761d313cac20 rank 1 nranks 4 cudaDev 1 nvmlDev 1 busId 4000 commId 0x273b00670b73d30e - Init COMPLETE
e7497f3c073c:6958:9750 [3] NCCL INFO comm 0x75ebdb2446d0 rank 3 nranks 4 cudaDev 3 nvmlDev 3 busId 6000 commId 0x273b00670b73d30e - Init COMPLETE
e7497f3c073c:6955:9751 [0] NCCL INFO comm 0x73d7493caf90 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId 3000 commId 0x273b00670b73d30e - Init COMPLETE
e7497f3c073c:6957:9752 [2] NCCL INFO comm 0x7ed4193cafe0 rank 2 nranks 4 cudaDev 2 nvmlDev 2 busId 5000 commId 0x273b00670b73d30e - Init COMPLETE
W0715 08:53:40.701000 127238712194880 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 6955 closing signal SIGTERM
W0715 08:53:40.704000 127238712194880 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 6956 closing signal SIGTERM
W0715 08:53:40.704000 127238712194880 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 6958 closing signal SIGTERM
E0715 08:53:42.146000 127238712194880 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: -9) local_rank: 2 (pid: 6957) of binary: /workspace/miniconda3/envs/Mantis/bin/python
Hello! Thanks for open-sourcing the code :)
I'm playing a bit with the code and trying to replicate the pre-training and fine-tuning results, but I'm encountering many problems.
The problems I'm having can be represented in two groups:
The LLaVA pre-training script mantis/train/scripts/pretrain_mllava.sh
uses the online hugging face dataset, but the majority of the data seems not present. In the TIGER-Lab/llava-data
dataset, the llava-pretrain
presents only 278670 images in 660 subfolders while the llava-finetune
has only 348679 images across the folders "coco, gqa, ocr_vqa, textvqa, vg". Regarding the llava-finetune, for the majority of the images in the CC3M I'm getting:
[PosixPath('.cache/huggingface/datasets/TIGER-Lab___llava-data/llava_pretrain/0.0.0/73ecf20404b651edea601a8b868b2c818f3a7abf5fda24c2d8b0993118c3aea4/train_images/images/00266/002667434.jpg')] does not exist
Error at 100759
Regarding LLaVA fine-tuning, the script raise an error of unsupported multi-images input ("This method only supports a single input, but get 4 inputs"), which seems to come from the LLaVA processor method _right_pad_inputs_with_attention_mask
. Why doesn't LLaVA support multi-input images? This error is raised both with llava
and mllava
for mllava_type
in the setting, and using the CLIP version.
The idefics2 script seems to work with the default settings, downloading the data and starting the run (I'm using the train_idefics2.sh
script, with default qlora finetuning). However, I have noticed that during training the memory usage of some GPUs (in my case the 0 and 2) spikes many times, going from 30% to 89%. I have tried to increase the per_device_train_batch_size
to more than 1 (I have 40GB a100 GPU) but I'm getting strange errors that I didn't get in the original settings (the only change is per_device_train_batch_size=4
in row 139):
Traceback (most recent call last):
File "Mantis/mantis/train/train_idefics2.py", line 255, in <module>
main(training_args, data_args, model_args)
File "Mantis/mantis/train/train_idefics2.py", line 229, in main
trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint)
File "miniconda3/envs/mantis/lib/python3.10/site-packages/transformers/trainer.py", line 1876, in train
return inner_training_loop(
File "miniconda3/envs/mantis/lib/python3.10/site-packages/transformers/trainer.py", line 2178, in _inner_training_loop
for step, inputs in enumerate(epoch_iterator):
File "miniconda3/envs/mantis/lib/python3.10/site-packages/accelerate/data_loader.py", line 454, in __iter__
current_batch = next(dataloader_iter)
File ".local/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 631, in __next__
data = self._next_data()
File ".local/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1346, in _next_data
return self._process_data(data)
File ".local/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1372, in _process_data
data.reraise()
File ".local/lib/python3.10/site-packages/torch/_utils.py", line 722, in reraise
raise exception
AssertionError: Caught AssertionError in DataLoader worker process 0.
Original Traceback (most recent call last):
File ".local/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
data = fetcher.fetch(index)
File ".local/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 54, in fetch
return self.collate_fn(data)
File "Mantis/mantis/train/data.py", line 753, in __call__
batch_encoding = self._right_pad_inputs_with_attention_mask(model_inputs=batch)
File "Mantis/mantis/train/data.py", line 741, in _right_pad_inputs_with_attention_mask
assert len(model_inputs) == 1, "This method only supports a single input, but get {} inputs".format(len(model_inputs))
AssertionError: This method only supports a single input, but get 4 inputs
This error seems related to the one I had with LLaVA which makes me thinking that probably we should modify/create a _right_pad_inputs_with_attention_mask
method for the idefict2 processor, as well as for llava, similar to the one made for MFuyu ?
Please, give some advice
Excellent work! BTW, Does the model support Chinese?
Nice work! Thanks for contribution.
We are carrying out instruction tuning experiments with Mantis-8B-siglip-llama3
. The pretraining and instruction finetuning with lora work fine, except for full param finetuning. The warning below came up and finetuning got stuck. I put this here for others reference.
Invalidate trace cache @ step 344: expected module 345, but got module 1
Referring to the issue, this might be due to how accelerate
or deepspeed
is installed. Noticing that there is no version specifications in setup.py
from this repo, may we ask the exact versions you use for fine-tuning, for some dependencies like torch
, accelerate
or deepspeed
?
Thanks in advance.
Hi,
Thank you for open-sourcing this great work. I appreciate the team's efforts in putting this together.
I have a question about the evaluation criteria in the mantis-eval, "short-answer" question specifically. It looks like the correctness of "short-answer" is judged by exact match between model's output and the reference answer, without further parsing(see the edit below). But the prompt template for this type of question also instructs the model to output both analysis and final answer.
In this case, I noticed that a model would give the correct answer (for example, "Yes") followed by some reasoning, but such an answer wouldn't be counted as correct because of how the exact match works.
Could you help me understand why it's written like this? Does it make sense to improve the matching rule? Thanks.
Edit:
I just saw that there is parsing on the model's output that only takes the outputs after "Final Answer: ". This makes much more sense. However, I noticed that sometimes a model would answer correctly but with more than one word.
Do you think it makes sense to loosen the matching criteria? Alternatively, I think it also makes sense to make the instruction more clear in the prompt template, for example, by adding one more sentence like "Answer the question in a single word or phrase."
Can 4 32G V100s support its training with Lora or Qlora?
Hi, wanna ask ,does mantis used image separator between images sending to LLM? From i can tell, llava doesn't have it and the data used in Mantis doesn't provide a str for separator too.
Also, do u think which way is better? If consider video frames input as well
I'm working on fine-tuning Idefics2 with multiple images in instruction
I follow this script for full fine-tuning: https://github.com/TIGER-AI-Lab/Mantis/blob/89d34077bd87b66eaadc13117add553e3a3d4c0b/mantis/train/scripts/train_idefics2_full.sh
Here is the command
NCCL_DEBUG=WARN accelerate launch --config_file=./accelerate_configs/accelerate_config_zero3.yaml \
--machine_rank 0 --main_process_ip 10.29.35.44 --main_process_port 12956 \
--num_machines 1 --num_processes 8 \
train_idefics2.py \
--model_name_or_path HuggingFaceM4/idefics2-8b \
--data_config_file custom_data_config.yaml \
--data_format chat \
--run_name 240523_idefics2_mantis \
--output_dir 240523_idefics2_mantis \
--num_train_epochs 3 \
--per_device_train_batch_size 1 \
--per_device_eval_batch_size 1 \
--gradient_accumulation_steps 2 \
--evaluation_strategy "steps" \
--save_strategy "steps" \
--save_steps 200 \
--eval_steps 200 \
--save_total_limit 5 \
--learning_rate 2e-5 \
--weight_decay 0.01 \
--warmup_ratio 0.03 \
--lr_scheduler_type "cosine" \
--logging_steps 10 \
--gradient_checkpointing True \
--dataloader_num_workers 5 \
--report_to wandb \
--do_train \
--lora_enabled False \
--qlora_enabled False \
--dora_enabled False \
--max_seq_len 512 \
--fp16 \
--attn_implementation eager
Error i got is
[rank0]: result = forward_call(*args, **kwargs)
[rank0]: File "/home/user/Mantis/mantis/models/idefics2/modeling_idefics2.py", line 1677, in forward
[rank0]: inputs_embeds = self.inputs_merger(
[rank0]: File "/home/user/Mantis/mantis/models/idefics2/modeling_idefics2.py", line 1564, in inputs_merger
[rank0]: new_inputs_embeds[special_image_token_mask] = reshaped_image_hidden_states
[rank0]: RuntimeError: shape mismatch: value tensor of shape [256, 4096] cannot be broadcast to indexing result of shape [192, 4096]
Any suggestions how to fix it?
Thanks in advance
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.