dvlab-research / mgm Goto Github PK
View Code? Open in Web Editor NEWOfficial repo for "Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models"
License: Apache License 2.0
Official repo for "Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models"
License: Apache License 2.0
command used:
python -m minigemini.serve.cli
--model-path work_dirs/Mini-Gemini/Mini-Gemini-13B-HD
--image-file image_1.png,image_2.png
Nice work, from the scripts provided, seems all optimize_vision_tower
either aux
were not used.
Then I have some puzzels:
Simply run the demo code as following fails:
python -m minigemini.serve.cli \
--model-path YanweiLi/Mini-Gemini-2B \
--image-file ./images/demo_gen.png
error message:
File "XXXXX/envs/minigemini/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 784, in from_pretrained
raise ValueError(
ValueError: Tokenizer class GemmaTokenizer does not exist or is not currently imported.
Update to newest transformers
fix this problem
Hi, was wondering whether the stage2 script would be applicable for further FT from stage 2 with a small custom dataset for domain transfer? Or do we have to rewrite a separate script to do this?
Thanks and appreciate any help given!
Regards,
Adriel
There's quite a few gradio issues:
First two were "fixed" by just commenting them out, however the third issue prevent gradio working at all as it immediately crashes the instance with that error.
/xxx//minigemini/serve/gradio_web_server.py:351: UserWarning: `layout` parameter is deprecated, and it has no effect
chatbot = gr.Chatbot(
Traceback (most recent call last):
File "/usr/lib64/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib64/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/xxx/minigemini/serve/gradio_web_server.py", line 472, in <module>
demo = build_demo(args.embed, concurrency_count=args.concurrency_count)
File "/xxx/minigemini/serve/gradio_web_server.py", line 371, in build_demo
gr.Markdown(function_markdown)
/xxx//minigemini/serve/gradio_web_server.py:351: UserWarning: `layout` parameter is deprecated, and it has no effect
chatbot = gr.Chatbot(
Traceback (most recent call last):
File "/usr/lib64/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib64/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/xxx//minigemini/serve/gradio_web_server.py", line 472, in <module>
demo = build_demo(args.embed, concurrency_count=args.concurrency_count)
File "/xxx//minigemini/serve/gradio_web_server.py", line 394, in build_demo
regenerate_btn.click(
TypeError: EventListenerMethod.__call__() got an unexpected keyword argument 'concurrency_limit'
File "/xxx//venv/lib64/python3.10/site-packages/fastapi/encoders.py", line 287, in jsonable_encoder
encoded_value = jsonable_encoder(
File "/xxx//venv/lib64/python3.10/site-packages/fastapi/encoders.py", line 331, in jsonable_encoder
return jsonable_encoder(
File "/xxx//venv/lib64/python3.10/site-packages/fastapi/encoders.py", line 287, in jsonable_encoder
encoded_value = jsonable_encoder(
File "/xxx//venv/lib64/python3.10/site-packages/fastapi/encoders.py", line 331, in jsonable_encoder
return jsonable_encoder(
File "/xxx//venv/lib64/python3.10/site-packages/fastapi/encoders.py", line 287, in jsonable_encoder
encoded_value = jsonable_encoder(
File "/xxx//venv/lib64/python3.10/site-packages/fastapi/encoders.py", line 318, in jsonable_encoder
if isinstance(obj, classes_tuple):
File "/usr/lib64/python3.10/abc.py", line 119, in __instancecheck__
return _abc_instancecheck(cls, instance)
RecursionError: maximum recursion depth exceeded in comparison
model_zoo/OpenAI/clip-vit-large-patch14-336
and model_zoo/OpenAI/openclip-convnext-large-d-320-laion2B-s29B-b131K-ft-soup
from https://huggingface.co/YanweiLi/Mini-Gemini-7B/blob/main/config.json#L31
What is the exact model path of the huggingface?
Hi could you tell me how I can do batch inferencing?
I have multiple images and a different prompt for eacg image, so is there a way i can get the output in one go ?
Traceback (most recent call last):
File "/checkpoint/binary/train_package/minigemini/train/train_mem.py", line 14, in
train(attn_implementation="flash_attention_2")
File "/checkpoint/binary/train_package/minigemini/train/train.py", line 1262, in train
trainer.train()
File "/root/.local/lib/python3.8/site-packages/transformers/trainer.py", line 1624, in train
return inner_training_loop(
File "/root/.local/lib/python3.8/site-packages/transformers/trainer.py", line 1961, in _inner_training_loop
tr_loss_step = self.training_step(model, inputs)
File "/root/.local/lib/python3.8/site-packages/transformers/trainer.py", line 2902, in training_step
loss = self.compute_loss(model, inputs)
File "/root/.local/lib/python3.8/site-packages/transformers/trainer.py", line 2925, in compute_loss
outputs = model(**inputs)
File "/opt/conda/envs/python3.8.13/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/.local/lib/python3.8/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
ret_val = func(*args, **kwargs)
File "/root/.local/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 1833, in forward
loss = self.module(*inputs, **kwargs)
File "/opt/conda/envs/python3.8.13/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/checkpoint/binary/train_package/minigemini/model/language_model/mini_gemini_gemma.py", line 87, in forward
) = self.prepare_inputs_labels_for_multimodal(
File "/checkpoint/binary/train_package/minigemini/model/mini_gemini_arch.py", line 328, in prepare_inputs_labels_for_multimodal
image_features = self.encode_images(images, images_aux)
File "/checkpoint/binary/train_package/minigemini/model/mini_gemini_arch.py", line 255, in encode_images
image_aux_features_raw = self.get_model().get_vision_tower_aux()(images_aux).to(
AttributeError: 'list' object has no attribute 'to'
我发现教程只有从开源大模型的权重微调的,如果要在你们的检查点启动使用什么方法呢
Hello, everyone, I am using MiniGemini evaluation on an image by typing command:
python -m minigemini.serve.cli --model-path ./Mini-Gemini-2B/ --image-file replaced_with_path_to_image
then following OSError emerged:
Traceback (most recent call last):
File "home_path/anaconda3/envs/minigeimini/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "home_path/anaconda3/envs/minigeimini/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/data1/code_path/Projects/github/MiniGemini/minigemini/serve/cli.py", line 237, in <module>
main(args)
File "/data1/code_path/Projects/github/MiniGemini/minigemini/serve/cli.py", line 56, in main
tokenizer, model, image_processor, context_len = load_pretrained_model(args.model_path, args.model_base, model_name, args.load_8bit, args.load_4bit, device=args.device)
File "/data1/code_path/Projects/github/MiniGemini/minigemini/model/builder.py", line 112, in load_pretrained_model
vision_tower.load_model()
File "/data1/code_path/Projects/github/MiniGemini/minigemini/model/multimodal_encoder/clip_encoder.py", line 33, in load_model
self.vision_tower = CLIPVisionModel.from_pretrained(self.vision_tower_name)
File "home_path/anaconda3/envs/minigeimini/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3144, in from_pretrained
raise EnvironmentError(
OSError: Error no file named pytorch_model.bin, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory model_zoo/OpenAI/clip-vit-large-patch14-336.
Some internet friends share their solutions: Downloading the corresponding "xxx.index.json" file, but I can't find any "xxx.index.json" file of "clip-vit-large-patch14-336" on the huggingface website.
I think maybe the relative path leads to the probelm, so I replace it with absolute path, but the same problem is shot.
My environment:
OS: ubuntu 22.04 64 bit
python: 3.10.14
others: other libraries are installed according to MiniGemini's official installation guide.
Does anybody have a solution for that, I will be very grateful for you, thank u.
when i was running minigemini.serve.cli to inference an image(minigemini_34b_hd),i got this error:AttributeError: 'MiniGeminiLlamaModel' object has no attribute 'vlm_uni_query_projector'.
Should i put all the following pretrainded models to the specific directory?or just need some of them.
Looking forward to your reply
Thanks
Hi, I try evaluating MMMU_val of 2B by running this command
bash /home/user/minigemini/scripts/gemma/eval/mmmu.sh
but get error log as follows while evaluation of MMMU_val on 7B, 13B and 34B models could run successfully.
Traceback (most recent call last):
File "/home/user/MiniGemini/minigemini/eval/MMMU/eval/run_llava.py", line 207, in <module>
main()
File "/home/user/MiniGemini/minigemini/eval/MMMU/eval/run_llava.py", line 158, in main
response = call_model_engine(args, sample, model, tokenizer, processor)
File "/home/user/MiniGemini/minigemini/eval/MMMU/eval/utils/model_utils_ind.py", line 53, in call_llava_engine_df
output_ids = model.generate(
File "/home/user/anaconda2/envs/minigemini/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/user/MiniGemini/minigemini/model/language_model/mini_gemini_gemma.py", line 144, in generate
return super().generate(
File "/home/user/anaconda2/envs/minigemini/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/user/anaconda2/envs/minigemini/lib/python3.10/site-packages/transformers/generation/utils.py", line 1648, in generate
result = self._beam_sample(
File "/home/user/anaconda2/envs/minigemini/lib/python3.10/site-packages/transformers/generation/utils.py", line 3402, in _beam_sample
outputs = self(
File "/home/user/anaconda2/envs/minigemini/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/user/anaconda2/envs/minigemini/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/user/MiniGemini/minigemini/model/language_model/mini_gemini_gemma.py", line 97, in forward
return super().forward(
File "/home/user/anaconda2/envs/minigemini/lib/python3.10/site-packages/transformers/models/gemma/modeling_gemma.py", line 1105, in forward
outputs = self.model(
File "/home/user/anaconda2/envs/minigemini/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/user/anaconda2/envs/minigemini/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/user/anaconda2/envs/minigemini/lib/python3.10/site-packages/transformers/models/gemma/modeling_gemma.py", line 891, in forward
causal_mask = self._update_causal_mask(attention_mask, inputs_embeds, cache_position)
File "/home/user/anaconda2/envs/minigemini/lib/python3.10/site-packages/transformers/models/gemma/modeling_gemma.py", line 984, in _update_causal_mask
causal_mask *= torch.arange(target_length, device=device) > cache_position.reshape(-1, 1)
RuntimeError: The size of tensor a (701) must match the size of tensor b (0) at non-singleton dimension 0
scripts/gemma/eval/mmmu.sh: line 31: MMMU/answers/Mini-Gemini-2B/merge.jsonl: No such file or directory
scripts/gemma/eval/mmmu.sh: line 35: MMMU/answers/Mini-Gemini-2B/merge.jsonl: No such file or directory
Traceback (most recent call last):
File "/home/user/MiniGemini/minigemini/eval/MMMU/eval/eval.py", line 31, in <module>
main()
File "/home/user/MiniGemini/minigemini/eval/MMMU/eval/eval.py", line 17, in main
out_samples = [json.loads(line) for line in open(args.result_file)]
FileNotFoundError: [Errno 2] No such file or directory: 'MMMU/answers/Mini-Gemini-2B/merge.jsonl'
Demo page is not working also domain is an ip adress.
Dear author:
Thanks for publishing the mini gemini paper. As Gemini 1.5 support up to one hour long video input for 1fps sampling, I wonder how to adopt your framework to support long video training and inference? thank you.
Thanks for sharing your great work. I have a small question about the attention process in the patch info mining module. I found that there is no positional embedding for the high-resolution tokens to indicate their position in the corresponding patch. Do you think adding positional embedding here would be helpful and have you tried this?
Currently, there is an option to using two image grid to double the input, but this introduce sort of heavy compute.
I want just make the input resolution slightly large, say from 336 -> 448, while keep the Convnext input resolution same (although I think currenctly it should larger if base vision tower are larger).
Could that be possible? Can u give me some advisor how to adopt it?
I have tried both the model worker and the cli, and both when passed 4bit loading just fail with the error message:
Loading pretrained weights (convnext_large_d_320).
Traceback (most recent call last):
File "/usr/lib64/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib64/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/xxx/minigemini/serve/cli.py", line 234, in <module>
main(args)
File "/xxx/minigemini/serve/cli.py", line 56, in main
tokenizer, model, image_processor, context_len = load_pretrained_model(args.model_path, args.model_base, model_name, args.load_8bit, args.load_4bit, device=args.device)
File "/xxx/minigemini/model/builder.py", line 124, in load_pretrained_model
model.get_model().initialize_uni_modules(model.config, for_eval=True)
File "/xxx/minigemini/model/mini_gemini_arch.py", line 213, in initialize_uni_modules
get_w(projector_weights, 'vision_tower.vision_tower', self.vision_tower, 'vision_tower')
File "/xxx/minigemini/model/mini_gemini_arch.py", line 209, in get_w
getattr(main_module, sub_module).to(device=device_type, dtype=weight_type)
File "/xxx/venv/lib64/python3.10/site-packages/transformers/modeling_utils.py", line 2460, in to
return super().to(*args, **kwargs)
File "/xxx/venv/lib64/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in to
raise TypeError('nn.Module.to only accepts floating point or complex '
TypeError: nn.Module.to only accepts floating point or complex dtypes, but got desired dtype=torch.uint8
loading with 8bit works, but OOMs on my hardware. (24+24 vram)
Hi, Nice work! but when i run the example code , i have encounter some problems,when i run python -m minigemini.serve.cli --model-path YanweiLi/Mini-Gemini-13B-HD --image-file Woman.jpg
,i got the error ValueError: Not find vision tower: model_zoo/OpenAI/clip-vit-large-patch14-336 ,how can i solve this problem thank u!
Congratulations on your great work and solid performance! However, we notice that your core idea and design are highly similar to our previsiou work LLaVA-HR, especially the dual visual pathways for MLLMs. We would like to see simple clarifications and discussions with our work in your paper. Thank you!
minigemini_instruction.json中有条目的image为llava/LLaVA-Pretrain/images/00013/000133305.jpg,有很多llava的图片,但是readme中没有提到要用llava的图片,是你们疏忽了吗
Hi, am alreaady tried using vit336 and convnext + Qwen LLM, which is great, and really got a good performance.
But when I try using another CLIP vit model with input size is 378, rest things are same (include traning data) the result are extremly poor.
To precisely:
Any reason for this? This is really weired, better and larger ViT got bad results.
Hi, the pretrained data used allava images both from laion nad vfan.
But the laion part image names are totally different from ALLava's images format.
I tried to found:
465440.jpeg
320609
they all used in minigemini_pretrain.json but can not be found in ALLava images folder.
ls -f images | grep 465440
46544031.jpeg
(base) ➜ allava_laion git:(main) ✗ ls -f images | grep 320609
132060956.jpeg
43206091.jpeg
why is that?
Hi, I follow the instructions to prepare the data and model and get the following error when evaluating the Mini-Gemini-2B.
AttributeError: 'OpenCLIPVisionTower' object has no attribute 'vision_stem'
How to reproduce:
export HF_ENDPOINT=https://hf-mirror.com
python -m minigemini.serve.cli \
--model-path ./Mini-Gemini/Mini-Gemini-2B \
--image-file ./images/demo_gen.png \
--debug \
Results:
The model does not generate anything since the stop_str
is set to be EMPTY STRING ''
. After I fixed this, I also found the chat history is not preserved in the prompt which results the multi-turn conversation results to be unexpected.
follew https://huggingface.co/datasets/laion/gpt4v-dataset. There are about 400-500 pictures that cannot be downloaded. Could you provide the complete version?
作者好,非常感谢你的开源。我有两个疑问想咨询下
1. convnext 的 drop path 问题
convnext 的 drop path 是 0.1,虽然你在训练时候设置的是不训练,但是这个路径还是会执行的。原则上在训练时候,convnext 要设置为 eval 模式吧? 我没有找到相关代码,有点奇怪,想了解下
2. 代码鲁棒性问题
如果是 clip+convnext 组合方式训练没有任何问题。但是我想尝试 siglip,发现在仅仅将 clip 换成 siglip 后(均值和方差也是两套),模型在 iter=2 时候就会出现 nan。注释如下
# token attention
embed_query = self.vlm_uni_query_projector(images)
embed_aux = self.vlm_uni_aux_projector(images_aux)
embed_value = self.vlm_uni_val_projector(images_aux)
# TODO siglip+convnext 在第一次 forward 后正常,但是 embed_att 会出现 nan
# TODO 导致第二次迭代时候 embed_value 会出现 nan,无法训练
# TODO 怀疑是特征不匹配,即使全部转换为 fp32 也会出现 nan, 需要进一步排查
embed_att = embed_query[:, :, None] @ (embed_aux.transpose(-1, -2) / (embed_aux.shape[-1] ** 0.5))
# print('=xxxx=', torch.any(torch.isnan(embed_query)).item(),
# torch.any(torch.isnan(embed_aux)).item(),
# torch.any(torch.isnan(embed_value)).item(),
# torch.any(torch.isnan(embed_att)).item())
embed_att = embed_att.nan_to_num()
embed_feat = (embed_att.softmax(-1) @ embed_value).mean(2)
# print('=xxcccxx=', torch.any(torch.isnan(embed_feat)).item())
image_features = images + embed_feat
return image_features
不知道作者对于这个问题是咋看的。有一个细节不一样:因为 siglip 输入是 384x384 的,输出尺度是 27x27,所以我必须要将 convnext 输入分辨率设置为 864, 这样才可以保证两者空间完全对齐。
期待您的回复。
针对llama2-7b模型,max token length为2048,按照stage2 的训练参数,一旦设置IMAGE_GRID=2和IMAGE_GLOBAL为True,
image_features = torch.cat([image_feat_global, image_features], dim=1)
image_aux_features = torch.cat([image_aux_feat_global, image_aux_features], dim=1)
两行最终得到的图像特征token数就变成了2880,这里token数不是超了吗?如果理解有误敬请指正。
Hi, I found currently the image generating way hard to make input image as reference:
any thought about it?
BTW, I found the pretrain stage of this loss is very high:
{'loss': 2.573, 'learning_rate': 0.0007701925673852566, 'epoch': 0.34}
{'loss': 2.8217, 'learning_rate': 0.0007698799612970509, 'epoch': 0.34}
{'loss': 2.5646, 'learning_rate': 0.0007695672062744539, 'epoch': 0.34}
{'loss': 2.6857, 'learning_rate': 0.0007692543024900611, 'epoch': 0.34}
{'loss': 2.6944, 'learning_rate': 0.0007689412501165496, 'epoch': 0.34}
{'loss': 2.6418, 'learning_rate': 0.0007686280493266786, 'epoch': 0.34}
{'loss': 2.6801, 'learning_rate': 0.0007683147002932893, 'epoch': 0.34}
{'loss': 2.7245, 'learning_rate': 0.0007680012031893049, 'epoch': 0.34}
{'loss': 2.6275, 'learning_rate': 0.0007676875581877296, 'epoch': 0.34}
In section 3.1 of the paper, Should "N`=H/4*W/4" be adjusted to "N`*N`=H/4*W/4"?
Hi, I'm trying to use MiniGemini outside of the demo environment, but am running into the following error when calling model.generate():
File "/app/backend/minigemini.py", line 104, in chat_with_images
output_ids = self.model.generate(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/app/MiniGemini/minigemini/model/language_model/mini_gemini_mixtral.py", line 142, in generate
return super().generate(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/transformers/generation/utils.py", line 1527, in generate
result = self._greedy_search(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/transformers/generation/utils.py", line 2411, in _greedy_search
outputs = self(
^^^^^
File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/accelerate/hooks.py", line 166, in new_forward
output = module._old_forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: MiniGeminiMixtralForCausalLM.forward() got an unexpected keyword argument 'output_router_logits'
I have MiniGemini-2B working, but Mixtral is still giving me some trouble. The only relevant reference I can find is: https://www.opensourceagenda.com/projects/transformers/versions, which mentions output_router_logits was removed in transformers 4.39.0.
I see a similar error with Mini-Gemini-7B:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/accelerate/hooks.py", line 166, in new_forward
output = module._old_forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: MiniGeminiLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position'
I'm using:
transformers 4.39.3
accelerate-0.29.1
torch 2.2.2
torchvision 0.17.2
I confess, the dependencies for MiniGemini are a real challenge to integrate, but I hope it can get resolved.
Dear author:
Thanks for your interesting work.
But I am still confused about following questions:
Looking forward to your reply~
Thanks!!
when performing finetune on top of the minigemini-8x7b-HD model using the following config:
#!/bin/bash
PRETRAIN_NAME=Mini-Gemini-8x7B-Pretrain
FINETUNE_NAME=Mini-Gemini-8x7B-HD
AUX_SIZE=1536
IMAGE_GRID=2
IMAGE_GLOBAL=True
LR_MULTI="model.mm_projector:2,model.vlm_uni:2"
# delete --hostfile hostfile_4 and change --per_device_train_batch_size if trained on single machine
deepspeed minigemini/train/train_mem.py \
--deepspeed ./scripts/zero3_offload.json \
--model_name_or_path Mini-Gemini-8x7B-HD \
--version mistral_instruct \
--data_path data/minigemini_instruction.json \
--image_folder data/figures \
--vision_tower model_zoo/OpenAI/clip-vit-large-patch14-336 \
--vision_tower_aux model_zoo/OpenAI/openclip-convnext-large-d-320-laion2B-s29B-b131K-ft-soup \
--image_grid $IMAGE_GRID \
--image_global $IMAGE_GLOBAL \
--mm_projector_type mlp2x_gelu \
--mm_vision_select_layer -2 \
--mm_use_im_start_end False \
--mm_use_im_patch_token False \
--image_aspect_ratio pad \
--group_by_modality_length True \
--image_size_aux $AUX_SIZE \
--bf16 True \
--output_dir ./work_dirs/experiment-2/$FINETUNE_NAME \
--num_train_epochs 5 \
--per_device_train_batch_size 8 \
--gradient_accumulation_steps 2 \
--save_strategy "steps" \
--save_steps 2 \
--save_total_limit 1 \
--learning_rate 8e-4 \
--lr_multi $LR_MULTI \
--weight_decay 0. \
--warmup_ratio 0.05 \
--lr_scheduler_type "cosine" \
--logging_steps 1 \
--tf32 True \
--model_max_length 8192 \
--gradient_checkpointing True \
--dataloader_num_workers 64 \
--lazy_preprocess True \
--report_to wandb
The final model only has 4 safetensors files instead of 20, why is that ? thanks.
Thank you for your great work!
Could you clarify where the data about Generation-related Instructions in Section 3.3 is located?
Dear author:
Thanks for your interesting work.
When I run the following command with the input What is unusual about this image?
, no image generated in the output:
python -m minigemini.serve.cli \
--model-path work_dirs/Mini-Gemini/Mini-Gemini-2B \
--image-file examples/extreme_ironing.jpg \
--gen
I wonder if the Mini-Gemini-2B model does not have the ability to generate images?
And if fine-tuning is needed, which datasets should be used to make the model output <h> ... </h>
?
Thanks!!
Hi, am using Qwen2 4b as LLM training the model, the template solving way I previous tried on LLaVa works OK, and no warnings appeared.
But when pretrain on minigemini, got unexpected loss result:
{'loss': 1.9597, 'learning_rate': 0.0007436649460805199, 'epoch': 0.36}
{'loss': 1.8565, 'learning_rate': 0.0007435934693368483, 'epoch': 0.36}
{'loss': 2.0595, 'learning_rate': 0.0007435219860653267, 'epoch': 0.36}
{'loss': 1.9304, 'learning_rate': 0.0007434504962678705, 'epoch': 0.36}
{'loss': 1.9705, 'learning_rate': 0.0007433789999463957, 'epoch': 0.36}
{'loss': 1.9536, 'learning_rate': 0.000743307497102818, 'epoch': 0.36}
{'loss': 1.9405, 'learning_rate': 0.0007432359877390538, 'epoch': 0.36}
{'loss': 1.8501, 'learning_rate': 0.0007431644718570192, 'epoch': 0.36}
{'loss': 1.8722, 'learning_rate': 0.000743092949458631, 'epoch': 0.36}
{'loss': 1.6984, 'learning_rate': 0.0007430214205458056, 'epoch': 0.36}
{'loss': 1.8309, 'learning_rate': 0.0007429498851204598, 'epoch': 0.36}
{'loss': 1.8681, 'learning_rate': 0.0007428783431845109, 'epoch': 0.36}
{'loss': 1.9383, 'learning_rate': 0.0007428067947398757, 'epoch': 0.36}
{'loss': 1.8059, 'learning_rate': 0.000742735239788472, 'epoch': 0.36}
{'loss': 1.8016, 'learning_rate': 0.0007426636783322172, 'epoch': 0.36}
{'loss': 2.0427, 'learning_rate': 0.0007425921103730288, 'epoch': 0.36}
{'loss': 1.899, 'learning_rate': 0.0007425205359128248, 'epoch': 0.36}
Looks like stucked at ~1.8, no longer decrease, what could be the reason for this? (the data all same)
(I tried using both chat template in pretrain and finetune, same as mini does)
When using clip.py for inference, I encountered the following error. How should I solve it?
File "/mnt/bn/codegen-finetune/projects/MiniGemini/minigemini/model/mini_gemini_arch.py", line 255, in encode_images
if images_aux is not None:
File "/root/miniconda3/envs/vl_model/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/envs/vl_model/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/mnt/bn/codegen-finetune/projects/MiniGemini/minigemini/model/multimodal_encoder/clip_encoder.py", line 58, in forward
image_features = self.image_forward(images)
File "/mnt/bn/codegen-finetune/projects/MiniGemini/minigemini/model/multimodal_encoder/clip_encoder.py", line 50, in image_forward
image_forward_outs = self.vision_tower(images.to(device=self.device, dtype=self.dtype), output_hidden_states=True)
File "/root/miniconda3/envs/vl_model/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/envs/vl_model/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 917, in forward
return self.vision_model(
File "/root/miniconda3/envs/vl_model/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/envs/vl_model/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 841, in forward
hidden_states = self.embeddings(pixel_values)
File "/root/miniconda3/envs/vl_model/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/envs/vl_model/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 187, in forward
embeddings = embeddings + self.position_embedding(self.position_ids)
RuntimeError: The size of tensor a (2305) must match the size of tensor b (50) at non-singleton dimension 1
command: python -m minigemini.serve.cli --model-path Mini-Gemini-34B-HD --image-file test_image.png
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/paperspace/MiniGemini/minigemini/serve/cli.py", line 241, in <module>
main(args)
File "/home/paperspace/MiniGemini/minigemini/serve/cli.py", line 200, in main
output_ids = model.generate(
File "/home/paperspace/MiniGemini/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/paperspace/MiniGemini/minigemini/model/language_model/mini_gemini_llama.py", line 183, in generate
return super().generate(
File "/home/paperspace/MiniGemini/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/paperspace/MiniGemini/venv/lib/python3.10/site-packages/transformers/generation/utils.py", line 1575, in generate
result = self._sample(
File "/home/paperspace/MiniGemini/venv/lib/python3.10/site-packages/transformers/generation/utils.py", line 2697, in _sample
outputs = self(
File "/home/paperspace/MiniGemini/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/paperspace/MiniGemini/venv/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
TypeError: MiniGeminiLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position'```
Hi I noticed that when doing batch inference with a static prompt like ‘Describe the image’ the model gives wrong output like ‘in detail’, like it is just doing sentence completion. Whereas if I try a more descriptive prompt, where I tell minigemini that it is a ‘prompt generator’ then it goes into sentence completion mode and gives me an okayish response.
However, I have the original image description also, so I tried adding those in the prompt and then asking the model to describe the image, given the information about the image. This works perfectly fine when I just use 1 image at a time.
But when doing batch processing I get completely garbage outputs. To do batch processing, I use padding to get the prompts to the same shape. But this gives me completely garbage output.
I do this by changing line 44 in / MiniGemini/minigemini/mm_utils.py
to
tokenizer(chunk, padding='max_length', max_length=max_len).input_ids for chunk in prompt.split('')
Could you give me any advice on how to do this effectively?
I downloaded sharegpt4v used fientuneing part data, but always got image not found, am using finetune stage.
Does finetune used sharegpt4v pretrain data?
sharegpt4v finetune just used very little data from SAM.
Shall we download the. sam_000000 - 0000050 whole 500GB images for it?
TypeError: pad_sequence(): argument 'padding_value' (position 3) must be float, not NoneType
Am using Qwen as LLM, got above error, what could be the reason? I have checked:
if tokenizer.pad_token_id == None:
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.unk_token
tokenizer.pad_token_id = tokenizer.encode(tokenizer.pad_token)
didnt make it work, any help could be possible?
基于提供的1 stage预训练模型Mini-Gemini-7B-Pretrain和默认的配置scripts/llama/train/stage_2_full_v7b_672_hr_1536.sh,单独使用minigemini_generation_pure_text.json 训练2 stage,loss在训练过程中(10个step后)突然降至0,在后续训练过程中持续为0,
@yanwei-li @yukang2017 @wcy1122 打印出中间变量,看到shift_logits为Nan导致loss出现为Nan的情况,这个正常吗?
There are some image sources missing in the preparation stage:
sam/images/: 19982
wikiart/images/: 500
share_textvqa/images/: 500
web-celebrity/images/: 498
web-landmark/images/: 500
Could you help add these datasets?
Hi, as previous issue raised, the data all wrong for any data from ALLaVA, they have changed the image names.
What even worse, even tough using url mapping the image names with latest ALLaVa's caption json, still some images were totally can not be found, in latest ALLaVa.
for example:
cat ALLaVA-Caption-LAION-4V.json | grep 'https://slideplayer.it/slide/553401/1/images/40/Delayed+relaxation+filling+pattern.jpg' -C 9
this url existed in minigemini but gone at latest ALLava images
Therefore, please help fix the data first, it emergency. Stops anyone want to reveal the minigemini training and scores.
Hi, the begin loss is a little high, does it normal?
{'loss': 6.7177, 'learning_rate': 0.0, 'epoch': 0.0}
{'loss': 6.7738, 'learning_rate': 0.0, 'epoch': 0.0}
{'loss': 6.801, 'learning_rate': 0.0, 'epoch': 0.0}
{'loss': 6.5226, 'learning_rate': 0.0, 'epoch': 0.0}
{'loss': 6.813, 'learning_rate': 7.633587786259541e-06, 'epoch': 0.0}
{'loss': 6.6355, 'learning_rate': 1.5267175572519083e-05, 'epoch': 0.0}
{'loss': 6.3212, 'learning_rate': 2.2900763358778628e-05, 'epoch': 0.0}
{'loss': 6.1449, 'learning_rate': 3.0534351145038166e-05, 'epoch': 0.0}
{'loss': 5.8368, 'learning_rate': 3.816793893129771e-05, 'epoch': 0.0}
{'loss': 5.9083, 'learning_rate': 4.5801526717557256e-05, 'epoch': 0.0}
{'loss': 5.7246, 'learning_rate': 5.3435114503816794e-05, 'epoch': 0.0}
{'loss': 5.6311, 'learning_rate': 6.106870229007633e-05, 'epoch': 0.0}
{'loss': 5.7906, 'learning_rate': 6.870229007633588e-05, 'epoch': 0.0}
{'loss': 5.6311, 'learning_rate': 7.633587786259542e-05, 'epoch': 0.0}
{'loss': 5.1357, 'learning_rate': 8.396946564885496e-05, 'epoch': 0.0}
{'loss': 4.9726, 'learning_rate': 9.160305343511451e-05, 'epoch': 0.0}
{'loss': 4.7747, 'learning_rate': 9.923664122137405e-05, 'epoch': 0.0}
{'loss': 4.8886, 'learning_rate': 0.00010687022900763359, 'epoch': 0.0}
{'loss': 4.7224, 'learning_rate': 0.00011450381679389313, 'epoch': 0.0}
{'loss': 4.4193, 'learning_rate': 0.00012213740458015266, 'epoch': 0.0}
{'loss': 4.5041, 'learning_rate': 0.00012977099236641222, 'epoch': 0.0}
{'loss': 4.3678, 'learning_rate': 0.00013740458015267177, 'epoch': 0.01}
{'loss': 4.3701, 'learning_rate': 0.0001450381679389313, 'epoch': 0.01}
{'loss': 4.505, 'learning_rate': 0.00015267175572519084, 'epoch': 0.01}
{'loss': 4.5022, 'learning_rate': 0.00016030534351145037, 'epoch': 0.01}
{'loss': 4.2254, 'learning_rate': 0.00016793893129770992, 'epoch': 0.01}
{'loss': 4.339, 'learning_rate': 0.00017557251908396944, 'epoch': 0.01}
{'loss': 4.5537, 'learning_rate': 0.00018320610687022902, 'epoch': 0.01}
{'loss': 4.2307, 'learning_rate': 0.00019083969465648857, 'epoch': 0.01}
How's the loss curve on your side? (am using a 4B llm model
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.