Weight size mismatch when load_pretrain for llava_onevision (0.5b model)

[zeal-up]

Weight size mismatch when load_pretrain for llava_onevision (0.5b model)

[SorryMaker2022]

Loading vision tower: /home/docker/.cache/huggingface/manually_hub/siglip-so400m-patch14-384

It seems that you're using your vision tower from a local path, which will then be loaded as a clip vision tower, not a siglip vision tower.

LLaVA-NeXT/llava/model/multimodal_encoder/builder.py

Lines 15 to 23 in e988491

	is_absolute_path_exists = os.path.exists(vision_tower)
	use_s2 = getattr(vision_tower_cfg, "s2", False)
	if is_absolute_path_exists or vision_tower.startswith("openai") or vision_tower.startswith("laion") or "ShareGPT4V" in vision_tower:
	if use_s2:
	return CLIPVisionTowerS2(vision_tower, args=vision_tower_cfg, **kwargs)
	else:
	return CLIPVisionTower(vision_tower, args=vision_tower_cfg, **kwargs)
	elif "siglip" in vision_tower:
	return SigLipVisionTower(vision_tower, vision_tower_cfg=vision_tower_cfg, **kwargs)

Exchanging the if and elif predicates may solve the problem. But note that this modification may have potential side-effect to other vision towers.

[Luodian]

#144

Can you check the discussion and update code to try again? It's weird you are loading with a CLIP model's architecture.

[jxgu1016]

same issue here

[JinhuiYE]

this one can work well. but I am unsure why returning with output_attentions will cause NAN and output ["!"]. there are not issue for 0.5B and 72B.

                    return_dict_in_generate = model.generate(
                        input_ids,
                        images=image_tensors,
                        attention_mask=attention_masks,
                        pad_token_id=tokenizer.pad_token_id,
                        use_cache=True,
                        # output_attentions=True, return_dict_in_generate=True,
                        **gen_kwargs
                        )