ChineseCLIPVisionModel in huggingface can used?

[ranck626]

ChineseCLIPVisionModel in huggingface can used?

[ranck626]

或者说，有什么办法能直只下载ChineseCLIP的image encoder。
我可以只针对image endcoder 进行微调

[ranck626]

好像是tokenizer的问题，换成AutoProcessor好使了

[ranck626]

from PIL import Image
import requests
from transformers import CLIPProcessor, ChineseCLIPVisionModel, AutoProcessor

model = ChineseCLIPVisionModel.from_pretrained("OFA-Sys/chinese-clip-vit-base-patch16")

processor = AutoProcessor.from_pretrained("OFA-Sys/chinese-clip-vit-base-patch16")

url = "https://clip-cn-beijing.oss-cn-beijing.aliyuncs.com/pokemon.jpeg"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(images=image, return_tensors="pt")

outputs = model(**inputs)
last_hidden_state = outputs.last_hidden_state
pooled_output = outputs.pooler_output # pooled CLS states
print(len(pooled_output))