staghado/vit.cpp

Inference Vision Transformer (ViT) in plain C/C++ with ggml

vit.cpp's Issues

/home/shared/vit.cpp/tests/benchmark.cpp:4:10: fatal error: nlohmann/json.hpp: No such file or directory

did you miss the dependency nlohmann/json.hpp?

Failed to load vit_large_patch14_clip_336.openai

converter

python convert-pth-to-ggml.py --model_name vit_large_patch14_clip_336.openai --ftype 1

runner

./bin/vit -t 4 -m ../ggml-model-f16.gguf -i ../assets/tench.jpg 

Any idea?

Implementation of vision models

Hi @staghado thank you for publishing this amazing implementation of ViT using ggml.

I was thinking of doing a similar implementation for another transformer-based model by following your codebase. However, I could not find good documentation of ggml to know about existing functionalities and how to use them. Also, certain concepts (e.g., https://github.com/staghado/vit.cpp/blob/main/main.cpp#L82-L91) are not seen in a Python-based inference script.

Can you please share your approach when you implemented this ggml-based vit code? What were the resources that helped you to build this project? I appreciate any help you can provide.

Batch forward

Hey @staghado thanks for the great work!

I'm wondering whether you have any idea about implementing the batched images forward given batch size > 1.

I notice the batch processing in llama.cpp seems pretty different. No batch dim is involved in the forward pass. I also propose a [discussion]
(ggerganov/llama.cpp#4371) there. Do you have any idea about this? Thanks in advance!

is it possible to use cuda

Hi,

is it possible to use cuda for inference?

Is it possible to build it for Android

It would be nice if we can try this out on Android. I followed instructions from https://github.com/ggerganov/ggml

I tried it and seems that we need to remove -march=native from CMakeLists.txt

-set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -O3 -march=native")
-set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -O3 -march=native")

However, after that I got an error

CANNOT LINK EXECUTABLE "/data/local/tmp/bin/vit": cannot locate symbol "__emutls_get_address" referenced by "/data/local/tmp/bin/vit"...