greyovo/clip-android-demo

A demo for running quantized CLIP model (ViT-B/32) on Android.

Hi there 👋, this is @greyovo :D

• 🌱 Currently studying for a master's degree in SCNU
• ⭐ Being interested in:
  • Mobile / front-end development and user experience (UI/UX)
  • Large Language Models & Knowledge Graphs & NLP
• 📖 Learning React, TypeScript and Android development.
• 🩵 In favourite of Flutter :)

Technology Stack:

• TypeScript, Python, Java, Kotlin, Dart, and HTML/CSS
• React, Vue, Flutter, Jetpack Compose, and Wechat Mini Program
• SpringBoot, and MySQL

clip-android-demo's Issues

Scoring accuracy is not at par with original CLIP model

For the image of the cat, we compared the scores from the ONNX-based model with those from clip.cpp.

It appears that the ONNX model doesn't provide accurate output. Although we understand that the scores may not match exactly, the relative order should ideally be the same.

clip.cpp output

clip-android-demo (ONNX) output #2

BPE Tokenzier geting stuck while encoding `pavement`

Hello,

Thank you you creating this CLIP demo app.

While playing around with the app, we noticed that the text tokenzier isn't working for the following text "cat on the pavement". It looks like the bpe function is getting suck in infinite loop while trying to encoding the token 'pavement'.

大佬，FC了

环境：
小米13ultra
安卓13
MIUI 14 V14.0.23.9.18.DEV
开发版
每次都闪退，发生在初始化索引相册时

java.io.FileNotFoundException: Failed to create image decoder with message 'invalid input'Input contained an error.
at android.database.DatabaseUtils.readExceptionWithFileNotFoundExceptionFromParcel(DatabaseUtils.java:151)
at android.content.ContentProviderProxy.openTypedAssetFile(ContentProviderNative.java:780)
at android.content.ContentResolver.openTypedAssetFileDescriptor(ContentResolver.java:2029)
at android.content.ContentResolver.openTypedAssetFile(ContentResolver.java:1934)
at android.content.ContentResolver.lambda$loadThumbnail$0(ContentResolver.java:4159)
at android.content.ContentResolver$$ExternalSyntheticLambda1.call(Unknown Source:10)
at android.graphics.ImageDecoder$CallableSource.createImageDecoder(ImageDecoder.java:550)
at android.graphics.ImageDecoder.decodeBitmapImpl(ImageDecoder.java:1870)
at android.graphics.ImageDecoder.decodeBitmap(ImageDecoder.java:1863)
at android.content.ContentResolver.loadThumbnail(ContentResolver.java:4158)
at android.content.ContentResolver.loadThumbnail(ContentResolver.java:4142)
at me.grey.picquery.domain.ImageSearcher.encodePhotoList(ImageSearcher.kt:116)
at me.grey.picquery.domain.ImageSearcher$encodePhotoList$1.invokeSuspend(Unknown Source:14)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:280)
at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:85)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source:1)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source:1)
at me.grey.picquery.domain.AlbumManager$encodeAlbums$2.invokeSuspend(AlbumManager.kt:103)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)
Suppressed: kotlinx.coroutines.internal.DiagnosticCoroutineContextException: [androidx.compose.ui.platform.MotionDurationScaleImpl@e39da66, androidx.compose.runtime.BroadcastFrameClock@1141a7, StandaloneCoroutine{Cancelling}@7840654, AndroidUiDispatcher@61c79fd]