akbartus/mobilesam-in-the-browser

Demonstration of MobileSAM in the browser enabled through ONNX runtime web

MobileSAM-in-the-Browser

Description / Rationale

This is the example implementation of MobileSAM in the browser.

Notes

• ONNX runtime web ver. 1.14.0 was used because it was the most compatible with MobileSAM. Unfortunately latest versions on ONNX runtime web did not show good results in segmentation tasks.
• When converting SAM to encoder and decoder some options need to be played with in order to achieve the best results.
• The example uses converted MobileSAM encoder in onnx format. It can be downloaded from here. There are also normal and quantized versions of the decoder, which can be found in "models" folder.

Tech Stack

The project was made possible thanks to SAM, MobileSAM and SAMExporter repositories. In particular, SAM's original model was used in conversion to ONNX and creating a decoder. MobileSAM's original file was used in converting to ONNX and creating encoder. SAMExporter was used in doing conversions to ONNX.

Demo

Demo: Demo

mobilesam-in-the-browser's Issues

Can you pull a request under the official MobileSAM project?

Thanks for your interest in MobileSAM, for which I am the first author.

You did great work! we have already made a demo on the hugging face and are currently building the official demo of MobileSAM as the original SAM. The official demo will kindly resemble what you have done in terms of the concept "MobileSAM in the browser", but it should be much more professional (somewhat like the original segment anything demo). Please kindly let us know whether you are willing to help with this process. Before the official version comes out, we can temporarily use your current version, which is already non-trivial progress.

Thanks again for your interest in our work.

The result will be spent, wouldn't the original Mobilesam be like this? What could be the problem?

The result will be spent, wouldn't the original Mobilesam be like this? What could be the problem?

[FEATURE REQUEST] Make it compatible with webgl textures

To be able to use on PlayCanvas, Babylonjs, Unity WebGL, etc.

Is there any way to speed up the time it takes to generate a mask

Hello, when running this project on my computer, it takes about 400-500ms for the model to generate mask by inference, which is far less than 4ms in the paper. I would like to ask what the reason is, is it because of the slow inference speed of onnx or the low execution efficiency of javascript? Is there any way to speed up the process of creating a mask

Trying to Run the Glitch Project: Uncaught (in promise) Error

I am a complete beginner with running ML models but I have been trying to run the Glitch project. Every time I upload an image it gives me two errors (see below imgae) :

1. Failed to load resource: net::ERR_CONNECTION_RESET

2. Uncaught (in promise) TypeError: Failed to fetch
   at e.OnnxruntimeWebAssemblySessionHandler.createSessionAllocate (ort.wasm.min.js:6:76144)
   at e.OnnxruntimeWebAssemblySessionHandler.loadModel (ort.wasm.min.js:6:76704)
   at async Object.createSessionHandler (ort.wasm.min.js:6:69742)
   at async h.create (ort.wasm.min.js:6:11814)
   at async handleImage (main.js:83:21)

How can I fix this?

no encoder.onnx

models/mobilesam.encoder.onnx does not exist

models/mobilesam.decoder.onnx does

js uses the non-existing encoder file.