bip-rep/sherpa

A mobile Implementation of llama.cpp

sherpa's Issues

How does this work?!?!?!?

I mean seriously, how the f$ck did you get LLama on Android to run? :O

I tried to look at the code myself, but I don't fully understand it. I think this function is where the actual magic happens: https://github.com/Bip-Rep/sherpa/blob/main/lib/lib.dart#L337

Could you give me a brief summary on how this works and what are the most important code parts?
This will probably also help other users a lot, which will help you get more people to use and develop your app better :D

How to run it on Mac?

It will be great if there is a guide to show how , thanks.

Does this support GGUF models?

Does this support GGUF models?

Model file too old

I have downloaded multiple ggm-model .bin files from hugging face and they keep returning this message after trying each model...

Is this app support Llama 2?

Has anyone tried running the Llama2 (https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML)model? After I load the model, the APP immediately crashes after sending out the prompt.

Generation never starts: "context is null"

Load model,
Type hello
Press send
Open log
Log says:
[isolate 08:13:02] llama loaded
[isolate 08:13:02] main found: true
[isolate 08:13:02] trying main
[isolate 08:13:02] trying main DONE Instance of 'llama_context_params'
[isolate 08:13:02] context is null

Generation never starts

Samsung Z Flip4
8gb of ram
Snapdragon 8+ Gen1
The demo shows a oneplus device, probably an issue with Samsung phones, I'll check later on a different device.

Share link to `ggml-model.bin`

In the README it says

Download the ggml-model.bin from Meta for research purposes.

And in the Android app it says

Please download the 7B ggml-model-q4 from the official link meta provided you.

Where can we find the official link?

Support GGML quantitized models

They are smaller and should be faster than the original .bin file from meta/HF

for example this 7B is too huge, and in 2 .bin files ??
https://huggingface.co/psmathur/orca_mini_v3_7b/tree/main

edit : Here you can find GGMLv3 quantitized versions of this model :
https://huggingface.co/TheBloke/orca_mini_v3_7B-GGML/tree/main

Pretty sure it's not supported though, as the readme just talks of .bin files
edit : well these are bin files too, let me try

Add support for MobileLLaMA/MobileVLM models

LLM model targeted for mobile devices, available in 1.4B and 2.7B
https://github.com/Meituan-AutoML/MobileVLM

Apparently only "MobileLLaMA" is out, and not "MobileVLM"

Models links :
https://huggingface.co/mtgv/MobileLLaMA-1.4B-Base
https://huggingface.co/mtgv/MobileLLaMA-1.4B-Chat

https://huggingface.co/mtgv/MobileLLaMA-2.7B-Base
https://huggingface.co/mtgv/MobileLLaMA-2.7B-Chat

how to make it faster

i installed the latest version and its a cool app but it so slow I'm running the vicuna 7b is there a way to make faster i have an 8gb ram phone and what other models does support and please link me to them

[Feature Request] Add ability to edit pre-prompt (and save them)

Would be great if wr could get a way to edit and save pre-prompt.

For inspiration, I like how Faraday app does it.
You can create agents, that have specific pre-prompts, model, model settings, etc.

Then you can create a conversation, or multiple ones, with them, like it instanciate them

Useful when you switch between pre-prompts, or just topics in general.

So I can have ine agent with a sprcific pre-prompt, saying he's an IT expert for example, and another saying he's a music theory expert, or whatever.

Useful when you want to delete your history and start a new conversation too, as we cannot delete messages or rollback as of right now

Building libllama.so file for Android

Hi,
How to build the libllama.so file for Android? The .so file in this repository is for 64 bit machines and my Android phone is 32 bit. I would like to build a so file to work for my phone. Please share the steps to build a libllama.so file.

Thanks

Support for GGMLv3?

The newest llama.cpp provides a new ggml encoding format, and no modern models support this version. Can you update the llama.cpp version? Thank you!

Add support for k-quant latest models

libllama.so is 64-bit instead of 32-bit error when trying to load the model on my Samsung Device

Hi,

I tried using Orca mini 3B q4_0 (https://huggingface.co/TheBloke/orca_mini_3B-GGML/tree/main) model with this repo on my Samsung Galaxy M13 Android Device with 6GB Ram.

I get the following error while trying to load the model.

Failed to load dynamic library '/data/user/0/com.biprep.sherpa/app_flutter/libllama.so': dlopen failed: "/data/data/com.biprep.sherpa/app_flutter/libllama.so" is 64-bit instead of 32-bit

Have u faced this error previously? or is there a different .so file I can use on my device?

Model file not working

Hi,
Can you please send me the link of the model file which works in this project. All the model files I have downloaded are not working in this project.

Thanks

Does it support GGUF (instead of GGML)?

New small models (like Mistral) are not available as GGML, but GGUF. Does Sherpa plan to support this?

This can work with Vicuna?

Hello! tks a lot for this app.
Can you make this compatible with vicuna model?

App crash on mtk 1080+ 8GB

Sherpa crashes after I ask for a question, no error messages are shown in the log
I also tried mlcchat app and it works, very slow but at least it works

Context Is null CPU tensor

With a Google pixel 6 pro and chip Google tensor:
[isolate 17:11:10] llama loaded
[isolate 17:11:10] main found: true
[isolate 17:11:10] trying main
[isolate 17:11:10] trying main DONE Instance of "Ilama_context_params
[isolate 17:11:10] context error : unknown tensor in model file

Maybe when compiling LLama.cpp the Google tensor Need a specific flag?

Support for Linux (Debian-class or other) ?

Hello,
is there a way to build sherpa for a Linux system ?

Thanks, Emmanuel.

It crashes

It Crashes While Generating A propmt

I am using a s21 fe with 8gb of ram and 8gb vram

what is the folder of 'download folder'?

it asks me to put the ggml-model.bin into the download folder, i wonder what is the path of this 'download folder' in phone