fabiomatricardi/tinillamagradiochat

Repo for a ChatBot, with Gradio, Streaming TinyLlama1.1BOpenOrca.gguf on CPU only

TiniLlamaGradioChat

Repo for a ChatBot, with Gradio, Streaming TinyLlama1.1BOpenOrca.gguf on CPU only

This is a ChatBot, with TinyLlama model, 1.1Billion parameters, distilled with Orca2 dataset.

Instructions

Virtual Environment

You can also find a requirements.txt file in the repo Create a Virtual Environment and activate it

Install dependencies

• langchain is required only if you want to use the conversation history capabilities.
• CTransformers is supported by LangChain

pip install ctransformers
pip install gradio
pip install langchain

Download the files

Download the python file Chat_tinyLlamaOpenOrca.py and the images 456322.webp and TinyLlama_logo.png Create a subfolder called models and download there the GGUF model file tinyllama-1.1b-1t-openorca.Q4_K_M.gguf.

Go in your terminal, and with the VENV activated run

python Chat_tinyLlamaOpenOrca.py

This is the result :-)

tinillamagradiochat's Issues

Local operation is normal

This is a great project. It is generated very quickly and takes up less than 1G of memory. Why do so few people like it?

Tiny LLM in Android?

Are there any successful cases of TinyLLM running on Android? Or do you know where there is a corresponding open source project for running llm on Android?