This repository contains a web application designed to execute relatively compact, locally-operated Large Language Models (LLMs).
Please follow these steps to install the software:
- Create a new conda environment:
conda create -n ollm python=3.10
conda activate ollm- Clone the software repository:
git clone https://github.com/Uminosachi/open-llm-webui.git
cd open-llm-webui- Install the necessary Python packages by executing:
pip install -r requirements.txt
- To enable Flash Attention in some models, if CUDA is available, install Flash Attention:
pip install packaging ninja pip install flash-attn --no-build-isolation
-
For Windows (with CUDA support):
- Install Visual Studio:
⚠️ Important: Make sure to selectDesktop development with C++during the installation process.
- Copy MSBuild extensions for CUDA as an administrator (adjust the CUDA version as necessary):
xcopy /e "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\extras\visual_studio_integration\MSBuildExtensions" "C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\BuildCustomizations"
- Configure the required environment variables for the build (adjust the CUDA version as necessary):
set PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\bin;%PATH% "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvars64.bat" set FORCE_CMAKE=1 && set CMAKE_ARGS="-DLLAMA_CUDA=on"
- Install the necessary Python packages (this process may take some time):
pip install ninja cmake scikit-build-core[pyproject] pip install --force-reinstall --no-cache-dir llama-cpp-python pip install -r requirements.txt
- Install Visual Studio:
-
For Linux (with CUDA support):
- Configure the required environment variables for the build (if not already set):
export PATH=/usr/local/cuda/bin:${PATH}
- Install the necessary Python packages:
CMAKE_ARGS="-DLLAMA_CUDA=on" pip install --force-reinstall --no-cache-dir llama-cpp-python pip install -r requirements.txt
- Configure the required environment variables for the build (if not already set):
-
For Mac OS (without CUDA support):
- Install the necessary Python packages:
BUILD_CUDA_EXT=0 pip install -r requirements.txt
- Rebuild the
bitsandbytespackage with the CPU option.pip uninstall bitsandbytes git clone https://github.com/TimDettmers/bitsandbytes.git cd bitsandbytes cmake -DCOMPUTE_BACKEND=cpu -S . make pip install .
- Known Issue: Running the LLaVA model on Mac results in an error.
- Install the necessary Python packages:
python ollm_app.py- Open http://127.0.0.1:7860/ in your browser.
To download the model:
- Launch this application.
- Click on the "Download model" button next to the LLM model ID.
- Wait for the download to complete.
| Provider | Model Names |
|---|---|
| Microsoft | Phi-3-mini-4k-instruct |
| gemma-2-9b-it, gemma-1.1-2b-it, gemma-1.1-7b-it | |
| NVIDIA | Llama3-ChatQA-1.5-8B |
| Qwen | Qwen2-7B-Instruct |
| Mistral AI | Mistral-7B-Instruct-v0.3 |
| Rakuten | RakutenAI-7B-chat, RakutenAI-7B-instruct |
| rinna | youri-7b-chat |
| TheBloke | Llama-2-7b-Chat-GPTQ, Kunoichi-7B-GPTQ |
- 📋 Note: By adding the repository paths of models to
model_manager/add_tfs_models.txt, they will be included in the list of Model IDs and displayed in the UI. - 🔍 Note: The downloaded model file will be stored in the
.cache/huggingface/hubdirectory of your home directory.
- Ensure you have obtained the necessary access rights via Hugging Face before downloading any models. Visit the following pages to obtain access:
- Before downloading any models, please log in via the command line using:
huggingface-cli login
| Provider | Model Names |
|---|---|
| Microsoft | Phi-3-mini-4k-instruct-q4.gguf, Phi-3-mini-4k-instruct-fp16.gguf |
| TheBloke | llama-2-7b-chat.Q4_K_M.gguf |
| QuantFactory | Meta-Llama-3-8B-Instruct.Q4_K_M.gguf |
- 🔍 File Placement: Place files with the
.ggufextension in themodelsdirectory within theopen-llm-webuifolder. These files will then appear in the model list on thellama.cpptab of the web UI and can be used accordingly. - 📝 Metadata Usage: If the metadata of a GGUF model includes
tokenizer.chat_template, this template will be used to create the prompts.
| Provider | Model Names |
|---|---|
| Microsoft | Phi-3-vision-128k-instruct |
| llava-hf | llava-v1.6-mistral-7b-hf, llava-v1.6-vicuna-7b-hf, llava-1.5-7b-hf |
| tinyllava | TinyLLaVA-Phi-2-SigLIP-3.1B |
| openbmb | MiniCPM-Llama3-V-2_5-int4, MiniCPM-Llama3-V-2_5 |
| SakanaAI | EvoVLM-JP-v1-7B |
| xtuner | llava-llama-3-8b-v1_1-transformers |
| cyberagent | llava-calm2-siglip |
- Enter your message into the "Input text" box. Adjust the slider for "Max new tokens" as needed.
- Under "Advanced options" adjust the settings for "Temperature", "Top k", "Top p", and "Repetition Penalty" as needed.
- If replacing the system message of the prompt, under "Advanced options" enable the checkbox and enter text.
- Press "Enter" on your keyboard or click the "Generate" button.
⚠️ Note: If the cloud-based model has been updated, it may be downloaded upon execution.
- If you click the "Clear chat" button, the chat history will be cleared.
- By enabling the
CPU executioncheckbox, the model will use the argumentdevice_map="cpu". - Some of the transformers models are loaded with the following 4-bit or 8-bit settings using the
bitsandbytespackage.
- Use the radio buttons in the
Default chat templateto select the template that will be used if the GGUF model lacks achat_template.
- You can upload an image to the LLaVA Image area of this tab and input a prompt related to the image.
- Some of the LLaVA models are loaded with the following 4-bit or 8-bit settings using the
bitsandbytespackage.
- When you enable the
Translate (ja->en/en->ja)checkbox:- Any input in Japanese will be automatically translated to English, and responses in English will be automatically translated back into Japanese.
⚠️ Note: Downloading the translation model for the first time may take some time.
| Developer | Model | License |
|---|---|---|
| Microsoft | Phi-3 | The MIT License |
| Gemma | Gemma Terms of Use | |
| NVIDIA | Llama3-ChatQA | Llama 3 Community License |
| Alibaba Group | Qwen2-7B-Instruct | Apache License 2.0 |
| Mistral AI | Mistral-7B-Instruct | Apache License 2.0 |
| Rakuten | RakutenAI | Apache License 2.0 |
| rinna | Youri | Llama 2 Community License |
| Meta AI | Llama 2 | Llama 2 Community License |
| Sanji Watsuki | Kunoichi-7B | CC-BY-NC-4.0 |
| Developer | Model | License |
|---|---|---|
| Hugging Face | llava-v1.6-mistral-7b-hf | Apache License 2.0 |
| Hugging Face | llava-v1.6-vicuna-7b-hf, llava-1.5-7b-hf | Llama 2 Community License |
| TinyLLaVA | TinyLLaVA-Phi-2-SigLIP-3.1B | Apache License 2.0 |
| OpenBMB | MiniCPM | MiniCPM Model License |
| Sakana AI | EvoVLM-JP-v1-7B | Apache License 2.0 |
| XTuner | llava-llama-3-8b-v1_1-hf | Llama 3 Community License |
| CyberAgent | llava-calm2-siglip | Apache License 2.0 |
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent