ossirytk/llama-cpp-langchain-chat

!!This repository is no longer being maintained!!

llama-cpp-langchain-chat

Lightweight Llama.cpp chatbot made with langchain and chainlit. This project mainly serves as a simple example of langchain chatbot and is a template for further langchain projects. Uses chainlit as a dropin UI chatbot so there is basically no ui code.

Prompt Support

Supports alpaca text prompts, v2 and tavern style json and yaml files and V2 and tavern png cards. Text prompts need to start with the instruct portion, but the response is appended to the text prompt template. Avatar images need to be in the same folder as the prompt file. V2 and Tavern png files get a copy of the image without exif data in the project temp file.

Card Format

'name' : 'char_name'
The name for the ai character. When using json or yaml, this is expected to correspond to avatar image. name.png or name.jpg.
'description' : 'char_persona'
The description for the character personality. Likes, dislikes, personality traits.
'scenario' : 'world_scenario'
Description of the scenario. This roughly corresponds to things like "You are a hr customer service having a discussion with a customer. Always be polite". etc.
'mes_example' : 'example_dialogue'
Example dialogue. The AI will pick answer patterns based on this
'first_mes' : 'char_greeting'
A landing page for the chat. This will not be included in the prompt.

See https://zoltanai.github.io/character-editor/ for a character card editor.

Configs

See the configuration params in .env.example file

Running the chatbot

You might want to build with cuda support. You need to pass FORCE_CMAKE=1 and CMAKE_ARGS="-DLLAMA_CUBLAS=on" to env variables

pipenv install - requirements.txt
pipenv shell
(optional for cuda support)$env:FORCE_CMAKE=1
(optional for cuda support)$env:CMAKE_ARGS="-DLLAMA_CUBLAS=on"
(optional for cuda support)pip install llama-cpp-python==0.2.6 --force-reinstall --upgrade --no-cache-dir --no-deps
cd src\llama_cpp_langchain_chat
chainlit run chat.py

OR

hatch env create
hatch shell
(optional for cuda support)$env:FORCE_CMAKE=1
(optional for cuda support)$env:CMAKE_ARGS="-DLLAMA_CUBLAS=on"
(optional for cuda support)pip install llama-cpp-python==0.2.6
--force-reinstall --upgrade --no-cache-dir --no-deps
cd src\llama_cpp_langchain_chat
chainlit run chat.py

The chatbot should open in your browser

llama-cpp-langchain-chat's Issues

I get the error: UnboundLocalError: cannot access local variable 'card' where it is not associated with a value

Hi, excellent work! I am very interested in trying this app. But i'm trying to use it as it is by default, but I'm getting this error:

llama-cpp-langchain-chat\src\llama_cpp_langchain_chat_init_.py", line 116, in parse_prompt
char_name = card["name"] if "name" in card else card["char_name"]
^^^^
UnboundLocalError: cannot access local variable 'card' where it is not associated with a value

I understand this is because I am not assigned an Charracter. Is there any possibility that by default they include a standard Character?
So it can be tested out of the box? Or maybe you can tell me where the error may be. Thank you so much

There was an error in the model response

Something seems to go wrong when the model responds. Please tell me how I can fix it.