francisbrero/local-llama

Create an FAQ bot that can be used in Slack and is trained on public + private data

FAQ_bot

Create an FAQ bot that can be used in Slack and is trained on public + private data

Setup

Virtual Environment

Start virtual environment

python3 -m venv env
source env/bin/activate

install requirements

pip install -r requirements.txt --upgrade

Deactivate virtual environment

deactivate

Configure environment variables

There is a .env_sample file. Rename it to .env and fill in the values.

VectorDB

We'll run ChromaDB locally using Docker.

Install Docker

Follow the instructions here.

brew install docker-compose

Install chroma

Follow the instructions here.

git clone [email protected]:chroma-core/chroma.git

Run chroma

Start docker

cd chroma
docker-compose up -d --build

You will likely run into an error which requires updating your docker config

nano ~/.docker/config.json

Remove the following line to the file and run the docker-compose command again

"credSstore": "desktop"

Run chroma

Start docker, then go to the chroma folder and run

docker-compose up -d --build

Test chroma

The DB server is exposed to port 8000 on the host machine and we should be able to access it from the host machine at localhost:8000 . We can hit the heartbeat URL http://localhost:8000/api/v1/heartbeat to make sure the server is up and running. If you see a JSON with a nanosecond EPOC timestamp, you're all set!

LLM

Install LLM

We will run llama2 locally. Original instructions from this article

Just in case, updated instructions for an M2 are also here. There's quite a bit that no longer works in their documentation.

Test LLM

../../llama.cpp/main -m ../../llama/llama-2-7b-chat/ggml-model-f16_q4_0.gguf \
        -t 8 \
        -n 128 \
        -p 'The first man on the moon was '

Run

Get all the conent from zendesk using their API

python ./lib/get_zendesk_content.py

Train the model

Create the bot

python ./lib/main.py