Examples of Chat Bots using Panels chat features: Traditional, LLMs, AI Agents, LangChain, OpenAI etc
Home Page: https://holoviz-topics.github.io/panel-chat-examples/
License: MIT License
I believe getting a token/ api key from a user and setting it as an environment variable is a bad practice and insecure.
Implementations like the below could quickly lead to one user using the api key of another user.
I'm working on the llama_and_mistral example.
The spinner never stops or hides after the download finishes.
This is True both for the current version in main branch and the below version.
"""
Demonstrates how to use the ChatInterface widget to create a chatbot using
Llama2.
"""
import panel as pn
from langchain.chains import LLMChain
from langchain.llms import CTransformers
from langchain.prompts import PromptTemplate
pn.extension()
MODEL_KWARGS = {
"llama": {
"model": "TheBloke/Llama-2-7b-Chat-GGUF",
"model_file": "llama-2-7b-chat.Q5_K_M.gguf",
},
"mistral": {
"model": "TheBloke/Mistral-7B-Instruct-v0.1-GGUF",
"model_file": "mistral-7b-instruct-v0.1.Q4_K_M.gguf",
},
}
llm_chains = pn.state.cache["llm_chains"]=pn.state.cache.get("llm_chains", {})
responses = pn.state.cache["responses"]=pn.state.cache.get("responses", {})
TEMPLATE = """<s>[INST] You are a friendly chat bot who's willing to help answer the
user:
{user_input} [/INST] </s>
"""
CONFIG = {"max_new_tokens": 256, "temperature": 0.5}
def _get_llm_chain(model, template=TEMPLATE, config=CONFIG):
llm = CTransformers(**MODEL_KWARGS[model], config=config)
prompt = PromptTemplate(template=template, input_variables=["user_input"])
return LLMChain(prompt=prompt, llm=llm)
# Cannot use pn.cache due to https://github.com/holoviz/panel/issues/4236
async def _get_response(contents: str, model: str)->str:
key = (contents, model)
if key in responses:
return responses[key]
llm_chain = llm_chains[model]
response = responses[key] = await llm_chain.apredict(user_input=contents)
return response
async def callback(contents: str, user: str, instance: pn.widgets.ChatInterface):
for model in MODEL_KWARGS:
if model not in llm_chains:
instance.placeholder_text = (
f"Downloading {model}, this may take a few minutes,"
f"or longer, depending on your internet connection."
)
llm_chains[model] = _get_llm_chain(model)
response = await _get_response(contents, model)
instance.send(response, user=model.title(), respond=False)
chat_interface = pn.widgets.ChatInterface(callback=callback, placeholder_threshold=0.1)
chat_interface.send(
"Send a message to get a reply from both Llama 2 and Mistral (7B)!",
user="System",
respond=False,
)
chat_interface.servable()When looking at the docs from my iPhone the videos have zero length and do not play.
Please make me a contributor to the project.
Its friction if I have to keep both this project and a seperate fork up to date.
I would like to be able to git push directly to new branches in this project for pull requests.
We should either get the codecov and pypi buttons working or remove them
hatch run panel serve docs/examples/**/*.py --static-dirs thumbnails=docs/assets/thumbnails --autoreload is a very long copy that takes time to write or copy paste.
Please implement short cut command. Is something like hatch serve possible?
Its nice to have a documentation page. But please use Panel colors (especially Panel blue). It will increase the user recognition of Panel and panel-chat-exampels.
On source cards, it'd be nice to have a clipboard icon.
LangChain are implementing their reference chat examples based on Streamlit here https://github.com/langchain-ai/streamlit-agent.
I think we should cover the same reference chat examples.
I think that with the introduction of Hatch its no longer clear if the instructions are for users or for developer contributors.
I would suggest moving the developer instructions a separate DEVELOPER_GUIDE.md file. And then focusing the README.md and docs/index.md file on users.
Then there is a clear seperation.
When serving authentication.py
I get
2023-09-30 06:43:57,923 Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x7f085a341f60>>, <Task finished name='Task-72' coro=<async_execute.<locals>.wrapper() done, defined at /home/jovyan/repos/private/panel-chat-examples/.venv/lib/python3.10/site-packages/panel/io/server.py:175> exception=TypeError("object function can't be used in 'await' expression")>)
Traceback (most recent call last):
File "/home/jovyan/repos/private/panel-chat-examples/.venv/lib/python3.10/site-packages/tornado/ioloop.py", line 738, in _run_callback
ret = callback()
File "/home/jovyan/repos/private/panel-chat-examples/.venv/lib/python3.10/site-packages/tornado/ioloop.py", line 762, in _discard_future_result
future.result()
File "/home/jovyan/repos/private/panel-chat-examples/.venv/lib/python3.10/site-packages/panel/io/server.py", line 181, in wrapper
state._handle_exception(e)
File "/home/jovyan/repos/private/panel-chat-examples/.venv/lib/python3.10/site-packages/panel/io/state.py", line 436, in _handle_exception
raise exception
File "/home/jovyan/repos/private/panel-chat-examples/.venv/lib/python3.10/site-packages/panel/io/server.py", line 179, in wrapper
return await func(*args, **kw)
File "/home/jovyan/repos/private/panel-chat-examples/.venv/lib/python3.10/site-packages/param/parameterized.py", line 2062, in _async_ref
raise e
File "/home/jovyan/repos/private/panel-chat-examples/.venv/lib/python3.10/site-packages/param/parameterized.py", line 2060, in _async_ref
self_.update({pname: await awaitable})
TypeError: object function can't be used in 'await' expressionFuthermore I would suggest
Hi - Is there anyways this can be embedded as a component in an existing React app already with a FastAPI backend?
Thanks
AISEE
Currently there are lots of dev dependencies in the default dependencies. The consequence is that it takes a very long time for hatch to initialize the default environment. For users just want to play around with the examples this is not a great experience.
The solution would be to create a dedicated dev environment with dev dependencies.
Trying to make videos out of the examples I've noticed some things that could make the examples more useful
chrome_pdf_qa.py to chat_with_pdf.py.chrome_pdf_qa.py could be improvedEnvironmentWidget to get OPENAI_API_KEY etc.send etc. before the keys and assets (like pdfs) have been provided.I will give it my take in #41.
For example this one failed https://github.com/holoviz-topics/panel-chat-examples/actions/runs/6628238862/job/18004877544
When serving the echo.py example, I see
TypeError: Cannot read properties of undefined (reading 'classList')
at n.toggle_scroll_buttonHi @MarcSkovMadsen, thanks so much for improving this example!!
https://github.com/holoviz-topics/panel-chat-examples/blob/main/docs/examples/langchain/langchain_pdf_assistant.py
I have a question: this example relies on this other file in the repo (from panel_chat_examples import EnvironmentWidgetBase). People can't just copy and paste this example and get the app running. I guess it's okay but might be difficult for new users to figure everything out. What do you think? @MarcSkovMadsen @ahuang11
We should add social meta tags as described in https://mrkeo.github.io/reference/meta-tags/ to make our pages more easily shareable.
An alternative would be to have Panel post one tweet per example. Then we could utilize twitter intent and linkedin sharing urls to like and repost the examples.
<a href="https://twitter.com/intent/tweet?text=🤖 Check out this {title} example&via=Panel_org&url=https://holoviz-topics.github.io/panel-chat-examples/{folder}/%23{anchor}&hashtags=Python,DataScience,LLM,ArtificialIntelligence" target="_blank">Twitter</a>,
<a href="https://www.linkedin.com/sharing/share-offsite/?url=https://holoviz-topics.github.io/panel-chat-examples/{folder}/%23{anchor}" target="_blank">LinkedIn</a>I believe this repository should be easily installable and useable. I think the examples should be at the center of things.
It should be easy
examples in the repositoryI think moving the examples to /docs/examples/ works against all of that.
I think forcing users to use Hatch works against that. Hatch is for developers. Not users.
Please move the examples back to /examples/.
Please consider if you really want to force users to use Hatch.
Invalid Markdown formatting like the below just makes it harder to read and maintain the docs.
Please use Markdownlint to identify issues like these. Thanks.
The video does not show chain of thought. Maybe the callback needs to be added in other places.
I don't think this repo will be released on PyPi so it's not possible to import without cloning this repo.
The components examples should link back to GitHub source code so users can copy/paste it to use.
We should prepare this repository to grow without having to change names, links etc. a lot.
Currently a lot of the examples have short, not very specific names. An example is chat.py.
This will be a problem when we try to panel serve ... all the examples.
We should make the names more specific and unique.
The Code Spaces works really great for viewing the code and some apps
But other apps do not work. For example apps that require token like most langchain and openai apps.
But for example mistral should be possible to get working. We should test the apps as I can see for example the example below fails which we should be able to avoid.
If I use for example https://raw.githubusercontent.com/mwaskom/seaborn-data/master/penguins.csv I get
An extra issue is that the full csv file is inserted in the chat directly
A much better user experience would be that only a part (.head, .describe) was inserted. Or that some card or modal was used such that the user could optionally explore the dataframe.
Update the github project description to something like
"Examples of Chat Bots using Panels chat features: Traditional, LLMs, AI Agents, LangChain, OpenAI etc."
"""
In the same directory:
git clone https://github.com/holoviz/lumen.git
mkdir data
cp lumen/doc/**/*.md data
cp lumen/doc/*.md data"""
from transformers import AutoTokenizer
from llama_index import (
SimpleDirectoryReader,
VectorStoreIndex,
ServiceContext,
)
from llama_index.llms import LlamaCPP
from llama_index.llms.llama_utils import (
messages_to_prompt,
completion_to_prompt,
)
from llama_index.embeddings import HuggingFaceEmbedding
from panel.chat import ChatInterface
BASE_REPO = "HuggingFaceH4/zephyr-7b-beta"
QUANTIZED_REPO = "TheBloke/zephyr-7B-beta-GGUF"
QUANTIZED_FILE = "zephyr-7b-beta.Q5_K_S.gguf"
DATA_DIR = "data"
def load_llm():
model_url = f"https://huggingface.co/{QUANTIZED_REPO}/resolve/main/{QUANTIZED_FILE}"
llm = LlamaCPP(
# You can pass in the URL to a GGML model to download it automatically
model_url=model_url,
# optionally, you can set the path to a pre-downloaded model instead of model_url
model_path=None,
temperature=0.1,
max_new_tokens=256,
context_window=3900,
# kwargs to pass to __call__()
generate_kwargs={},
# kwargs to pass to __init__()
# set to at least 1 to use GPU
model_kwargs={"n_gpu_layers": 1},
messages_to_prompt=messages_to_prompt,
completion_to_prompt=completion_to_prompt,
verbose=True,
)
return llm
def load_tokenizer():
return AutoTokenizer.from_pretrained(BASE_REPO)
def create_query_engine(llm):
embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")
service_context = ServiceContext.from_defaults(
llm=llm,
embed_model=embed_model,
)
documents = SimpleDirectoryReader(DATA_DIR, required_exts=[".md"]).load_data()
index = VectorStoreIndex.from_documents(documents, service_context=service_context)
query_engine = index.as_query_engine(streaming=True)
return query_engine
async def respond(contents, user, instance):
conversation = [
{
"role": "system",
"content": "You are a friendly chatbot who will help the user.",
},
{"role": "user", "content": contents},
]
formatted_contents = tokenizer.apply_chat_template(
conversation, tokenize=False, add_generation_prompt=True
)
response = query_engine.query(formatted_contents)
message = None
for chunk in response.response_gen:
message = instance.stream(chunk, message=message, user="Assistant")
llm = load_llm()
tokenizer = load_tokenizer()
query_engine = create_query_engine(llm)
chat_interface = ChatInterface(callback=respond, callback_exception="verbose")
chat_interface.send(
"This uses Llama Index to search through Lumen docs. Try asking a question, "
"like: 'How do I add a hvplot view to penguins.csv?'",
user="System",
respond=False,
)
chat_interface.servable()I've now access to Jupyter-ai in my JupyterHub. Its provides an interface to a RAG.
They have implemented a pretty amazing UI where its really, really easy to learn. A user just needs to write /learn path/to/documents in the input widget. And it will be learned in a split second via a very efficient implementation using Dask. See https://github.com/jupyterlab/jupyter-ai/blob/ec75ba827945c48b6e707c6cc4835eb5c561b616/packages/jupyter-ai/jupyter_ai/chat_handlers/learn.py#L138.
You should try it out and learn from their implementation @ahuang11 if you ever want to implement a chat interface for Panel or HoloViz. You can add markdown, python code, notebooks etc. in a split second and start using.
I would love for us to have a similar reference implementation. Would be so useful.
I was wondering about the organisation of the new menu. What is the sort order and where does it come from?
My initial thought was that it should be sorted automatically. That would be helpful. Especially as it grows.
I tried to design my own apply_template function, but I found that there's one from transformers.
https://github.com/holoviz-topics/panel-chat-examples/blob/main/docs/examples/mistral/mistral_with_memory.py
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")
chat = [
{"role": "system", "content": "You are a helpful bot."},
{"role": "user", "content": "Hello, how are you?"},
{"role": "assistant", "content": "I'm doing great. How can I help you today?"},
{"role": "user", "content": "I'd like to show off how chat templating works!"},
]
print(tokenizer.apply_chat_template(chat, tokenize=False))
This repository could grow over time. Maybe it could need for example an assets or docs folder.
To prepare for the future please move all examples to an examples folder.
That would avoid broken links later on if we share links directly to an example on twitter, linkedin, discourse etc.
Currently the github pipeline installs everything before running lint checks. This takes + 3 mins.
It would be very useful to get the feedback of failing lint check faster (30secs). And a pipeline that fails fast would also help save resources.
Please refactor pipeline such that it fails as fast as possible and uses as few resources as possible.
I'm trying to install and serve the project following the new instructions using hatch
When I run
hatch run panel serve docs/examples/**/*.py --static-dirs thumbnails=docs/assets/thumbnails --autoreloadIt fails
Environment `default` is incompatible: cannot locate Python: 3.9I don't have and cannot get Python 3.9 installed for enterprise reasons. I have python 3.10.
Please make the installation more flexible to support at least Python 3.9-3.11. Thanks.
If I look in the current main branch, I see that the autogenerated .md files are out of sync with the code. For example they refer to pn.widgets.Chat....
I can also see that the deployed documentation will not be affected because the are generated by hatch run docs-build.
But as a contributor I am affected as my PRs can quickly look complicated because I might have rebuilt the docs.
I propose we remove the autogenerated files from the repository and then add them to .gitignore.
In whitphx/transformers.js.py#23 I'm contributing an example of using transformers_js_py.
We should add an example here. But we should make the example work both in normal Python and in Pyodide to show users how that can be done.
When released as a Pyodide app we should add a link to the app in the Panel section of the README of https://github.com/whitphx/transformers.js.py
Its best practice to use async functions and methods for chat interfaces. This would also help us when we deploy them.
I can see some examples use sync sleep or openai methods. One example is echo_stream.py. We should convert them to async examples.
We should produce assets like thumbnails and gifs. I believe they should be stored in HoloViz CDN instead of in repository. At least the gifs should.
For now we can store them in this issue.
Instead of using the bokeh native design I think we should use a more modern one. Either bootstrap or material. It will communicate better.
I can see though that the bootstrap theme lacks a bit of margin at the top.
Material looks better
Currently I cannot test all examples in the github build pipelines because an OPENAI_API_KEY is not available.
Please provide a key in the pipeline or specify how we should test those examples. Thanks.
Example: #12
https://github.com/holoviz-topics/panel-chat-examples/actions/runs/6440192051/job/17488753601?pr=12
For now I will disable testing the specific examples such that I can get my PR in before another major change to the repo happens. This will help me avoid spending a day of refactoring of my PRs.
With probably some work it is possible to mock the OPENAI api. See openai/openai-python#398 (comment)
ImportError: create_pandas_dataframe_agent has been moved to langchain experimental. See https://github.com/langchain-ai/langchain/discussions/11680for more information.
Please update your import statement from: `langchain.agents.create_pandas_dataframe_agent` to `langchain_experimental.agents.create_pandas_dataframe_agent`.
Right now you have to browse around and read the code to identify the example you need. Or maybe even run them to figure out how they work.
I think we should make them more discoverable. I would propose
.png, mp4 or .gif. This could be added to the README or in a seperate gallery.md document.It would be very cool if we got the applications deployed. For example to Hugging Face spaces.
As we don't have resources to cover api keys we would probably need to ask users for their api keys.
Instead of creating our own code and copying it into each app Panel could provide the functionality. I've made a feature request here holoviz/panel#5562.
I believe this repository has the potential to become useful to lots of AI Chat interested people. And it could develop rapidly if the processes for accepting contributions are in place.
I would suggest
holoviz-topics organisation on GithubRight now the videos created automatically by Playwright are not fully ready for usage because
.webm files not supported by iphone. They must be converted to .mp4.It would probably also be nice (for social media) if the videos started with a panel-chat-examples logo, text or similar. Maybe we could even add some nice audio.
Would be very useful to fully automate this. Would probably have to use ffmpeg and python. I It might also be helpful to get Playwright to output to one folder instead of subfolders for each file. See https://playwright.dev/python/docs/videos.
Something like this, maybe not use os.
"""
Demonstrates how to use the `ChatInterface` to create a chatbot using
[Mistral](https://docs.mistral.ai) through
[CTransformers](https://github.com/marella/ctransformers).
"""
import os
import panel as pn
from ctransformers import AutoConfig, AutoModelForCausalLM, Config
pn.extension()
llms = pn.state.cache["llms"] = pn.state.cache.get("llms", {})
MODEL_FILE = "mistral-7b-openorca.Q4_K_M.gguf"
MODEL_URL = f"https://huggingface.co/TheBloke/Mistral-7B-OpenOrca-GGUF/resolve/main/{MODEL_FILE}"
CURL_CMD = f"curl -C - -o {MODEL_FILE} {MODEL_URL}"
async def callback(contents: str, user: str, instance: pn.chat.ChatInterface):
if "mistral" not in llms:
instance.placeholder_text = "Downloading model; please wait..."
config = AutoConfig(
config=Config(
temperature=0.5, max_new_tokens=2048, context_length=2048, gpu_layers=1
),
)
if not os.path.exists(MODEL_FILE):
return_code = os.system(CURL_CMD)
if return_code != 0:
raise RuntimeError(f"Could not download {MODEL_URL}")
llms["mistral"] = AutoModelForCausalLM.from_pretrained(
MODEL_FILE,
config=config,
local_files_only=True,
)
llm = llms["mistral"]
response = llm(contents, stream=True)
message = ""
for token in response:
message += token
yield message
chat_interface = pn.chat.ChatInterface(
callback=callback,
callback_user="Mistral",
reset_on_send=True,
)
chat_interface.send(
"Send a message to get a reply from Mistral!", user="System", respond=False
)
chat_interface.servable()The code can be found here run-llama/llama-hub#703
We could either add the app as an example with code or have an awesome chat examples section linking to llama hub for this one, the blogposts by Andrew and Sophia, The videos about autogen usage etc.
Let me know what you think @ahuang11 ?
Sophia has created a very nice video for panel-chat-examples
I think we should include the video in the README.md and index.md files.
Some things that could improve the video are
panel-chat-examples on the first screen. That is the thing users will see before they play the video.Other ideas
When running the example it takes really a long time to download the model. And when its done I get
Could not find module 'C:\Users\USERNAME\AppData\Local\hatch\env\virtual\panel-chat-examples\yC6ANKqz\panel-chat-examples\Lib\site-packages\ctransformers\lib\cuda\ctransformers.dll' (or one of its dependencies). Try using the full path with constructor syntax."""
Demonstrates how to use the ChatInterface widget to create a chatbot using
Mistral through CTransformers.
"""
import panel as pn
from ctransformers import AutoModelForCausalLM
pn.extension(design="material")
llms = pn.state.cache["llms"] = pn.state.cache.get("llms", {})
async def callback(contents: str, user: str, instance: pn.widgets.ChatInterface):
if "mistral" not in llms:
instance.placeholder_text = "Downloading model; please wait..."
llms["mistral"] = AutoModelForCausalLM.from_pretrained(
"TheBloke/Mistral-7B-Instruct-v0.1-GGUF",
model_file="mistral-7b-instruct-v0.1.Q4_K_M.gguf",
gpu_layers=1,
)
llm = llms["mistral"]
response = llm(contents, stream=True, max_new_tokens=1000)
message = ""
for token in response:
message += token
yield message
chat_interface = pn.widgets.ChatInterface(callback=callback, callback_user="Mistral")
chat_interface.send(
"Send a message to get a reply from Mistral!", user="System", respond=False
)
chat_interface.servable()We need working ui tests, images and videos for these examples.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.