the-full-stack/ask-fsdl

Document Q&A over The Full Stack's Corpus

ask-fsdl's Issues

Add answer for "what can you do?"

This is a common question for chatbots, and the current prompt doesn't set the model up to give a good answer.

try out Anthropic models

ideally, we'd make it possible to swap them easily

make Discord bot respectable

switch to command instead of prefixes, other assorted changes

resolve flakiness of the yt.lemnoslife endpoint

it's not an official endpoint, so it's unsurprising that it isn't super robust

could add retry logic to the modal.Function calls or start using the official API -- which is both painful and requires an API key

Control context length

The response to this message, which asked what is the size of gpt-4, timed out. The Modal logs indicate that it was a 500 due to an InvalidRequestError from exceeding the context length:

While we could play with chunk length and number of retrieved chunks to try and prevent this from happening, I think it's better to make the max_tokens parameters for the completion requests adaptive up to a minimum length and truncating the sources if they are too long.

automate Notion -> Mongo connection

private Colab here with turbojank implementation

add pulumi provisioning of mongoDB

Atlas Mongo is a Pulumi provider, so it's at least possible

should only happen after #7

add a readme

Add blogposts as sources

add a Makefile

Cmd+K for FSDL website

automate ingestion of emoji feedback

jank implementation of bulk feedback ingestion in a private Colab here

could probably be done by adjusting the Discord bot a bit -- it should be able to respond to reaction events, which could even be cleaner

add in more videos

from us:

• guest talks from LLMBC
• LangChain tool talk
• my LangChain webinars?

bring in videos from elsewhere? need to make sure to have a fallback for chapterless videos

run similarity searches for example questions to determine threshold

right now, just set heuristically

abstract LLM choice, make it configurable

the same way we've abstracted vecstore and docstore

add chapter chunking to YouTube videos

and fall back to our current strategy if the chapters aren't available

chapters provide better metadata for chunks, see #16

Add metadata to chunks

Chunks should include contextual info at the top, like the document and section title, when they're presented to the LM.

This can be stored as metadata and inserted at prompt construction time.

error out faster

probably due to not handling Discord's async behavior correctly, we only error out on timeout, even once the server has 500'd

we should at the very least return the 500s once we have them.

error handling in general is minimal. retry logic with backoff would be even better, but let's hold off. maybe the new Gradio SDK would give us those features easily?

add cloud deployment of Discord bot to the Makefile

could do AWS by hand or could try out Pulumi -- relatively simple.

For starters, just need to share a handful of secrets and pull down the repo on a single instance, then run some make commands.

make the MongoDB default database and collection configurable, add a dev version

add LLM Bootcamp material

MVP: update the list of YouTube videos and rerun notebook ETL
Stretch: move the ETL into Modal and run it from the notebook

switch to an async request with aiohttp

example from pycord docs: https://docs.pycord.dev/en/stable/faq.html#what-does-blocking-mean

waiting on the backend is the majority of the execution time, so this is pretty important

improve transcript quality

the transcripts are autogenerated by YouTube. an afternoon of annotation -- with either Descript or Whisper + GPT-4's help -- would make them much cleaner and that makes them more useful to folks watching the videos as well.

Merge LLM paper corpus into main corpus

The LLM paper corpus code lives elsewhere, should bring it in here.

add gantry logging -- ins/outs synchronous, feedbacks async

use message IDs (plus other metadata if needed) as identifiers/keys so that feedbacks can be matched

return emoji counts

deliver a better devX

the developer experience is just OK. let's make it awesome.

note: some of these are more suggestion than requirement

Everywhere

• add some minimal tests
• formalize into a devcontainer so we can run on codespaces
  • configure linting better
  • improve Python environment setup -- pyenv virtualenv seems to be on the out, see VS Code docs
• modularize the Makefile into reusable scripts
• modularize the env? as it gets more complicated, will become necessary

Backend

Modal

• use modal serve in development so that the backend endpoint isn't polluted -- also gets faster updates
  • consider making it "mandatory", ie set by the ENV=dev flag
• update the debug command to use modal shell
• change secret name for dev vs prod
• allow local configuration to separate dev and prod, #41
• #73

MongoDB

• better handling of dev/prod database separation, #41
• better setup for ad hoc querying #42
• #54
• write a notebook for expectation testing/data doc generation for the docstore
  • important: connect to vector index so that vector index exploration feeds nicely into it

FastAPI

• Make sure we have a good Swagger UI
  • linking docs across multiple "apps", e.g. the / mount and the /gradio mount
• Look into QoL tools for FastAPI -- Postman? OpenAPI tooling?
• Review other best practices for FastAPI design, focused on DevX
• Improve the Gradio interface

LLM and Vector Index

Testing

• Add a MockLLM that fits LangChain interface? #48
  • reduces costs for unit tests and non-e2e integration tests

Evaluation

• Try out AutoEvaluator
• Make at least one simple LLM evaluation test, e.g. "No relevant sources found" or questions about capabilities
• Automate ingress of emoji feedback, #47

Tooling

• Vector index exploration in W&B, #5
• LangChain tracing for debugging

Frontend

• Testing of Discord bot -- health check, maybe a query
• make the /health a "hidden command" so we can use it in prod
  - look into interactions library for Discord to see if it's cleaner, especially for streaming responses
• #49

add instructions and checks for the Discord auth setup

right now, we just point folks to the Python SDK's docs, but it's an involved process

automate Discord bot setup

ideally, we would literally provision the bot for people with Pulumi -- but it's probably easier to get them to copy-paste info into a CLI

consider rearchitecting in light of new pricing and features from OpenAI

https://openai.com/blog/function-calling-and-other-api-updates

tl;dr: 16k context windows for turbo, better system prompting (move from user prompt to system?), tool use that might actually work

Move back to `ask-fsdl` collection name

Switched to ask-fsdl-llm while we were maintaining two separate instances, should switch it back

write our own chain for qa with sources

this blocks #16

make document-store error: pymongo.errors.InvalidURI (reopen)

I could not find a way to reopen so I opened new.
"reopen" #57 (comment)

git --no-pager log -n 1

commit d7a12fb (HEAD -> main, upstream/main, origin/main)
Author: Charles Frye [email protected]
Date: Sun Jun 18 02:08:14 2023 +0000

removes git branch selection now that pulumi is in main

make document-store

🥞: Loaded config from .env.dev

python -m pip install -qqq -r requirements.txt

🥞: If you haven't gotten a Modal token yet, run make modal-token

Verifying token against https://api.modal.com
Token verified successfully
Token written to /home/ido/.modal.toml
bash tasks/send_secrets_to_modal.sh
Created a new secret 'mongodb-fsdl' with the keys 'MONGODB_USER', 'MONGODB_URI', 'MONGODB_PASSWORD'

Use it in to your Modal app using:

@stub.function(secret=modal.Secret.from_name("mongodb-fsdl"))
def some_function():
os.getenv("MONGODB_USER")
os.getenv("MONGODB_URI")
os.getenv("MONGODB_PASSWORD")

Created a new secret 'openai-api-key-fsdl' with the key 'OPENAI_API_KEY'

Use it in to your Modal app using:

@stub.function(secret=modal.Secret.from_name("openai-api-key-fsdl"))
def some_function():
os.getenv("OPENAI_API_KEY")

🥞: GANTRY_API_KEY not set. Logging will not be available.

Created a new secret 'gantry-api-key-fsdl' with the key 'GANTRY_API_KEY'

Use it in to your Modal app using:

@stub.function(secret=modal.Secret.from_name("gantry-api-key-fsdl"))
def some_function():
os.getenv("GANTRY_API_KEY")

🥞: See docstore.py and the ETL notebook for details

tasks/run_etl.sh --drop --db fsdl-dev --collection ask-fsdl

🥞: Dropping collection ask-fsdl in fsdl-dev

✓ Initialized. View app at https://modal.com/apps/ap-mcra9lE85EmUQPHM7Nuj2V
✓ Created objects.
├── 🔨 Created web => https://ido777--ask-fsdl-hook-dev.modal.run
├── 🔨 Created mount /home/ido/gGPT/ask-fsdl/vecstore.py
├── 🔨 Created mount /home/ido/gGPT/ask-fsdl/docstore.py
├── 🔨 Created mount /home/ido/gGPT/ask-fsdl/utils.py
├── 🔨 Created mount /home/ido/gGPT/ask-fsdl/prompts.py
├── 🔨 Created mount /home/ido/gGPT/ask-fsdl/app.py
├── 🔨 Created mount /home/ido/gGPT/ask-fsdl/utils.py
├── 🔨 Created mount /home/ido/gGPT/ask-fsdl/vecstore.py
├── 🔨 Created qanda_langchain.
├── 🔨 Created create_vector_index.
├── 🔨 Created drop_docs.
├── 🔨 Created cli.
└── 🔨 Created fastapi_app => https://ido777--ask-fsdl-dev.modal.run
Traceback (most recent call last):
File "/pkg/modal/_container_entrypoint.py", line 330, in handle_input_exception
yield
File "/pkg/modal/_container_entrypoint.py", line 403, in call_function_sync
res = fun(*args, **kwargs)
File "/root/app.py", line 177, in drop_docs
docstore.drop(collection, db)
File "/pkg/docstore.py", line 17, in drop
collection = get_collection(collection, db, client)
File "/pkg/docstore.py", line 44, in get_collection
db = get_database(db, client)
File "/pkg/docstore.py", line 59, in get_database
client = client or connect()
File "/pkg/docstore.py", line 86, in connect
client = pymongo.MongoClient(connection_string, connect=True, appname="ask-fsdl")
File "/usr/local/lib/python3.10/site-packages/pymongo/mongo_client.py", line 639, in init
res = uri_parser.parse_uri(
File "/usr/local/lib/python3.10/site-packages/pymongo/uri_parser.py", line 461, in parse_uri
raise InvalidURI('Bad database name "%s"' % dbase)
pymongo.errors.InvalidURI: Bad database name "/user:password@fsdl"

The real user and password value were replaced

update the setup instructions

after completing #51

VectorDB to wanDB

https://twitter.com/_ScottCondron/status/1620347174692454400

run the Discord bot in Modal

at first, I thought this was not possible because bots needed to be "always on" to respond to events, which didn't jive with Modal's serverless style, so I put it on EC2, but Modal's web endpoints seem to serve as webhooks, which should work

this would simplify the automation a lot!

but even if we can do everything on Modal, it's nice to have trad cloud infrastructure in the repo, even if only as an option among others.

set up Modal Secret creation from .env

see this makefile for an example

Look into Q&A chain details to look for improvements

use abstract links for papers so that they have nice embeds

eg arXiv abs links, as opposed to pdf links

How to connect discord question & answers with Github issues?

How to connect discord question & answers with Github issues?

add make commands to check for presence of secrets, check for presence of document DB

QoL improvement for folks who want to use/adapt this

add QoL tooling for ad hoc work with JSON, e.g. `jq`

this is weirdly hard -- JSON doesn't show up nicely in VS Code Jupyter; JSON in the terminal has been a PITA with shell-specific nonsense

move ETL into Modal and orchestrate it from the notebook

much cleaner and better demonstrates the "isolated environment" features of Modal

the CLI will look weird in a notebook, probably, but we can work around that

add a Mermaid diagram of the architecture to the README

automatically generating it will allow it to keep up with changes in the structure and to survive if we move to a cookiecutter style

make document-store error: pymongo.errors.InvalidURI

I tried to install it with no luck.
I progressed a bit and found some errors I the guide.
I will open a PR for the errors I already overcome, but now I am stuck on

make document-store

ExecutionError: Could not deserialize remote exception due to local error:
No module named 'pymongo'
This can happen if your local environment does not have the remote exception definitions.
Here is the remote traceback:
Traceback (most recent call last):
File "/pkg/modal/_container_entrypoint.py", line 330, in handle_input_exception
yield
File "/pkg/modal/_container_entrypoint.py", line 403, in call_function_sync
res = fun(*args, **kwargs)
File "/root/app.py", line 176, in drop_docs
docstore.drop(collection, db)
File "/pkg/docstore.py", line 17, in drop
collection = get_collection(collection, db, client)
File "/pkg/docstore.py", line 44, in get_collection
db = get_database(db, client)
File "/pkg/docstore.py", line 59, in get_database
client = client or connect()
File "/pkg/docstore.py", line 81, in connect
client = pymongo.MongoClient(connection_string, connect=True, appname="ask-fsdl")
File "/usr/local/lib/python3.10/site-packages/pymongo/mongo_client.py", line 639, in init
res = uri_parser.parse_uri(
File "/usr/local/lib/python3.10/site-packages/pymongo/uri_parser.py", line 461, in parse_uri
raise InvalidURI('Bad database name "%s"' % dbase)
pymongo.errors.InvalidURI: Bad database name "/<my_user>:@fsdl"

add a function in docstore for general query execution

would be great if it can run off of a file or a Python object, but I'd settle for Python object

get_documents would then call this function

actually provision ssh keys

right now, we assume folks have a key pair called fsdl-webserver-keys, which is unfair

unfortunately, you cannot provision ec2 keyPairs directly from pulumi-aws -- you need to generate a key pair first and then register it

get rid of some of the garbage in the collection from the PDF ingestion

Report here.

See #5 for more comments on using the embedding exploration in W&B.

use a text index with a prefix regex to demonstrate the principle of indices

the source field is a useful one to search over, usually with regex

prefix-matching regexes in Mongo can use lexical text indices

vector search is a type of index, so we want to have more (simpler) indices to point to when teaching this material

Add lab notebooks as sources

handle Gantry secret like other secrets