Tooling to build LLM applications: prompt templating and composition, agents, LLM memory, and other instruments for builders of AI applications.
Home Page: https://zmedelis.github.io/bosquet/
License: Eclipse Public License 1.0
clj -Sdeps '{:deps {io.github.zmedelis/bosquet {:mvn/version "0.3.6"}}}' -M -e "(require 'nextjournal.clerk)"
throws
Execution error (NoSuchFieldError) at clojure.tools.analyzer.jvm.utils__init/load (REPL:259).
__thunk__0__
The Clerk maintainer points to outside Clerk.
https://clojurians.slack.com/archives/C035GRLJEP8/p1694292968954859
to continue from #22 regarding memory:
After I have read here:
https://archive.pinecone.io/learn/langchain-conversational-memory/
I see know that there are conceptually two very different types of memory.
The article above talks about "conversational memory", so a memory which lives only during a longer lasting conversation
between a person and a LLM.
What I have tried is more about giving an LLM access to a "long term persistent memory"
gen/complete-templategen/complete-template which takes in results of above and and other instructions to summarize thisBoth might / might not rely on vector databases.
Both have in common that gen/complete-template is called several times:
The latest version of bosquet does not work with Azure OpenAI any more.
(it did work in the past)
There are various issue with passing "impl", "keys" and so on.
bosquet does not show them in any form, I assume that Selmers just renders "empty" string when a tag fails
I was looking with interest at your library and I was wondering if "applying LLMs on large documents" is on our feature list ?
I have done this ones ad hoc, so I mean:
bosquet/src/bosquet/system.clj
Line 37 in 95d410a
fails with NPE, whne ":cohere-api-key" is not given in config.edn.
(even when cohere is not used)
I think the config for various LLMs should be full optional
It looks like something broke in the latest release and I can no longer require bosquet.llm.generator. It worked in the previous version:
$ clj -Sdeps '{:deps {io.github.zmedelis/bosquet {:mvn/version "v2023.10.11"}}}}'
Downloading: net/minidev/json-smart/maven-metadata.xml from central
Clojure 1.11.1
user=> (require '[bosquet.llm.generator :as g])
WARNING: Unable to resolve "#ref [:config :azure-openai-api-endpoint]" at [[:llm/openai :provider/azure] :api-endpoint :form 0]
2023-11-10T19:01:43.560Z xps INFO [bosquet.system:72] - Initializing Bosquet resources:
2023-11-10T19:01:43.586Z xps INFO [bosquet.system:65] - * Short term memory with ({:encoder :memory.encoding/as-is})
2023-11-10T19:01:43.590Z xps INFO [bosquet.system:51] - * OpenAI API service (openai)
nil
user=> but not in latest:
$ clj -Sdeps '{:deps {io.github.zmedelis/bosquet {:mvn/version "v2023.10.31"}}}}'
Clojure 1.11.1
user=> (require '[bosquet.llm.generator :as g])
Execution error (FileNotFoundException) at java.io.FileInputStream/open0 (FileInputStream.java:-2).
system.edn (No such file or directory)
user=> (.printStackTrace *e)
Syntax error macroexpanding at (system.clj:63:3).
at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3719)
at clojure.lang.Compiler$DefExpr.eval(Compiler.java:457)
at clojure.lang.Compiler.eval(Compiler.java:7199)
at clojure.lang.Compiler.load(Compiler.java:7653)
at clojure.lang.RT.loadResourceScript(RT.java:381)
at clojure.lang.RT.loadResourceScript(RT.java:372)
at clojure.lang.RT.load(RT.java:459)
at clojure.lang.RT.load(RT.java:424)
at clojure.core$load$fn__4621.invoke(core.clj:6161)
at clojure.core$load.invokeStatic(core.clj:6160)
at clojure.core$load.doInvoke(core.clj:6144)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invokeStatic(core.clj:5933)
at clojure.core$load_one.invoke(core.clj:5928)
at clojure.core$load_lib$fn__4563.invoke(core.clj:5975)
at clojure.core$load_lib.invokeStatic(core.clj:5974)
at clojure.core$load_lib.doInvoke(core.clj:5953)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invokeStatic(core.clj:669)
at clojure.core$apply.invoke(core.clj:662)
at clojure.core$load_libs.invokeStatic(core.clj:6017)
at clojure.core$load_libs.doInvoke(core.clj:6000)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invokeStatic(core.clj:669)
at clojure.core$apply.invoke(core.clj:662)
at clojure.core$require.invokeStatic(core.clj:6106)
at clojure.core$require.doInvoke(core.clj:6038)
at clojure.lang.RestFn.invoke(RestFn.java:619)
at bosquet.complete$eval2675$loading__1607__auto____2676.invoke(complete.clj:1)
at bosquet.complete$eval2675.invokeStatic(complete.clj:1)
at bosquet.complete$eval2675.invoke(complete.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:7194)
at clojure.lang.Compiler.eval(Compiler.java:7183)
at clojure.lang.Compiler.load(Compiler.java:7653)
at clojure.lang.RT.loadResourceScript(RT.java:381)
at clojure.lang.RT.loadResourceScript(RT.java:372)
at clojure.lang.RT.load(RT.java:459)
at clojure.lang.RT.load(RT.java:424)
at clojure.core$load$fn__1742.invoke(core.clj:6161)
at clojure.core$load.invokeStatic(core.clj:6160)
at clojure.core$load.doInvoke(core.clj:6144)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invokeStatic(core.clj:5933)
at clojure.core$load_one.invoke(core.clj:5928)
at clojure.core$load_lib$fn__1684.invoke(core.clj:5975)
at clojure.core$load_lib.invokeStatic(core.clj:5974)
at clojure.core$load_lib.doInvoke(core.clj:5953)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invokeStatic(core.clj:669)
at clojure.core$apply.invoke(core.clj:662)
at clojure.core$load_libs.invokeStatic(core.clj:6017)
at clojure.core$load_libs.doInvoke(core.clj:6000)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invokeStatic(core.clj:669)
at clojure.core$apply.invoke(core.clj:662)
at clojure.core$require.invokeStatic(core.clj:6106)
at clojure.core$require.doInvoke(core.clj:6038)
at clojure.lang.RestFn.invoke(RestFn.java:805)
at bosquet.llm.generator$eval2669$loading__1607__auto____2670.invoke(generator.clj:1)
at bosquet.llm.generator$eval2669.invokeStatic(generator.clj:1)
at bosquet.llm.generator$eval2669.invoke(generator.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:7194)
at clojure.lang.Compiler.eval(Compiler.java:7183)
at clojure.lang.Compiler.load(Compiler.java:7653)
at clojure.lang.RT.loadResourceScript(RT.java:381)
at clojure.lang.RT.loadResourceScript(RT.java:372)
at clojure.lang.RT.load(RT.java:459)
at clojure.lang.RT.load(RT.java:424)
at clojure.core$load$fn__1742.invoke(core.clj:6161)
at clojure.core$load.invokeStatic(core.clj:6160)
at clojure.core$load.doInvoke(core.clj:6144)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invokeStatic(core.clj:5933)
at clojure.core$load_one.invoke(core.clj:5928)
at clojure.core$load_lib$fn__1684.invoke(core.clj:5975)
at clojure.core$load_lib.invokeStatic(core.clj:5974)
at clojure.core$load_lib.doInvoke(core.clj:5953)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invokeStatic(core.clj:669)
at clojure.core$apply.invoke(core.clj:662)
at clojure.core$load_libs.invokeStatic(core.clj:6017)
at clojure.core$load_libs.doInvoke(core.clj:6000)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invokeStatic(core.clj:669)
at clojure.core$apply.invoke(core.clj:662)
at clojure.core$require.invokeStatic(core.clj:6106)
at clojure.core$require.doInvoke(core.clj:6038)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at user$eval2530.invokeStatic(NO_SOURCE_FILE:1)
at user$eval2530.invoke(NO_SOURCE_FILE:1)
at clojure.lang.Compiler.eval(Compiler.java:7194)
at clojure.lang.Compiler.eval(Compiler.java:7149)
at clojure.core$eval.invokeStatic(core.clj:3215)
at clojure.core$eval.invoke(core.clj:3211)
at clojure.main$repl$read_eval_print__9206$fn__9209.invoke(main.clj:437)
at clojure.main$repl$read_eval_print__9206.invoke(main.clj:437)
at clojure.main$repl$fn__9215.invoke(main.clj:458)
at clojure.main$repl.invokeStatic(main.clj:458)
at clojure.main$repl_opt.invokeStatic(main.clj:522)
at clojure.main$main.invokeStatic(main.clj:667)
at clojure.main$main.doInvoke(main.clj:616)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at clojure.lang.Var.applyTo(Var.java:705)
at clojure.main.main(main.java:40)
Caused by: java.io.FileNotFoundException: system.edn (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
at clojure.java.io$fn__11617.invokeStatic(io.clj:229)
at clojure.java.io$fn__11617.invoke(io.clj:229)
at clojure.java.io$fn__11569$G__11523__11576.invoke(io.clj:69)
at clojure.java.io$fn__11629.invokeStatic(io.clj:258)
at clojure.java.io$fn__11629.invoke(io.clj:254)
at clojure.java.io$fn__11569$G__11523__11576.invoke(io.clj:69)
at clojure.java.io$fn__11591.invokeStatic(io.clj:165)
at clojure.java.io$fn__11591.invoke(io.clj:165)
at clojure.java.io$fn__11530$G__11519__11537.invoke(io.clj:69)
at clojure.java.io$reader.invokeStatic(io.clj:102)
at clojure.java.io$reader.doInvoke(io.clj:86)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at aero.core$read_config_into_tagged_literal.invokeStatic(core.cljc:194)
at aero.core$read_config_into_tagged_literal.invoke(core.cljc:191)
at aero.core$read_config.invokeStatic(core.cljc:426)
at aero.core$read_config.invoke(core.cljc:414)
at clojure.lang.AFn.applyToHelper(AFn.java:156)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3714)
... 105 more
nil
user=> Hi Žygimantas, a minor thing... bosquet.llm.openai/->error fails (with the message Additional data must be non-nil.) when both if its if branch return nil.
I came across this scenario when I was trying to use a local Llama2 service (with a OpenAI API). but I had incorrect values in system.edn ...which resulted in wkok.openai-clojure.core throwing something that caused the scenario in bosquet.llm.openai/->error.
Cheers, Ash
bb native:image results in
Error: Detected a started Thread in the image heap. Thread name: main. Threads running in the image generator are no longer running at image runtime. Object has been initialized in a core JDK class that is not instrumented for class initialization tracking. Therefore, a stack trace cannot be provided.
Please try to infer from the source code how the culprit object got instantiated.
The object was probably created by a class initializer and is reachable from a static field. You can request class initialization at image runtime by using the option --initialize-at-run-time=<class-name>. Or you can write your own initialization methods and call them explicitly from your main entry point.
Trace: Object was reached by
reading field clojure.lang.Var$TBox.thread of constant
clojure.lang.Var$TBox@7ab8739d: clojure.lang.Var$TBox@7ab8739d
indexing into array java.lang.Object[]@65ee555d: [Ljava.lang.Object;@65ee555d
reading field clojure.lang.PersistentHashMap$BitmapIndexedNode.array of constant
...
Finished generating 'bllm' in 1m 23s.
Generating fallback image...
Warning: Image 'bllm' is a fallback image that requires a JDK for execution (use --no-fallback to suppress fallback image generation and to print more detailed information why a fallback image was necessary).
openai-clojure supports both for "completion", see here:
https://cljdoc.org/d/net.clojars.wkok/openai-clojure/0.4.0/doc/usage-azure-openai
Using this in v 0.3.6
(def template "
Answer the following question by outputting the letters A, B, C, D, and E in order of the most likely to be correct to the to least likely to be correct.
{{question}}
A) {{option-a}}
{% llm-generate model=text-davinci-003 var-name=play %}
")
(bg/complete-template template
{:question "Which of the following statements accurately describes the impact of Modified Newtonian Dynamics (MOND) on the observed \"missing baryonic mass\" discrepancy in galaxy clusters?"
:option-a "MOND is a theory that reduces the observed missing baryonic mass in galaxy clusters by postulating the existence of a new form of matter called \"fuzzy dark matter.\""
{:play :llm/azure})config.edn
{:azure-openai-api-key <correct key>
:azure-openai-api-endpoint <correct endpoint>}
I am not able to switch to use azure open AI. I get exception on missing/wrong OpenAI key.
I debugged this intensively but could not figure out, if my configuration or teh code is teh problem.
I'm thinking about using this project but it contains quite a bit of bloat that I don't know if I need.
I already call ChatGPT's API using different libraries where I pass secrets and I can have more than one secret at a time.
You already have tests that mock the APIs of the GPT model here: https://github.com/zmedelis/bosquet/blob/main/test/bosquet/generator_test.clj#L36.
This test uses with-redefs to change the model into anything but this means that if I'd do it then all CLI parsing and additional libraries would be a wasted pull from repositories. Maybe it would even fail the startup if there is no key.
I'd like to know whether you'll want to support user's models by overriding that callback and without loading API keys from config.edn. For instance if user would be able to override the complete function without loading OpenAI's keys then they could support a wide variety of models that they choose (e.g. locally run custom models).
Do you think you'll want to do it with this library?
This could mean that the system would need to be loaded by the user and passed around to the function calls as it would contain the reference to complete function.
As they often cost money or are slow, we might want to have a configurable caching
(a user supplied cache function ? maybe a default impl)
Should get easier when custom model functions are implemented #8
Very early on, the prompt palette was imagined as a repository of prompts to do all sorts of text analysis and generation tasks. Prompt engineering and models change too fast to have any fixed repository of prompts.
Having a repository of smaller bits of prompt elements might still make sense. The intended use case for those "elemental" prompt items would be for Bosquet to inject them into the user prompt when instructed. Output format spec would be a good use case.
For example in https://zmedelis.github.io/bosquet/notebook/papers/chain_of_density/index.html
prompt instructs
Answer in {{FORMAT|default:JSON}}. The {{FORMAT|default:JSON}} should be a list (length 5) of dictionaries whose keys
are \"Missing-Entities\" and \"Denser-Summary\".
Thus when the user requests generation with wkk/output-format :json a prompt element with something like the above could be automatically appended to the prompt.
An interesting case would be ``wkk/output-format :clojure` where the output is coerced into Clojure code.
Another issue with EDN/JSON is examples, those could be generated from the supplied Malli schema
Dear author! Many thanks for open-sourcing this very top quality package!
I’m curious if GitHub Copilot support is on your radar? I have an idea to leverage Copilot to iteratively enhance and contribute to the code I’m developing, and it appears to be a perfect fit for the agent.
Best regards,
Karol
While I was looking into fixes for #8 #9
and getting OpenAI and Azure OpenAI working
and assuming that prompt templates should be shareable across users
I think that an different, more flexible, approach is needed to separate prompts and the model(s) setup (which has indeed secrete keys which should not go into code)
I want to be able to have a fixed template such as:
{:text-analyzer/summarize-to-sentence
"Provide a summary of the following {{text-type|default:text}} in a single sentence:
Text:
{{text}}
Summary:
{% llm-generate max-tokens=60 var-name=completion %}"
}
and a user of this template can "configure" which model (and which model setup including keys)
are used without changing the template as such.
This can IMHO not work via using environement variables. For different models with different setups this is impossible.
I am currently trying to get bosquet set up.
When I have the following line in deps.edn:
io.github.zmedelis/bosquet {:mvn/version "v2023.10.11"}
I get this upon startup (same as #35, thought I would have expected this to be fixed in this version):
Execution error (NoSuchFieldError) at clojure.tools.analyzer.jvm.utils__init/load (REPL:259).
__thunk__0__
Full report at:
/var/folders/df/0xb_85b51dx2p538kv3mf1pw0000gn/T/clojure-2323675351862269305.edn
When I have v2023.10.31, clerk is able to start up fine, but the following namespace:
(ns llm
(:require [nextjournal.clerk :as clerk]
[bosquet.wkk :as wkk]
[bosquet.llm.generator :as g]
[bosquet.system :as system]
))
Fails with
Unhandled java.lang.Exception
namespace 'bosquet.system' not found
Note that this is only caused by the last two imports; importing bosquet.wkk is fine. This persists when this library is installed as io.github.zmedelis/bosquet {:git/sha "00a1534f2dd9d274ae55f9f771f016522017e291"}.
This is the command I am using to run clerk (ignore the escapes): clj -M -e \"(require '[nextjournal.clerk :as clerk]) (clerk/serve! {:browser true :watch-paths [\\\"notebooks\\\" \\\"src/clj\\\" \\\"deps.edn\\\"]})\"
I also have a config.edn and deps.edn and no other clojure configuration.
Line 6 in 95d410a
API documentation
https://docs.anthropic.com/claude/reference/getting-started-with-the-api
Possible interface
(chat llm-service context user-message)
context holds the initial system prompt or whatever else is needed/allowed by the current chained prompt definition
(def context
{:bosquet.conversation/system
"The following is a friendly conversation between a human and an AI.
The AI is talkative and provides lots of specific details from its context. If the
AI does not know the answer to a question, it truthfully says it does not know."})As conversation turns are made, history is kept under :bosquet.conversation/history in the same growing context. History is recorded in ChatML (example):
[{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Who won the world series in 2020?"},
{"role": "assistant", "content": "The Los Angeles Dodgers won the World Series in 2020."},
{"role": "user", "content": "Where was it played?"}]Possible shape of calls and data
(def context (chat oai context "What is 2+2?"))
=>
{:bosquet.conversation/system "The following is a friendly conversation between a human and an AI.
The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question,
it truthfully says it does not know."
:bosquet.conversation/history
[{:role :user
:content "What is 2+2?"}
{:role :assistant
:content "4"}]
:bosquet.conversation/asistant-last-response "4"}This call creates a new context where conversation history is added to :bosquet.conversation/history. New call with new user message will be added to it
(chat oai context "Where was Albert Einstein born?" :bosquet.conversation/system :bosquet.conversation/history)
=>
{:bosquet.conversation/system "The following is a friendly conversation between a human and an AI.
The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question,
it truthfully says it does not know."
:bosquet.conversation/history
[{:role :user
:content "What is 2+2?"}
{:role :assistant
:content "4"}
{:role :user
:content "Where was Albert Einstein born?"}
{:role :assistant
:content "He was born in Germany."}]
:bosquet.conversation/asistant-last-response
"He was born in Germany."}Chat will quickly outgrow the LLM context window. Memory must be added
(chat llm-service context memory user-message)
where the memory parameter would specify what kind of memory to use.
TODO spec in other issue
Memory output will be added to context under :bosquet.conversation/memory
When Bosquet executes chat, it should send memory to LLM, if not present history data.
Implement a Calculator Tool to be used by Agents. Just as the sole currently existing Wikipedia search tool it should follow a defined protocol. The main goal here is to see if current Tool abstraction can cover different types of tools.
The current implementation has system.edn hidden from modifications and extensions. There is no straightforward way to define a new LLM component, memory, etc. Modifying system components should be exposed to the user
Steps to achieve it:
#ref/ig is one example.#ref upfront is not a great user experience.system.edn stays as it is and acts as a baseline, users can merge in custom-system.edn. Something based on Aero #include functionality might be useful here.I’m running Bosquet against a local, small (3.8GB) Llama2 model that is wrapped by an OpenAI API service.
Some of the examples that are bundled with Bosquet, fail with this set-up.
It seems like Llama2's responses aren’t what Bosquet’s parsing expects.
This isn't surprising because:
the responses from this small Llama2 model are often poor; and
Bosquet hasn't yet been tweaked to work with Llama2's responses.
I take the code from math_generate_code.clj and run generate for question1.
The :answer in Llama2's response often contains a mixture of postfix and infix.
E.g. " (+ 40 * (200 - 40))"
But this problem is probably caused by the poor abilities of this small LLM, i.e. this is (1) above.
And after examining other problems in more detail, I'd say that they're all caused by (1).
And Rupert suggested the same.
So this isn't a Bosquet issue intrinsically.
However, Žygimantas suggested recording this as an issue
since he might do some work to make Bosquet operate better against Llama2.
I'll outline my Bosquet->Llama2 set-up in case it helps someone:
python3 -m llama_cpp.server --model llama-2-7b-chat.ggmlv3.q2_K.gguf
:llama2-openai-api-key "<needed by Bosquet but not by the OpenAI API service>"
:llama2-openai-api-endpoint "http://localhost:8000/v1"
...seems like Bosquet expects an API key even when the service itself doesn't require it(?).
;; Config for an OpenAI API fronted LLM provided by a local Llama2 based service
[:llm/openai :provider/llama2] {:api-key #ref [:config :llama2-openai-api-key]
:api-endpoint #ref [:config :llama2-openai-api-endpoint]
:impl :openai}
;; Default LLM service to use if this is not specified in the
;; `generate` call options
:llm/default-llm [:llm/openai :provider/llama2]
One key required of using LLMs with longer texts is to "find for a given text similar texts somehow".
Having this would allow templates such as:
Given the following text snippets
{% for t in similar %}
{{t}}
{% endfor %}
answer the following question:
{{question}}
{% llm-generate }
together with code such as:
(def question "What is the capital of Germany")
(def similar-snippets (xxxxx/similar question ...)) ;; not existing function
(gen/complete-template capital-template {:question question
:similar similar-snippets})Fn "similar" would simply return a list of semantic similar texts to a given text (likely using embeddings and a vector database)
More then one implementation possible.
In this way we would
not sure, if this requires any new code in bosquet.
A concrete implementation of "similar" depends for sure on the underlying vector store, if used
using 0.3.6 I get a exception requiring the 'bosquet.generator" namespace:
(require '[bosquet.generator :as bg])
Execution error (FileNotFoundException) at selmer.tags/loading (tags.clj:1).
Could not locate selmer/node__init.class, selmer/node.clj or selmer/node.cljc on classpath.
deps.edn:
{:deps {io.github.zmedelis/bosquet {:mvn/version "0.3.6"}}}
on Linux
I was reflecting about the minimal tooling we need to work on larger texts with LLMs.
In my view we need 3 things:
I am not sure if 3) is existing in the Clojure world. I believe there are some vector database having a java binding, at least Milvus does:
https://github.com/milvus-io/milvus-sdk-java/blob/master/examples/main/java/io/milvus/GeneralExample.java
Ideally bosquet would support various vector databases, maybe via an abstraction
Any thoughts ?
I hot-fixed a bit the code related to #30,
and came to a point where it indeed tried a call to Azure, which (correctly) gave an exception :
#error {
:cause nil
:via
[{:type clojure.lang.ExceptionInfo
:message Interceptor Exception:
:data {:execution-id #uuid "bee7c345-b673-4b5d-86b9-f9a54ff3eb59", :stage :leave, :interceptor :wkok.openai-clojure.sse/perform-sse-capable-request, :type java.net.ConnectException, :exception #error {
:cause nil
:via
[{:type java.net.ConnectException
:message nil
:at [jdk.internal.net.http.HttpClientImpl send HttpClientImpl.java 573]}
{:type java.net.ConnectException
:message nil
:at [jdk.internal.net.http.common.Utils toConnectException Utils.java 1047]}
{:type java.nio.channels.UnresolvedAddressException
:message nil
:at [sun.nio.ch.Net checkAddress Net.java 149]}]
:trace
[[sun.nio.ch.Net checkAddress Net.java 149]
[sun.nio.ch.Net checkAddress Net.java 157]
[sun.nio.ch.SocketChannelImpl checkRemote SocketChannelImpl.java 816]
[sun.nio.ch.SocketChannelImpl connect SocketChannelImpl.java 839]
[jdk.internal.net.http.PlainHttpConnection lambda$connectAsync$0 PlainHttpConnection.java 183]
[java.security.AccessController doPrivileged AccessController.java 569]
[jdk.internal.net.http.PlainHttpConnection connectAsync PlainHttpConnection.java 185]
[jdk.internal.net.http.Http1E
This was wrongly handled here:
bosquet/src/bosquet/llm/openai.clj
Line 79 in 95d410a
therefore "hidden" and resulted in this exception:
#error {
:cause "Additional data must be non-nil."
:via
[{:type java.lang.IllegalArgumentException
:message "Additional data must be non-nil."
:at [clojure.lang.ExceptionInfo <init> "ExceptionInfo.java" 31]}]
:trace
[[clojure.lang.ExceptionInfo <init> "ExceptionInfo.java" 31]
[clojure.lang.ExceptionInfo <init> "ExceptionInfo.java" 22]
[clojure.core$ex_info invokeStatic "core.clj" 4807]
[clojure.core$ex_info invoke "core.clj" 4807]
[bosquet.llm.openai$complete invokeStatic "openai.clj" 82]
[bosquet.llm.openai$complete invoke "openai.clj" 42]
[bosquet.llm.openai.OpenAI generate "openai.clj" 98]
So something goes wrong in the exception handling above. (L79)
I'd like to get a prompt response from an LLM, send it to an API, then send the API response back into the LLM (and possibly repeat). Is that possible with Bosquet?
For example:
Firt -> First
As there is no JVM code for GPT token ccount, I was wondering if we could not add an token-count-fn which uses a "hosted token count function".
I found one here:
https://rapidapi.com/galim-galim-default/api/openai-gpt-pricing-calculator
A "user" of this function needed to subscribe to rapidapi to use it, but 100 calls per day are for free.
In practice the method signature of a rapidapi-gpt-token-count-fn would then taken the users API key as input and then simply make an http call :
curl --request POST \
--url https://openai-gpt-pricing-calculator.p.rapidapi.com/chat \
--header 'X-RapidAPI-Host: openai-gpt-pricing-calculator.p.rapidapi.com' \
--header 'X-RapidAPI-Key: <<users key>>> \
--header 'content-type: application/json' \
--data '{
"model": "gpt-3.5-turbo",
"content": "Hey there. How are you doing today? Answer with a Steve Jobs style."
}'
which returns the number of tokens.
What do you think ?
I think we should add something to easily use non-openai models.
Main reason being that both OpenAI and AzureOpenAI are not really open or are even paid.
Ideally we should be able to use the same code (using bosquet) and we can easely "swap" the model
I am thinking about incorporating existing tools from langchain ecosystem, which is growing very fast. They have mature products, like AutoGPT and BabyAGI.
It would be nice if we could extend/reuse some of that in bosquet.
For instance, we certainly need google search. Currently I am using babashka to call python google-search client.
One option would be use a python interop via libpython_clj.
Also, we can integrate vector db APIs, like activeloop, because they have serverless REST api.
That way I could keep my babashka script which populates deeplake and then use bosquet for the rest of my needs.
I know that Vald Vector DB has clojure client, but, unfortunately, langchain tools do not integrate with Vald. I hope they will add it in the near future. In the meantime, we could just build AWS lambda clojure-vald client. Calling AWS Lambda from langchain is easy.
I think now that we need as well a vector database abstraction in bosquet.
This might allow to to have "vector operations" inside templates, similar to what is now possible with `llm-generate".
(But I am not sure on this).
I do know that we would need to support 2 abstract operations (at least to start with):
Not sure if we need both, or 2) is enough
just following "Getting start" guide, I've modified config.edn on my project root to have the correct api-key. curl works. using lein/clj and neither works. Maybe the docs are a bit outdated?
Stacktrace:
1. Unhandled clojure.lang.ExceptionInfo
LLM service is not configured
{:service nil}
complete.clj: 48 bosquet.complete/complete
complete.clj: 39 bosquet.complete/complete
tag.clj: 46 bosquet.template.tag/gen-tag
tag.clj: 41 bosquet.template.tag/gen-tag
tags.clj: 533 selmer.tags/tag-handler/fn/fn
AFn.java: 154 clojure.lang.AFn/applyToHelper
AFn.java: 144 clojure.lang.AFn/applyTo
AFunction.java: 31 clojure.lang.AFunction$1/doInvoke
RestFn.java: 408 clojure.lang.RestFn/invoke
node.clj: 19 selmer.node.FunctionNode/render_node
parser.clj: 143 selmer.parser/render-template-with-values
parser.clj: 119 selmer.parser/render-template-with-values
parser.clj: 163 selmer.parser/render-with-values
parser.clj: 158 selmer.parser/render-with-values
RestFn.java: 425 clojure.lang.RestFn/invoke
read.clj: 68 bosquet.template.read/fill-slots
read.clj: 63 bosquet.template.read/fill-slots
generator.clj: 40 bosquet.llm.generator/complete-template
generator.clj: 19 bosquet.llm.generator/complete-template
generator.clj: 133 bosquet.llm.generator/generate
generator.clj: 125 bosquet.llm.generator/generate
generator.clj: 126 bosquet.llm.generator/generate
generator.clj: 125 bosquet.llm.generator/generate
REPL: 4 ai2-test.core/eval38483
REPL: 4 ai2-test.core/eval38483
Compiler.java: 7194 clojure.lang.Compiler/eval
Compiler.java: 7149 clojure.lang.Compiler/eval
core.clj: 3215 clojure.core/eval
core.clj: 3211 clojure.core/eval
interruptible_eval.clj: 87 nrepl.middleware.interruptible-eval/evaluate/fn/fn
AFn.java: 152 clojure.lang.AFn/applyToHelper
AFn.java: 144 clojure.lang.AFn/applyTo
core.clj: 667 clojure.core/apply
core.clj: 1990 clojure.core/with-bindings*
core.clj: 1990 clojure.core/with-bindings*
RestFn.java: 425 clojure.lang.RestFn/invoke
interruptible_eval.clj: 87 nrepl.middleware.interruptible-eval/evaluate/fn
main.clj: 437 clojure.main/repl/read-eval-print/fn
main.clj: 437 clojure.main/repl/read-eval-print
main.clj: 458 clojure.main/repl/fn
main.clj: 458 clojure.main/repl
main.clj: 368 clojure.main/repl
RestFn.java: 1523 clojure.lang.RestFn/invoke
interruptible_eval.clj: 84 nrepl.middleware.interruptible-eval/evaluate
interruptible_eval.clj: 56 nrepl.middleware.interruptible-eval/evaluate
interruptible_eval.clj: 152 nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
AFn.java: 22 clojure.lang.AFn/run
session.clj: 218 nrepl.middleware.session/session-exec/main-loop/fn
session.clj: 217 nrepl.middleware.session/session-exec/main-loop
AFn.java: 22 clojure.lang.AFn/run
Thread.java: 1583 java.lang.Thread/run
Between gen/complete-template:
(def synopsis
(gen/complete-template
synopsis-template
{:title "Mr. X" :genre "crime"}
{:play :llm/openai
:review :llm/cohere}))
and gen/complete
(def review (gen/complete play
{:title "Mr. X" :genre "crime"}
{:synopsis {:llm-generate {:impl :openai :api-key "<my-key>"}}
:review {:llm-generate {:impl :openai :api-key "<my-key>"}}}))
Maybe we mix here the "new style" and the "old style".
This relates as well to #30
So maybe the code in 0.3.6 is still in a transition to the new style, as introduced by the change to "Config management for LLM components"
So I will wait for your feedback, before looking deeper into #30.
I think, that the code as in 0.3.6 cannot work for Azure with "gen/complete-template"
Integration should be done with https://github.com/danielsz/cohere-clojure
Docu is here:
https://huggingface.co/docs/api-inference/detailed_parameters#text-generation-task
This 'task' seem to match bosques concept of "completion" best.
If I understand hugging fcase ocrrectly, this would then allow to use > 14000 models from bosquet:
https://huggingface.co/models?pipeline_tag=text-generation
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.