billxbf/rewoo

Decoupling Reasoning from Observations for Efficient Augmented Language Models

👋 Welcome to my space of hobby projects!

What's New 👉

🎹 | > > _ _ _ _ _ _ _ _

ArXplorer | A recommender of daily papers from arXiv, customized with your Prompt. [Code]

Gentopia & GentPool | An Agent [Framework] & [Platform] for the hardcore builders.

ReWOO | Discussing tool-use efficiency and reasoning ability of LLMs. [Code] [Paper]

rewoo's Issues

add gpt-4

putting gpt-4 as a base model currently throws the error:

Traceback (most recent call last):
File "repos/ReWOO/run.py", line 88, in
main(args)
File "repos/ReWOO/run.py", line 75, in main
method = PWS_Base(planner_model=args.planner_lm, solver_model=args.sovler_lm,
File "repos/ReWOO/algos/PWS.py", line 130, in init
super().init(available_tools=available_tools,
File "repos/ReWOO/algos/PWS.py", line 23, in init
self.planner_token_unit_price = get_token_unit_price(planner_model)
File "repos/ReWOO/utils/util.py", line 40, in get_token_unit_price
raise ValueError("Model not found")
ValueError: Model not found

Got error from SerpAPI: Google hasn't returned any results for this query.

Thanks for the awesome tool. Very excited to see what opportunities this opens up.

I'm getting "Got error from SerpAPI: Google hasn't returned any results for this query." but I can see in the serpapi history that it is running the searches. Any Ideas? What info would be useful?

Running on Debian 11 linux and Python 3.9

traceback (most recent call last):
  File "/home/username/dev/ReWOO/run.py", line 90, in <module>
    main(args)
  File "/home/username/dev/ReWOO/run.py", line 79, in main
    response = method.run(task)
  File "/home/username/dev/ReWOO/algos/PWS.py", line 43, in run
    self._get_worker_evidences()
  File "/home/username/dev/ReWOO/algos/PWS.py", line 106, in _get_worker_evidences
    self.worker_evidences[e] = WORKER_REGISTRY[tool].run(tool_input)
  File "/home/username/dev/ReWOO/nodes/Worker.py", line 24, in run
    evidence = tool.run(input)
  File "/usr/local/lib/python3.9/dist-packages/langchain/utilities/serpapi.py", line 85, in run
    return self._process_response(self.results(query))
  File "/usr/local/lib/python3.9/dist-packages/langchain/utilities/serpapi.py", line 131, in _process_response
    raise ValueError(f"Got error from SerpAPI: {res['error']}")
ValueError: Got error from SerpAPI: Google hasn't returned any results for this query.

Add custom API endpoint

This can be especially helpful for those in China as an example,
or connect external servers that use Gpt-4 for example

SerpApi

Is it possible to use ReWOO without a SerpApi key - since it seem to be pretty expensive after a low number of API calls?

--solver_lm throwing error

You have a typo for solver_lm, its written as sovler_lm

usage: run.py [-h] [--method {direct,cot,react,rewoo}] [--exemplar EXEMPLAR]
[--toolset TOOLSET [TOOLSET ...]] [--base_lm BASE_LM] [--planner_lm PLANNER_LM]
[--sovler_lm SOVLER_LM] [--print_trajectory] [--key_path KEY_PATH]
run.py: error: unrecognized arguments: --solver_lm gpt-3.5-turbo

why do we need the worker phase?

using gpt-4 i am able to generate plans that are interdependent as a json that i can immediately execute by looping through them, at this point, we don't need the worker phase right? I can just pass the result of the execution straight to the solver?

Planner7B Model for Solver

You said we can pass planner7b lora for run.py which use the merged model as base model. you used condition in nodes/Planner.py you formatted the prompt into instruction and input if lora model was used. but you didn't used same condition in nodes/Solver.py. you formatted the prompt like this prompt = self.prefix + input + "\n" + worker_log + self.suffix + input + '\n'and call_llm() method with this prompt. if i use lora model as base llm. it check whether the model is lora model merged if so it requires the prompt to be list of intruction and input as you did in Planner.py. @billxbf Please Look into This ASAP. Correct me if I'm Wrong.

RateLimitError

Running with either:
python run.py --method rewoo --toolset Google LLM --base_lm gpt-3.5-turbo
Or Gradio results in the following error:
RateLimitError: You exceeded your current quota, please check your plan and billing details.

1. I have 3 other apps that use the same key and it works
2. I changed the model to use text-davinci-003 AND gpt-3.5-turbo with no change
3. Added logging to verify my key was correctly being read from ./keys/openai.key file and the model was being passed correctly
4. Verified that my quota had not been exceeded

So key is correct, Quota is not exceeded and model declaration is correct.
Is the framework spamming faster than the API endpoint allows?

bootstrapping

Can you provide a more detailed explanation on how to use the bootstrapping method to sample the correct 2000 Planner instruction data from 4000 samples? Were these data not generated by any tool and only split into steps? How were the final results obtained from these steps? Were they all generated by LLM?
Thank you

Token price human error issue

Hi, I read the paper ReWOO very impressive.

I think I found some human error when counting the token cost on the PWS.py

ReWOO/algos/PWS.py

Lines 66 to 68 in 9cd0283

	result["token_cost"] = self.planner_token_unit_price * (planner_response["prompt_tokens"] + planner_response["completion_tokens"]) \
	+ self.solver_token_unit_price * (solver_response["prompt_tokens"] + solver_response["completion_tokens"]) \
	+ self.tool_token_unit_price * (self.tool_counter.get("LLM_token", 0) + self.tool_counter.get("Calculator_token", 0))

OpenAI count the input token and output token because they have different pricing policy, but these codes just count the input and output token as same.
Maybe I can make a mistake, but I want you to know the issue.

Thanks a lot for the nice research!!

openai function calling

the syntax of the prompt is rather unique #E1 for example, I wonder if this is on purpose? Because it is rather limiting. For example, I want to introduce a tool called email that expects a JSON object of {subject: string, body: string, email: string} is there a way to adjust the prompt to accommodate this case? Or would this affect the efficacy of the framework?

This is especially useful if paired with openai function calling that already supports multiple parameters function calls

langchain version

what is the right langchain version you use, I install 0.0.18 and get wrong