danikhan632/guidance_api

An Extension for oobabooga/text-generation-webui

guidance_api's Issues

Wrong port in example and actual code (9000 - 9555)

README.md uses wrong port in the example usage: 9000

The actual port used by this API is 9555.

How do I set this extension up?

Fairly new to the world of local llms so apologies if this question has an obvious answer.

What I've attempted:
I pulled the repo into the text-generation-webui extensions folder, then launched a server with $ python server.py --n-gpu-layers 30 --load-in-4bit --api --model /wizard/wizard-mega-13B.ggmlv3.q5_0.bin --extensions guidance_api

but when I try to run the example I get the error

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[4], line 5
      3 import re,sys
      4 oobabooga_api_url = "http://127.0.0.1:5000/"
----> 5 guidance.llm = guidance.llms.TGWUI(oobabooga_api_url, chat_mode=False)
      7 character_maker = guidance("""The following is a character profile for an RPG game in JSON format.
      8 '''json
      9 {
   (...)
     14 
     15 }""")
     17 # generate a character

AttributeError: module 'guidance.llms' has no attribute 'TGWUI'

Can anyone advise on how to properly configure the guidance_api extension.

[BUG] Missing 'util' module

I think we're missing a file in the most recent commit.

2023-06-24 07:15:57 ERROR:Failed to load the extension "guidance_api".
Traceback (most recent call last):
File "/home/bilwis/text-generation-webui/modules/extensions.py", line 34, in load_extensions
exec(f"import extensions.{name}.script")
File "", line 1, in
File "/home/bilwis/text-generation-webui/extensions/guidance_api/script.py", line 11, in
from .guidance_gen import GuidanceGenerator
File "/home/bilwis/text-generation-webui/extensions/guidance_api/guidance_gen.py", line 3, in
from .util import build_parameters
ModuleNotFoundError: No module named 'extensions.guidance_api.util'

Thank you for your work, btw, can't wait to try it out!

Thank you!

Thank you very much for this, guidance is an amazing tool and this will massively assist in using it alongside all the goodness text-generation-webui brings together.

If you don't mind me asking, how does it integrate guidance with text-generation-webui, I have a rough idea of how guidance works but I'm totally curious how guidance is integrated into the text-generation-webui pipeline.

AttributeError: 'NoneType' object has no attribute 'config'

Hi,

Thank you for your work on this project. I really need Guidance support.

I'm having issues with loading your extension, I installed the requirements, but I'm getting the following error:

2023-09-03 13:24:32 INFO:Loading the extension "guidance_api"...
starting guidance server
2023-09-03 13:24:32 ERROR:Failed to load the extension "guidance_api".
Traceback (most recent call last):
  File "E:\Apps\oobabooga_windows\oobabooga_windows\text-generation-webui\modules\extensions.py", line 40, in load_extensions
    extension.setup()
  File "E:\Apps\oobabooga_windows\oobabooga_windows\text-generation-webui\extensions\guidance_api\script.py", line 134, in setup
    gen =GuidanceGenerator()
  File "E:\Apps\oobabooga_windows\oobabooga_windows\text-generation-webui\extensions\guidance_api\guidance_gen.py", line 36, in __init__
    self.data= setup_model_data()
  File "E:\Apps\oobabooga_windows\oobabooga_windows\text-generation-webui\extensions\guidance_api\model_info.py", line 13, in setup_model_data
    config_dict = vars(shared.model.config)
AttributeError: 'NoneType' object has no attribute 'config'

Failed to load extension 'NoneType' object has no attribute 'lower'

2023-07-15 22:07:19 ERROR:Failed to load the extension "guidance_api".
Traceback (most recent call last):
  File "O:\Workspace\AI\LLM\ooba\text_generation_webui_01\text-generation-webui\modules\extensions.py", line 41, in load_extensions
    extension.setup()
  File "O:\Workspace\AI\LLM\ooba\text_generation_webui_01\text-generation-webui\extensions\guidance_api\script.py", line 134, in setup
    gen =GuidanceGenerator()
  File "O:\Workspace\AI\LLM\ooba\text_generation_webui_01\text-generation-webui\extensions\guidance_api\guidance_gen.py", line 36, in __init__
    self.data= setup_model_data()
  File "O:\Workspace\AI\LLM\ooba\text_generation_webui_01\text-generation-webui\extensions\guidance_api\model_info.py", line 71, in setup_model_data
    if "falcon" in shared.args.model.lower():
AttributeError: 'NoneType' object has no attribute 'lower'

I don't use falcon, so I just commented that whole block out and it launched after that (and after installing extra requirements manually which are mentioned in another issue)

repetition_penalty_range missing from util.py generate_params

repetition_penalty_range attribute is missing from util.py generate_params which causes an error in newer versions of ooba

Missing dependencies in requirements.txt

It looks like there are some dependencies missing from requirements.txt:

pygtrie
platformdirs

Could be more, these are just the ones I had missing