New api in oobabooga

[CyberTimon]

New api in oobabooga

[DataBassGit]

Hi @CyberTimon

Just heard. @nickmitchko just sent a push to handle the new api. You will need to adjust your configs to match, but it looks like it will work.

[nickmitchko]

@DataBassGit streaming won't work, but the standard API seems to work.

If you use AutoGPT, (this fork), the following code needs to be updated: ...

local_llm.py

import json
import requests
from autogpt.config.config import Config


class OoobaboogaAPI: 
    """This class implements the singleton pattern in Python."""  
    _instance = None  
    def __new__(cls, *args, **kwargs):  
        if cls._instance is None or cls._instance.__class__!= cls:  
            cls._instance = super(OoobaboogaAPI, cls).__new__(cls)  
        return cls._instance

    def __init__(self, CFG: Config):
        self.host = CFG.oobabooga_api_host
        

    def generate_text(self, input: str, **manual_params) -> None:
        
        request = {
            'prompt': input,
            'max_new_tokens': 250,
            'do_sample': True,
            'temperature': 1.3,
            'top_p': 0.1,
            'typical_p': 1,
            'repetition_penalty': 1.18,
            'top_k': 40,
            'min_length': 0,
            'no_repeat_ngram_size': 0,
            'num_beams': 1,
            'penalty_alpha': 0,
            'length_penalty': 1,
            'early_stopping': False,
            'seed': -1,
            'add_bos_token': True,
            'truncation_length': 2048,
            'ban_eos_token': False,
            'skip_special_tokens': True,
            'stopping_strings': []
        } | manual_params
        # params = {
        #     'max_new_tokens': 200,
        #     'do_sample': True,
        #     'temperature': 0.72,
        #     'top_p': 0.73,
        #     'typical_p': 1,
        #     'repetition_penalty': 1.1,
        #     'encoder_repetition_penalty': 1.0,
        #     'top_k': 0,
        #     'min_length': 0,
        #     'no_repeat_ngram_size': 0,
        #     'num_beams': 1,
        #     'penalty_alpha': 0,
        #     'length_penalty': 1,
        #     'early_stopping': False,
        #     'seed': -1,
        #     'add_bos_token': True,
        #     'truncation_length': 2048,
        #     'ban_eos_token': False,
        #     'skip_special_tokens': True,
        #     'stopping_strings': [],
        # } | manual_params 
        
        # payload = json.dumps([input, params])
        # post_payload = {
        #     "data": [
        #         payload
        #     ]
        # }

        # print(f"POST PAYLOAD: {post_payload}")

        response = requests.post(f"{self.host}/api/v1/generate", json=request).json()
        print(f"RESPONSE: {response}")

        # reply = response["data"][0]
        reply = response['results'][0]['text']
        print(f"REPLY: {reply}")
        return reply

[DataBassGit]

@nickmitchko

We're not using AutoGPT. We were based on BabyAGI, but that code has basically been completely replaced at this point. I will review this code at some point, but we are not currently focused on getting streaming APIs working. If you do manage to get streaming working, however, feel free to send another PR. Thanks for your help!

[DataBassGit]

Nick's code was merged and should resolve this issue.