togethercomputer/openapi

An OpenAPI API specifications file describes an API in its entirety and are typically written in YAML or JSON. including: Endpoints: which are available (/users) and operations on each (GET /users, POST /users) Authentication methods. Operation parameters for each operation (Input and output)

openapi's Issues

What exactly are logprobs in the API?

The logprobs.token_logprobs section is described as "List of token log probabilities".

This seems to be incorrect, because the first number in a list is always a positive float and only subsequent numbers are negative.
My best guess is that perhaps the first number is probability and subsequent are logprobs, but I guess this is not intended?

My code:

from openai import OpenAI
import os

TOGETHER_API_KEY = os.environ.get("TOGETHER_API_KEY")

client = OpenAI(
    api_key=TOGETHER_API_KEY,
    base_url='https://api.together.xyz',
)
model = "mistralai/Mixtral-8x7B-Instruct-v0.1"

chat_completion = client.chat.completions.create(
    messages=[{"role": "user", "content": "but why?"}],
    model=model,
    max_tokens=4,
    logprobs=1,
    temperature=1,
)

print(chat_completion.choices[0].logprobs)

Example results (each line is a separate run):

ChoiceLogprobs(content=None, tokens=[' "', 'But', ' why', '?"'], token_logprobs=[0.69970703, -0.039855957, -8.34465e-07, -0.009651184])
ChoiceLogprobs(content=None, tokens=[' I', ' apolog', 'ize', ','], token_logprobs=[0.09472656, -0.50146484, 0, -0.0001360178])

After changing max_tokens to 2:

ChoiceLogprobs(content=None, tokens=[' I', ' believe'], token_logprobs=[0.09472656, -1.4804688])
ChoiceLogprobs(content=None, tokens=[' "', 'But'], token_logprobs=[0.69970703, -0.038635254])

How exactly is OpenAI format (list of messages) translated to model prompts?

From what I understand, e.g. chat LLaMAs receive as a prompt something that has tags like [INST] and <<SYS>>. Few questions:

1. Where can I read about how together.ai exactly translates OpenAI chat format to this llama-format?
2. Is there a way to see the final text sent to the model?
3. I noticed that for LLaMA-2 chat models (e.g. meta-llama/Llama-2-7b-chat-hf), the first token of every response is always (a single white space), which is pretty weird, because then responses usually start with two white spaces. E.g. this code:

from openai import OpenAI
import os

TOGETHER_API_KEY = os.environ.get("TOGETHER_API_KEY")

client = OpenAI(
    api_key=TOGETHER_API_KEY,
    base_url='https://api.together.xyz',
)
model = "meta-llama/Llama-2-7b-chat-hf"

chat_completion = client.chat.completions.create(
    messages=[{"role": "user", "content": "but why?\n\n\n"}],
    model=model,
    max_tokens=2,
    logprobs=1,
    temperature=1,
)

print(chat_completion.choices[0])

returns

Choice(finish_reason='length', index=0, logprobs=ChoiceLogprobs(content=None, tokens=[' ', ' There'], token_logprobs=[0.99902344, -0.15124512]), message=ChatCompletionMessage(content='  There', role='assistant', function_call=None, tool_calls=None))

(first token is a whitespace and there's a whitespace in the second There token).

This behaviour is not a problem, but I'd like to understand where this comes from.