How exactly is OpenAI format (list of messages) translated to model prompts?

[johny-b]

How exactly is OpenAI format (list of messages) translated to model prompts?

[orangetin]

This can be achieved by specifying "echo": true! Here's an example:

curl https://api.together.xyz/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TOGETHER_API_KEY" \
  -d '{
    "model": "mistralai/Mistral-7B-Instruct-v0.1",
    "messages": [{"role": "user", "content": "but why?\n\n\n"}],
    "max_tokens": 2,
    "logprobs": 1,
    "temperature": 1,
    "echo": true
  }'

Response:

{
  "id": "86ef1ec809836444-SJC",
  "object": "chat.completion",
  "created": 1712210868,
  "model": "mistralai/Mistral-7B-Instruct-v0.1",
  "prompt": [
    {
      "text": "<s>[INST] but why?\n\n\n [/INST]",
      "logprobs": {
        "token_ids": [
          1,
          28792,
          16289,
          28793,
          562,
          2079,
          28804,
          13,
          13,
          13,
          733,
          28748,
          16289,
          28793
        ],
        "tokens": [
          "<s>",
          "[",
          "INST",
          "]",
          "but",
          "why",
          "?",
          "\n",
          "\n",
          "\n",
          "[",
          "/",
          "INST",
          "]"
        ],
        "token_logprobs": [
          null,
          -20.015625,
          -11.859375,
          -9.1796875,
          -9.328125,
          -3.8691406,
          -2.8476562,
          -0.65234375,
          -0.6142578,
          -4.0664062,
          -10.7578125,
          -5.6484375,
          -11.4453125,
          -8.9140625
        ]
      }
    }
  ],
  "choices": [
    {
      "finish_reason": "length",
      "logprobs": {
        "token_ids": [
          4003,
          349
        ],
        "tokens": [
          " Here",
          " is"
        ],
        "token_logprobs": [
          -6.78125,
          -1.9003906
        ]
      },
      "index": 0,
      "message": {
        "role": "assistant",
        "content": " Here is"
      }
    }
  ],
  "usage": {
    "prompt_tokens": 15,
    "completion_tokens": 2,
    "total_tokens": 17
  }
}

Notice .prompt.text contains the converted message. Additionally, passing in `"logprobs: 1" w/ echo returns prompt logprobs with the response :)