Validation error on `list[str]` return annotation for Anthropic models

[jackmpcollins]

Validation error on `list[str]` return annotation for Anthropic models

[mnicstruwig]

I've opened a PR on litellm here, which solves the problem (at least for me while testing Anthropic locally) : BerriAI/litellm#2640.

[mnicstruwig]

This issue appears to occasionally fail slightly differently (with the result being returned as an empty string) when I manually handle the function call:

@chatprompt(
    SystemMessage("<instructions>You are a helpful model that precisely follows instructions. What is on the menu? You can use the get_menu function. Return your answer as a list of strings.</instructions>"),
    AssistantMessage(function_call),
    FunctionResultMessage(content=result, function_call=function_call),
    functions=[get_menu],
    model=LitellmChatModel(model="anthropic/claude-3-sonnet-20240229")
)
def on_the_menu_final_response() -> list[str]: ...

on_the_menu_final_response()

But from the litellm verbose output, it looks like a parsing issue on magentic's side, (since the result appears to be present):

Request to litellm:
litellm.completion(model='anthropic/claude-3-sonnet-20240229', messages=[{'role': 'system', 'content': '<instructions>You are a helpful model that precisely follows instructions. What is on the menu? You can use the get_menu function. Return your answer as a list of strings.</instructions>'}, {'role': 'assistant', 'content': None, 'tool_calls': [{'id': '18ae8c69-fc48-414e-bc65-41b6a85c7b9b', 'type': 'function', 'function': {'name': 'get_menu', 'arguments': '{}'}}]}, {'role': 'tool', 'tool_call_id': '18ae8c69-fc48-414e-bc65-41b6a85c7b9b', 'content': '{"value":"On the menu today we have pizza, chips and burgers."}'}], stop=None, stream=True, tools=[{'type': 'function', 'function': {'name': 'get_menu', 'parameters': {'properties': {}, 'type': 'object'}}}, {'type': 'function', 'function': {'name': 'return_list_of_str', 'parameters': {'properties': {'value': {'items': {'type': 'string'}, 'title': 'Value', 'type': 'array'}}, 'required': ['value'], 'type': 'object'}}}])


self.optional_params: {}
kwargs[caching]: False; litellm.cache: None
Final returned optional params: {'stream': True, 'tools': [{'type': 'function', 'function': {'name': 'get_menu', 'parameters': {'properties': {}, 'type': 'object'}}}, {'type': 'function', 'function': {'name': 'return_list_of_str', 'parameters': {'properties': {'value': {'items': {'type': 'string'}, 'title': 'Value', 'type': 'array'}}, 'required': ['value'], 'type': 'object'}}}]}
self.optional_params: {'stream': True, 'tools': [{'type': 'function', 'function': {'name': 'get_menu', 'parameters': {'properties': {}, 'type': 'object'}}}, {'type': 'function', 'function': {'name': 'return_list_of_str', 'parameters': {'properties': {'value': {'items': {'type': 'string'}, 'title': 'Value', 'type': 'array'}}, 'required': ['value'], 'type': 'object'}}}]}


POST Request Sent from LiteLLM:
curl -X POST \
https://api.anthropic.com/v1/messages \
-H 'accept: application/json' -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -H 'x-api-key: sk-ant-api03-1-sSgKgEh9hdpu-_7kwe8NvyJhT225WzzbSF_6mpZYab4RIOM-VGdWOIY_kBAVFxoGOBUSG-FrA********************' \
-d '{'model': 'claude-3-sonnet-20240229', 'messages': [{'role': 'user', 'content': [{'type': 'text', 'text': '.'}]}, {'role': 'assistant', 'content': [{'type': 'text', 'text': '<function_calls>\n<invoke>\n<tool_name>get_menu</tool_name>\n<parameters>\n</parameters>\n</invoke>\n</function_calls>'}]}, {'role': 'user', 'content': [{'type': 'text', 'text': '<function_results>\n<result>\n<tool_name>None</tool_name>\n<stdout>\n{"value":"On the menu today we have pizza, chips and burgers."}\n</stdout>\n</result>\n</function_results>'}]}], 'system': "<instructions>You are a helpful model that precisely follows instructions. What is on the menu? You can use the get_menu function. Return your answer as a list of strings.</instructions>In this environment you have access to a set of tools you can use to answer the user's question.\n\nYou may call them like this:\n<function_calls>\n<invoke>\n<tool_name>$TOOL_NAME</tool_name>\n<parameters>\n<$PARAMETER_NAME>$PARAMETER_VALUE</$PARAMETER_NAME>\n...\n</parameters>\n</invoke>\n</function_calls>\n\nHere are the tools available:\n<tools>\n<tool_description>\n<tool_name>get_menu</tool_name>\n<description>\n\n</description>\n<parameters>\n<parameter>\n<properties>{}</properties><type>object</type>\n</parameter>\n</parameters>\n</tool_description>\n<tool_description>\n<tool_name>return_list_of_str</tool_name>\n<description>\n\n</description>\n<parameters>\n<parameter>\n<properties>{'value': {'items': {'type': 'string'}, 'title': 'Value', 'type': 'array'}}</properties><required>['value']</required><type>object</type>\n</parameter>\n</parameters>\n</tool_description>\n</tools>", 'max_tokens': 256}'


_is_function_call: True
RAW RESPONSE:
{"id":"msg_01GJkpBZTzZmU8Wmn2CbBjQ1","type":"message","role":"assistant","content":[{"type":"text","text":"<function_calls>\n<invoke>\n<tool_name>return_list_of_str</tool_name>\n<parameters>\n<value>[\"pizza\", \"chips\", \"burgers\"]</value>\n</parameters>\n</invoke>\n</function_calls>"}],"model":"claude-3-sonnet-20240229","stop_reason":"end_turn","stop_sequence":null,"usage":{"input_tokens":429,"output_tokens":65}}


raw model_response: {"id":"msg_01GJkpBZTzZmU8Wmn2CbBjQ1","type":"message","role":"assistant","content":[{"type":"text","text":"<function_calls>\n<invoke>\n<tool_name>return_list_of_str</tool_name>\n<parameters>\n<value>[\"pizza\", \"chips\", \"burgers\"]</value>\n</parameters>\n</invoke>\n</function_calls>"}],"model":"claude-3-sonnet-20240229","stop_reason":"end_turn","stop_sequence":null,"usage":{"input_tokens":429,"output_tokens":65}}
_is_function_call: True; stream: True
INSIDE ANTHROPIC STREAMING TOOL CALLING CONDITION BLOCK
type of model_response.choices[0]: <class 'litellm.utils.Choices'>
type of streaming_choice: <class 'litellm.utils.StreamingChoices'>
Returns anthropic CustomStreamWrapper with 'cached_response' streaming object
RAW RESPONSE:
<litellm.utils.CustomStreamWrapper object at 0x136445a90>


PROCESSED CHUNK PRE CHUNK CREATOR: ModelResponse(id='chatcmpl-27757066-f728-4224-98c0-7662d8f14529', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=None, role='assistant', function_call=None, tool_calls=[ChatCompletionDeltaToolCall(id='call_a7282793-1bb8-4bac-8c34-d6a3c9340e14', function=Function(arguments='{"value": "[\\"pizza\\", \\"chips\\", \\"burgers\\"]"}', name='return_list_of_str'), type='function', index=0)]), logprobs=None)], created=1711103544, model=None, object='chat.completion.chunk', system_fingerprint=None, usage=Usage()); custom_llm_provider: cached_response
completion obj content: None
model_response finish reason 3: None; response_obj={'text': None, 'is_finished': True, 'finish_reason': None, 'original_chunk': ModelResponse(id='chatcmpl-27757066-f728-4224-98c0-7662d8f14529', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=None, role='assistant', function_call=None, tool_calls=[ChatCompletionDeltaToolCall(id='call_a7282793-1bb8-4bac-8c34-d6a3c9340e14', function=Function(arguments='{"value": "[\\"pizza\\", \\"chips\\", \\"burgers\\"]"}', name='return_list_of_str'), type='function', index=0)]), logprobs=None)], created=1711103544, model=None, object='chat.completion.chunk', system_fingerprint=None, usage=Usage())}
_json_delta: {'content': None, 'role': 'assistant', 'function_call': None, 'tool_calls': [{'id': 'call_a7282793-1bb8-4bac-8c34-d6a3c9340e14', 'function': {'arguments': '{"value": "[\\"pizza\\", \\"chips\\", \\"burgers\\"]"}', 'name': 'return_list_of_str'}, 'type': 'function', 'index': 0}]}
model_response.choices[0].delta: Delta(content=None, role='assistant', function_call=None, tool_calls=[ChatCompletionDeltaToolCall(id='call_a7282793-1bb8-4bac-8c34-d6a3c9340e14', function=Function(arguments='{"value": "[\\"pizza\\", \\"chips\\", \\"burgers\\"]"}', name='return_list_of_str'), type='function', index=0)]); completion_obj: {'content': None}
self.sent_first_chunk: False
PROCESSED CHUNK POST CHUNK CREATOR: ModelResponse(id='chatcmpl-27757066-f728-4224-98c0-7662d8f14529', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=None, role='assistant', function_call=None, tool_calls=[ChatCompletionDeltaToolCall(id='call_a7282793-1bb8-4bac-8c34-d6a3c9340e14', function=Function(arguments='{"value": "[\\"pizza\\", \\"chips\\", \\"burgers\\"]"}', name='return_list_of_str'), type='function', index=0)]), logprobs=None)], created=1711103544, model='claude-3-sonnet-20240229', object='chat.completion.chunk', system_fingerprint=None, usage=Usage())

[mnicstruwig]

Some digging seems to reveal that there is a slight difference in how OpenAI models and Anthropic models choose to return function calls via litellm (I'm not sure if this is from the models or litellm):

Here is OpenAI:

from litellm import completion

tools = [{'type': 'function', 'function': {'name': 'get_menu', 'parameters': {'properties': {}, 'type': 'object'}}}, {'type': 'function', 'function': {'name': 'return_list_of_str', 'parameters': {'properties': {'value': {'items': {'type': 'string'}, 'title': 'Value', 'type': 'array'}}, 'required': ['value'], 'type': 'object'}}}]
messages = [
    {"role": "user", "content": "<instructions>You are a helpful model that precisely follows instructions. What is on the menu? Return your answer as a list of strings.</instructions>"},
    {"role": "user", "content": "Today on the menu there is pizza, chips and burgers."}
]

response = completion(
    #model="anthropic/claude-3-sonnet-20240229",
    model="openai/gpt-3.5-turbo",
    messages=messages,
    tools=tools,
    tool_choice="auto",
)

>>> import json
>>> json.loads(response.choices[0].message.tool_calls[0].function.arguments)
{'value': ['pizza', 'chips', 'burgers']}  # <-- this is parsed correctly into a list of strings

And with Anthropic:

>>> import json
>>> json.loads(response.choices[0].message.tool_calls[0].function.arguments)
{'value': "\n['pizza', 'chips', 'burgers']\n"}  # <-- this doesn't have a correct output

It seems as if there function call arguments are being parsed differently from the XML by litellm -- I'm going to open an issue there.

Let me know what you think, or if you have any other ideas.

[jackmpcollins]

Should be fixed in litellm PR BerriAI/litellm#2748 Waiting on litellm release of 1.34.18 which will include this before testing. Looks like both JSON array/output and XML items will now be parsed, if not we will need to prompt Claude to return whichever format is supported.