Adding a support for Cohere Command-R, Anthropic Claude, and Gemini APIs

[nus-apr]

Adding a support for Cohere Command-R, Anthropic Claude, and Gemini APIs

[yuntongzhang]

We are currently working on the LiteLLM support. It should be ready sometime next week.

[IvoP1]

It would be good to add AWS Bedrock API to that list. It supports Claude, LLama, etc...

[andr0s]

Just add litellm support. Everything already included there. No need to add support for each and every LLM and provider

[stevensu1977]

Just add litellm support. Everything already included there. No need to add support for each and every LLM and provider

LiteLLM is a great project, but some models have no native support for function calling , etc. Amazon Bedrock Claude3

[andr0s]

@stevensu1977 well, the whole point is to have the ability to run open source and/or free things. Or experiment with different LLMS. For example, I'm really curios about Llama 3 70b and that big llm from databrix - in my tests on other projects, they showed pretty decent results, despite being dirt cheap or even free (depending on where you run them). I think others also willing to experiment or reduce costs, otherwise people would just be happy with ChatGPT and Claude 3, and this ticket wouldn't even exist.
Another problem with close source models that I noticed - sometimes they change things and it just breaks everything. I'm using Cursor IDE with ChatGPT and the same model suddenly started to perform much worse, basically breaking the whole code, removing tons of my code and adding stubs like "... unchanged code... " which was extremely annoying. I think it wouldn't happen if the model and the backend layer above it would be exactly the same, which is guaranteed only when using open source solutions

[yuntongzhang]

ACR now supports Claude models through LiteLLM and Llama3 through Ollama (added in #32). Feel free to try it out :)

[rezzie-rich]

can you please add support for any open LLM through liteLLM or ollama? currently, for code generation, codeQwen 1.5 7b is ranked as high as GPT-4 and it's only a 7b model so it can easily run locally pretty quickly.