majacinka/crewai-experiments

Experiments with local as well as models available through an api

CREWAI experiments

---

For my experiments with CrewAI, I decided to try 3 different projects, starting from the easiest to the most complex. The aim of the experiments was to have a team of AI agents do following work for me:

1. Examine my Startup idea
2. Build AI newsletter with Google SERP
3. Build AI newsletter with Reddit Scraper
4. Email classifier [WIP]

For my experiements I've tried following LLMs:

---

Available through API calls:

1. OpenAI -> GPT-4
2. Gemini Pro

Local Models through Ollama + rating of how they performed:

1. Mistral 7B

• Nice, coherent results
• Didn't understand that it should use scraping tool for the output
• Result is a bunch of generic text from training data

2. Mistral 7B instruct

• Nice, coherent results, a lot of emojis
• Didn't use any scraping tool for the output
• Result is a bunch of generic text from training data.

3. Open Chat 3.5 7B

• The best and most "newsletter-y" results
• But again, didn't use any tool, so generic content

4. Nous Hermes 7B

• Ok results
• didn't use any tool
• generic content

5. Open Hermes 2.5 7B

• The tone and style of writing is great
• but generic content
• didn't understand that it needs to use tools

6. Starling 7B

• Ok results
• didn't use any tool
• generic content

7. Llama 2 13B

• The only model that "understood" what the task is
• but the text wasn't coherent enough, didn's sound like a newsletter

8. Llama 2 13B chat

• Didn't understand the task or produce any output

9. Llama 2 13B text

• Didn't understand the task or produce any output

10. Llama 2 7B

• not coherent
• didn't use any tool
• no output

11. Llama 2 7B text

• No actual output
• didn't use any tool
• generic content

12. Llama 2 7B chat

• Didn't use any tool
• generic content

13. Phi-2

• The smallest model ran into biggest problems
• Lost track of what it's suppose to do, no output

crewai-experiments's Issues

How do I change the default GPT4-0613 model to "gpt-3.5-turbo-1106"

How do I change the default GPT4-0613 model to "gpt-3.5-turbo-1106" in reddit_newsletter.py?

Look forward to the enthusiastic person to solve the doubt, thank you

The startup_crew.py didn't find the openai key

Did not find openai_api_key, please add an environment variable OPENAI_API_KEY which contains it, or pass openai_api_key as a named parameter. (type=value_error)

To Load GPT-4

api = os.environ.get("sk-proj-fNk......")

something wrong with code of task

@majacinka could you please help me to find out the problem?

Req for CrewAI Dev assistance

Hello Maya,

I saw your video about CrewAI and it was great!!
Would you be available for providing some CrewAI Dev consulting?

Please connect with me on LinkedIn if that might be possible: https://www.linkedin.com/in/tbendien/

I hope to hear from you!

Tom

InternLLM

https://huggingface.co/internlm

TEST THIS MODEL PLEASE

BIGGER CONTEXT WINDOW
BETTER REASONING
STRONG IN FUNCTION CALLING!

How to use just gemini-pro model

I imported the .env file with gemini api key, and un # to load gemini model to agents but I'm still getting an error, could you upload or show how to just use gemini-pro model.

Exception has occurred: ValidationError
1 validation error for Agent
  Value error, 1 validation error for ConversationSummaryMemory
llm
  Can't instantiate abstract class BaseLanguageModel with abstract methods agenerate_prompt, apredict, apredict_messages, generate_prompt, invoke, predict, predict_messages (type=type_error) [type=value_error, input_value={'role': 'Market Research...***'), temperature=0.1)}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.5/v/value_error
  File "/home/dongwook/Project/personal/test.py", line 23, in <module>
    marketer = Agent(
pydantic_core._pydantic_core.ValidationError: 1 validation error for Agent
  Value error, 1 validation error for ConversationSummaryMemory
llm
  Can't instantiate abstract class BaseLanguageModel with abstract methods agenerate_prompt, apredict, apredict_messages, generate_prompt, invoke, predict, predict_messages (type=type_error) [type=value_error, input_value={'role': 'Market Research...***'), temperature=0.1)}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.5/v/value_error

What is the conclusion then ?

Have you found at least one LLM capable of calling tools ? Because I've tested mistral, dolphin-mixtral and llam3:8B and all of them are lost !

If I read correctly your benchmark none of your LLMs achieved it either.

So now what ? We dump CrewAI ? XD
Honnestly I don't see the point of using a framework that can't call tools.