This project goal is to create a simple comparison between the top LLMs that has multi-linguil capabilities with Arabic currently: LLAMA3.1-405B, Command-R+, GPT4o, AceGPTv2-30B.
I collected small dataset to test broad range of model capabilities in Tasks like math quizes, following instructions and rules, summarization, ner.
I tried to make the tasks challenging because they were very small in number.
- Collect data
- Add parser for OpenAI GPT4o, GPT4o-mini
- Add parser for LLAMA3.1-405B
- Add parser for AceGPTv2-30B
- Add parser for Command-R+
- Add adapter for each task
- Qualitative analysis
- Quantitative analysis (still not sure how yet for some tasks it will be easy like word count, json formatting, for others like summarization it will be hard)
Still not available