bely66/llm-arabic-benchmark

Overview

This project goal is to create a simple comparison between the top LLMs that has multi-linguil capabilities with Arabic currently: LLAMA3.1-405B, Command-R+, GPT4o, AceGPTv2-30B.

Data

I collected small dataset to test broad range of model capabilities in Tasks like math quizes, following instructions and rules, summarization, ner.

I tried to make the tasks challenging because they were very small in number.

Todo

• Collect data
• Add parser for OpenAI GPT4o, GPT4o-mini
• Add parser for LLAMA3.1-405B
• Add parser for AceGPTv2-30B
• Add parser for Command-R+
• Add adapter for each task
• Qualitative analysis
• Quantitative analysis (still not sure how yet for some tasks it will be easy like word count, json formatting, for others like summarization it will be hard)

Results

Still not available