damo-nlp-sg/auto-arena-llms

This is the repo for paper "Auto Arena of LLMs: Automating LLM Evaluations with Agent Peer-battles and Committee Discussions".

Project Website | Paper | Leaderboard

How to use the repository

Prepare the environment:

1. Set up the environment using: conda env create -f env.yml
2. Activate the environment with: conda activate LLM_Eval
3. Make sure you have the environment variables listed in APIs

Before including any participants, make sure:

1. The participant's calling function is written in the "generate_response" function inside "utils/api_utils.py".
2. The participant's MMLU score is included in "data/MMLU.csv". If the participant doesn't have an MMLU score, fill in "-1" and it will be initially paired with the median candidate.

To run a tournament, first change the "player_names" variable in the python files to adjust the tournament members. Example commands:

1. English: python run_tournament.py --tournament_dir data/main_tour_40
2. Chinese: python run_tournament.py --tournament_dir data/main_tour_40_zh --language zh

To add a new participant to a finished tournament, here is an example command:

1. python run_tournament_add_participant.py --tournament_dir data/main_tour_40 --add_participant SenseChat_5

To run debates between a pair of selected models, here is an example command:

1. python run_LD_pair.py --model_a Qwen/Qwen1.5-72B-Chat --model_b claude-3-haiku-20240307

We release 2 questions per category as demos for each debate on the website. The full results are not pushed to the website. An analysis including all figures in the papers was run in notebook result_analysis.ipynb.

auto-arena-llms's Issues

请教下环境安装问题

在参考readme安装环境时（conda env create -f env.yml），出现下列报错：
Channels:

• conda-forge
• defaults
  Platform: linux-64
  Collecting package metadata (repodata.json): done
  Solving environment: failed
  Channels:
• conda-forge
• defaults
  Platform: linux-64
  Collecting package metadata (repodata.json): done
  Solving environment: failed

LibMambaUnsatisfiableError: Encountered problems while solving:

• nothing provides requested zlib ==1.2.13 h5a0b063_0
• nothing provides requested zeromq ==4.3.5 h313beb8_0
• nothing provides requested xz ==5.4.6 h80987f9_0
• nothing provides requested wheel ==0.41.2 py39hca03da5_0
• nothing provides requested typing_extensions ==4.9.0 py39hca03da5_1
• nothing provides requested typing-extensions ==4.9.0 py39hca03da5_1
• nothing provides requested types-pytz ==2022.4.0.0 py39hca03da5_1
• nothing provides requested tornado ==6.3.3 py39h80987f9_0
• nothing provides requested tk ==8.6.12 hb8d0fd4_0
• nothing provides requested sqlite ==3.41.2 h80987f9_0
• nothing provides requested sniffio ==1.3.0 py39hca03da5_0
• nothing provides requested setuptools ==68.2.2 py39hca03da5_0
• nothing provides requested readline ==8.2 h1a28f6b_0
• nothing provides requested pyzmq ==25.1.2 py39h313beb8_0
• nothing provides requested pytz ==2023.3.post1 py39hca03da5_0
• nothing provides requested python ==3.9.19 hb885b13_0
• nothing provides requested pydantic-core ==2.14.6 py39hf0e4da2_0
• nothing provides requested pydantic ==2.5.3 py39hca03da5_0
• nothing provides requested pip ==23.3.1 py39hca03da5_0
• nothing provides requested pandas-stubs ==2.1.4.231227 py39hca03da5_0
• nothing provides requested pandas ==2.2.1 py39h313beb8_0
• nothing provides requested openssl ==3.2.1 h0d3ecfb_1
• nothing provides requested numpy-base ==1.26.4 py39ha9811e2_0
• nothing provides requested numpy ==1.26.4 py39h3b2db8e_0
• nothing provides requested numexpr ==2.8.7 py39hecc3335_0
• nothing provides requested ncurses ==6.4 h313beb8_0
• nothing provides requested llvm-openmp ==14.0.6 hc6e5704_0
• nothing provides requested libsodium ==1.0.18 h27ca646_1
• nothing provides requested libopenblas ==0.3.21 h269037a_0
• nothing provides requested libgfortran5 ==11.3.0 h009349e_28
• nothing provides requested libgfortran ==5.0.0 11_3_0_hca03da5_28
• nothing provides requested libffi ==3.4.4 hca03da5_0
• nothing provides requested libcxx ==14.0.6 h848a8c0_0
• nothing provides requested jupyter_core ==5.5.0 py39hca03da5_0
• nothing provides __osx needed by ipykernel-6.29.3-pyh3cd1d5f_0
• nothing provides requested idna ==3.4 py39hca03da5_0
• nothing provides requested httpx ==0.26.0 py39hca03da5_0
• nothing provides requested httpcore ==1.0.2 py39hca03da5_0
• nothing provides requested h11 ==0.14.0 py39hca03da5_0
• nothing provides requested exceptiongroup ==1.2.0 py39hca03da5_0
• nothing provides requested distro ==1.8.0 py39hca03da5_0
• nothing provides requested debugpy ==1.6.7 py39h313beb8_0
• nothing provides requested ca-certificates ==2024.3.11 hca03da5_0
• nothing provides requested bottleneck ==1.3.7 py39hbda83bc_0
• nothing provides requested anyio ==4.2.0 py39hca03da5_0
• nothing provides requested annotated-types ==0.6.0 py39hca03da5_0

Could not solve for environment specs
The following packages are incompatible
├─ annotated-types ==0.6.0 py39hca03da5_0 does not exist (perhaps a typo or a missing channel);
├─ anyio ==4.2.0 py39hca03da5_0 does not exist (perhaps a typo or a missing channel);
├─ bottleneck ==1.3.7 py39hbda83bc_0 does not exist (perhaps a typo or a missing channel);
├─ ca-certificates ==2024.3.11 hca03da5_0 does not exist (perhaps a typo or a missing channel);
├─ debugpy ==1.6.7 py39h313beb8_0 does not exist (perhaps a typo or a missing channel);
├─ distro ==1.8.0 py39hca03da5_0 does not exist (perhaps a typo or a missing channel);
├─ exceptiongroup ==1.2.0 py39hca03da5_0 does not exist (perhaps a typo or a missing channel);
├─ h11 ==0.14.0 py39hca03da5_0 does not exist (perhaps a typo or a missing channel);
├─ httpcore ==1.0.2 py39hca03da5_0 does not exist (perhaps a typo or a missing channel);
├─ httpx ==0.26.0 py39hca03da5_0 does not exist (perhaps a typo or a missing channel);
├─ idna ==3.4 py39hca03da5_0 does not exist (perhaps a typo or a missing channel);
├─ ipykernel ==6.29.3 pyh3cd1d5f_0 is not installable because it requires
│ └─ __osx, which is missing on the system;
├─ jupyter_core ==5.5.0 py39hca03da5_0 does not exist (perhaps a typo or a missing channel);
├─ libcxx ==14.0.6 h848a8c0_0 does not exist (perhaps a typo or a missing channel);
├─ libffi ==3.4.4 hca03da5_0 does not exist (perhaps a typo or a missing channel);
├─ libgfortran5 ==11.3.0 h009349e_28 does not exist (perhaps a typo or a missing channel);
├─ libgfortran ==5.0.0 11_3_0_hca03da5_28 does not exist (perhaps a typo or a missing channel);
├─ libopenblas ==0.3.21 h269037a_0 does not exist (perhaps a typo or a missing channel);
├─ libsodium ==1.0.18 h27ca646_1 does not exist (perhaps a typo or a missing channel);
├─ llvm-openmp ==14.0.6 hc6e5704_0 does not exist (perhaps a typo or a missing channel);
├─ ncurses ==6.4 h313beb8_0 does not exist (perhaps a typo or a missing channel);
├─ numexpr ==2.8.7 py39hecc3335_0 does not exist (perhaps a typo or a missing channel);
├─ numpy-base ==1.26.4 py39ha9811e2_0 does not exist (perhaps a typo or a missing channel);
├─ numpy ==1.26.4 py39h3b2db8e_0 does not exist (perhaps a typo or a missing channel);
├─ openssl ==3.2.1 h0d3ecfb_1 does not exist (perhaps a typo or a missing channel);
├─ pandas-stubs ==2.1.4.231227 py39hca03da5_0 does not exist (perhaps a typo or a missing channel);
├─ pandas ==2.2.1 py39h313beb8_0 does not exist (perhaps a typo or a missing channel);
├─ pip ==23.3.1 py39hca03da5_0 does not exist (perhaps a typo or a missing channel);
├─ pydantic-core ==2.14.6 py39hf0e4da2_0 does not exist (perhaps a typo or a missing channel);
├─ pydantic ==2.5.3 py39hca03da5_0 does not exist (perhaps a typo or a missing channel);
├─ python ==3.9.19 hb885b13_0 does not exist (perhaps a typo or a missing channel);
├─ pytz ==2023.3.post1 py39hca03da5_0 does not exist (perhaps a typo or a missing channel);
├─ pyzmq ==25.1.2 py39h313beb8_0 does not exist (perhaps a typo or a missing channel);
├─ readline ==8.2 h1a28f6b_0 does not exist (perhaps a typo or a missing channel);
├─ setuptools ==68.2.2 py39hca03da5_0 does not exist (perhaps a typo or a missing channel);
├─ sniffio ==1.3.0 py39hca03da5_0 does not exist (perhaps a typo or a missing channel);
├─ sqlite ==3.41.2 h80987f9_0 does not exist (perhaps a typo or a missing channel);
├─ tk ==8.6.12 hb8d0fd4_0 does not exist (perhaps a typo or a missing channel);
├─ tornado ==6.3.3 py39h80987f9_0 does not exist (perhaps a typo or a missing channel);
├─ types-pytz ==2022.4.0.0 py39hca03da5_1 does not exist (perhaps a typo or a missing channel);
├─ typing-extensions ==4.9.0 py39hca03da5_1 does not exist (perhaps a typo or a missing channel);
├─ typing_extensions ==4.9.0 py39hca03da5_1 does not exist (perhaps a typo or a missing channel);
├─ wheel ==0.41.2 py39hca03da5_0 does not exist (perhaps a typo or a missing channel);
├─ xz ==5.4.6 h80987f9_0 does not exist (perhaps a typo or a missing channel);
├─ zeromq ==4.3.5 h313beb8_0 does not exist (perhaps a typo or a missing channel);
└─ zlib ==1.2.13 h5a0b063_0 does not exist (perhaps a typo or a missing channel).

请教下大佬，感谢！

Paper picture drawing problem

I would like to ask, how was this picture in the paper drawn?