2024-04-30: we have released our first finetuned 6.7B chat model aigcode/AIGCodeGeek-DS-6.7B and provided code for EvalPlus evaluation in this repo.
This repository contains scripts and code to run EvalPlus benchmarks on AIGCodeGeek series models.
EvalPlus includes HumanEval(+) and MBPP(+) datasets for evaluating code completion preformance.
generations/
stores the generation files and results/
gives the corresponding eval results:
generations
├── humaneval
│ └── aigcode--AIGCodeGeek-DS-6.7B_temp_0.0
├── mbpp
└── aigcode--AIGCodeGeek-DS-6.7B_temp_0.0
results
├── humaneval
│ └── aigcodev1_temp_0.0.txt
└── mbpp
└── aigcodev1_temp_0.0.txt
Both Base and Plus results are reported:
Model | HumanEval | HumanEval+ | MBPP | MBPP+ |
AIGCodeGeek-DS-6.7B | 82.3 | 76.2 | 77.7 | 64.4 |
pip install evalplus --upgrade
pip install -r requirements.txt
The evaluation is done with a single A100 GPU. Here are scripts:
bash run_humaneval.sh
bash run_mbpp.sh