Overlap between Magicoder-Evol-Instruct-110K and HumanEval

[ise-uiuc]

Overlap between Magicoder-Evol-Instruct-110K and HumanEval

[UniverseFly]

Hi @Truth-In-Lies, thanks for your comprehensive analysis, which is very valuable. Here is how we view the contamination problem: since it is intrinsically challenging to formulate a precise definition of contamination (https://arxiv.org/abs/2311.04850), and as you pointed out -- paraphrasing, our decontamination pipeline exactly follows existing good practices, i.e., StarCoder to just remove exact match.

To truly understand the model performance, our approach is to evaluate the models on as more benchmarks as possible. I also want to mention two additional and independent evaluation efforts, LiveCodeBench -- a contamination free benchmark of fresh coding tasks, and EvoEval -- evolving existing benchmarks to formulate new coding problems. They all showed Magicoder's good performance.

To sum up, we believe a comprehensive evaluation is the way to resolve the contamination issue and it is possible as you said the model may overfit to one specific benchmark suite due to paraphrasing (e.g., HumanEval).

Hope this answers your question!

[Truth-In-Lies]

Thanks so much for your answer and for your amazing work. I think that HumanEval (and MBPP) has been widely leaked on the web, which makes it less useful as a benchmark. I've noticed that many models tend to generate problems similar to those in HumanEval. Maybe it's time for new benchmarks to take their place. Thanks again for your time and expertise!