evaluation code

[shawnricecake]

evaluation code

[shawnricecake]

Yes, this is what we used.

Thanks, that helps a lot! I will implement this code!

[JLTastet]

Yes, we used the evaluation pipeline provided by the organisers of the BabyLM challenge (like most submissions probably did).

The code can be a bit fragile, but if you follow the installation instructions, it should in principle work. Also note that depending on the specific versions of dependencies (in particular transformers), you may obtain slightly different numerical results.

[shawnricecake]

Thanks for your response! by the way, did you use the finetune code in that repo on GLUE and MSGS dataset when writing the paper?

[JLTastet]

Yes, this is what we used.

[shawnricecake]

Hi, can I know why you use F1 score for MRPC and QQP and use MCC for others in (Super)GLUE dataset?

[JLTastet]

Hi. I just double-checked, and for this dataset the only reason is that this is what was specified by the challenge organisers in order to have consistent scores between the various submissions. I am not sure why they made this choice, but if I remember correctly, the evaluation code should compute all the metrics under the hood.

Note that unless specified otherwise, what is reported in fig. 3 is the accuracy, with the only exceptions being QQP and MRPC for which the $F_1$ score is reported, and CoLA for which the MCC is reported.