Par_bn and mam_adapter got "eval_matthews_correlation': 0.0" on GLUE-Cola

[wenjingk-xilinx]

Par_bn and mam_adapter got "eval_matthews_correlation': 0.0" on GLUE-Cola

[calpt]

Hey @wenjingk-xilinx,

From the script parameters you shared, it seems you're using a bert-large checkpoint. For successful training, you might need to tune the training hyperparameters for this model a bit. E.g. by lowering the learning rate or enabling warmup steps to help the model converge.

I was able to get a Matthews coefficient of ~0.546 after 1 epoch with your training setup (par_bn config) by lowering the learning rate to 5e-5 and setting --warmup_steps 200.

Hope this helps!

[wenjingk-xilinx]

Hi @calpt ,
Yeah, I tried with lower learning rate lr=5e-6, then get "eval_matthews_correlation=0.502" for one epoch. Thanks for your quick reply!