Some questions about training settings

[Youggls]

Some questions about training settings

[Youggls]

Your experimental setup turns out to be quite different than ours. Firstly, we recommend training with total_effective_batch_size >= 256 for a steady loss curve. Since you are training with a single GPU, and assuming you can't increase PER_DEVICE_TRAIN_BATCH_SIZE to more than 8 without memory overflow, set gradient_accumulation_steps to 32 to reach the target total_effective_batch_size. Secondly, please stick to the default learning_rate of 1 with the lr_scheduler_type of "transformer" to reproduce our results. Please let us know if this resolves your issues.

Thanks a lot! I will try to retrain the model as your instruction.

[Youggls]

The per_lang_batch_size along with total_effective_batch_size dictates how many language pairs we sample from in a batch. In our case, we use total_effective_batch_size=256 with per_lang_batch_size=32, which means we sample mini-batches from 8 language pairs in a batch. So, for your previous settings, you'd have to set per_lang_batch_size=4, to mimic our settings for a total_effective_batch_size of 32.

However, since the batch size is quite small, you may not get the results mentioned in our paper. Let us know how it turns out. Please feel free to comment if you have any questions regarding the paper/experiment settings.

Thanks! I successfully reproduce your result with your instruction! Really nice work!

[abhik1505040]

Hi,
Thank you for appreciating our work. Our training hyperparameters are detailed in the trainer.sh script. However, the default training_batch_size hyperparameter assumes you are running the script with 8 GPUs. You would need to set this according to your training setup so that the total_effective_batch_size equals 256. If you can provide additional details like your previous experimental setup, arguments provided to trainer.sh etc., it'll be easier to figure out the problem.

[Youggls]

Hi, Thank you for appreciating our work. Our training hyperparameters are detailed in the trainer.sh script. However, the default training_batch_size hyperparameter assumes you are running the script with 8 GPUs. You would need to set this according to your training setup so that the total_effective_batch_size equals 256. If you can provide additional details like your previous experimental setup, arguments provided to trainer.sh etc., it'll be easier to figure out the problem.

Thanks for your explanation, I use bash trainer.sh --ngpus 1 --training_type m2o --pivot_lang english to train the model. My training settings basically follow your settings in train.sh. I set lr_scheduler_type to "constant_with_warmup. Firstly I run train.sh, I set learning_rate as default(1.0), and I quickly found that the training loss curve fluctuates a lot. So I set learning_rate to 5e-6. Others settings are same with the settings provided in train.sh. Given that I only use single GPU to train the model, the total_effective_batch_size equals 32 finally.

[abhik1505040]

Your experimental setup turns out to be quite different than ours. Firstly, we recommend training with total_effective_batch_size >= 256 for a steady loss curve. Since you are training with a single GPU, and assuming you can't increase PER_DEVICE_TRAIN_BATCH_SIZE to more than 8 without memory overflow, set gradient_accumulation_steps to 32 to reach the target total_effective_batch_size. Secondly, please stick to the default learning_rate of 1 with the lr_scheduler_type of "transformer" to reproduce our results. Please let us know if this resolves your issues.

[Youggls]

I'm sorry to disturb you, I just found there is a hyperparameter named per_lang_batch_size in trainer.sh. This parameter seems to influence sampling stage. Should I change the parameter to fit my previous settings?

[abhik1505040]

The per_lang_batch_size along with total_effective_batch_size dictates how many language pairs we sample from in a batch. In our case, we use total_effective_batch_size=256 with per_lang_batch_size=32, which means we sample mini-batches from 8 language pairs in a batch. So, for your previous settings, you'd have to set per_lang_batch_size=4, to mimic our settings for a total_effective_batch_size of 32.

However, since the batch size is quite small, you may not get the results mentioned in our paper. Let us know how it turns out. Please feel free to comment if you have any questions regarding the paper/experiment settings.