Problem with xe_costs_en_fr_en loss when reloading checkpoint

[facebookresearch]

Problem with xe_costs_en_fr_en loss when reloading checkpoint

[pbordes]

Ok I got it! thank you very much

[glample]

Hi,

How are you doing the reload? Are you reloading a pretrained model? Or a checkpoint.pth file? In the first case, what you observe is possible (even though the difference seems surprisingly large), since the optimizer weights will also be reset, but in the case of a checkpoint, the loss should be the same before and after the training interruption, since the optimizer is also reloaded.

Maybe could you try to print the weights before and after reloading (both for the model and the optimizer) to see if they are identical?

[pbordes]

Hello,
Thank you for your quick answer ! I trained entirely a model (until the stopping criterion) and now I am trying to reload it: do you mean that when a model is trained until the end then the optimizer weights are not saved ? However, the penultimate function called in the main.py is end_epoch() which calls save_checkpoint(), and this function saves optimizer weights ...

[glample]

What is the name of the reloaded file? If it's something like best-bleu_en_fr.pth it will only contain the model. The checkpoint.pth file is to reload experiments that have crashed or have been interrupted for various reasons, and for which you want to resume the training.