Failed to rerun predictor model

[unbabel]

Failed to rerun predictor model

[captainvera]

However, I'll take this chance to clarify some details about model re-training. I believe our documentation is slightly misleading as it is.

There are two ways to continue to train a model: 1 - Resuming and 2 - Loading

Resuming
When you are resuming, you are retrieving the previous state of the trainer. This includes everything related to the training process including, for example, optimizer parameters. The use case for this is things like preemptible servers where your training might stop and you might need to resume from the latest checkpoint. As such, you have to increase your total number of epochs if you want to continue training.

When resuming, Kiwi will first look for the trainer in your output-dir and then in runs/uuid if you don't specify the output-dir. In your case, since you state the output-dir there's no need to state the run-uuid.

Loading
On the other hand, if you just want to load model weights and continue training with new optimizer/trainer parameters you can just pass the load-model flag pointing at the model you'd like to load.

[captainvera]

Hello @damyana79

Thanks for reporting this issue! I'll look into this and get back to you soon.

[captainvera]

Hey @damyana79

I haven't been able to reproduce your issue with resuming. It seems to be happening because Kiwi cannot load the trainer and model from your previous run.
This means that for some reason it is looking in the wrong place. Would you mind sharing your output.log? This should be stored in the same folder as your model

[captainvera]

This behaviour as been clarified and improved with #36.

I will now close this. Feel free to re-open if your problem continues!