🐛 [BUG] Fail to restart and append: 'Trainer' object has no attribute 'iepoch'

[mir-group]

🐛 [BUG] Fail to restart and append: 'Trainer' object has no attribute 'iepoch'

[simonbatzner]

Hi @davidleocadio, which command did you run for the restart: It will not work if you run nequip-train with a restart config, you'll have to use nequip-restart for a restart. I also see you're running nequip 0.3.2., can you try this with the latest version 0.3.3?

Regarding restart vs requeue: requeue is an option that automatically figures out whether it's your first run or a restarted ran, so if it's your first run, it trains a new model, if it's a restarted ran, it restarts an existing one. Requeue is helpful for queueing systems in which you might get interrupted and your job is restarted, in this case you don't need to take care of anything.

I find it quite useful, here's a config I usually use:

nequip-requeue config.yaml

and then config.yaml has the following lines (you can also find an example under configs/requeue.yaml):

root: example-root
run_name: example-run_name
workdir: example-workdir

requeue: true
append: true    

[davidleocadio]

Hi @simonbatzner. I was training my networks using the procedure and code outlined in the Developer's Tutorial, and I was under the impression I could keep training the networks that way and simply modify the config.yaml with requeue: True ; append: True

But now I realize it's meant to work by training the networks like you comment above.

Also, with nequip 0.3.3, it all seems to work now.

Thanks for your help.