KeyError: 'generator' when continuing from checkpoint

[rosinality]

KeyError: 'generator' when continuing from checkpoint

[rosinality]

Sorry, [DIGITS].model file only saves running average of generator. Generators and discriminators are saved in the train-step-X.model file when train phase changed. Could you use that checkpoints?

[Ghostbeach]

Thank you for your quick answer. So the problem probably is, there is no train-step-X.model existing, because the process wasnt far enough executed to make the jump to 16x resolution, where it would be safed as far as i understand. Is there a easy way to modify the code, that training stops get safed more often than only on resolution jumps?

[Ghostbeach]

Or maybe i'll just need to get better hardware ( :

[rosinality]

Hmm I thought 20000 iterations are enough to change phases. You can copy these lines https://github.com/rosinality/style-based-gan-pytorch/blob/master/train.py#L100 to https://github.com/rosinality/style-based-gan-pytorch/blob/master/train.py#L241 this to save states for resume training. Sorry for confusion.

[MohammedAlghamdi]

Thank you for sharing your code.

I think also you need to reset --init_size to where it stopped training when resuming training otherwise it would restart from the 8 by 8.