Saved checkpoints corrupted?

[TalalWasim]

Saved checkpoints corrupted?

[linziyi96]

Do you encounter problems loading the CLIP backbone checkpoint or the video model checkpoint we provided? Would you mind sharing the md5sum of the checkpoint file and the PyTorch version you are using? Thanks.

[TalalWasim]

I have trouble loading the checkpoint saved by the training code when I run training. The one that is saved at each 5k iterations.

9f2669f93083aa5ee442394990c14c79 checkpoint-45000.pth

torch version: 1.11.0

[linziyi96]

So you mean you cannot open each saved checkpoint during training (including earlier checkpoints like checkpoint-5000.pth, checkpoint-10000.pth…)? Could you try saving and loading a toy model in a standalone script under the same environment? It looks like an odd problem and we probably need a sample of the faulty checkpoints for further testing…

…

On Wed, Sep 7, 2022 at 6:50 PM TalalWasim ***@***.***> wrote: I have trouble loading the checkpoint saved by the training code when I run training. The one that is saved at each 5k iterations. 9f2669f93083aa5ee442394990c14c79 checkpoint-45000.pth torch version: 1.11.0 — Reply to this email directly, view it on GitHub <#9 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AM3OG24IC2W3JI52YQWN3ELV5BXQ7ANCNFSM6AAAAAAQGUIJFY> . You are receiving this because you commented.Message ID: ***@***.***>

[liubo0902]

I have trouble loading the checkpoint saved by the training code when I run training. The one that is saved at each 5k iterations.

9f2669f93083aa5ee442394990c14c79 checkpoint-45000.pth

torch version: 1.11.0

Using multi-gpu training?
According to the checkpoint saving code, the saved checkpoint may be corrupted when using ddp training mode.