Only can train 1 epoch?

[yyangmei]

Only can train 1 epoch?

[YvanYin]

Hi, can you remove all the training codes part, just load the data in a loop to check if it can work?
For example:
for i, data in enumerate(train_dataloader): print(i)

[yyangmei]

Thank you,I remove all training codes and it can work.
But it can't work with training codes.
I want to train my own model, what can I do to solve it?

[YvanYin]

Hi, I have not faced this problem before.
What is your PyTorch version and CUDA version?
Where is the process stuck at?
Besides, our code will do the evaluation every 2000 iterations. Can you check if it is in the evaluation?

[yyangmei]

I use ipdb to check the code and I find when start to train the secend epoch the code can't pass for i, data in enumerate(train_dataloader).
My pytorch version is 1.1.0 and CUDA version is 9.0.

[YvanYin]

When you remove the network forward codes, is it can work?
Can for i, data in enumerate(train_dataloader) successfully run to the second epoch and successfully load the data?

[yyangmei]

Yes, if I remove the forward codes it can work and load data successfully on the secend epoch.

[YvanYin]

Hi, I think the problem maybe is the GPU memory. You can set a smaller batch size and monitor the GPU memory. You can see that without the forward codes it can smoothly load all data.

[yyangmei]

Thank you so much!
I have solved the problem by reconfiguring the environment.

[tnaderi]

Hi, Thank you for sharing your code. I have the same problem. It seems that it is able to train the model but when it is done with first epoch or get into evaluation it stop working just stay idle! I do not know how to fix this problem. Could you please help me in this regard, thank you