Error in model training

[githubharald]

Error in model training

[ritzyag]

I could solve the tensorflow error message "2019-03-02 12:39:29.292787: E tensorflow/core/common_runtime/bfc_allocator.cc:373] tried to deallocate nullptr" by making a new virtual env and installing packages from scratch.

But the retraining of model from checkpoint still gives very high loss and very low accuracy. I do not understand why. When I have restored the weights from checkpoint then the training should start from the restored checkpoint and the loss should further decrease (on train data). But the loss shoots out of range in the second batch of first epoch itself. Please resolve this bug.

Training again from scratch every time is very monotonous.

Thank You!

[githubharald]

set learning rate to a fixed number, e.g. 0.0001:
https://github.com/githubharald/SimpleHTR/blob/master/src/Model.py#L212

[ritzyag]

Firstly, I would like to extend my heartful thanks to you for being so considerate in taking the time out to reply to our issues. It really means a lot! 💯

Now coming back to what you suggested I believe that might help. But if that is the issue then the loss in the first batch should also be very high. But when the retraining starts then the loss is very low for the first batch. Why? The learning rate is consistent for the first ten batches and so must the losses be during the first ten batches. Am I missing something?