OOM on Ubuntu 14.04 with GTX670

[dennybritz]

OOM on Ubuntu 14.04 with GTX670

[dennybritz]

If your OOM you may need to make the model smaller by changing the hyperparameters (see https://github.com/dennybritz/chatbot-retrieval/blob/master/udc_hparams.py).

It will resume training if you set the model_dir parameter to where it saved the model previously (see code in train.py).

[JoeyTeng]

Is there anyway to reduce the usage of GPU memory but instead, save some data in memory such that I can train a big model with limited GPU memory?

And anyway, thanks:)

[dennybritz]

Try training with a smaller batch size (also in hparams). That means you matrix multiplications are smaller and need less memory.

[JoeyTeng]

Oh, I see... I wonder why OOM occur only after certain steps of calculation? Does the GPU memory save all the previous batches of data? Or to be specific, what are the data saved in GPU memory:)

[dennybritz]

I'm not sure. It could be related to the sequence length of the examples. E.g. if you have a really long sentence and the RNN needs to backpropagate through it you may run out of memory for that example.