Memory allocation/leak issue

[jcjohnson]

Memory allocation/leak issue

[ostrosablin]

Judging by your description, that sounds like kernel-mode memory leak, since keeping allocated memory after process ending would be not safe and would open the vector to perform DoS attack.

It could be, for instance, a faulty driver allocating excess memory and not freeing it up. Try using

sudo slabtop

It should give info on kernel-mode memory usage. It should not normally take up more than several hundreds of MBs (my machine keeps 200). If it takes a GB or more, there must be something bad going in kernel-space (nvidia drivers leak kernel memory?).

[buckohfive]

Interesting. The total space used under slabtop isn't much (~150MB) but running the sample.lua repeatedly caused hundreds of thousands of kmalloc-32 objects to be allocated and not destroyed (right now, my dev box has 1.47M kmalloc-32 objects).

[buckohfive]

Further testing. I can cause this memory leak to occur using either char-rnn or torch-rnn. Using keras (Theano), I don't get the same issue with similar networks.

I also note that the memory leak is consistent - 110 objects (ish - sometimes 111, sometimes 109) of 128 bytes each, regardless of network size.

Weird. I'm not sure how to proceed from here.

[buckohfive]

This is a cutorch error most likely. In sample.lua, if i halt execution error() after "require 'cutorch'", I can cause the memory leak. Before "require 'cutorch'", no leak.

I'll close this issue. It isn't torch-rnn :)

[buckohfive]

I put this over in cutorch, FYI: torch/cutorch#379