l0sg/relational-rnn-pytorch

An implementation of DeepMind's Relational Recurrent Neural Networks (NeurIPS 2018) in PyTorch.

relational-rnn-pytorch's Issues

why not get logits from memory[:,-1, :] instead of func(memory[:,:mem_slots,:])

It is also reasonable

About Original Nth Farthest

Hey!

I've been wondering if you have tried the original Nth Farthest code (from Sonnet) on a 16GB Ram GPU. I keep running into memory errors no matter what I do (on a Volta GPU).

Wondering if you have any clue. (Sorry this is not directly related to your repository), just wondering if you got the original Sonnet version to work.

Thanks!

slow system memory leak of RMC model

Issue: while doing WikiText-103 benchmarks, the system memory usage of RMC increases very slowly. GPU VRAM remains stable.

Setup: anaconda, python 3.6, conda binary of PyTorch 0.4.1. Both CUDA 9.0 + CuDNN 7.1.2 and CUDA 9.2 + CuDNN 7.1.4 have same issues.

Problem: nn.CrossEntropyLoss wrapped inside the forward() of RelationalMemory may be the suspect. Removing it and calculating the loss in the training loop appears to remove the memory leak. But then the VRAM usage will be imbalanced if using multi-GPU with DataParallel.

Maybe related to reference cycle issue link, but it's fixed long ago and some toy examples (nn.CrossEntropyLoss inside the class) showed no leak.

Things tried: gc.collect() and del gc.gargabe[:] here and there. Didn't help.

Possible solution: Just giving up VRAM usage optimization. Have to use adaptive softmax anyway if using large-vocab dataset.