GPU memory leak

[ncullen93]

GPU memory leak

[benwu232]

I met similar problem while using predict_loader. Still debugging.

[arunpatala]

I have defaulted the tensor to Cuda. Changing that seems to have fixed the issue

[ncullen93]

Hey sorry I've been out of town. Are you using regularizers or constraints?

[arunpatala]

I think train and val are using separate memory for input variables. May be they should be merged in a single loop

[recastrodiaz]

@ncullen93 I experienced the same issue as @arunpatala when running predict() and predict_loader() methods on large GPU datasets. A fix I found is to move all output predictions to cpu() before running torch.cats(). I have implemented this fix in one of my branches, happy to make a PR if you like the idea: https://github.com/recastrodiaz/torchsample/blob/19e80210868632597e14ba400b4ff72ac17e5697/torchsample/modules/module_trainer.py#L424

Whilst training, I have also experienced RAM Out Of Memory errors when using fit() and shuffle=True. I bypassed this issue by building a TensorDataset and a DataLoader with shuffle=True.

One option would be to make the fit() method internally call the fit_loader() function. This would avoid the need to manually create batches, and make the fit() method more flexible by allowing us to take advantage of the extra DataLoader settings such as shuffle, pin_memory, num_workers, batch_sampler, etc.