GPU memory hog

[juliagpu]

GPU memory hog

[jekbradbury]

Never freeing except on OOM is also the current behavior of other imperative deep learning frameworks (Chainer, PyTorch) because the latency of cudaMalloc is just so terrible that they want to avoid it at all costs.

[maleadt]

I was thinking of either freeing upon alloc, or an idle task (if free latency is high too), possibly with async API calls to reduce latency even further. Alternatively, if that turns out too slow, we can have a different limit specified by eg. environment variable.