Benchmark peak GPU memory used

[soumith]

Benchmark peak GPU memory used

[soumith]

while i really want to see this as well, there's no easy way to do this, afaik. if someone knows how to measure GPU memory consumed exactly (other than running nvidia-smi at the exact time and hoping to catch a glimpse), i'd love to know.

[cancan101]

Theano has theano.sandbox.cuda.cuda_ndarray.cuda_ndarray.mem_info():
(https://github.com/Theano/Theano/blob/master/theano/sandbox/cuda/cuda_ndarray.cu#L3418)

I am not sure about the other frameworks. Also you have to make that function call.

[soumith]

right the Cuda API has cudaMemGetInfo, but the problem is that you dont know the peak memory usage while the kernels are running, you only get the mem usage before and after (or by chance in between, if the call itself is asynchronous and the timing is just right).

[soumith]

the functions that launch the kernels might have mallocs and frees before and after launching the kernel, we would want to be first-party to the mallocs before the frees happen.

[nouiz]

This can be run only between theano fct call. So you won't know the peak.

I see 2 ways to do this:

1. give the formula for the extra memory requested for the gpu.
2. run experiments bigger and bigger until it crash with missing memory.
   Then you compare the biggest one that was working.

On Thu, Jul 31, 2014 at 1:31 PM, Alex Rothberg [email protected]
wrote:

Theano has theano.sandbox.cuda.cuda_ndarray.cuda_ndarray.mem_info():
(
https://github.com/Theano/Theano/blob/master/theano/sandbox/cuda/cuda_ndarray.cu#L3418
)

I am not sure about the other frameworks. Also you have to make that
function call.

—
Reply to this email directly or view it on GitHub
#9 (comment)
.

[soumith]

i like @nouiz 's formula approach, we can calculate it from the method used.

[soumith]

@nouiz I can add the formulae, could you help out in adding time benchmarks for the backprop calls for Theano, i have finished backprop numbers for all the others.

[nouiz]

I'm pretty buzy, I don't know when I can do it. What do you need exactly, a
benchmark with the full forward/backward or just the backward? Is there an
example I can follow?

On Thu, Jul 31, 2014 at 1:50 PM, Soumith Chintala [email protected]
wrote:

@nouiz https://github.com/nouiz I can add the formulae, could you help
out in adding time benchmarks for the backprop calls for Theano, i have
finished backprop numbers for all the others.

—
Reply to this email directly or view it on GitHub
#9 (comment)
.

[soumith]

What do you need exactly, a benchmark with the full forward/backward or just the backward?

Adding backward timings to the current benchmark is what I'm looking for:
https://github.com/soumith/convnet-benchmarks/blob/master/theano/pylearn2_benchmark.py

[nicholas-leonard]

Okay so you want one timing for the entirety of the forward/backward.

[nicholas-leonard]

Oh, an just so we are clear, does this includes an inplace parameter update? (in torch, backward or backwardUpdate vs just updateGradInput).

[soumith]

Okay so you want one timing for the entirety of the forward/backward.

Correct.

Oh, an just so we are clear, does this includes an inplace parameter update?

backward (so updateGradInput + accGradParameters)

Thanks nick!

[nicholas-leonard]

French:
Ok donc Fred ce sera benchmark qui inclu le forward, ainsi qu'un update des parametres utilisant les gradient. Et le update peut être fait inplace.

[nicholas-leonard]

Okay so the theano backward is complete: #11 @f0k.