GPU memory utilization puzzle

[akshatdewan]

GPU memory utilization puzzle

[albertz]

How do you measure the GPU memory usage? nvidia-smi will not be helpful here, as TF just takes all the GPU memory, because it uses its own custom memory allocator. You should use tf_log_memory_usage = True, and look at the log output. But even then, TF might behave different depending on the GPU. Maybe search a bit whether other people observe similar behavior, maybe someone reported sth similar in the TF issue tracker.

[akshatdewan]

Ah, okay, thanks!
Yes, I was using nvidia-smi for GPU memory usage.
It makes sense now:
For GTX 1080:
mem_usage:GPU:0: Stats(mean=6.8GB, std_dev=0.0B, min=6.8GB, max=6.8GB, num_seqs=4742, avg_data_len=1)
For V100:
mem_usage:GPU:1: Stats(mean=6.8GB, std_dev=0.0B, min=6.8GB, max=6.8GB, num_seqs=2126, avg_data_len=1)

Thanks a lot!