llama3 with torch.compile used more memory

[songh11]

llama3 with torch.compile used more memory

[gante]

@zucchini-nlp In my experience the spikes are hardware-dependent, even when two devices have the same spare memory available.

@songh11 "You may might also notice that the second time we run our model with torch.compile is significantly slower than the other runs, although it is much faster than the first run. This is because the "reduce-overhead" mode runs a few warm-up iterations for CUDA graphs." (source)

[gante]

Hi @songh11 👋

If you check the documentation regarding torch.compile, especially relative to the "reduce-overhead" flag, you'll see an explanation :)

[zucchini-nlp]

Interestingly I didn't get sudden memory spike after second generation and after 5 steps the memory remained around 16GB 🤔 . My specs are:

PyTorch version: 2.3.0+cu121
CUDA used to build PyTorch: 12.1
OS: Ubuntu 20.04.6 LTS (x86_64) 
GPU: NVIDIA A100-SXM4-80GB

[songh11]

Interestingly I didn't get sudden memory spike after second generation and after 5 steps the memory remained around 16GB 🤔 . My specs are:

PyTorch version: 2.3.0+cu121
CUDA used to build PyTorch: 12.1
OS: Ubuntu 20.04.6 LTS (x86_64) 
GPU: NVIDIA A100-SXM4-80GB

NVIDIA RTX A5000, I think the second generation is also for warm-up.

[songh11]

Hi @songh11 👋

If you check the documentation regarding torch.compile, especially relative to the "reduce-overhead" flag, you'll see an explanation :)

Many thanks, another question I have is why does the second generate use more memory

[songh11]

@zucchini-nlp In my experience the spikes are hardware-dependent, even when two devices have the same spare memory available.

@songh11 "You may might also notice that the second time we run our model with torch.compile is significantly slower than the other runs, although it is much faster than the first run. This is because the "reduce-overhead" mode runs a few warm-up iterations for CUDA graphs." (source)

Thank you for your reply. I can use default to pass.