CUDA out of memory

[Garret012034]

CUDA out of memory

[pascal-pfeiffer]

LLM Studio needs quite a bit of GPU memory to fine-tune the large language models.
We employ state of the art techniques to reduce the amount of VRAM required for training.
Make sure to use LoRA, gradient checkpointing, low batch size (1) and int4 quantization.
May I ask which model you were trying to fine-tune and which GPU you have? With 10 GB VRAM and the above settings, it may be possible to fine-tune a 7B model, but not larger.

[Garret012034]

Thank you for your response. The model trained successfully. It's during execution, opening the 'Chat' tab, that it fails. I see this error in Ubuntu: torch.cuda.OutOfMemoryError. On the 'Config' tab, it shows Lora = True, Gradient Checkpointing = True, Batch Size = 2, and Backbone Dtype = int4.

Maybe I need to change the batch size to 1?

I'm running dual Nvidia GeForce RTX 3080s.

[pascal-pfeiffer]

I am trying to reproduce what could be different in the chat than for fine-tune. Both use the same logic, and locally, I see the memory peaking at around 15 GB for both when loading the checkpoints.
The batch_size is not used during inference/chat. It seems that the model exceeds max VRAM when loading the checkpoint.

[psinger]

There is a small memory leak after training, so if the weights are very tight it could oom.
Did you try restarting the whole application and then navigating to the Chat tab?

[Garret012034]

Yes, I did restart the application and restarted the machine just to be sure.

[maxjeblick]

Maybe we can init the model with empty weights before loading the checkpoint, haven't tested the workflow yet.
https://huggingface.co/docs/accelerate/v0.11.0/en/big_modeling