LoRA + FSDP -- issue

[declare-lab]

LoRA + FSDP -- issue

[chiayewken]

Hi, this may not work well, as it will unfreeze the model layers, causing much higher memory usage compared to normal lora. Unfortunately, lora does not seem to play well with fsdp in general, so I'm not sure if there is an easy fix: https://github.com/huggingface/peft#caveats

To speed up the training, it is recommended not to use fsdp if the model can fit on a single gpu, as it will cause gpu communication overhead. Instead, you can try using data parallel mode (assuming you have 8 GPUs):

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python training.py
--output_dir outputs/model/xl \
--use_lora \
--train_epochs 2 \
--data_path data/train.json \
--model_name_or_path "google/flan-t5-xl" \
--train_batch_size 1 \
--gradient_accumulation_steps 16

[ngun7]

Hello thanks for response, I tried and got OOM. So I reduced my batch_size to 4 still same error, whereas using fsdp instead of lora training works on 64 batch size.