Fail to train with ORPOTrainer under multi-GPUs setting

[blaze7451]

Fail to train with ORPOTrainer under multi-GPUs setting

[blaze7451]

@alvarobartt hi, thx for your detailed explanation at another post, looks like I misuse device_map="auto" for training, I will follow your instruction to try later. big thx again.

[alvarobartt]

Hi here too @blaze7451 can you share with me a Gist with the full fine-tuning script so that I can provide you the FSDP configuration and command to run after testing that myself? But as mentioned in the other issue, if you use accelerate with FSDP and remove device_map="auto" you shouldn't encounter this issue. Thanks in advance!

[huangxinping]

@alvarobartt Hi buddy, thx. I will try again later.

[huangxinping]

@blaze7451 Hey, I'm facing the same issue, do you know how to fix it?

[Aisuko]

hi @alvarobartt, thanks for your response. I hit this issue too, from the same article. I have some experience with device_map="auto" to fine-tune models like phi-2, Gemma with multiple GPUs. It is really useful when the single GPU does not have enough memory to load the model and training data. Do you have any suggestion here that we can use ORPO on multiple GPUs in notebook environment(like kaggle)? Thank you.

[alvarobartt]

But the issue for you @Aisuko is that you're not able to fine-tune Llama 3B with device_map="auto" right? Or is it that you're just unable to use trl.ORPOTrainer + device_map="auto"?

[Aisuko]

But the issue for you @Aisuko is that you're not able to fine-tune Llama 3B with device_map="auto" right? Or is it that you're just unable to use trl.ORPOTrainer + device_map="auto"?

Hi @alvarobartt thanks for your quick replying. Let's make it clearly, can we use trl.ORPOTrainer to fine-tune model on multiple GPUs with device_map="auto" parameter?

[alvarobartt]

Hi @alvarobartt thanks for your quick replying. Let's make it clearly, can we use trl.ORPOTrainer to fine-tune model on multiple GPUs with device_map="auto" parameter?

Hi @Aisuko as mentioned before I don't have much experience on relying on device_map="auto" to fine-tune models, indeed I thought it was not even supported, but it seems it is with some tweaks. After reading some issues I realised that @younesbelkada answered a similar question for the KTOTrainer that can be easily solved apparently, please read #1560 (comment)