[FEATURE] Support full fine-tuning for DPO modeling

[tmostak]

[FEATURE] Support full fine-tuning for DPO modeling

[tmostak]

Thanks @pascal-pfeiffer for the explanation, that makes sense. Will try with LORA for now on top of an SFT step and see if that yields improvements.

[pascal-pfeiffer]

Thank you for raising the request.
DPO needs to compare the logits of the fine-tune with the base logits. For that we leverage LoRA, as we can easily disable and enable the extra layers without needing to have two copies of the same model in the memory. Allowing full tunes, would require twice as much GPU memory and additional code complexity which we believe is not worth the potential gain.

We suggest to do an SFT fine-tune first and use DPO as a second step. The need for full fine-tuning usually stems from a big difference to the pre-training. Doing SFT first, and then DPO removes this need from DPO.