LoRA for trasnformers are typically only applied to Linear layers, how to achieve this with this package？

[TrueNobility303]

LoRA for trasnformers are typically only applied to Linear layers, how to achieve this with this package？

[davisyoshida]

@TrueNobility303 Can you elaborate on the question? The example given here should be a standard application of LoRA.

[TrueNobility303]

Thanks a lot for your lorax implementation! Now I know more about how to use it from your example

My original question is: given a transformer, how can I set only the Q, and V in attention layers to be LORA_FULL, while the other parameters LORA_FREEZE?

I think I have found the solution to my original question from the example you gave.
It seems that I should define my decision_fn and create a pytree with the same shape as params indicating how each parameter should be handled as. Then apply

    lora_spec = lorax.simple_spec(model.params, decision_fn=decision_fn, tune_vectors=True)
    lora_params = lorax.init_lora(model.params, lora_spec, jax.random.PRNGKey(0))

Thank you so much again for your sharing!

[davisyoshida]

Yup! Basically you can just specify which parameters you want it to affect in the decision function.