IA3 implementation doesn't add parameters for feedforward layers

[r-three]

IA3 implementation doesn't add parameters for feedforward layers

[muqeeth]

Hi, thank you for reporting the issue. The (IA)^3 implemented in this project is based on T5 1.1 models. The specific backbone model we utilized is T0, which is a multitask trained T5 1.1 LM adapted model. You can find T5 1.1 models on Hugging Face under the name google/t5_v_1_base. There are differences in architecture and pretraining data between T5 and T5 1.1 models. As a result, the configuration includes wi_1 for feedforward layers, whereas T5 has w_i. If you plan to use (IA)^3 for T5 models, you can modify the config accordingly.

[dionman]

how can this json be written for the case of falcon-7b?

[SumanthRH]

Hi @dionman, from the way I understand it, the current $(IA)^3$ code is a thin wrapper around a LoRA implementation. The implementation works only for T5 1.1 - like models with a certain feedforward architecture. For a standard transformer architecture like in BERT, GPT-2, the learned vector should get injected differently for the feedforward block vs the attention block (key, value layers). This is what I got from the paper atleast.

As of this writing, HuggingFace's PEFT has integrated $(IA)^3$. (My PR here: huggingface/peft#578). That should be simple to use for your use case!

Also, @muqeeth, do check out the current PEFT implementation!

[dionman]

Thanks @SumanthRH. Is this also handling the altered loss function and the normalization proposed in the T-Few recipe?

[SumanthRH]

No it does not! PEFT only implements $(IA)^3$, not the full TFew recipe!