How to load the original model LM head in the new version of Adapters?

[Guitaricet]

How to load the original model LM head in the new version of Adapters?

[hSterz]

Hello @Guitaricet, yes you are correct the add_causal_lm_head adds a randomly initialized language modelling head. What you can do here is you can use the original transformers class and initialize it afterwards:

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
adapters.init(model)

This should solve the problem (let me know if it does not).

[Guitaricet]

Thank you, everything works now!