How to finetune Llama-65b with this project？

[microsoft]

How to finetune Llama-65b with this project？

[martinjaggi]

you might use this fork of the project, which we just released. it supports full finetuning of Llama 1,2 and Falcon:
https://github.com/epfLLM/Megatron-LLM

[clumsy]

you might use this fork of the project, which we just released. it supports full finetuning of Llama 1,2 and Falcon:

https://github.com/epfLLM/Megatron-LLM

Hm, in this fork LLaMA simply extends GPT model?

I saw another fork with seemingly full implementation: https://github.com/ProjectD-AI/LLaMA-Megatron-DeepSpeed

[martinjaggi]

Hm, in this fork LLaMA simply extends GPT model?

yes

I saw another fork with seemingly full implementation: https://github.com/ProjectD-AI/LLaMA-Megatron-DeepSpeed

not sure about the status of that one. it doesn't seem to support Llama 2

[clumsy]

@martinjaggi do you plan on contributing the new bits to Megatron-LM? E.g. post_ln? Especially since other features are already present, like rotary_positional_embeddings, rms_norm

[martinjaggi]

we'd be happy to, but would maybe need some help from the team here to do so