Will Megatron-DeepSpeed rebase from https://github.com/NVIDIA/Megatron-LM in the future?

[microsoft]

Will Megatron-DeepSpeed rebase from https://github.com/NVIDIA/Megatron-LM in the future?

[jiayulu]

I was about to post the same question, any plans to merge in newer commits from upstream? I did a manual merge but there's just way too many conflicts.

[conglongli]

@delock @jiayulu We are planning to revise our examples including those in Megatron-DeepSpeed, but we have not finalized the plan yet. Because of that, there is no planned rebase in near future.

[delock]

Thanks for clarification @conglongli

[delock]

@conglongli is there any update of the plan? In particular about plan for flash attention. Does Megatron-DeepSpeed plan to support flash attention training from picking from Megatron-LM, or plan to support through DeepSpeed op builder?

[conglongli]

@delock We had an internal discussion about your interest on fast attention (this is less related to Megatron-DeepSpeed plan), and my colleague Tunji @tjruwase would like to talk to you to learn more about this. Could you contact him next week? I guess you already have his contact. Thanks.

[conglongli]

@delock @jiayulu We have just synced with Megatron-LM and merged 1100+ commits. Please take a look at the new main branch. We actually replaced the main branch by renaming branches, so you might have to clone the repo again because it's a bit tricky to update local repo that already cloned https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-branches-in-your-repository/renaming-a-branch#updating-a-local-clone-after-a-branch-name-changes. There could still exist bugs, and we very welcome bug report/bug fix contributions.