Add script for merging expert models via weight averaging

[huu4ontocord]

Add script for merging expert models via weight averaging

[kenhktsui]

@mrcabbage972 I am interested to help! We could use lm-evaluation-harness to benchmark the merged model.
The seedLM EleutherAI/pythia-1b-deduped will be a great baseline.

[mrcabbage972]

@kenhktsui Great, please assign the ticket to yourself!

Regarding lm-evaluation-harness, can you please create a separate issue for that and add the details (e.g. on which tasks we are going to test)?

[kenhktsui]

@mrcabbage972 I had added the evaluation ticket.

For the merge, let's align and define terminology as I see there are different implementations so that we could assign different tickets to different contributors:

• c-BTM - which is a weighted logits of next token prediction
• element-wise averaging/ blending of model parameters
• mixture-of-experts

[mrcabbage972]

@kenhktsui Let's keep this ticket as element-wise averaging.
I created a separate one for c-BTM.

[kenhktsui]

@mrcabbage972 I think ticket has been done by Concedo and TeH_Venom. I would like to work on the c-BTM ticket.

[mrcabbage972]

@kenhktsui The version of Concedo's script that I saw only merges two experts, we need a solution to merge N.

To close the ticket, I think what is needed is a PR that:

1. adds the script to the repo
2. Extends it to support merging of N experts
3. Adds a section in the readme with usage instructions

If you prefer to focus on the c-BTM ticket, I can take this one.

[mrcabbage972]

May be able to load the models layer by layer