Expert merging: c-BTM

[mrcabbage972]

Expert merging: c-BTM

[kenhktsui]

I would also work on this too.
@NourFahmy There are two steps that we could split 😃

• Clustering of original training data used in different expert
• Inference code (that weight next token prediction logits according to proximity of input to cluster centroid)

[mrcabbage972]

@NourFahmy @kenhktsui
Check out Minho's adapation of the clustering step from the cBTM repo.

[NourFahmy]

Hi @kenhktsui - happy to take on inference and support where need be on clustering, and to fill any gaps from Minho's efforts.

I've put up a PR here

I've made the following assumptions I can easily fix:

• both the embedded context & current token will be passed to the script
• only the conditional probability of the token at time t given the context is needed, as per formula 2 in the paper

kindly inform if anything else is needed!

cc: @mrcabbage972