Since there is this paper MiniLLM I do not see any problem of having an general framework for Knowledge Distillation. The only limitation is that 2 models should share the same Tokenizer, and maybe booth have to have the same CausalLM Denoising objective, but I thing it would be enough that they predict token by token.
n1o / llmdistilery Goto Github PK
View Code? Open in Web Editor NEWLicense: MIT License