skunkworksai/hydra-moe

hydra-moe's Issues

Simplify the process of Finetuning new experts and adding them to the MoE arch

Support / Sponsorig computing power

Hi, i had a similiar idea in mind and like to see there is already a group doing lora based moe.
I would like to contribute to that project with computing power, datasets and coding capaticity :)

Looking forward to talk

UI to run MoE inference and Base Model inference side by side

work on this started already at https://github.com/fearnworks/hydra-moe/tree/webui_prototype

Changes centered around :
https://github.com/fearnworks/hydra-moe/blob/webui_prototype/moe.py
https://github.com/fearnworks/hydra-moe/blob/webui_prototype/server.py

automate safety checks

Lets use pyupio/safety to autoscan all pip packages for vulnerabilities, especially as more ppl come along

https://github.com/pyupio/safety

How can I be part of the Discord server? Your link doesn't work.

Gating needs to be standardized to enable flexible switching of expert adapters from a larger db of adapters

Do you plan to publish the technical documentation of this project

update all dependencies and docker to latest possible

future proof the project from dependency hell

Improve the Multilora PEFT class extension code

(@sumo already has an implementation and will push it shortly)

License

Hi,
Great work on this repo! Would you mind adding a license?
Thanks!

Immediate To-Dos:

Immediate To-Dos:
Improve the Multilora PEFT class extension code ( @sumo already has an implementation and will push it shortly)
Gating needs to be standardized to enable flexible switching of expert adapters from a larger db of adapters (likely through centroid/similarity measures)
UI to run MoE inference and Base Model inference side by side (w streaming and display of selected experts during inference)
simplifying the process of Finetuning new experts and adding them to the MoE arch