[Question] what are the usages of multiway_network.py?

[yiqiwang8177]

[Question] what are the usages of multiway_network.py?

[NormXU]

From my understanding, they are actually two different tricks.

Feedforward_network.py line 55 is used to initialize X-MoE in FFN, each FFN can have, let's say, 32 experts.

While BeitV3.py line 32 have two sets of FFN parameters, one is for text modality, the other is for image modality. They can be merged by the top FFN, which is named VL-FFN in the BERT-v3 paper.

Overall, X-MoE is for expanding the width of an encoder for a single modality, while the multiway is designed for multi-modalities

[shumingma]

Yes, moe_count is only used for the X-MoE implementation, while the multiway is for multimodal modeling as described in BEiT-3 paper.

We will update the README to make it clear.