xy980523/moec_model

code for paper - MoEC: Mixture of Expert Clusters

moec_model's Issues

No module named 'unilm.data.bert_mlm_loader'

Description:
I am trying to run some code and I am running into the following error:

ModuleNotFoundError: No module named 'unilm.data.bert_mlm_loader'

I checked the unilm/data/ directory and there is no bert_mlm_loader file present. Can someone please help me with this issue?

Expected behavior:
The script should run without any errors.

Actual behavior:
I encounter the error ModuleNotFoundError: No module named 'unilm.data.bert_mlm_loader'

Code hangs at generate_moe..py dist.init_process_group() during inference

I am running inference using the script provided in the readme.md file, and the code seems to be getting stuck at the dist.init_process_group() function.

I am using a single A100 GPU to run the code.

Here is the relevant code snippet:

import torch.distributed as dist
import random
port = random.randint(10000, 20000)
distributed_backend = 'nccl'
distributed_init_method = "tcp://localhost:{port}".format(port=port)

dist.init_process_group(
    backend=distributed_backend,
    init_method=distributed_init_method,
    world_size=1,
    rank=0,
)

I have verified that the port variable is being set correctly, and I am not seeing any error messages. However, the code just seems to hang at this point and never progresses.

Can you suggest any potential causes for this issue, or any steps I can take to debug the problem? Any help would be greatly appreciated.