NCCL_ERROR_SYSTEM_ERROR: unhandled system error

[Fhrozen]

NCCL_ERROR_SYSTEM_ERROR: unhandled system error

[keisukefukuda]

Hi @Fhrozen , thanks for trying ChainerMN.
First, which ChainerMN version are you using? (1.3 or master?)

About the issue of NCCL, NCCL is actually a black box library and the error message unhandled system error does not contain much information. I will discuss it with my colleagues anyways.

On the second issue, please try Open MPI 2.x series (2.1.3 would be a good choice). Open MPI 3.x has a bug on GPU-Direct communication (issue number 3792 on open-mpi/ompi repository, as indicated in our issue #221 ).
From the error messages on the first line, it seems you use UCX BTL component. The ompi bug is in openib component, so I'm not sure it really fixes the issue, but it worth trying.

Thanks,
Keisuke

[keisukefukuda]

Also, will you try again with the environmental variable NCCL_DEBUG=INFO ?
It is not very useful in many cases, but better than nothing in this case.

Thanks.
Keisuke

[Fhrozen]

@keisukefukuda Thank you for your support.
I am using the current version from pip 1.3.
I am running now some test with NCCL and MPI to check the full functionallity of the last. I also running some test with Open MPI 2.x. will update you as soon as I get any new information.