Adding allreduce for ndarray

[Hakuyume]

Adding allreduce for ndarray

[kuenishi]

Glad to hear that. So far I'll be closing this issue for cleaning up the milestone list, but feel free to reopen this when you think you need more.

[Hakuyume]

And this is my case (similar to that of batch_normalization).
https://github.com/Hakuyume/chainercv/blob/ssd-multi-gpu-train/chainercv/links/model/ssd/multibox_loss.py#L91-L95

I had to handle MPI module manually.

[kuenishi]

@Hakuyume as #237 has been merged, this issue will be addressed at 1.3.0 release. So far only communicator will support allreduce, but do you think you need functions like AllGather ?

To see real use case, the URL of your code ( https://github.com/Hakuyume/chainercv/blob/ssd-multi-gpu-train/chainercv/links/model/ssd/multibox_loss.py#L91-L95 ) has been lost 404 so could you provide latest allreduce code if you still have it somewhere open?

[Hakuyume]

@kuenishi Thank you for supporting allreduce!

So far only communicator will support allreduce, but do you think you need functions like AllGather ?

For my case, function classes are not required.

To see real use case, the URL of your code ( https://github.com/Hakuyume/chainercv/blob/ssd-multi-gpu-train/chainercv/links/model/ssd/multibox_loss.py#L91-L95 ) has been lost 404 so could you provide latest allreduce code if you still have it somewhere open?

I'm sorry for removing the branch. Here is the latest code.
https://github.com/chainer/chainercv/blob/master/chainercv/links/model/ssd/multibox_loss.py#L91-L95

[Hakuyume]

@kuenishi Does #237 support reducing cupy.ndarray? According this line, it takes only numpy.ndarray.

[kuenishi]

Yes, it is not just documented, but _memory_utility.array_to_buffer_object() automatically handles both cupy and numpy array objects. It is tested over hierarchical and naive communicator but should work with other communicators. With pure_nccl communicator it needs stream synchronization before IIRC.

[Hakuyume]

Thank you.

[kuenishi]

Actually it wasn't implemented for cupy.ndarray and now #293 should fix it.

[ankahira]

The allreduce function doesn't seem to use NCCL or am I missing something?

[keisukefukuda]

Hi @ankahira ,
First, this repository is an old one. ChainerMN was merged into Chainer.
Also, Chainer is no longer maintained.

Besides, ChainerMN's PureNcclCommunicator uses NCCL. https://github.com/chainer/chainer/blob/master/chainermn/communicators/pure_nccl_communicator.py#L180