openucx/torch-ucc

pytorch ucc plugin

torch-ucc's Issues

Compilation error

Hi when I want to build torch-ucc, I got this error in torch_ucc.hpp:

error: expected ")" before opType 
OpType opType,

All output of compile:
https://paste.ofcode.org/G7KRkwGchkz4iFgNNiMYAh

Build env

• GCC 7.4.0
• python 3.6.1
• cuda 10.1
• ucx-v1.11.1
• ucc github latest
• torch-1.6.0-cu101

Compilation error

When I try to build I get a compilation error.

/home/torch-ucc/include/torch_ucc.hpp:20:10: fatal error: c10d/ProcessGroup.hpp: No such file or directory

Seems this header file is in the pytorch project and it's complaining this header file is missing. I run the build command in torch-ucc directory with

sudo UCX_HOME=/opt/ucx WITH_XCCL=/opt/xccl/install WITH_CUDA=/usr/local/cuda python setup.py clean --all install

Do I do anything wrong in the process of building?

Thanks!

Segfault when dist.all_reduce(torch.ones(2, device=torch.device("cuda", rank)) on DGX-1

Code to repro: https://gist.github.com/froody/c9e6749e877e0f0edfe412d079d73719
Command to repro: TORCH_UCC_COLL_BACKEND=xccl python bug.py
Output: https://gist.github.com/froody/f1c8107f7bd1a1650cdd47d64406a7d5

pytorch: 1.6.0
torch-ucc: a277d7d (master)
ucx: 1.9.0
cuda: 10.1