starting bps server
starting bps scheduler
[01:17:42] byteps/server/server.cc:451: BytePS server engine uses 4 threads, consider increasing BYTEPS_SERVER_ENGINE_THREAD for higher performance
[01:17:42] src/postoffice.cc:25: Creating Van: 0
[01:17:42] src/./zmq_van.h:299: Start ZMQ recv thread
[01:17:42] byteps/server/server.cc:451: BytePS server engine uses 4 threads, consider increasing BYTEPS_SERVER_ENGINE_THREAD for higher performance
[01:17:42] src/postoffice.cc:25: Creating Van: 0
[01:17:42] src/./zmq_van.h:299: Start ZMQ recv thread
NCCL version 2.11.4+cuda11.6
[01:17:48] src/postoffice.cc:25: Creating Van: 0
[01:17:48] src/./zmq_van.h:299: Start ZMQ recv thread
[01:17:48] 3rdparty/ps-lite/include/dmlc/logging.h:276: [01:17:48] src/./zmq_van.h:248: there is no socket to node 9
Stack trace returned 10 entries:
[bt] (0) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x42016) [0x7fb8e2eb8016]
[bt] (1) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x42391) [0x7fb8e2eb8391]
[bt] (2) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x677a3) [0x7fb8e2edd7a3]
[bt] (3) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x6793c) [0x7fb8e2edd93c]
[bt] (4) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x59274) [0x7fb8e2ecf274]
[bt] (5) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x61ff5) [0x7fb8e2ed7ff5]
[bt] (6) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x630b1) [0x7fb8e2ed90b1]
[bt] (7) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x63985) [0x7fb8e2ed9985]
[bt] (8) /lib/x86_64-linux-gnu/libstdc++.so.6(+0xcda93) [0x7fb8e2d2ca93]
[bt] (9) /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7fb8e38a5609]
terminate called after throwing an instance of 'dmlc::Error'
what(): [01:17:48] src/./zmq_van.h:248: there is no socket to node 9
Stack trace returned 10 entries:
[bt] (0) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x42016) [0x7fb8e2eb8016]
[bt] (1) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x42391) [0x7fb8e2eb8391]
[bt] (2) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x677a3) [0x7fb8e2edd7a3]
[bt] (3) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x6793c) [0x7fb8e2edd93c]
[bt] (4) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x59274) [0x7fb8e2ecf274]
[bt] (5) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x61ff5) [0x7fb8e2ed7ff5]
[bt] (6) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x630b1) [0x7fb8e2ed90b1]
[bt] (7) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x63985) [0x7fb8e2ed9985]
[bt] (8) /lib/x86_64-linux-gnu/libstdc++.so.6(+0xcda93) [0x7fb8e2d2ca93]
[bt] (9) /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7fb8e38a5609]
Aborted (core dumped)
stop bps scheduler
On worker machine 1. Machine 2 keeps waiting:
[01:17:42] byteps/server/server.cc:451: BytePS server engine uses 4 threads, consider increasing BYTEPS_SERVER_ENGINE_THREAD for higher performance
[01:17:42] src/postoffice.cc:25: Creating Van: 0
[01:17:42] src/./zmq_van.h:299: Start ZMQ recv thread
NCCL version 2.11.4+cuda11.6
[01:17:47] src/postoffice.cc:25: Creating Van: 0
[01:17:47] src/./zmq_van.h:299: Start ZMQ recv thread
[01:04:44] byteps/server/server.cc:451: BytePS server engine uses 4 threads, consider increasing BYTEPS_SERVER_ENGINE_THREAD for higher performance
[01:04:44] src/postoffice.cc:25: Creating Van: 0
[01:04:44] src/./zmq_van.h:299: Start ZMQ recv thread
NCCL version 2.11.4+cuda11.6
[01:04:50] src/postoffice.cc:25: Creating Van: 0
[01:04:50] 3rdparty/ps-lite/include/dmlc/logging.h:276: [01:04:50] src/postoffice.cc:32: Check notnull: Environment::Get()->find("DMLC_ROLE")
Stack trace returned 10 entries:
[bt] (0) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/torch/c_lib.cpython-38-x86_64-linux-gnu.so(+0x5e246) [0x7f22032cd246]
[bt] (1) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/torch/c_lib.cpython-38-x86_64-linux-gnu.so(+0x5e5ca) [0x7f22032cd5ca]
[bt] (2) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/torch/c_lib.cpython-38-x86_64-linux-gnu.so(+0xb48dc) [0x7f22033238dc]
[bt] (3) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/torch/c_lib.cpython-38-x86_64-linux-gnu.so(+0xb8bd4) [0x7f2203327bd4]
[bt] (4) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/torch/c_lib.cpython-38-x86_64-linux-gnu.so(byteps::common::BytePSGlobal::GetOrInitPS()+0xa6) [0x7f22032de356]
[bt] (5) /home/lab/lq470379/miniconda3/lib/python3.8/lib-dynload/../../libffi.so.7(+0x69dd) [0x7f22fbe449dd]
[bt] (6) /home/lab/lq470379/miniconda3/lib/python3.8/lib-dynload/../../libffi.so.7(+0x6067) [0x7f22fbe44067]
[bt] (7) /home/lab/lq470379/miniconda3/lib/python3.8/lib-dynload/_ctypes.cpython-38-x86_64-linux-gnu.so(_ctypes_callproc+0x6d4) [0x7f22fbe5d5a4]
[bt] (8) /home/lab/lq470379/miniconda3/lib/python3.8/lib-dynload/_ctypes.cpython-38-x86_64-linux-gnu.so(+0x13c95) [0x7f22fbe5dc95]
[bt] (9) /home/lab/lq470379/miniconda3/bin/python3(_PyObject_MakeTpCall+0x3bf) [0x55e0d397a13f]
terminate called after throwing an instance of 'dmlc::Error'
what(): [01:04:50] src/postoffice.cc:32: Check notnull: Environment::Get()->find("DMLC_ROLE")
Stack trace returned 10 entries:
[bt] (0) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/torch/c_lib.cpython-38-x86_64-linux-gnu.so(+0x5e246) [0x7f22032cd246]
[bt] (1) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/torch/c_lib.cpython-38-x86_64-linux-gnu.so(+0x5e5ca) [0x7f22032cd5ca]
[bt] (2) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/torch/c_lib.cpython-38-x86_64-linux-gnu.so(+0xb48dc) [0x7f22033238dc]
[bt] (3) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/torch/c_lib.cpython-38-x86_64-linux-gnu.so(+0xb8bd4) [0x7f2203327bd4]
[bt] (4) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/torch/c_lib.cpython-38-x86_64-linux-gnu.so(byteps::common::BytePSGlobal::GetOrInitPS()+0xa6) [0x7f22032de356]
[bt] (5) /home/lab/lq470379/miniconda3/lib/python3.8/lib-dynload/../../libffi.so.7(+0x69dd) [0x7f22fbe449dd]
[bt] (6) /home/lab/lq470379/miniconda3/lib/python3.8/lib-dynload/../../libffi.so.7(+0x6067) [0x7f22fbe44067]
[bt] (7) /home/lab/lq470379/miniconda3/lib/python3.8/lib-dynload/_ctypes.cpython-38-x86_64-linux-gnu.so(_ctypes_callproc+0x6d4) [0x7f22fbe5d5a4]
[bt] (8) /home/lab/lq470379/miniconda3/lib/python3.8/lib-dynload/_ctypes.cpython-38-x86_64-linux-gnu.so(+0x13c95) [0x7f22fbe5dc95]
[bt] (9) /home/lab/lq470379/miniconda3/bin/python3(_PyObject_MakeTpCall+0x3bf) [0x55e0d397a13f]
Traceback (most recent call last):
File "/home/lab/lq470379/miniconda3/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/lab/lq470379/miniconda3/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/lab/lq470379/miniconda3/lib/python3.8/site-packages/torch/distributed/launch.py", line 340, in <module>
main()
File "/home/lab/lq470379/miniconda3/lib/python3.8/site-packages/torch/distributed/launch.py", line 326, in main
sigkill_handler(signal.SIGTERM, None) # not coming back
File "/home/lab/lq470379/miniconda3/lib/python3.8/site-packages/torch/distributed/launch.py", line 301, in sigkill_handler
raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/lab/lq470379/miniconda3/bin/python3', '-u', './main.py', '--local_rank=0', '--model', 'vgg16', '--epochs', '5', '--batch-size', '32', '--speed_test']' died with <Signals.SIGABRT: 6>.
Killing subprocess 4181307
on worker machines 3,4. Other machines (1,2) keep waiting like above.
I don't know what is going wrong. Could you help?