zhuangwang93/espresso

Hi-Speed DNN Training with Espresso: Unleashing the Full Potential of Gradient Compression with Near-Optimal Usage Strategies (EuroSys '23)

Reproduce Espresso Experiments

This repository contains the system code and scripts that help run the Espresso experiments from our EuroSys '23 paper.

Prerequisites

• CUDA == 11.1
• PyTorch >= 1.8.0
• NCCL >= 2.8.3

What machines?

All of our experiments require multiple GPU machines. We expect each GPU machine has eight V100 GPUs and the GPU memory is 32GB. NVLink-based (100Gbps TCP) and PCIe-only (25Gbps TCP) GPU machines are needed for Figure 11 and Figure 12. If resources are not available, it is fine to perform experiments with 4 GPU machines and use 25Gbps TCP for NVLink-based experiments. It is also fine to perform PCIe-only experiments on NVLink-based machines with export NCCL_P2P_DISABLE=1

Installation

Install Espresso on each GPU machine. Please make sure the machines can successfully install BytePS.

# In case you need to install PyTorch
pip3 install torch==1.8.0+cu111 torchvision==0.9.0+cu111 torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html

git clone https://github.com/zhuangwang93/Espresso.git --recursive
cd Espresso

# install dependencies and datasets for training
bash install.sh

AE file

Follow the instructions in espresso_EuroSys_AE.pdf to reproduce the experimental results in our paper.

End-to-end training throughput

The DNN models used by Espresso are in examples. There are six DNN models and each of them has a folder under examples/

Follow the instructions in this directory to reproduce the experiments results in Figure 11 and Figure 12.

espresso's Issues

Zeromq there is no socket to node or other dmlc errors

Hello
I want to run E2 (pcie) from artifact (I followed the instructions in examples folder) on a simple setup. Each machine has one gpu and I want a worker process and a server process to run in each machine. If I run it on two workers with:
bash run_pcie_models.sh 2 0 and bash run_pcie_models.sh 2 1 I get the error:

starting bps server
starting bps scheduler
[01:17:42] byteps/server/server.cc:451: BytePS server engine uses 4 threads, consider increasing BYTEPS_SERVER_ENGINE_THREAD for higher performance
[01:17:42] src/postoffice.cc:25: Creating Van: 0
[01:17:42] src/./zmq_van.h:299: Start ZMQ recv thread
[01:17:42] byteps/server/server.cc:451: BytePS server engine uses 4 threads, consider increasing BYTEPS_SERVER_ENGINE_THREAD for higher performance
[01:17:42] src/postoffice.cc:25: Creating Van: 0
[01:17:42] src/./zmq_van.h:299: Start ZMQ recv thread
NCCL version 2.11.4+cuda11.6
[01:17:48] src/postoffice.cc:25: Creating Van: 0
[01:17:48] src/./zmq_van.h:299: Start ZMQ recv thread
[01:17:48] 3rdparty/ps-lite/include/dmlc/logging.h:276: [01:17:48] src/./zmq_van.h:248: there is no socket to node 9

Stack trace returned 10 entries:
[bt] (0) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x42016) [0x7fb8e2eb8016]
[bt] (1) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x42391) [0x7fb8e2eb8391]
[bt] (2) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x677a3) [0x7fb8e2edd7a3]
[bt] (3) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x6793c) [0x7fb8e2edd93c]
[bt] (4) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x59274) [0x7fb8e2ecf274]
[bt] (5) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x61ff5) [0x7fb8e2ed7ff5]
[bt] (6) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x630b1) [0x7fb8e2ed90b1]
[bt] (7) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x63985) [0x7fb8e2ed9985]
[bt] (8) /lib/x86_64-linux-gnu/libstdc++.so.6(+0xcda93) [0x7fb8e2d2ca93]
[bt] (9) /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7fb8e38a5609]


terminate called after throwing an instance of 'dmlc::Error'
  what():  [01:17:48] src/./zmq_van.h:248: there is no socket to node 9

Stack trace returned 10 entries:
[bt] (0) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x42016) [0x7fb8e2eb8016]
[bt] (1) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x42391) [0x7fb8e2eb8391]
[bt] (2) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x677a3) [0x7fb8e2edd7a3]
[bt] (3) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x6793c) [0x7fb8e2edd93c]
[bt] (4) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x59274) [0x7fb8e2ecf274]
[bt] (5) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x61ff5) [0x7fb8e2ed7ff5]
[bt] (6) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x630b1) [0x7fb8e2ed90b1]
[bt] (7) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/server/c_lib.cpython-38-x86_64-linux-gnu.so(+0x63985) [0x7fb8e2ed9985]
[bt] (8) /lib/x86_64-linux-gnu/libstdc++.so.6(+0xcda93) [0x7fb8e2d2ca93]
[bt] (9) /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7fb8e38a5609]


Aborted (core dumped)
stop bps scheduler

On worker machine 1. Machine 2 keeps waiting:

[01:17:42] byteps/server/server.cc:451: BytePS server engine uses 4 threads, consider increasing BYTEPS_SERVER_ENGINE_THREAD for higher performance
[01:17:42] src/postoffice.cc:25: Creating Van: 0
[01:17:42] src/./zmq_van.h:299: Start ZMQ recv thread
NCCL version 2.11.4+cuda11.6
[01:17:47] src/postoffice.cc:25: Creating Van: 0
[01:17:47] src/./zmq_van.h:299: Start ZMQ recv thread

but if I run it on 4 workers using: bash run_pcie_models.sh 4 0 and bash run_pcie_models.sh 4 1 and bash run_pcie_models.sh 4 2 and bash run_pcie_models.sh 4 3 I get:

[01:04:44] byteps/server/server.cc:451: BytePS server engine uses 4 threads, consider increasing BYTEPS_SERVER_ENGINE_THREAD for higher performance
[01:04:44] src/postoffice.cc:25: Creating Van: 0
[01:04:44] src/./zmq_van.h:299: Start ZMQ recv thread
NCCL version 2.11.4+cuda11.6
[01:04:50] src/postoffice.cc:25: Creating Van: 0
[01:04:50] 3rdparty/ps-lite/include/dmlc/logging.h:276: [01:04:50] src/postoffice.cc:32: Check  notnull: Environment::Get()->find("DMLC_ROLE") 

Stack trace returned 10 entries:
[bt] (0) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/torch/c_lib.cpython-38-x86_64-linux-gnu.so(+0x5e246) [0x7f22032cd246]
[bt] (1) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/torch/c_lib.cpython-38-x86_64-linux-gnu.so(+0x5e5ca) [0x7f22032cd5ca]
[bt] (2) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/torch/c_lib.cpython-38-x86_64-linux-gnu.so(+0xb48dc) [0x7f22033238dc]
[bt] (3) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/torch/c_lib.cpython-38-x86_64-linux-gnu.so(+0xb8bd4) [0x7f2203327bd4]
[bt] (4) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/torch/c_lib.cpython-38-x86_64-linux-gnu.so(byteps::common::BytePSGlobal::GetOrInitPS()+0xa6) [0x7f22032de356]
[bt] (5) /home/lab/lq470379/miniconda3/lib/python3.8/lib-dynload/../../libffi.so.7(+0x69dd) [0x7f22fbe449dd]
[bt] (6) /home/lab/lq470379/miniconda3/lib/python3.8/lib-dynload/../../libffi.so.7(+0x6067) [0x7f22fbe44067]
[bt] (7) /home/lab/lq470379/miniconda3/lib/python3.8/lib-dynload/_ctypes.cpython-38-x86_64-linux-gnu.so(_ctypes_callproc+0x6d4) [0x7f22fbe5d5a4]
[bt] (8) /home/lab/lq470379/miniconda3/lib/python3.8/lib-dynload/_ctypes.cpython-38-x86_64-linux-gnu.so(+0x13c95) [0x7f22fbe5dc95]
[bt] (9) /home/lab/lq470379/miniconda3/bin/python3(_PyObject_MakeTpCall+0x3bf) [0x55e0d397a13f]


terminate called after throwing an instance of 'dmlc::Error'
  what():  [01:04:50] src/postoffice.cc:32: Check  notnull: Environment::Get()->find("DMLC_ROLE") 

Stack trace returned 10 entries:
[bt] (0) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/torch/c_lib.cpython-38-x86_64-linux-gnu.so(+0x5e246) [0x7f22032cd246]
[bt] (1) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/torch/c_lib.cpython-38-x86_64-linux-gnu.so(+0x5e5ca) [0x7f22032cd5ca]
[bt] (2) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/torch/c_lib.cpython-38-x86_64-linux-gnu.so(+0xb48dc) [0x7f22033238dc]
[bt] (3) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/torch/c_lib.cpython-38-x86_64-linux-gnu.so(+0xb8bd4) [0x7f2203327bd4]
[bt] (4) /home/lab/lq470379/.local/lib/python3.8/site-packages/byteps-0.2.5-py3.8-linux-x86_64.egg/byteps/torch/c_lib.cpython-38-x86_64-linux-gnu.so(byteps::common::BytePSGlobal::GetOrInitPS()+0xa6) [0x7f22032de356]
[bt] (5) /home/lab/lq470379/miniconda3/lib/python3.8/lib-dynload/../../libffi.so.7(+0x69dd) [0x7f22fbe449dd]
[bt] (6) /home/lab/lq470379/miniconda3/lib/python3.8/lib-dynload/../../libffi.so.7(+0x6067) [0x7f22fbe44067]
[bt] (7) /home/lab/lq470379/miniconda3/lib/python3.8/lib-dynload/_ctypes.cpython-38-x86_64-linux-gnu.so(_ctypes_callproc+0x6d4) [0x7f22fbe5d5a4]
[bt] (8) /home/lab/lq470379/miniconda3/lib/python3.8/lib-dynload/_ctypes.cpython-38-x86_64-linux-gnu.so(+0x13c95) [0x7f22fbe5dc95]
[bt] (9) /home/lab/lq470379/miniconda3/bin/python3(_PyObject_MakeTpCall+0x3bf) [0x55e0d397a13f]


Traceback (most recent call last):
  File "/home/lab/lq470379/miniconda3/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/lab/lq470379/miniconda3/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/lab/lq470379/miniconda3/lib/python3.8/site-packages/torch/distributed/launch.py", line 340, in <module>
    main()
  File "/home/lab/lq470379/miniconda3/lib/python3.8/site-packages/torch/distributed/launch.py", line 326, in main
    sigkill_handler(signal.SIGTERM, None)  # not coming back
  File "/home/lab/lq470379/miniconda3/lib/python3.8/site-packages/torch/distributed/launch.py", line 301, in sigkill_handler
    raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/lab/lq470379/miniconda3/bin/python3', '-u', './main.py', '--local_rank=0', '--model', 'vgg16', '--epochs', '5', '--batch-size', '32', '--speed_test']' died with <Signals.SIGABRT: 6>.
Killing subprocess 4181307

on worker machines 3,4. Other machines (1,2) keep waiting like above.

I don't know what is going wrong. Could you help?

Nan or infinite loss

I'm running vgg and resnet experiments baseline scripts and for any compression method, after a few iterations, the loss goes to infinity (nan) so the accuracy does not increase. Any clues why this happens?