Giter Site home page Giter Site logo

deepreduce's Introduction

DeepReduce

A Sparse-tensor Communication Framework for Distributed Deep Learning

Abstrat: Sparse tensors appear frequently in distributed deep learning, either as a direct artifact of the deep neural network’s gradients, or as a result of an explicit sparsification process. Existing communication primitives are agnostic to the peculiarities of deep learning; consequently, they impose unnecessary communication overhead. We introduce DeepReduce, a versatile framework for the compressed communication of sparse tensors, tailored for distributed deep learning. DeepReduce decomposes sparse tensors in two sets, values and indices, and allows both independent and combined compression of these sets. We support a variety of common compressors, such as Deflate for values, or run-length encoding for indices. We also propose two novel compression schemes that achieve superior results: curve fitting-based for values and bloom filter-based for indices. DeepReduce is orthogonal to existing gradient sparsifiers and can be applied in conjunction with them, transparently to the end-user, to significantly lower the communication overhead. As proof of concept, we implement our approach on TensorFlow and PyTorch. Our experiments with large real models demonstrate that DeepReduce transmits fewer data and imposes lower computational overhead than existing methods, without affecting the training accuracy.

Prerequisites

The code is built with following libraries:

Benchmarks

We use the following benchmarks to run our experiments:

Usage

For the usage of GRACE and environment setup etc., please check the guides here.

First, create a GRACE instance from params. params should include parameters for both GRACE and DeepReduce. The valid parameter options for DeepReduce is listed as below:

'''
'deepreduce': None, 'value', 'index', 'both'
'value': None, 'polyfit', ...(other custom methods)
'index': None, 'bloom', ...(other custom methods)
'''
from grace_dl.dist.helper import grace_from_params
params = {'compressor': 'topk', 'memory': 'residual', 'communicator': 'allgather', 'compress_ratio': 0.01, 'deepreduce':'index', 'index':'bloom'}
grc = grace_from_params(params)

Once you get a desired GRACE instance, warp the compressor by DeepReduce. After that, you can use DeepReduce in the same way as GRACE.

deepreduce_wrapper = {'value': ValueCompressor,
                      'index': IndexCompressor,
                      'both': DeepReduce}
DReduce = deepreduce_wrapper[deepreduce](grc.compressor, params)
grc.compressor = DReduce

Scripts & Data

We provide the bash scripts to reproduce the experiments in our paper. All experiment results are also available at WANDB database.

Citation

If you find this useful, please cite our work as:

Kostopoulou K, Xu H, Dutta A, et al. DeepReduce: A Sparse-tensor Communication Framework for Distributed Deep Learning[J]. arXiv preprint arXiv:2102.03112, 2021.

Here's a BibTeX blurb:

@misc{kostopoulou2021deepreduce,
      title={DeepReduce: A Sparse-tensor Communication Framework for Distributed Deep Learning}, 
      author={Kelly Kostopoulou and Hang Xu and Aritra Dutta and Xin Li and Alexandros Ntoulas and Panos Kalnis},
      year={2021},
      eprint={2102.03112},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}

Publications

  • DeepReduce: A Sparse-tensor Communication Framework for Distributed Deep Learning [arXiv]

deepreduce's People

Contributors

hangxu0304 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.