hoangt/ck-caffe

Collective Knowledge repository for evaluating and optimising performance of Caffe

Collective Knowledge repository for optimising Caffe-based designs

CK-Caffe is an open framework for collaborative and reproducible optimisation of convolutional neural networks. It's based on the Caffe framework from the Berkeley Vision and Learning Center (BVLC) and the Collective Knowledge framework from the cTuning Foundation. In essence, CK-Caffe is simply a suite of convenient wrappers for building, evaluating and optimising performance of Caffe.

As outlined in our vision, we invite the community to collaboratively design and optimize convolutional neural networks to meet the performance, accuracy and cost requirements for deployment on a range of form factors - from sensors to self-driving cars. To this end, CK-Caffe leverages the key capabilities of CK to crowdsource experimentation across diverse platforms, CNN designs, optimization options, and so on; exchange experimental data in a flexible JSON-based format; and apply leading-edge predictive analytics to extract valuable insights from the experimental data.

Examples

Compare accuracy of 4 CNNs

In this Jupyter notebook, we compare the Top-1 and Top-5 accuracy of 4 CNNs:

• AlexNet
• SqueezeNet 1.0
• SqueezeNet 1.1
• GoogleNet

on the Imagenet validation set (50,000 images).

We have thus independently verified that on this data set SqueezeNet matches (and even slightly exceeds) the accuracy of AlexNet.

The experimental data is stored in the main CK-Caffe repository under 'experiment'.

Compare performance of 4 CNNs on Chromebook 2

This notebook investigates effects on inference performance of varying the batch size:

• across the same 4 CNNs;
• with 4 BLAS libraries:
  • [CPU] OpenBLAS 0.2.18 (one thread per core);
  • [GPU] clBLAS 2.4 (OpenCL 1.1 compliant);
  • [GPU] CLBlast dev (35623cd > 0.8.0);
  • [GPU] CLBlast dev (35623cd > 0.8.0) with Mali-optimized overlay (641bb07);
• on the Samsung Chromebook 2 platform:
  • [CPU] quad-core ARM Cortex-A15 (@ 1900 MHz);
  • [GPU] quad-core ARM Mali-T628 (@ 600 MHz);
  • [GPU] OpenCL driver 6.0 (r6p0); OpenCL standard 1.1.

Finally, this notebook compares the best performance per image across the CNNs and BLAS libraries. When using OpenBLAS, SqueezeNet 1.1 is 2 times faster than SqueezeNet 1.0 and 2.4 times faster than AlexNet, broadly in line with expectations set by the SqueezeNet paper.

When using OpenCL BLAS libraries, however, SqueezeNet 1.0 is not necessarily faster than AlexNet, despite roughly 500 times reduction in the weights' size. This suggests that an optimal network design for a given task may depend on the software stack as well as on the hardware platform. Moreover, design choices may well shift over time, as software matures and new hardware becomes available. That's why we believe it is necessary to leverage community effort to collectively grow design and optimisation knowledge.

The experimental data and visualisation notebooks are stored in a separate repository which can be obtained as follows:

ck pull repo:ck-caffe-explore-batch-size-chromebook2 \
    --url=https://github.com/dividiti/ck-caffe-explore-batch-size-chromebook2.git

Contributors

• Anton Lokhmotov, dividiti
• Grigori Fursin, dividiti / cTuning foundation
• Unmesh Bordoloi, General Motors

License

• BSD (3 clause)

Status

Under development.

Installing CK-Caffe

Before installing CK-Caffe on the target system, several libraries and programs should be installed. So far, instructions for the following Linux flavours are available:

• Ubuntu
• Gentoo
• Yocto

Conventions

In this guide, shell commands prefixed with '$' should be run as user, whereas commands prefixed with '#' should be run as root (or as user with 'sudo').

For example, to install the 'pip' package manager and then Jupyter on Ubuntu, run as root:

# apt install python-pip
# pip install jupyter

or as user:

$ sudo apt install python-pip
$ sudo -H pip install jupyter

[Ubuntu] Installing CK-Caffe dependencies

Installing the dependencies is recommended via 'apt install' (for standard Ubuntu packages), or 'pip install' (for standard Python packages, typically of more recent versions than those available via 'apt install'). This can be simply done by opening a Linux shell and copying-and-pasting commands from cells below.

[Ubuntu] Installing core CK dependencies

Collective Knowledge has only two dependencies: Python (2.x and 3.x) and Git, which can be installed as follows:

# apt install  \
    python-dev \
    git

[Ubuntu] Installing common dependencies

Some CK packages and Caffe require common Linux utilities (e.g. make, cmake, wget), which can be installed as follows:

# apt install \
    coreutils \
    build-essential \
    make \
    cmake \
    wget \
    python-pip

[Ubuntu] Installing Caffe dependencies

The BVLC Caffe framework has quite a few dependencies. If you've already run Caffe on your machine, it's likely that you've already satisfied all of them. If not, however, you can easily install them in one gollop as follows:

# apt install \
    libboost-all-dev \
    libgflags-dev \
    libgoogle-glog-dev \
    libhdf5-serial-dev \
    liblmdb-dev \
    libleveldb-dev \
    libprotobuf-dev \
    protobuf-compiler \
    libsnappy-dev \
    libopencv-dev
# pip install \
    protobuf

[Ubuntu] Installing optional dependencies

# apt install \
    libatlas-base-dev \
# pip install \
    jupyter \
    pandas numpy scipy matplotlib \
    scikit-image scikit-learn \
    pyyaml

[Ubuntu] Checking all dependencies

You can check all the dependencies on an Ubuntu system by running this notebook. (View the output of this notebook on an Odroid XU3 board here.)

[Ubuntu] Installing CK

Please proceed to installing CK.

[Gentoo] Installing CK-Caffe dependencies

Installing the dependencies is recommended via 'emerge' (for standard Gentoo packages), or 'pip install' (for standard Python packages, typically of more recent versions than those available via 'emerge'). This can be simply done by opening a Linux shell and copying-and-pasting commands from cells below.

[Gentoo] Installing core CK dependencies

Collective Knowledge has only two dependencies: Python (2.x and 3.x) and Git, which can be installed as follows:

# emerge  \
    dev-lang/python \
    dev-vcs/git

[Gentoo] Installing common dependencies

Some CK packages and Caffe require common Linux utilities (e.g. make, cmake, wget), which can be installed as follows:

# emerge \
    sys-devel/gcc \
    sys-devel/make \
    dev-util/cmake \
    net-misc/wget \
    dev-python/pip

[Gentoo] Installing Caffe dependencies

The BVLC Caffe framework has quite a few dependencies. If you've already run Caffe on your machine, it's likely that you've already satisfied all of them. If not, however, you can easily install them in one gollop as follows:

# emerge \
    dev-libs/boost \
    dev-util/boost-build \
    dev-cpp/gflags \
    dev-cpp/glog \
    sci-libs/hdf5 \
    dev-db/lmdb \
    dev-libs/leveldb \
    dev-libs/protobuf \
    app-arch/snappy \
    media-libs/opencv
# pip install \
    protobuf

[Gentoo] Installing optional dependencies

# emerge \
    sci-libs/atlas
# pip install \
    jupyter \
    pandas numpy scipy matplotlib \
    scikit-image scikit-learn \
    pyyaml

[Gentoo] Installing CK

Please proceed to installing CK.

[Yocto] Installing CK-Caffe dependencies

NB: This section is work-in-progress.

NB: Not all CK-Caffe dependencies can be automatically installed on Yocto.

Add the following Bitbake layers to your bblayers.conf (e.g. build/build/conf/bblayers.conf):

BBLAYERS ?= " \
  ${TOPDIR}/../poky/meta \
  ${TOPDIR}/../poky/meta-yocto \
  ${TOPDIR}/../poky/meta-yocto-bsp \
  ...
  ${TOPDIR}/../meta-linaro/meta-linaro-toolchain \
  ${TOPDIR}/../meta-openembedded/meta-oe \
  ${TOPDIR}/../meta-openembedded/meta-networking \
  ${TOPDIR}/../meta-openembedded/meta-multimedia \
  ${TOPDIR}/../meta-openembedded/meta-python \
  "

[Yocto] Installing core CK dependencies

Collective Knowledge has only two dependencies: Python (2.x and 3.x) and Git.

Add the following to your image recipes:

# Core CK dependencies.
IMAGE_INSTALL_append... = " \
    python \
    git \
"

[Yocto] Installing common dependencies

Some CK packages and Caffe require common Linux utilities (e.g. make, cmake, wget).

Add the following to your image recipes:

# Common CK-Caffe dependencies.
IMAGE_INSTALL_append... = " \
    gcc \
    make \
    cmake \
    wget \
    zlib \
    python-setuptools \
    python-pip \
"

[Yocto] Installing Caffe dependencies

The BVLC Caffe framework has quite a few dependencies. If you've already run Caffe on your machine, it's likely that you've already satisfied all of them. If not, however, you may need to install some of them manually.

[Yocto] Installing Caffe dependencies automatically

You can install some of the Caffe dependencies automatically by adding the following to your image recipes:

# Caffe dependencies.
IMAGE_INSTALL_append... = " \
    boost \
    libunwind \
    glog \
    protobuf \
    leveldb \
    opencv \
    opencv-samples \
    libopencv-core \
    libopencv-highgui \
    libopencv-imgproc \
    libopencv-features2d \
    libopencv-calib3d \
    libopencv-flann \
    libopencv-ocl \
"

NB: This list probably overapproximates the real dependencies.

Also, run:

# pip install \
    protobuf

[Yocto] Installing Caffe dependencies manually

[Yocto] hdf5

TODO

[Yocto] lmdb

$ cd tmp/
$ wget https://github.com/LMDB/lmdb/archive/LMDB_0.9.18.tar.gz
$ tar xvzf LMDB_0.9.18.tar.gz
$ cd lmdf-LMDB-0.9.18/libraries/liblmdb
$ make && make install

NB: Installs to /usr/local/ by default.

[Yocto] gflags

$ cd /tmp
$ wget https://github.com/gflags/gflags/archive/v2.1.2.tar.gz
$ cd gflags-2.1.2
$ mkdir -p build && cd build
$ cmake .. -DCMAKE_CXX_FLAGS=-fPIC

[Yocto] snappy

$ cd /tmp
$ wget https://github.com/google/snappy/releases/download/1.1.3/snappy-1.1.3.tar.gz
$ tar xvzf snappy-1.1.3.tar.gz
$ cd snappy-1.1.3
$ ./configure
$ make -j4 && make install

NB: Installs to /usr/local/ by default.

[Yocto] Installing CK

Please proceed to installing CK.

Installing CK

Clone CK from GitHub into e.g. '$HOME/CK':

$ git clone https://github.com/ctuning/ck.git $HOME/CK

Add the following to your '$HOME/.bashrc' and run 'source ~/.bashrc' after that:

# Collective Knowledge.
export CK_ROOT=${HOME}/CK
export CK_REPOS=${HOME}/CK_REPOS
export CK_TOOLS=${HOME}/CK_TOOLS
export PATH=${HOME}/CK/bin:$PATH

Install the Python interface to CK:

$ cd $HOME/CK && sudo python setup.py install

Test that both the command line and Python interfaces work:

$ ck version
V1.8.2dev
$ python -c "import ck.kernel as ck; print (ck.__version__)"
V1.8.2dev

Installing CK Caffe

We are now ready to install and run CK-Caffe:

$ ck pull repo:ck-caffe --url=https://github.com/dividiti/ck-caffe
$ ck run program:caffe

Sample run

TBD

Misc hints

Creating dataset subsets

The ILSVRC2012 validation dataset contains 50K images. For quick experiments, you can create a subset of this dataset, as follows. Run:

$ ck install package:imagenet-2012-val-lmdb-256

When prompted, enter the number of images to convert to LMDB, say, N = 100. The first N images will be taken.

Setting environment variables

To set environment variables for running the program, use e.g.:

$ ck run program:caffe --env.CK_CAFFE_BATCH_SIZE=1 --env.CK_CAFFE_ITERATIONS=10

Our publications related to this project and crowd-tuning

• All references with BibTex