ivotron/mlperf-workflows

Popperized MLPerf benchmark workflows

mlperf-benchmarks

Steps to configure the machine

1. Clone the repository.

git clone https://github.com/blkswanio/mlperf-benchmarks

2. Install docker, cuda-runtime and nvidia-docker on the machine.

cd mlperf-benchmarks/
./install_cuda_docker.sh

3. Install the popper tool.

pip install popper

You can also install it in a virtualenv.

Running the benchmarks

Object Detection

cd object_detection/
popper run -f main.yml -c settings.py

Sentiment Analysis

cd sentiment_analysis/
popper run -f main.yml -c settings.py

Single Stage Detector

cd single_stage_detector/
popper run -f main.yml -c settings.py

RNN translator

cd rnn_translator/
popper run -f main.yml -c settings.py

Translation

cd translation/
popper run -f main.yml -c settings.py

Benchmark output

results/
|----- ssd/
|      |--result_1.txt
|      |--result_2.txt
|      |--result_3.txt
|      |--result_4.txt
|      |--result_5.txt
|----- maskrcnn/
|----- gnmt/
|----- transformer/
|----- systems/
       |--system_details.json

On running the benchmarks, a directory structure like the one above would be generated in the root of the repository.

mlperf-workflows's Issues

Fix READMES

Fix .dockerignores and .gitignores

reorg folders

• rename output/ to results/
• create a training/open/ folder on the root of the project and move all the benchmarks within that folder.

add reporting step

add a reporting step to each benchmark that follows the result reporting policies https://github.com/mlperf/training_policies/blob/master/training_rules.adoc#run-results

Check for more than 1 run continuously

add dataset verification steps

for each benchmark, add a verification step that checks the checksum of the tarball to ensure that it has been correctly downloaded

make download scripts idempotent

For the download step:

1. obtain the list of files from zip/tar files
2. compare that with existing files.
3. skip download if all files are already present

move install commands from scripts to Dockerfile

Some benchmarks (e.g. object_detection) contain apt-get instructions on the scripts they use. We can move these to the Dockerfile to avoid having to re-run apt-get everytime the workflow runs.

This will also result in having any install step in the workflow being removed

create workflows for the 'closed' mlperf division

this is an umbrella issue for keeping track of workflows that comply with the Closed division outline in the MLPerf guidelines

Popperize other benchmarks

Add non cuda alternatives

Update README

allow for multiple results

instead of overwriting the results folder everytime a benchmark gets executed, we can instead obtain a timestamp when the benchmark starts, and create a folder using that timestamp. One cool thing would be to create a folder symlink within the results folder that points to the latest (e.g. a results/open/latest that points to the folder containing the latest execution)

Generate machine details in according to mlperf defined format

https://github.com/mlperf/policies/blob/master/submission_rules.adoc#56-system_desc_idjson-metadata

add an ssd-small inference workflow

take the ssd-small inference benchmark and create an end-to-end workflow for it, containing all required steps that adhere to the mlperf policies

Fix translation benchmark

1. Remove tensorflow/model dir after benchmarking in translation benchmark.
2. Confirm the type of model used.

Switch to global storage of output

Refactor object detection