facebook/fai-pep

Facebook AI Performance Evaluation Platform

fai-pep's Issues

What's the magic behind "--platform android"

Hello, I'm trying to get benchmarks from my phone running MobilenetV2.

I was reading this document and got one question.

Just passing "--platform=android" is enough to run benchmark binaries? How does this tell where's my phone to the executed binary file? I thought I should set up some "adb" kind of things, but I couldn't find any mention of "adb". Now I am wondering if FAI-PEP has some android simulators that runs binary..

Could you tell me where should I refer to if I want to run FAI-PEP on the phone I connect?

Wiki needs update?

Hello all,
It seems that the Wiki of this project needs an update.
For example, "~/caffe2/pytorch" in https://github.com/facebook/FAI-PEP/wiki/Run-FAI-PEP-for-the-first-time has already been invalid because caffee2 is part of pytorch now.

If I use pytorch/caffe2 instead, the build.sh for caffe2@Android cannot generate caffe2_benchmark binary file as expected for the latest pytorch repo.

Thank you!

When running multiple tests, failures can be "swallowed"

We run multiple tests from a single config, and introduced an error in the second test in the group. This failure was not noticed, because each test reuses the same temp directory so report.json from the previous run is reused. That means that even though the run fails, we report data for the previous run.

Failed to run shuffleNet on host

I have tried to run shufflenet and modified inputs to gpu_0/data getting following issue
INFO 12:24:56 subprocess_with_logger.py: 24: Running: /tmp/FAI-PEP/libraries/python/imagenet_test_map.py --image-dir /tmp/imagenet/val --label-file /tmp/FAI-PEP/libraries/python/labels.txt --output-image-file /tmp/tmpLwDpGk/caffe2/host/images.txt --output-label-file /tmp/tmpLwDpGk/caffe2/host/labels.txt --shuffle
INFO 12:24:56 subprocess_with_logger.py: 24: Running: awk (NR>050000/1000)&&(NR<=050000/1000+50000/1000) {print > "/tmp/tmpLwDpGk/caffe2/host/inputs/labels_0.txt"} /tmp/tmpLwDpGk/caffe2/host/labels.txt
INFO 12:24:56 subprocess_with_logger.py: 24: Running: /tmp/config/exec/caffe2/host/incremental/2019/5/23/90182a7332997fb0edf666abc4b554b83a1670d1/convert_image_to_tensor --input_image_file /tmp/tmpLwDpGk/caffe2/host/inputs/labels_0.txt --output_tensor /tmp/tmpLwDpGk/caffe2/host/images_tensor.pb --batch_size 1 --scale 256,-1 --crop 224,224 --preprocess normalize,mean,std --report_time json|Caffe2Observer
INFO 12:24:57 hdb.py: 27: push /tmp/tmpLwDpGk/caffe2/host/images_tensor.pb to /tmp/tmpJ2TNu5/6ea951fe0a41/images_tensor.pb
INFO 12:24:57 hdb.py: 27: push /tmp/config/model_cache/caffe2/shufflenet/model.pb to /tmp/tmpJ2TNu5/6ea951fe0a41/model.pb
INFO 12:24:57 hdb.py: 27: push /tmp/config/model_cache/caffe2/shufflenet/model_init.pb to /tmp/tmpJ2TNu5/6ea951fe0a41/model_init.pb
{u'softmax': u'/tmp/tmpJ2TNu5/6ea951fe0a41/output/softmax.txt'}
INFO 12:24:57 subprocess_with_logger.py: 24: Running: /tmp/tmpJ2TNu5/6ea951fe0a41/caffe2_benchmark --net /tmp/tmpJ2TNu5/6ea951fe0a41/model.pb --init_net /tmp/tmpJ2TNu5/6ea951fe0a41/model_init.pb --warmup 0 --iter 50 --input gpu_0/data --input_file /tmp/tmpJ2TNu5/6ea951fe0a41/images_tensor.pb --input_type float --output gpu_0/softmax --text_output true --output_folder /tmp/tmpJ2TNu5/6ea951fe0a41/output

/tmp/tmpLwDpGk/caffe2

/tmp/tmpLwDpGk/caffe2/output
INFO 12:25:00 hdb.py: 38: pull /tmp/tmpJ2TNu5/6ea951fe0a41/output/softmax.txt to /tmp/tmpLwDpGk/caffe2/output/softmax.txt
INFO 12:25:00 hdb.py: 40: directory /tmp/tmpJ2TNu5/6ea951fe0a41/output
INFO 12:25:00 hdb.py: 46: filenames /tmp/tmpJ2TNu5/6ea951fe0a41/output/
INFO 12:25:00 benchmark_driver.py: 64: Exception caught when running benchmark
INFO 12:25:00 benchmark_driver.py: 65: [Errno 2] No such file or directory: u'/tmp/tmpJ2TNu5/6ea951fe0a41/output/softmax.txt'
ERROR 12:25:00 benchmark_driver.py: 69: Traceback (most recent call last):

As gpu_0 is prefixed in every node in shufflenet checkpoint.

What should be the directory the framework repo resides for run_bench.py

I tried benchmarking/run_bench.py -b specifications/models/tflite/mobilenet_v2/mobilenet_v2_1.4_224.json but don't know what should I enter at the prompt "Please enter the directory the framework repo resides" - should it be the local PyTorch or TF Lite repo directory? An example of this?

Contribution

Sorry for being creating an issue in this repo. Are you guys working on Django ?

Problems when running the experiment with docker

When trying to run the experiment with docker for TFLite example, I encountered the following error:

+ python /tmp/FAI-PEP/benchmarking/run_bench.py -b /tmp/FAI-PEP/specifications/models/tflite/mobilenet_v2/mobilenet_v2_0.35_96.json --config_dir /tmp/config
usage: run_bench.py [-h] [--app_id APP_ID] [-b BENCHMARK_FILE] [--lab]
                    [--logger_level {info,warning,error}] [--remote]
                    --root_model_dir ROOT_MODEL_DIR [--token TOKEN]
                    [-c CUSTOM_BINARY] [--pre_built_binary PRE_BUILT_BINARY]
                    [--user_string USER_STRING]
run_bench.py: error: argument --root_model_dir is require

Does anyone know how to resolve this? Did I missed anything before running the example?

Change dictionary syntax to use get

FAI-PEP/ailab/benchmark/templatetags/nvd3_tags.py

Lines 44 to 54 in 6f74b68

	if 'x_is_date' not in kw_extra:
	kw_extra['x_is_date'] = False
	if 'x_axis_format' not in kw_extra:
	kw_extra['x_axis_format'] = "%d %b %Y"
	if 'color_category' not in kw_extra:
	kw_extra['color_category'] = "category20"
	if 'tag_script_js' not in kw_extra:
	kw_extra['tag_script_js'] = True
	if 'chart_attr' not in kw_extra:
	kw_extra['chart_attr'] = {}
	# set the container name

The above syntax can be changed to the following.

kw_extra['x_is_date'] = kw_extra.get('x_is_date', False)
kw_extra['x_axis_format'] = kw_extra.get('x_axis_format', "%d %b %Y")
kw_extra['color_category'] = kw_extra.get('color_category', "category20")
kw_extra['tag_script_js'] = kw_extra.get('tag_script_js', True)
kw_extra['chart_attr'] = kw_extra.get('chart_attr', {})

UnboundLocalError: local variable 'abs_name' referenced before assignment

When I first run the command,

python ${FAI_PEP_DIR}/benchmarking/run_bench.py -b "${BENCHMARK_FILE}" --config_dir "${CONFIG_DIR}"

I encounter this message.
"UnboundLocalError: local variable 'abs_name' referenced before assignment"

However, when I ran the same command again, it disappears. I doubt .md5 autogeneration somehow broke at the first time.

What is meant by "without a battery" in top level README.md ?

In top level README one of the bullets about performance metrics says,

"energy/power : the energy per inference and average power of running the the ML model on a phone without battery"

I assume the "the the" is a typing error that should be "the".

But what about "without battery" ? If inference engine hardware in mobile phones could truly run without a battery that would be very low power indeed !! Is this a typo or if not, what is meant by this ?

-- jS

Example usage for iOS

Can you show an example on how to use the system with iOS?
Does the system compiles the .ipa that get sent to the iOS device or do we have to provide it?

Checking if the key exists in dictionary instead of using dict.get(key, None)

FAI-PEP/benchmarking/driver/benchmark_driver.py

Lines 35 to 36 in a26fa88

	if "shared_libs" in info:
	minfo["shared_libs"] = info["shared_libs"]

FAI-PEP/benchmarking/driver/benchmark_driver.py

Lines 42 to 43 in a26fa88

	if "shared_libs" in info:
	cinfo["shared_libs"] = info["shared_libs"]

Some places in this file FAI-PEP/benchmarking/driver/benchmark_driver.py has check if a key is in the dictionary and then tries to access it. It may be inefficient and less clean to go through
the dictionary twice. Can I raise a PR to change these to

minfo["shared_libs"] = info.get("shared_libs", "") 
cinfo["shared_libs"] = info.get("shared_libs", "")

[Proposal]Replace bazel build command with the binary provided from tensorflow document

bazel build command in specifications/frameworks/tflite/android/build.sh requires a lot of dependencies, like appropriate version of bazel, Android SDK and NDK. It's burdensome.

I found that the resulted binary is provided in here which is form https://www.tensorflow.org/lite/performance/measurement.

I commented out

#   --config=android_arm \
#   --cxxopt='--std=c++11' \
#   tensorflow/lite/tools/benchmark:benchmark_model

these lines and saved the downloaded binary in {tensorflow_dir}/bazel-bin/tensorflow/lite/tools/benchmark/benchmark_model

Then,

python ${FAI_PEP_DIR}/benchmarking/run_bench.py -b "${BENCHMARK_FILE}" --config_dir "${CONFIG_DIR}"

executed without problems.

Below is my configuration.

{
  \"--commit\": \"master\",
  \"--exec_dir\": \"${CONFIG_DIR}/exec\",
  \"--framework\": \"tflite\",
  \"--local_reporter\": \"${CONFIG_DIR}/reporter\",
  \"--model_cache\": \"${CONFIG_DIR}/model_cache\",
  \"--platforms\": \"android\",
  \"--remote_repository\": \"origin\",
  \"--repo\": \"git\",
  \"--repo_dir\": \"${REPO_DIR}\",
  \"--tmp_model_dir\": \"${CONFIG_DIR}/tmp_model_dir\",
  \"--root_model_dir\": \"${CONFIG_DIR}/root_model_dir\",
  \"--screen_reporter\": null
}

Path Manipulation

FAI-PEP/libraries/python/onnx_to_caffe2.py

Line 33 in 0295286

args = parser.parse_args()

FAI-PEP/libraries/python/onnx_to_caffe2.py

Line 38 in 0295286

with open(args.caffe2_init, "wb") as f:

Allowing user input to control paths used in file system operations could enable an attacker to access or modify otherwise protected system resources

Failed to run benchmark scripts in Android

Hi there,

I just new to here. So I typed by following the tutorial:

benchmarking/run_bench.py -b specifications/models/caffe2/shufflenet/shufflenet.json --platforms android

After a long time compiling, all of compile and link tasks are finished in the build_android folder in my pytorch repo. But it throws an error:

cmake unknown rule to install xxxx

It looks the caffe2_benchmark executable has been generated by failed to copied to the install folder, so I manually copied to the folder, namely:
/home/new/.aibench/git/exec/caffe2/android/2019/4/5/fefa6d305ea3e820afe64cec015d2f6746d9ca88
Then I modified repo_driver.py to avoid compiling again and run function _runBenchmarkSuites
But failed :

In file included from ../third_party/zstd/lib/common/pool.h:20:0,
from ../third_party/zstd/lib/common/pool.c:14:
../third_party/zstd/lib/common/zstd_internal.h:382:37: error: unknown type name ‘ZSTD_dictMode_e’; did you mean ‘FSE_decode_t’?
ZSTD_dictMode_e dictMode,
^~~~~~~~~~~~~~~
FSE_decode_t

Questions:

1. Any suggestions on how to run the tutorial correctly?
2. How to avoid the long time compiling for each time running

benchmarking/run_bench.py -b specifications/models/caffe2/shufflenet/shufflenet.json --platforms android

thanks!

General tutorial on running FAI-PEP

It would be great if you could add a general tutorial which allows practitioners to benchmark all sorts of models. The idea behind FAI-PEP is really good but all tutorials are geared towards image-classification models.

Is there a visualization component to understand models and their output?