mlcommons / inference Goto Github PK
View Code? Open in Web Editor NEWReference implementations of MLPerf™ inference benchmarks
Home Page: https://mlcommons.org/en/groups/inference
License: Apache License 2.0
Reference implementations of MLPerf™ inference benchmarks
Home Page: https://mlcommons.org/en/groups/inference
License: Apache License 2.0
I see the model url is https://zenodo.org/record/1492892/files/big-lstm.tar.gz
To clarify, would language modeling task base on this implementation, or base on Transformer language modeling, or both?
Is maskrcnn part of v0.5 benchmark? Where is the code under inference/cloud?
how does one go about disabling logging in the loadgen. Is there a compiler flag for that?
thanks
When we enable the pytorch SSD-MobileNet defined in "inference/edge/object_detection/ssd_mobilenet/pytorch/ssd_mobilenet_v1.py", we find out it just supports the batch_size=1 now.
In "ssd_mobilenet_v1.py" line153 (box_regression = box_regression.squeeze(0)), the batch_size information is removed by squeeze(0), and the successive "decode_boxes" function is performed on box_regression without batch_size information. Finally, line 167 (boxes = boxes[None]) expand the dims with batch_size=1.
If we want to get throughput, we need to revise the code to enable this model work well for batch_size>1, right?
Although I manually added the --count 500 in the starting shell script I still see in the log file (mlperf_log_detail.txt) GeneratedQueries: "queries" : 60000:
Trace:
"pid": 2101, "tid": 140124207314688, "ts": 253977ns : ERROR : SingleStream only supports a samples_per_query of 1.
"pid": 2101, "tid": 140124207314688, "ts": 307040ns : Starting verification mode:
"pid": 2101, "tid": 140124207314688, "ts": 310222ns : ERROR : Unsupported scenario. Only MultiStream supported.
"pid": 2101, "tid": 140124207314688, "ts": 310741ns : Starting performance mode:
"pid": 2101, "tid": 140124207314688, "ts": 381222999ns : GeneratedQueries: "queries" : 60000, "samples per query" : 1, "duration" : 60000000000
Don't know why it is behaving like this. :(
My command:
sudo make mlperf_loadgen_pymodule
Stack Trace:
cd third_party/ninja && python configure.py --bootstrap
bootstrapping ninja...
wrote build.ninja.
bootstrap complete. rebuilding...
[25/25] LINK ninja
# Generate gn's ninja build file
python third_party/gn/build/gen.py
# Build gn
third_party/ninja/ninja -C third_party/gn/out
ninja: Entering directory `third_party/gn/out'
ninja: no work to do.
# Copy gn to third_party/gn, where depot_tools expects it.
cp third_party/gn/out/gn* third_party/gn/.
third_party/gn/gn gen out/MakefileGnProj
Done. Made 24 targets from 41 files in 81ms
third_party/ninja/ninja -C out/MakefileGnProj loadgen_pymodule_wheel_src
ninja: Entering directory `out/MakefileGnProj'
[41/42] ACTION //:loadgen_pymodule_wheel_src(//build/toolchain/linux:x64)
FAILED: obj/dist/mlperf_loadgen-0.5a0.tar.gz
python gen/loadgen_pymodule_setup_src.py bdist_wheel
running bdist_wheel
running build
running build_ext
building 'mlperf_loadgen' extension
creating build
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/gen
creating build/temp.linux-x86_64-2.7/gen/loadgen
creating build/temp.linux-x86_64-2.7/gen/loadgen/bindings
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -Igen -Igen/third_party/pybind/include -I/usr/include/python2.7 -c gen/loadgen/bindings/python_api.cc -o build/temp.linux-x86_64-2.7/gen/loadgen/bindings/python_api.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
In file included from /usr/include/c++/5/forward_list:35:0,
from gen/third_party/pybind/include/pybind11/detail/common.h:140,
from gen/third_party/pybind/include/pybind11/pytypes.h:12,
from gen/third_party/pybind/include/pybind11/cast.h:13,
from gen/third_party/pybind/include/pybind11/attr.h:13,
from gen/third_party/pybind/include/pybind11/pybind11.h:44,
from gen/third_party/pybind/include/pybind11/functional.h:12,
from gen/loadgen/bindings/python_api.cc:6:
/usr/include/c++/5/bits/c++0x_warning.h:32:2: error: #error This file requires compiler and library support for the ISO C++ 2011 standard. This support must be enabled with the -std=c++11 or -std=gnu++11 compiler options.
#error This file requires compiler and library support \
^
In file included from gen/third_party/pybind/include/pybind11/pytypes.h:12:0,
from gen/third_party/pybind/include/pybind11/cast.h:13,
from gen/third_party/pybind/include/pybind11/attr.h:13,
from gen/third_party/pybind/include/pybind11/pybind11.h:44,
from gen/third_party/pybind/include/pybind11/functional.h:12,
from gen/loadgen/bindings/python_api.cc:6:
gen/third_party/pybind/include/pybind11/detail/common.h:355:1: warning: identifier ‘constexpr’ is a keyword in C++11 [-Wc++0x-compat]
inline static constexpr int log2(size_t n, int k = 0) { return (n <= 1) ? k : log2(n >> 1, k + 1); }
^
gen/third_party/pybind/include/pybind11/detail/common.h:367:5: warning: identifier ‘static_assert’ is a keyword in C++11 [-Wc++0x-compat]
static_assert(sizeof(std::shared_ptr<int>) >= sizeof(std::unique_ptr<int>),
^
gen/third_party/pybind/include/pybind11/detail/common.h:433:5: warning: identifier ‘nullptr’ is a keyword in C++11 [-Wc++0x-compat]
value_and_holder get_value_and_holder(const type_info *find_type = nullptr, bool throw_if_missing = true);
^
In file included from gen/third_party/pybind/include/pybind11/pytypes.h:12:0,
from gen/third_party/pybind/include/pybind11/cast.h:13,
from gen/third_party/pybind/include/pybind11/attr.h:13,
from gen/third_party/pybind/include/pybind11/pybind11.h:44,
from gen/third_party/pybind/include/pybind11/functional.h:12,
from gen/loadgen/bindings/python_api.cc:6:
gen/third_party/pybind/include/pybind11/detail/common.h:597:1: warning: identifier ‘decltype’ is a keyword in C++11 [-Wc++0x-compat]
using is_template_base_of = decltype(is_template_base_of_impl<Base>::check((intrinsic_t<T>*)nullptr));
^
In file included from gen/third_party/pybind/include/pybind11/cast.h:13:0,
from gen/third_party/pybind/include/pybind11/attr.h:13,
from gen/third_party/pybind/include/pybind11/pybind11.h:44,
from gen/third_party/pybind/include/pybind11/functional.h:12,
from gen/loadgen/bindings/python_api.cc:6:
gen/third_party/pybind/include/pybind11/pytypes.h:238:5: warning: identifier ‘noexcept’ is a keyword in C++11 [-Wc++0x-compat]
object(object &&other) noexcept { m_ptr = other.m_ptr; other.m_ptr = nullptr; }
^
In file included from gen/third_party/pybind/include/pybind11/pybind11.h:46:0,
from gen/third_party/pybind/include/pybind11/functional.h:12,
from gen/loadgen/bindings/python_api.cc:6:
gen/third_party/pybind/include/pybind11/detail/class.h:87:38: warning: missing terminating " character
PyObject *result = PyRun_String(R"(\
^
gen/third_party/pybind/include/pybind11/detail/class.h:95:10: warning: missing terminating " character
)", Py_file_input, d.ptr(), d.ptr()
^
In file included from gen/third_party/pybind/include/pybind11/pytypes.h:12:0,
from gen/third_party/pybind/include/pybind11/cast.h:13,
from gen/third_party/pybind/include/pybind11/attr.h:13,
from gen/third_party/pybind/include/pybind11/pybind11.h:44,
from gen/third_party/pybind/include/pybind11/functional.h:12,
from gen/loadgen/bindings/python_api.cc:6:
gen/third_party/pybind/include/pybind11/detail/common.h:298:7: error: expected nested-name-specifier before ‘ssize_t’
using ssize_t = Py_ssize_t;
^
gen/third_party/pybind/include/pybind11/detail/common.h:299:7: error: expected nested-name-specifier before ‘size_t’
using size_t = std::size_t;
^
gen/third_party/pybind/include/pybind11/detail/common.h:302:1: warning: scoped enums only available with -std=c++11 or -std=gnu++11
enum class return_value_policy : uint8_t {
^
gen/third_party/pybind/include/pybind11/detail/common.h:302:34: warning: scoped enums only available with -std=c++11 or -std=gnu++11
enum class return_value_policy : uint8_t {
^
gen/third_party/pybind/include/pybind11/detail/common.h:355:15: error: ‘constexpr’ does not name a type
inline static constexpr int log2(size_t n, int k = 0) { return (n <= 1) ? k : log2(n >> 1, k + 1); }
^
gen/third_party/pybind/include/pybind11/detail/common.h:355:15: note: C++11 ‘constexpr’ only available with -std=c++11 or -std=gnu++11
gen/third_party/pybind/include/pybind11/detail/common.h:358:15: error: ‘constexpr’ does not name a type
inline static constexpr size_t size_in_ptrs(size_t s) { return 1 + ((s - 1) >> log2(sizeof(void *))); }
^
gen/third_party/pybind/include/pybind11/detail/common.h:358:15: note: C++11 ‘constexpr’ only available with -std=c++11 or -std=gnu++11
gen/third_party/pybind/include/pybind11/detail/common.h:366:1: error: ‘constexpr’ does not name a type
constexpr size_t instance_simple_holder_in_ptrs() {
^
gen/third_party/pybind/include/pybind11/detail/common.h:366:1: note: C++11 ‘constexpr’ only available with -std=c++11 or -std=gnu++11
gen/third_party/pybind/include/pybind11/detail/common.h:386:70: error: ‘instance_simple_holder_in_ptrs’ was not declared in this scope
void *simple_value_holder[1 + instance_simple_holder_in_ptrs()];
^
gen/third_party/pybind/include/pybind11/detail/common.h:436:12: error: ‘constexpr’ does not name a type
static constexpr uint8_t status_holder_constructed = 1;
^
gen/third_party/pybind/include/pybind11/detail/common.h:436:12: note: C++11 ‘constexpr’ only available with -std=c++11 or -std=gnu++11
gen/third_party/pybind/include/pybind11/detail/common.h:437:12: error: ‘constexpr’ does not name a type
static constexpr uint8_t status_instance_registered = 2;
^
gen/third_party/pybind/include/pybind11/detail/common.h:437:12: note: C++11 ‘constexpr’ only available with -std=c++11 or -std=gnu++11
gen/third_party/pybind/include/pybind11/detail/common.h:433:72: error: ‘nullptr’ was not declared in this scope
value_and_holder get_value_and_holder(const type_info *find_type = nullptr, bool throw_if_missing = true);
^
gen/third_party/pybind/include/pybind11/detail/common.h:440:14: error: expected constructor, destructor, or type conversion before ‘(’ token
static_assert(std::is_standard_layout<instance>::value, "Internal error: `pybind11::detail::instance` is not standard layout!");
^
In file included from gen/third_party/pybind/include/pybind11/pytypes.h:12:0,
from gen/third_party/pybind/include/pybind11/cast.h:13,
from gen/third_party/pybind/include/pybind11/attr.h:13,
from gen/third_party/pybind/include/pybind11/pybind11.h:44,
from gen/third_party/pybind/include/pybind11/functional.h:12,
from gen/loadgen/bindings/python_api.cc:6:
gen/third_party/pybind/include/pybind11/detail/common.h:449:38: error: expected unqualified-id before ‘using’
template <bool B, typename T = void> using enable_if_t = typename std::enable_if<B, T>::type;
^
and a very long stack trace downwards. Thanks in advance!
The main https://github.com/mlperf/inference link doesn't have any README on it. We need to have some important details recorded there: e.g., dir structure, how to run one or all benchmarks. Pointer back to rules documents etc.
Hi, I find there are four scenarios in loadgen including SingleStream, MultiStream, Sever and Offline. However, the loadgen code now just supports the first three modes while for the Offline mode it just provides a API without function implementation. Do you plan to update the Offline model in loadgen? Or it should be implemented by the submitter?
root@ubuntu1:/home/data/mlperf_inference/mlperf_inference# make mlperf_loadgen
cd third_party/ninja && python configure.py --bootstrap
bootstrapping ninja...
wrote build.ninja.
bootstrap complete. rebuilding...
[25/25] LINK ninja
# Generate gn's ninja build file
python third_party/gn/build/gen.py
# Build gn
third_party/ninja/ninja -C third_party/gn/out
ninja: Entering directory `third_party/gn/out'
ninja: no work to do.
# Copy gn to third_party/gn, where depot_tools expects it.
cp third_party/gn/out/gn* third_party/gn/.
third_party/gn/gn gen out/MakefileGnProj
ERROR at //build/config/linux/pkg_config.gni:103:17: Script returned non-zero exit code.
pkgresult = exec_script(pkg_config_script, args, "value")
^----------
Current dir: /home/data/mlperf_inference/mlperf_inference/out/MakefileGnProj/
Command: python /home/data/mlperf_inference/mlperf_inference/build/config/linux/pkg-config.py glib-2.0 gmodule-2.0 gobject-2.0 gthread-2.0
Returned 1.
stderr:
File "/home/data/mlperf_inference/mlperf_inference/build/config/linux/pkg-config.py", line 57
print "You must specify an architecture via -a if using a sysroot."
^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print("You must specify an architecture via -a if using a sysroot.")?
See //build/config/linux/BUILD.gn:89:3: whence it was called.
pkg_config("glib") {
^-------------------
See //build/config/compiler/BUILD.gn:218:18: which caused the file to be included.
configs += [ "//build/config/linux:compiler" ]
^------------------------------
Makefile:22: recipe for target 'mlperf_loadgen' failed
make: *** [mlperf_loadgen] Error 1
Does anyone have this problem?
Updating repo "ck-env" ...
Local path: /root/CK/ck-env
URL: https://github.com/ctuning/ck-env
cd /root/CK/ck-env
git pull
fatal: No remote repository specified. Please, specify either a URL or a
remote name from which new revisions should be fetched.
WARNING: repository update likely failed OR IN A DIFFERENT BRANCH/CHECKOUT (git exit code: 256)
How do I solve this problem???
@psyhtest
root@ubuntu:/home/mlperf# ck compile program:image-classification-tf-cpp
CK error: [program] can't find path to CK entry "module:program" (?:032630d041b4fd8a:?)!
How can I fix it? Thanks!
just capturing the discussion outcome from today's meeting.
@guschmue
Hi, @jveejay
When C++ demo will update? And will QSL be provided in loadgen?
BR
need change. cloud/image_classification/python/main.py:286. just want to follow up. @guschmue
I see higher accuracy than reported in readme.
export DATA_DIR=/../CK-TOOLS/dataset-imagenet-ilsvrc2012-val/
$ ./run_local.sh tf resnet50 gpu --accuracy
Accuracy qps=129.15, mean=0.006666, time=387.14, acc=73.87, queries=50000, tiles=50.0:0.0060,80.0:0.0065,90.0:0.0089,95.0:0.0118,99.0:0.0120,99.9:0.0122
INFO 17:27:41 subprocess_with_logger.py: 24: Running: awk (NR>050000/1000)&&(NR<=050000/1000+50000/1000) {print > "/var/tmp/tmpL8e5eT/caffe2/host/inputs/labels_0.txt"} /var/tmp/tmpL8e5eT/caffe2/host/labels.txt
INFO 17:27:41 subprocess_with_logger.py: 73: awk: cannot open /var/tmp/tmpL8e5eT/caffe2/host/labels.txt (No such file or directory)
INFO 17:27:41 subprocess_with_logger.py: 24: Running: {convert_image_to_tensor} --input_image_file /var/tmp/tmpL8e5eT/caffe2/host/inputs/labels_0.txt --output_tensor /var/tmp/tmpL8e5eT/caffe2/host/images_tensor.pb --batch_size 1 --scale 256,-1 --crop 224,224 --preprocess normalize,mean,std --report_time json|Caffe2Observer
ERROR 17:27:41 subprocess_with_logger.py: 85: Unknown exception <type 'exceptions.OSError'>: {convert_image_to_tensor} --input_image_file /var/tmp/tmpL8e5eT/caffe2/host/inputs/labels_0.txt --output_tensor /var/tmp/tmpL8e5eT/caffe2/host/images_tensor.pb --batch_size 1 --scale 256,-1 --crop 224,224 --preprocess normalize,mean,std --report_time json|Caffe2Observer
INFO 17:27:41 hdb.py: 27: push /var/tmp/tmpL8e5eT/caffe2/host/images_tensor.pb to /var/tmp/tmprFiKA9/ubuntu/images_tensor.pb
INFO 17:27:41 benchmark_driver.py: 64: Exception caught when running benchmark
INFO 17:27:41 benchmark_driver.py: 65: [Errno 2] No such file or directory: u'/var/tmp/tmpL8e5eT/caffe2/host/images_tensor.pb'
ERROR 17:27:41 benchmark_driver.py: 69: Traceback (most recent call last):
File "/home/caffer/Downloads/shuff_tmp/tmp/FAI-PEP/benchmarking/driver/benchmark_driver.py", line 38, in runOneBenchmark
data = _runOnePass(minfo, mbenchmark, framework, platform)
File "/home/caffer/Downloads/shuff_tmp/tmp/FAI-PEP/benchmarking/driver/benchmark_driver.py", line 106, in _runOnePass
framework.runBenchmark(info, benchmark, platform)
File "/home/caffer/Downloads/shuff_tmp/tmp/FAI-PEP/benchmarking/frameworks/caffe2/caffe2.py", line 38, in runBenchmark
platform)
File "/home/caffer/Downloads/shuff_tmp/tmp/FAI-PEP/benchmarking/frameworks/framework_base.py", line 132, in runBenchmark
if input_files else None
File "/home/caffer/Downloads/shuff_tmp/tmp/FAI-PEP/benchmarking/platforms/platform_base.py", line 102, in copyFilesToPlatform
copy_files)
File "/home/caffer/Downloads/shuff_tmp/tmp/FAI-PEP/benchmarking/platforms/platform_base.py", line 90, in copyFilesToPlatform
self.util.push(files, target_file)
File "/home/caffer/Downloads/shuff_tmp/tmp/FAI-PEP/benchmarking/platforms/host/hdb.py", line 34, in push
shutil.copyfile(src, tgt)
File "/usr/lib/python2.7/shutil.py", line 96, in copyfile
with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: u'/var/tmp/tmpL8e5eT/caffe2/host/images_tensor.pb'
INFO 17:27:41 benchmark_driver.py: 83: No data collected for on Linux-4.18.0-17-generic-x86_64-with-Ubuntu-18.04-bionic--IntelR-CoreTM-i5-8250U-CPU--1.60GHz (ubuntu). The run may be failed for c50ded8
INFO 17:27:41 harness.py: 226: ======= user and harness error =======
Will Weight files be provided in demos?
Why is https://github.com/mlperf/inference/tree/master/v0.5/tools/upscale_coco sitting in a separate tools dir? Should we reorg it into the classification_and_detection directory?
With offline mode, the metric should be reported as samples per second rather than QPS, as there is only 1 query in the offline mode.
cat mlperf_log_summary.txt
SUT name : PySUT
Scenario : Offline
Mode : Performance
QPS: xxxxxxx
Result is : INVALID
Min duration satisfied : NO
Min queries satisfied : Yes
When I try to run the shufflenet, I cannot find the file of u'/Users/huaminli/.aibench/git/exec/caffe2/host/2019/1/2/eeb14675f12a19822d3a13f3d203c6396ee6cfdd/caffe2_benchmark'
I think that additional support for python should be added as the current code require to invoke the run from C. For instance if a part of the processing involves python (e.g. tokenization), why this should be precluded by the fact it has to be invoked from C?
Also, python support can ease the integration with the future reference benchmarks code.
currently this link does not show steps for v1 explicitly.
https://github.com/mlperf/inference/blob/master/edge/object_classification/mobilenets/README.md
The step to install the coco dataset from ck in the steps listed out in the edge/object_detection readme doesn't work.
Specifically, ck install package:dataset-coco-2014
.
We are trying to opt performance of SSD-ResNet34 inference 1200x1200 on tensorflow.
We did some explore and need check the accuracy.
So is there any trained model?
And if not, will there be any in future?
Hi all, I am getting the below error when running Cloud Inference - Sentiment Analysis Mxnet Implementation benchmark, some help?
command: python eval.py --model cnn --eval pretrained/seq2cnn_model --batch-size 1 --calc_accuracy
test: 39.84 % 0.8604417670682731
test: 39.88 % 0.8603811434302908
test: 39.92 % 0.8603206412825651
test: 39.96 % 0.8603603603603603
Finshed running 10000 batches
test: 40.0 % 0.8604
Traceback (most recent call last):
File "eval.py", line 104, in <module>
main()
File "eval.py", line 74, in main
batch.data[0]=batch.data[0].as_in_context(xpu)
AttributeError: type object 'StopIteration' has no attribute 'data'
Hi all, I want to run the deep speech inference using T4 and V100 GPUs models but I got the below warning and error:
Warning
warnings.warn(incorrect_binary_warn % (d, name, 9000, CUDA_VERSION))
/usr/local/lib/python2.7/dist-packages/torch/cuda/__init__.py:114: UserWarning:
Found GPU3 Tesla V100-SXM2-16GB which requires CUDA_VERSION >= 9000 for
optimal performance and fast startup time, but your PyTorch was compiled
with CUDA_VERSION 8000. Please install the correct PyTorch binary
using instructions from http://pytorch.org
Error:
RuntimeError: CUDNN_STATUS_MAPPING_ERROR
So, I decided to build the docker image using Pytorch with Cuda 9.0 or Cuda 10.0, none of them worked. Some suggestion?
The MLPerf inference models are residing in https://zenodo.org/communities/mlperf and there are duplicates in there. We need to clean-up the repo, along with GitHub.
Hi all, it looks like the sample inference/cloud/single_stage_detector was changed, I ran it before without issues but now it is throwing the below error :
THCudaCheck FAIL file=/pytorch/torch/csrc/cuda/Module.cpp line=34 error=35 : CUDA driver version is insufficient for CUDA runtime version
Traceback (most recent call last):
File "infer.py", line 156, in <module>
Using seed = 1
main()
File "infer.py", line 151, in main
torch.cuda.set_device(args.device)
File "/opt/conda/lib/python3.6/site-packages/torch/cuda/__init__.py", line 264, in set_device
torch._C._cuda_setDevice(device)
RuntimeError: cuda runtime error (35) : CUDA driver version is insufficient for CUDA runtime version at /pytorch/torch/csrc/cuda/Module.cpp:34
real 0m0.765s
user 0m1.120s
sys 0m2.000s
Seems like the implementation includes only cnn part. Is this implementation based
on https://arxiv.org/pdf/1805.07745.pdf?
I'm using tensorflow-gpu version ot Turing T4 for inferencing benchmarking.
Is the benchmark really using GPU for inferencing?
I got the code from mlperf/inference/cloud/translation/gnmt/tensorflow/.
I increased the # of iterations for 1 to 500000 in run_task.py.
It seems that the GPU is not being used for inferencing. Initially for first few seconds GPU utilization is 30%. Then nvidia-smi reports GPU utilization to be zero for most of the time for 500000 iterations.
It takes about 525 seconds to complete the run. For the first 125 or seconds, nvidia-smi shows any GPU utilization. After that nvidia-smi shows 0 utilization. Is the benchmark really using GPU for inferencing?
Here is is excerpt form the log of the run:
dynamic_seq2seq/decoder/output_projection/kernel:0, (1024, 32316),
2019-03-08 14:08:50.113076: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0
2019-03-08 14:08:50.113130: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-03-08 14:08:50.113139: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0
2019-03-08 14:08:50.113145: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0: N
2019-03-08 14:08:50.113335: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3318 MB memory) -> physical GPU (device: 0, name: GRID T4-4Q, pci bus id: 0000:02:01.0, compute capability: 7.5)
loaded infer model parameters from /home/ml/inference-master/cloud/translation/gnmt/tensorflow/ende_gnmt_model_4_layer/translate.ckpt, time 1.01s
decoding to output /home/ml/inference-master/cloud/translation/gnmt/tensorflow/nmt/data/output/g_nmt-out
Iterations 500000
done, num sentences 3003, num translations per input 1, time 35s, Fri Mar 8 14:09:27 2019.
done, num sentences 3003, num translations per input 1, time 35s, Fri Mar 8 14:09:27 2019.
done, num sentences 3003, num translations per input 1, time 35s, Fri Mar 8 14:09:27 2019.
done, num sentences 3003, num translations per input 1, time 35s, Fri Mar 8 14:09:27 2019.
done, num sentences 3003, num translations per input 1, time 35s, Fri Mar 8 14:09:27 2019.
done, num sentences 3003, num translations per input 1, time 35s, Fri Mar 8 14:09:27 2019.
Excerpt from nvidia-smi log: GPU utilization drops from 26% to 0 and then stays 0 for the rest of the run.
Fri Mar 8 14:09:27 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.92 Driver Version: 410.92 CUDA Version: 10.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GRID T4-4Q On | 00000000:02:01.0 Off | N/A |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 26% Default |
+-------------------------------+----------------------+----------------------+
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 26% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 26% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 25% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 24% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 24% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 23% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 22% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 21% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 21% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 20% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 19% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 18% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 18% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 17% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 16% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 15% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 14% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 14% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 13% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 12% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 11% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 11% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 10% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 9% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 8% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 8% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 7% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 6% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 5% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 5% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 4% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 3% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 2% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 2% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 1% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 0% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 0% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 0% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 0% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 0% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 0% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 0% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 0% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 0% Default |
| N/A N/A P0 N/A / N/A | 3851MiB / 4096MiB | 0% Default |
Thanks,
-Uday
When will the reference implementations for the inference of the model be added?
The weight provenance is not recorded for all the benchmarks and this is something that the README files should include. Need to have BM owners do this.
Hi,
I am try to run the ssd-resnet34 model in pytorch, but I got many issues on this model.
the out_size here cause miss understand. https://github.com/mlperf/inference/blob/master/cloud/single_stage_detector/pytorch/ssd_r34.py#L22
if the input size is 1200x1200, the out_size is 150, is 38 also works on 150 in function _build_additional_features
https://github.com/mlperf/inference/blob/master/cloud/single_stage_detector/pytorch/ssd_r34.py#L44
is the stride in loc and conf always 3?
https://github.com/mlperf/inference/blob/master/cloud/single_stage_detector/pytorch/ssd_r34.py#L36
the loc and conf shape miss match the comments, For SSD 300, shall return nbatch x 8732 x {nlabels, nlocs} results
https://github.com/mlperf/inference/blob/master/cloud/single_stage_detector/pytorch/ssd_r34.py#L143
the value I got is
(Pdb) print(ploc.shape)
torch.Size([1, 4, 1098])
(Pdb) print(plabel.shape)
torch.Size([1, 81, 1098])
Hi there,
I have the same error than when compiling by the training side, i.e. 👍
---> 5cb90201dd79
Step 13/14 : RUN cd ctcdecode; pip install .
---> Running in d406e114fed2
Processing /tmp/ctcdecode
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-req-build-Sq4p2m/setup.py", line 55, in
os.path.join(this_file, "build.py:ffi")
File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
_setup_distribution = dist = klass(attrs)
File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 272, in init
_Distribution.init(self,attrs)
File "/usr/lib/python2.7/distutils/dist.py", line 287, in init
self.finalize_options()
File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 327, in finalize_options
ep.load()(self, ep.name, value)
File "/usr/local/lib/python2.7/dist-packages/cffi/setuptools_ext.py", line 204, in cffi_modules
add_cffi_module(dist, cffi_module)
File "/usr/local/lib/python2.7/dist-packages/cffi/setuptools_ext.py", line 49, in add_cffi_module
execfile(build_file_name, mod_vars)
File "/usr/local/lib/python2.7/dist-packages/cffi/setuptools_ext.py", line 25, in execfile
exec(code, glob, glob)
File "/tmp/pip-req-build-Sq4p2m/build.py", line 28, in
'third_party/boost_1_67_0.tar.gz')
File "/tmp/pip-req-build-Sq4p2m/build.py", line 19, in download_extract
tar = tarfile.open(dl_path)
File "/usr/lib/python2.7/tarfile.py", line 1678, in open
raise ReadError("file could not be opened successfully")
tarfile.ReadError: file could not be opened successfully
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-req-build-Sq4p2m/
The command '/bin/sh -c cd ctcdecode; pip install .' returned a non-zero code: 1
I run shufflenet network by executing ./run.sh imagenet group1 or ./run.sh imagenet group2. The accuracy of different groups is the same, why did it turn out this result?
For group1(./run.sh group1):
NET: shufflenet METRIC: accuracy ID: 0
PLATFORM: Linux-4.18.0-17-generic-x86_64-with-Ubuntu-18.04-bionic--IntelR-CoreTM-i5-8250U-CPU--1.60GHz HASH: ubuntu
FRAMEWORK: caffe2 COMMIT: 0ebe252c9c81f64a3604b7b340550353c0537566 TIME: 2019-05-08 15:58:19
NET latency: value median 33488.19351 MAD: 1049.27826
image_preprocess convert: value median 2865.79500 MAD: 463.04500
preprocess data_pack: value median 74817.60000 MAD: 1008.20000
shufflenet total_number_of_top1_corrects: value median 33234.00000 MAD: 0.00000
shufflenet total_percent_of_top1_corrects: value median 66.46800 MAD: 0.00000
INFO 20:09:58 harness.py: 235: ======= success =======
INFO 20:09:58 repo_driver.py: 360: One benchmark run successful for 0ebe252c9c81f64a3604b7b340550353c0537566
For group8: (./run.sh group8)
NET: shufflenet METRIC: accuracy ID: 0
PLATFORM: Linux-4.18.0-17-generic-x86_64-with-Ubuntu-18.04-bionic--IntelR-CoreTM-i5-8250U-CPU--1.60GHz HASH: ubuntu
FRAMEWORK: caffe2 COMMIT: 0ebe252c9c81f64a3604b7b340550353c0537566 TIME: 2019-05-08 15:58:19
NET latency: value median 14934.82733 MAD: 212.27360
image_preprocess convert: value median 1397.55500 MAD: 229.61500
preprocess data_pack: value median 31351.20000 MAD: 324.00000
shufflenet total_number_of_top1_corrects: value median 33234.00000 MAD: 0.00000
shufflenet total_percent_of_top1_corrects: value median 66.46800 MAD: 0.00000
INFO 11:20:34 harness.py: 235: ======= success =======
INFO 11:20:34 repo_driver.py: 360: One benchmark run successful for 0ebe252c9c81f64a3604b7b340550353c0537566
hi all, how can I run cloud Language Modeling inference benchmark with GPU support?. Thanks, Vilmara
in the file https://github.com/mlperf/inference/blob/master/cloud/language_modeling/README it is specified to start benchmark.py with python, but urllib.request does not work. It only exists with python3.
Hi all,
Is the load generator ready to use to run automatically the MLPerf inferences? If so, what are the models already integrated to it?
Also, I have tried to build the LoadGen as a python module and as a C++ library but I got errors with both methods:
To build as a python module:
$ /mlperf_inference/loadgen# sudo -H pip install dist/mlperf_loadgen-0.5a0-cp27-cp27mu-linux_x86_64.whl
ERROR: mlperf_loadgen-0.5a0-cp27-cp27mu-linux_x86_64.whl is not a supported wheel on this platform.
To build as a C++ library:
$ /mlperf_inference# make mlperf_loadgen
cd third_party/ninja && python configure.py --bootstrap
bootstrapping ninja...
warning: A compatible version of re2c (>= 0.11.3) was not found; changes to src/*.in.cc will not affect your build.
wrote build.ninja.
bootstrap complete. rebuilding...
[25/25] LINK ninja
# Generate gn's ninja build file
python third_party/gn/build/gen.py
# Build gn
third_party/ninja/ninja -C third_party/gn/out
ninja: Entering directory `third_party/gn/out'
[1/246] CXX tools/gn/inherited_libraries.o
FAILED: tools/gn/inherited_libraries.o
clang++ -MMD -MF tools/gn/inherited_libraries.o.d -I/home/mlperf_inference/third_party/gn -I/home/mlperf_inference/third_party/gn/out -DNDEBUG -O3 -fdata-sections -ffunction-sections -D_FILE_OFFSET_BITS=64 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -pthread -pipe -fno-exceptions -fno-rtti -fdiagnostics-color -std=c++14 -Wno-c++11-narrowing -c /home/mlperf_inference/third_party/gn/tools/gn/inherited_libraries.cc -o tools/gn/inherited_libraries.o
/bin/sh: 1: clang++: not found
[2/246] CXX base/callback_internal.o
..... a bunch of FAILED messages
Some help how to fix the issue?
In run.sh of shufflenet network, it's described as:
# The downloaded images should be in the following directory structure
# ${IMAGENET_DIR}/labels.txt
# ${IMAGENET_DIR}/val/n*
I cannot find the labels.txt. Although, I find a seem-like text file named ILSVRC2012_validation_ground_truth.txt, but the format is not as requested in imagenet_test_map.py.
Can you provide the path to downloads labels.txt?
Just a friendly reminder that coco 2014 dataset using in the following is removed.
https://github.com/mlperf/inference/tree/master/edge/object_detection/ssd_resnet50
Can try to use this to get the data:
ck install package --tags=dataset,coco,2014
When I try to run inference on with with a NCHW model I get the following error.
2019-05-04 13:45:18.523575: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x562c16a7e780 executing computations on platform Host. Devices:
2019-05-04 13:45:18.523609: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
ERROR:main:execute_parallel thread: Cannot feed value of shape (1, 224, 224, 3) for Tensor 'data:0', which has shape '(1, 3, 224, 224)'
ERROR:main:execute_parallel thread: Cannot feed value of shape (1, 224, 224, 3) for Tensor 'data:0', which has shape '(1, 3, 224, 224)'
I do understand it is complaining about the NCHW format while it expects the NHWC format. Is changing the python/main.py enough?
SUPPORTED_DATASETS = {
"imagenet":
(imagenet.Imagenet, dataset.pre_process_vgg, dataset.post_process_offset1,
{"image_size": [3, 224, 224]}),
"imagenet_mobilenet":
(imagenet.Imagenet, dataset.pre_process_mobilenet, dataset.post_process_argmax_offset,
{"image_size": [224, 224, 3]}),
}
Changing the model could be one option, but I want to make mlperf work with NCHW
Any help is much appreciated!
Latest verified:
date 5/15
commit: fd037e6
tested commands:
under cloud/image_classification
./run_local.sh tf cpu --time 100
./run_local.sh tf gpu --time 100
--time 100 allows you to run a shorter version of the code
what to expect:
Hi , I'm following those instructions: https://github.com/mlperf/inference/tree/master/cloud/translation/gnmt/pytorch
While in this command :"bash download_dataset.sh"
I'm having this Error:"
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 6727: ordinal not in range(128)"
Any Idea?
Thanks.
Hi, could you please provide the TensorFlow Resnet50 v1.5 Pre-trained in NCHW data format to run the inference located at inference/cloud/image_classification/?
Also could you please confirm what is the precision mode of the pre-trained model https://zenodo.org/record/2535873/files/resnet50_v1.pb
Thanks,
Vilmara
root@ubuntu:/home/mlperf# ck install package:lib-tensorflow-1.13.1-src-static
CK error: [package] can't find path to CK entry "package:lib-tensorflow-1.13.1-src-static" (?:1dc07ee0f4742028:?)!
@psyhtest
hi all, I am running the inference single state detector benchmark with GPU support and I got the below error:
$ ./run_and_time.sh
stdbuf: missing operand
Try 'stdbuf --help' for more information.
real 0m0.001s
user 0m0.000s
sys 0m0.000s
/opt/conda/lib/python3.6/site-packages/torch/nn/_reduction.py:49: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
warnings.warn(warning.format(ret))
Traceback (most recent call last):
File "infer.py", line 144, in <module>
Using seed = 1
loading annotations into memory...
Done (t=0.70s)
creating index...
index created!
loading model checkpoint /mlperf/pretrained/resnet34-ssd300.pth
main()age: 1/4952
File "infer.py", line 141, in main
eval_ssd300_mlperf_coco(args)
File "infer.py", line 127, in eval_ssd300_mlperf_coco
coco_eval(ssd300, val_coco, cocoGt, encoder, inv_map, args.threshold,args.device)
File "infer.py", line 67, in coco_eval
ploc, plabel = model(inp)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
result = self.forward(*input, **kwargs)
File "/mlperf/pytorch/ssd300.py", line 125, in forward
layers = self.model(data)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
result = self.forward(*input, **kwargs)
File "/mlperf/pytorch/base_model.py", line 71, in forward
layer1_activation = self.layer1(data)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
result = self.forward(*input, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
input = module(input)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
result = self.forward(*input, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 320, in forward
self.padding, self.dilation, self.groups)
RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same
It looks like the input tensors and the model tensors are different. Some help?
Hi, I have a few questions of using the trace generator:
Would be good to get some sense on the run-to-run variations. Usually how much is it?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.