pfnet-research/menoh

Menoh: fast DNN inference library with multiple programming language support

Notice: Menoh is no longer maintained. Part of its functionality is inherited by chainer-compiler.

Menoh

Menoh is DNN inference library with C API.

Menoh is released under MIT License.

DISCLAIMER: Menoh is still experimental. Use it at your own risk. In particular not all operators in ONNX are supported, so please check whether the operators used in your model are supported. We have checked that VGG16 and ResNet50 models converted by onnx-chainer work fine.

Document

This codebase contains C API and C++ API.

Goal

• DNN Inference with CPU
• ONNX support
• Easy to use.

Related Projects

• Chainer model to ONNX : onnx-chainer
• C# wrapper : menoh-sharp
• Go wrapper : go-menoh
  • (unofficial wrapper gomenoh by kou-m san has been merged)
• Haskell wrapper : menoh-haskell
• Node.js wrapper : node-menoh
• Ruby wrapper : menoh-ruby
• Rust wrapper : menoh-rs
  • There is also unofficial Rust wrapper by Y-Nak san
• Java wrapper : menoh-java
• [Unofficial] ROS interface by Akio Ochiai san : menoh_ros
• [Unofficial] OCaml wrapper by wkwkes san : Menohcaml

Installation using package manager or binary packages

• For Windows users, prebuild libraries are available (see release) and Nuget package is available.
• For macOS user, Homebrew tap repository is available.
• For Ubuntu user, binary packages are available.

  $ curl -LO https://github.com/pfnet-research/menoh/releases/download/v1.1.1/ubuntu1604_mkl-dnn_0.16-1_amd64.deb
  $ curl -LO https://github.com/pfnet-research/menoh/releases/download/v1.1.1/ubuntu1604_menoh_1.1.1-1_amd64.deb
  $ curl -LO https://github.com/pfnet-research/menoh/releases/download/v1.1.1/ubuntu1604_menoh-dev_1.1.1-1_amd64.deb
  $ sudo apt install ./ubuntu1604_*_amd64.deb
  

  If you are using Ubuntu 18.04, please replace 1604 with 1804.

Installation from source

Requirements

• MKL-DNN Library (0.14 or later)
• Protocol Buffers (2.6.1 or later)

Build

Execute following commands in root directory.

python scripts/retrieve_data.py
mkdir build && cd build
cmake ..
make

See BUILDING.md for details.

Installation

Execute following command in build directory created at Build section.

make install

Run VGG16 example (it can run ResNet-50 as well)

Execute following command in root directory.

./example/vgg16_example_in_cpp

Result is here

vgg16 example
-18.1883 -26.5022 -20.0474 13.5325 -0.107129 0.76102 -23.9688 -24.218 -21.6314 14.2164 
top 5 categories are
8 0.885836 n01514859 hen
7 0.104591 n01514668 cock
86 0.00313584 n01807496 partridge
82 0.000934658 n01797886 ruffed grouse, partridge, Bonasa umbellus
97 0.000839487 n01847000 drake

You can also run ResNet-50

./example/vgg16_example_in_cpp -m ../data/resnet50.onnx

Please give --help option for details

./example/vgg16_example_in_cpp --help

Run test

Setup chainer

Then, execute following commands in root directory.

python scripts/gen_test_data.py
cd build
cmake -DENABLE_TEST=ON ..
make
./test/menoh_test.out

Current supported operators

Activation functions

• Elu
• LeakyRelu
• Relu
• Softmax
• Tanh

Array manipulations

• Concat

Neural network connections

• Conv
• ConvTranspose
• FC

Mathematical functions

• Abs
• Add
• Sqrt
• Sum

Normalization functions

• BatchNormalization
• LRN

Spatial pooling

• AveragePool
• GlobalAveragePool
• GlobalMaxPool
• MaxPool

License

Menoh is released under MIT License. Please see the LICENSE file for details.

Pre-trained models downloaded via retrieve_data.py were converted by onnx-chainer. The original models were downloaded via ChainerCV. Check scripts/generate_vgg16_onnx.py and scripts/generate_resnet50_onnx.py and see the LICENSE of ChainerCV about each terms of use of the pre-trained models.

menoh's Issues

Different pre-processing of VGG16 example

Menoh's VGG16 example applies center crop to images and does not subtract mean, where that of Chainer substract mean and does not crop. If this difference is not essential, it is better to follow Chainer's style.

'cmake -DCMAKE_BUILD_TYPE=Debug' causes dyld error on macOS Mojave

$ cmake .. -DCMAKE_BUILD_TYPE=Debug -DENABLE_TEST=ON -DENABLE_EXAMPLE=OFF && make 
..
[100%] Linking CXX executable vgg16_benchmark
[100%] Built target vgg16_benchmark
dyld: __dyld section not supported in /Users/sakai/Menoh/build/test/menoh_test
Abort trap: 6

I think this is a bug of -pg flag on macOS LLVM environment.

c.f. https://stackoverflow.com/questions/52519907/dyld-section-not-supported-after-mojave-upgrade

Add profiler

I'd like to know memory consumption and computation time for each layer. It would be helpful for tuning networks.

Remove menoh_onnx_viewer

menoh_onnx_viewer does not have enough features and there is no reason to include it in Menoh repo so it should be replaced with Netron

Consider switching to c++11

I noticed that current AWS Lambda did not support libstdc++.so with GLIBCXX_3.4.20 or greater, which means all dependencies must be compiled with gcc < v5.0. (C++14 is not supported)
It is a great disadvantage for Menoh since AWS Lambda is very popular now...

One of the obvious workarounds would be to put new version of libstdc++.so and all of its dependencies inside the Lambda deployment package, however, that increases the size of the package unnecessarily, and it makes creating such package a lot more difficult for developers.

I believe, future AWS Lambda would allow gcc >= v5.0 in the future, but such plan is unknown. A similar problem could occur in other use cases than AWS Lambda also. In order to increase usability/applicability of Menoh, we may need to consider making Menoh-core C++11 compatible.

Note: Other dependencies, such as mkl-dnn, protobuf, can be compiled with gcc < v5.0 with no problem.

`No target "protobuf::protoc"` error with CMake 3.9.2

I encountered No target "protobuf::protoc" error with CMake 3.9.2.
This also prevents us to build Menoh on Travis-CI Trusty environment.

$ PATH=/home/sakai/cmake-3.9.2-Linux-x86_64/bin:/home/sakai/local/bin:$PATH cmake .. -DCMAKE_PREFIX_PATH=/home/sakai/local -DENABLE_TEST=OFF -DENABLE_BENCHMARK=OFF -DENABLE_EXAMPLE=OFF -DMKLDNN_INCLUDE_DIR=/home/sakai/local/include -DMKLDNN_LIBRARY=/home/sakai/local/lib/libmkldnn.so.0.16.0
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- CMAKE_BUILD_TYPE is unset, defaulting to Release
-- Build type: Release
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Found Protobuf: /home/sakai/local/lib/libprotobuf.so;-lpthread (found version "3.6.1") 
-- Found Protobuf: /home/sakai/local/lib/libprotobuf.so;-lpthread;-lpthread (found version "3.6.1") 
-- Adding external/onnx
-- Patching to external/onnx
-- 
-- ******** Summary ********
--   CMake version         : 3.9.2
--   CMake command         : /home/sakai/cmake-3.9.2-Linux-x86_64/bin/cmake
--   System                : Linux
--   C++ compiler          : /usr/bin/c++
--   C++ compiler version  : 5.4.0
--   CXX flags             : 
--   Build type            : Release
--   Compile definitions   : 
--   CMAKE_PREFIX_PATH     : /home/sakai/local
--   CMAKE_INSTALL_PREFIX  : /usr/local
--   CMAKE_MODULE_PATH     : /home/sakai/menoh/cmake
-- 
--   ONNX version          : 1.2.2
--   ONNX NAMESPACE        : onnx
--   ONNX_BUILD_TESTS      : OFF
--   ONNX_BUILD_BENCHMARKS : OFF
--   ONNX_USE_LITE_PROTO   : OFF
-- 
--   Protobuf compiler     : /home/sakai/local/bin/protoc
--   Protobuf includes     : /home/sakai/local/include
--   Protobuf libraries    : /home/sakai/local/lib/libprotobuf.so;-lpthread;-lpthread
--   BUILD_ONNX_PYTHON     : OFF
-- Found MKLDNN: /home/sakai/local/lib/libmkldnn.so.0.16.0 (Required is at least version "0.14") 
-- Adding menoh
-- Adding include
-- Configuring done
CMake Error at external/onnx/CMakeLists.txt:232 (add_custom_command):
  Error evaluating generator expression:

    $<TARGET_FILE:protobuf::protoc>

  No target "protobuf::protoc"
Call Stack (most recent call first):
  external/onnx/CMakeLists.txt:253 (relative_protobuf_generate_cpp)


CMake Error at external/onnx/CMakeLists.txt:232 (add_custom_command):
  Error evaluating generator expression:

    $<TARGET_FILE:protobuf::protoc>

  No target "protobuf::protoc"
Call Stack (most recent call first):
  external/onnx/CMakeLists.txt:247 (relative_protobuf_generate_cpp)


CMake Error at external/onnx/CMakeLists.txt:232 (add_custom_command):
  Error evaluating generator expression:

    $<TARGET_FILE:protobuf::protoc>

  No target "protobuf::protoc"
Call Stack (most recent call first):
  external/onnx/CMakeLists.txt:247 (relative_protobuf_generate_cpp)


CMake Error at external/onnx/CMakeLists.txt:232 (add_custom_command):
  Error evaluating generator expression:

    $<TARGET_FILE:protobuf::protoc>

  No target "protobuf::protoc"
Call Stack (most recent call first):
  external/onnx/CMakeLists.txt:253 (relative_protobuf_generate_cpp)


CMake Error at external/onnx/CMakeLists.txt:232 (add_custom_command):
  Error evaluating generator expression:

    $<TARGET_FILE:protobuf::protoc>

  No target "protobuf::protoc"
Call Stack (most recent call first):
  external/onnx/CMakeLists.txt:253 (relative_protobuf_generate_cpp)


CMake Error at external/onnx/CMakeLists.txt:232 (add_custom_command):
  Error evaluating generator expression:

    $<TARGET_FILE:protobuf::protoc>

  No target "protobuf::protoc"
Call Stack (most recent call first):
  external/onnx/CMakeLists.txt:253 (relative_protobuf_generate_cpp)


CMake Error at external/onnx/CMakeLists.txt:232 (add_custom_command):
  Error evaluating generator expression:

    $<TARGET_FILE:protobuf::protoc>

  No target "protobuf::protoc"
Call Stack (most recent call first):
  external/onnx/CMakeLists.txt:247 (relative_protobuf_generate_cpp)


CMake Error at external/onnx/CMakeLists.txt:232 (add_custom_command):
  Error evaluating generator expression:

    $<TARGET_FILE:protobuf::protoc>

  No target "protobuf::protoc"
Call Stack (most recent call first):
  external/onnx/CMakeLists.txt:247 (relative_protobuf_generate_cpp)


CMake Error at external/onnx/CMakeLists.txt:232 (add_custom_command):
  Error evaluating generator expression:

    $<TARGET_FILE:protobuf::protoc>

  No target "protobuf::protoc"
Call Stack (most recent call first):
  external/onnx/CMakeLists.txt:247 (relative_protobuf_generate_cpp)


CMake Error at external/onnx/CMakeLists.txt:232 (add_custom_command):
  Error evaluating generator expression:

    $<TARGET_FILE:protobuf::protoc>

  No target "protobuf::protoc"
Call Stack (most recent call first):
  external/onnx/CMakeLists.txt:253 (relative_protobuf_generate_cpp)


CMake Error at external/onnx/CMakeLists.txt:232 (add_custom_command):
  Error evaluating generator expression:

    $<TARGET_FILE:protobuf::protoc>

  No target "protobuf::protoc"
Call Stack (most recent call first):
  external/onnx/CMakeLists.txt:247 (relative_protobuf_generate_cpp)


CMake Error at external/onnx/CMakeLists.txt:232 (add_custom_command):
  Error evaluating generator expression:

    $<TARGET_FILE:protobuf::protoc>

  No target "protobuf::protoc"
Call Stack (most recent call first):
  external/onnx/CMakeLists.txt:253 (relative_protobuf_generate_cpp)


-- Generating done
-- Build files have been written to: /home/sakai/menoh/build

mingw packages for mkl-dnn and menoh

I made mingw packages for mkl-dnn and menoh.

mkl-dnn
https://github.com/msakai/MINGW-packages/tree/mkl-dnn-0.14/mingw-w64-mkl-dnn

menoh
https://github.com/msakai/MINGW-packages/tree/menoh-1.0.1/mingw-w64-menoh

Binary packages:
https://www.dropbox.com/sh/vkomvo22fsy25h5/AAAOycfx-DP0EFLPiueMxDJIa?dl=0

Modify dependency of unexported modules

Current menoh_onnx_viewer and menoh_test depend on unexported modules. We have to modify them depending on only exported modules.
See #21

Assertion failed: (found->second.dims().size() == 2 || found->second.dims().size() == 4)

When I run menoh_test with cmake -DCMAKE_BUILD_TYPE=Debug, it failed with assertion error.

$ ./test/menoh_test 
...
[ RUN      ] OperatorTest.mkldnn_with_generic_fallback_test_relu
Assertion failed: (found->second.dims().size() == 2 || found->second.dims().size() == 4), function operator(), file /home/sakai/Menoh/menoh/mkldnn_with_generic_fallback/backend/mkldnn/mkldnn_context.cpp, line 63.
Abort trap: 6

Segmentation fault on Travis CI environment

I encountered a segmentation fault of menoh_test on Travis CI environment.
https://travis-ci.org/msakai/menoh/builds/394931222

$ ./test/menoh_test
Running main() from gtest_main.cc
[==========] Running 99 tests from 8 test cases.
[----------] Global test environment set-up.
[----------] 2 tests from NumpyIOTest
[ RUN      ] NumpyIOTest.load_test
[       OK ] NumpyIOTest.load_test (10 ms)
[ RUN      ] NumpyIOTest.load_as_arr_test
[       OK ] NumpyIOTest.load_as_arr_test (10 ms)
[----------] 2 tests from NumpyIOTest (20 ms total)
[----------] 4 tests from ArrayTest
[ RUN      ] ArrayTest.test_array_construct_from_data_handle
[       OK ] ArrayTest.test_array_construct_from_data_handle (0 ms)
[ RUN      ] ArrayTest.test_array_construct
[       OK ] ArrayTest.test_array_construct (0 ms)
[ RUN      ] ArrayTest.test_array_copy
[       OK ] ArrayTest.test_array_copy (0 ms)
[ RUN      ] ArrayTest.test_array_move_copy
[       OK ] ArrayTest.test_array_move_copy (0 ms)
[----------] 4 tests from ArrayTest (0 ms total)
[----------] 1 test from NodeTest
[ RUN      ] NodeTest.optional_attribute_float_access_test
[       OK ] NodeTest.optional_attribute_float_access_test (0 ms)
[----------] 1 test from NodeTest (0 ms total)
[----------] 7 tests from GraphTest
[ RUN      ] GraphTest.construct_test
[       OK ] GraphTest.construct_test (0 ms)
[ RUN      ] GraphTest.attribute_access_test
[       OK ] GraphTest.attribute_access_test (0 ms)
[ RUN      ] GraphTest.extract_needed_node_list_test1
/home/travis/.travis/job_stages: line 57: 18875 Segmentation fault      (core dumped) ./test/menoh_test
The command "./test/menoh_test" exited with 139.

Does anyone has an idea what's going on?

Operator requests

operator	requiring models
Sigmoid	YOLOv2, (NStep)LSTM
Transpose	YOLOv2, Super resolution (#56 (comment)), GGNN (#56 (comment))
Reshape	YOLOv2
Mul	SSD, (NStep)LSTM
Conv (dilation > 1)	SSD
Upsample	YOLOv3*, FPN*
Gather	GGNN (#56 (comment))
Identity	GGNN (#56 (comment))
Conv (group > 1)	MobileNet

variable not found error on a very simple model

I experimented with the following simple model (add1.onnx.zip), but I encountered menoh variable not found error: 103249637448 error.

ONNX version is 3
opset_import_size is 1

4 
domain is 
model version is 0
producer name is Chainer
producer version is 4.2.0
parameter list
name: 103249637448 dims: 2 2  values: min_value: 1 max_value: 1 

node list
node num is 1
0:Add
	input0: 4393462696
	input1: 103249637448
	output0: 111895884240

My code is as follows:

#include <menoh/menoh.h>
#include <stdio.h>

#define ERROR_CHECK(statement)                              \
    {                                                       \
        menoh_error_code ec = statement;                    \
        if(ec) {                                            \
            printf("%s\n", menoh_get_last_error_message()); \
            return 0;                                       \
        }                                                   \
    }

int main()
{
    const char* in_name = "4393462696";
    const char* out_name = "111895884240";

    menoh_model_data_handle model_data;
    ERROR_CHECK(
      menoh_make_model_data_from_onnx("add1.onnx", &model_data));

    menoh_variable_profile_table_builder_handle vpt_builder;
    ERROR_CHECK(menoh_make_variable_profile_table_builder(&vpt_builder));

    ERROR_CHECK(menoh_variable_profile_table_builder_add_input_profile_dims_2(
      vpt_builder, in_name, menoh_dtype_float, 2, 2));

    ERROR_CHECK(menoh_variable_profile_table_builder_add_output_profile(
      vpt_builder, out_name, menoh_dtype_float));

    menoh_variable_profile_table_handle variable_profile_table;
    ERROR_CHECK(menoh_build_variable_profile_table(vpt_builder, model_data,
                                                   &variable_profile_table));

    ERROR_CHECK(menoh_model_data_optimize(model_data, variable_profile_table));

    menoh_model_builder_handle model_builder;
    ERROR_CHECK(
      menoh_make_model_builder(variable_profile_table, &model_builder));

    menoh_model_handle model;
    ERROR_CHECK(
      menoh_build_model(model_builder, model_data, "mkldnn", "", &model));
}

Is there something wrong with this code or the model?

error: no type named 'format' in 'mkldnn::memory'

I have mkl-dnn-1.0.1 installed, and clang8 complains:

In file included from /usr/ports/math/menoh/work/menoh-1.1.1-169-gc00e8d5/menoh/composite_backend/backend/mkldnn/memory_cache.cpp:1:
In file included from /usr/ports/math/menoh/work/menoh-1.1.1-169-gc00e8d5/menoh/composite_backend/backend/mkldnn/memory_cache.hpp:10:
/usr/ports/math/menoh/work/menoh-1.1.1-169-gc00e8d5/menoh/composite_backend/backend/mkldnn/memory_conversion.hpp:12:49: error: no type named 'format' in 'mkldnn::memory'
            bool is_data_format(mkldnn::memory::format format);
                                ~~~~~~~~~~~~~~~~^
/usr/ports/math/menoh/work/menoh-1.1.1-169-gc00e8d5/menoh/composite_backend/backend/mkldnn/memory_conversion.hpp:15:29: error: no type named 'format' in 'mkldnn::memory'
            mkldnn::memory::format extract_format(mkldnn::memory const& m);
            ~~~~~~~~~~~~~~~~^
/usr/ports/math/menoh/work/menoh-1.1.1-169-gc00e8d5/menoh/composite_backend/backend/mkldnn/memory_conversion.hpp:25:29: error: no type named 'format' in 'mkldnn::memory'
            mkldnn::memory::format ndims_to_data_memory_format(int ndims);
            ~~~~~~~~~~~~~~~~^
/usr/ports/math/menoh/work/menoh-1.1.1-169-gc00e8d5/menoh/composite_backend/backend/mkldnn/memory_conversion.hpp:26:29: error: no type named 'format' in 'mkldnn::memory'
            mkldnn::memory::format ndims_to_weight_memory_format(int ndims);
            ~~~~~~~~~~~~~~~~^
/usr/ports/math/menoh/work/menoh-1.1.1-169-gc00e8d5/menoh/composite_backend/backend/mkldnn/memory_conversion.hpp:30:60: error: no type named 'format' in 'mkldnn::memory'
                                           mkldnn::memory::format format,
                                           ~~~~~~~~~~~~~~~~^
/usr/ports/math/menoh/work/menoh-1.1.1-169-gc00e8d5/menoh/composite_backend/backend/mkldnn/memory_conversion.hpp:34:60: error: no type named 'format' in 'mkldnn::memory'
                                           mkldnn::memory::format format,
                                           ~~~~~~~~~~~~~~~~^

Non-deterministic and suspicious errors

I ran the following program with the attached ggnn.onnx in ggnn.zip.

#include <menoh/menoh.h>
#include <stdio.h>

#define ERROR_CHECK(statement)                              \
    {                                                       \
        menoh_error_code ec = statement;                    \
        if(ec) {                                            \
            printf("%s\n", menoh_get_last_error_message()); \
            return 1;                                       \
        }                                                   \
    }

int main()
{
    menoh_model_data_handle model_data;
    ERROR_CHECK(
      menoh_make_model_data_from_onnx("ggnn.onnx", &model_data));

    menoh_variable_profile_table_builder_handle vpt_builder;
    ERROR_CHECK(menoh_make_variable_profile_table_builder(&vpt_builder));

    ERROR_CHECK(menoh_variable_profile_table_builder_add_input_profile_dims_4(
      vpt_builder, "111980687600", menoh_dtype_float, 32, 4, 51, 51));
    ERROR_CHECK(menoh_variable_profile_table_builder_add_input_profile_dims_4(
      vpt_builder, "111980687712", menoh_dtype_float, 32, 51, 16, 1));
    ERROR_CHECK(menoh_variable_profile_table_builder_add_output_profile(
      vpt_builder, "112043339960", menoh_dtype_float));

    menoh_variable_profile_table_handle variable_profile_table;
    ERROR_CHECK(menoh_build_variable_profile_table(vpt_builder, model_data,
                                                   &variable_profile_table));

    return 0;
}

The results are non-deterministic and the "menoh variable not found error" errors raise the doubt of bug.

$ gcc `pkg-config menoh --libs --cflags` test.c
$ ./a.out 
menoh unsupported operator error: Transpose
$ ./a.out 
menoh variable not found error: 112043186328
$ ./a.out 
menoh variable not found error: 112043186328
$ ./a.out 
menoh variable not found error: 112043244120
$ ./a.out 
menoh variable not found error: 112043244400
$ ./a.out 
menoh unsupported operator error: Identity
$ ./a.out 
menoh unsupported operator error: Transpose
$ ./a.out 
menoh unsupported operator error: Identity
$ ./a.out 
menoh unsupported operator error: Transpose
$ ./a.out 
menoh unsupported operator error: Transpose
$ ./a.out 
menoh unsupported operator error: Transpose
$ ./a.out 
menoh unsupported operator error: Identity
$ ./a.out 
menoh unsupported operator error: Transpose
$ ./a.out 
menoh unsupported operator error: Transpose
$ 

cmake script does nt check gcc version

I know the doc/md file correctly requires gcc 4.9 but in case someone tries to use an older gcc then cmake continues and then some (hard to understand for newcomers) compilation errors happen at make/compilation time.

Should nt the main cmake script check gcc verison ?

example:

if (CMAKE_COMPILER_IS_GNUCC)
if (CMAKE_CXX_COMPILER_VERSION VERSION_LESS 4.9)
message(FATAL_ERROR "blablabla")
endif()
endif()

make install fails for static library build

If -DBUILD_SHARED_LIBS=OFF is specified, make install fails with following error:

$ cmake .. -DBUILD_SHARED_LIBS=OFF -DENABLE_TEST=OFF -DENABLE_EXAMPLE=ON && make
-- The C compiler identification is AppleClang 10.0.0.10001044
-- The CXX compiler identification is AppleClang 10.0.0.10001044
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- CMAKE_BUILD_TYPE is unset, defaulting to Release
-- Build type: Release
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - found
-- Found Threads: TRUE  
-- Found Protobuf: /usr/local/lib/libprotobuf.dylib (found version "3.6.1") 
-- Found Git: /usr/bin/git (found version "2.17.2 (Apple Git-113)") 
-- Adding external/onnx
fatal: /Users/sakai/menoh/external/onnx: '/Users/sakai/menoh/external/onnx' is outside repository
-- 
-- ******** Summary ********
--   CMake version         : 3.12.4
--   CMake command         : /usr/local/Cellar/cmake/3.12.4/bin/cmake
--   System                : Darwin
--   C++ compiler          : /Library/Developer/CommandLineTools/usr/bin/c++
--   C++ compiler version  : 10.0.0.10001044
--   CXX flags             :  -Wnon-virtual-dtor
--   Build type            : Release
--   Compile definitions   : 
--   CMAKE_PREFIX_PATH     : 
--   CMAKE_INSTALL_PREFIX  : /usr/local
--   CMAKE_MODULE_PATH     : /Users/sakai/menoh/cmake
-- 
--   ONNX version          : 1.4.1
--   ONNX NAMESPACE        : menoh_onnx
--   ONNX_BUILD_TESTS      : OFF
--   ONNX_BUILD_BENCHMARKS : OFF
--   ONNX_USE_LITE_PROTO   : OFF
--   ONNXIFI_DUMMY_BACKEND : OFF
-- 
--   Protobuf compiler     : /usr/local/bin/protoc
--   Protobuf includes     : /usr/local/include
--   Protobuf libraries    : /usr/local/lib/libprotobuf.dylib
--   BUILD_ONNX_PYTHON     : OFF
-- Adding benchmark
-- Adding example
-- Found OpenCV: /usr/local (found version "3.4.3") 
-- Adding menoh
-- Found MKLDNN: /usr/local/lib/libmkldnn.dylib (Required is at least version "0.14") 
-- Adding include
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/sakai/menoh/build
Scanning dependencies of target gen_onnx_proto
[  1%] Running gen_proto.py on onnx/onnx.in.proto
Processing /Users/sakai/menoh/external/onnx/onnx/onnx.in.proto
Writing /Users/sakai/menoh/build/external/onnx/onnx/onnx_menoh_onnx.proto
Writing /Users/sakai/menoh/build/external/onnx/onnx/onnx_menoh_onnx.proto3
Writing /Users/sakai/menoh/build/external/onnx/onnx/onnx.pb.h
generating /Users/sakai/menoh/build/external/onnx/onnx/onnx_pb.py
..(snip)..
bash-3.2$ make install
[  2%] Built target gen_onnx_proto
[ 35%] Built target menoh_objlib
[ 44%] Built target onnx_proto
[ 88%] Built target onnx
[ 89%] Built target menoh
[ 91%] Built target vgg16_benchmark
[ 94%] Built target general_cnn_example_in_cpp
[ 96%] Built target mkldnn_vgg16_example_in_cpp
[ 98%] Built target mkldnn_vgg16_example_in_c
[100%] Built target menoh_test_target
Install the project...
-- Install configuration: "Release"
CMake Error at cmake_install.cmake:36 (file):
  file INSTALL cannot find "/Users/sakai/menoh/MENOH_ONNX_LIB-NOTFOUND".


make: *** [install] Error 1

I think this is because libmenoh_onnx.a and libmenoh_onnx_proto.a are not built yet when find_library is executed.

menoh/CMakeLists.txt

Lines 104 to 115 in d07eebb

	set(BUILD_SHARED_LIBS_SAVED "${BUILD_SHARED_LIBS}")
	set(BUILD_SHARED_LIBS OFF)
	set(ONNX_NAMESPACE "menoh_onnx" CACHE INTERNAL "onnx namespace")
	add_subdirectory(${EXTERNAL_DIR}/onnx EXCLUDE_FROM_ALL) # Note: BUILD_SHARED_LIBS must be OFF in this place
	if(NOT ${BUILD_SHARED_LIBS_SAVED}) # install libmenoh_onnx and lib_menoh_onnx_proto
	set_target_properties(onnx_proto PROPERTIES OUTPUT_NAME "menoh_onnx_proto")
	set_target_properties(onnx PROPERTIES OUTPUT_NAME "menoh_onnx")
	find_library(MENOH_ONNX_PROTO_LIB "menoh_onnx_proto" "${CMAKE_CURRENT_BINARY_DIR}/external/onnx")
	find_library(MENOH_ONNX_LIB "menoh_onnx" "${CMAKE_CURRENT_BINARY_DIR}/external/onnx")
	install(FILES "${MENOH_ONNX_LIB}" "${MENOH_ONNX_PROTO_LIB}" DESTINATION "lib")
	endif()
	set(BUILD_SHARED_LIBS "${BUILD_SHARED_LIBS_SAVED}")

Add metadata_props accessor

ONNX format has metadata_props which can be used for notating pre-process and post-process of the model.
const char* get_metadata_prop(model_data_handle, char *key) seems suitable.

Prepare precompiled DLL for Windows users

menoh-sharp has already had precompiled DLL.

Reduce building time in CI

The building time for each commit is time consuming (it takes 20~40 mins on Travis CI and Appveyor). It (potentially) hinders faster iterative development.

Prepare docker file

menoh-ruby and menoh-yolo has docker file.
menoh also should have docker file.

Pre-reorder of fixed parameters

Currently mkldnn backend reorders fixed parameters (e.g. weights of Conv) at runtime. It should be done once at build time.

Installing menoh_onnx_viewer

menoh_onnx_viewer is a convenient tool for using Menoh.
Is there any reason make install not installing menoh_onnx_viewer into bin directory?

Coverity Scan

How about setting up Coverity Scan?
https://scan.coverity.com/

Coverity Scan is famous static analysis tool for C/C++ and several other languages.
It can be used for open source project for free.

Feature Request: function for interrupting menoh_model_run()

Because menoh_model_run() can take long time, it is desirable to have a function for canceling the computation from other threads.

In Ruby binding, I'm trying to release the GVL (giant VM lock, aka GIL or global interpreter lock) while executing menoh_model_run() in this PR. But doing so requires to pass a function to cancel the execution safely. If I don't pass proper function, the program will not respond for Control+C or other shutdown events. (see https://github.com/ruby/ruby/blob/v2_5_3/thread.c#L1367-L1451 for detail)

I don't know the cases of other languages with GVL (e.g CPython). But other such language may suffer the same issue.

Feature request: function for computing element size of a dtype

It would be nice to have a function for computing element size of a given dtype, something like the following.

menoh_error_code MENOH_API menoh_dtype_size(menoh_dtype dtype, int32_t *dst_size);

It is useful for binding libraries, to allocate buffer or to copy data for example, in a general way without knowing each dtype (so that it is safe even if a new dtype is introduced in the future Menoh).

Example for calling run multiple times

There is no example which shows how to call run() multiple times. So many users get confused that they have to construct the model for each run(). There is no need to do so.
Constructed models can be called run() multiple times.

add mkdir to retrieve_data.py, and update README.md

This library is very interesting!
I want to use this library, but an error occurred.
Can I send you a easy pull requests to fix this bug?
I will add mkdir method to retrieve_data.py

I read how to build this library at README.md.
I wonder if it is not sh retrieve_data.py but python retrieve_data.py.
Thanks.

wget https://www.dropbox.com/s/bjfn9kehukpbmcm/VGG16.onnx?dl=1 -O ./data/VGG16.onnx
Traceback (most recent call last):
  File "retrieve_data.py", line 13, in <module>
    wget('https://www.dropbox.com/s/bjfn9kehukpbmcm/VGG16.onnx?dl=1', './data/VGG16.onnx')
  File "retrieve_data.py", line 11, in wget
    urllib.request.urlretrieve(address,  target)
  File "/Users/nishimurataichi/.pyenv/versions/anaconda3-4.1.0/lib/python3.5/urllib/request.py", line 197, in urlretrieve
    tfp = open(filename, 'wb')
FileNotFoundError: [Errno 2] No such file or directory: './data/VGG16.onnx'

Required version of ProtocolBuffers

I encountered following error when I tried to compile Menoh on Travis-CI Trusty environment that uses protobuf-compiler_2.5.0-9ubuntu1_amd64.deb.

...
-- Found Protobuf: /usr/lib/x86_64-linux-gnu/libprotobuf.so;-lpthread;-lpthread (found version "2.5.0") 
onnx/onnx.proto:393:5: Expected "required", "optional", or "repeated".
onnx/onnx.proto:393:17: Missing field number.
onnx/onnx.proto:443:3: Expected "required", "optional", or "repeated".
onnx/onnx.proto:443:15: Missing field number.
...
/home/travis/build/msakai/menoh/menoh/onnx.cpp:15:10: fatal error: external/onnx/onnx/onnx.pb.h: No such file or directory
 #include <external/onnx/onnx/onnx.pb.h>
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [menoh/CMakeFiles/menoh.dir/onnx.cpp.o] Error 1
make[1]: *** [menoh/CMakeFiles/menoh.dir/all] Error 2
make: *** [all] Error 2

It would be nice if required version of ProtocolBuffers is mentioned in README.md

Multiple different contexts issue variable_not_found error

While setting invalid name in the input profile throws variable_not_found_error and the log message contains the model's input name, setting in the output profile throws the same error but the log message contains the invalid output name. This inconsistency comes from that the former is the result of searching by model's input name and the latter is the result of searching by required output name.
This inconsistency confuses the developers of bindings.
We should split error into two different error (e.g. input_not_found_error, required_output_not_found_error).

Please allow to use the external onnx

I have onnx as a package on FreeBSD, and would like to just use it instead of bundling it.

Could you please create the cmake option that would allow to use external onnx?

Current BatchNormalziation OP does not support dynamic parameters

see #145
We should make it taking dynamic parameters or checking the prameter static.

Unhelpful error message when wrong version of protobuf

Hi
Menoh fails to compile probably because my local protobuf is too old :

$ cmake
...
-- Found Protobuf: /usr/lib64/libprotobuf.so;-lpthread (found version "2.5.0")
-- Found Protobuf: /usr/lib64/libprotobuf.so;-lpthread;-lpthread (found version "2.5.0")
...
cmake exits successfully

$ make
Scanning dependencies of target gen_onnx_proto
[ 1%] Running gen_proto.py on onnx/onnx.in.proto
Processing /home/wtambellini/repos/menoh/external/onnx/onnx/onnx.in.proto
Writing /home/wtambellini/repos/menoh/Debug/external/onnx/onnx/onnx.proto
Writing /home/wtambellini/repos/menoh/Debug/external/onnx/onnx/onnx.proto3
generating /home/wtambellini/repos/menoh/Debug/external/onnx/onnx/onnx_pb.py
[ 2%] Running C++ protocol buffer compiler on /home/wtambellini/repos/menoh/Debug/external/onnx/onnx/onnx.proto
onnx/onnx.proto:393:5: Expected "required", "optional", or "repeated".
onnx/onnx.proto:393:17: Missing field number.
onnx/onnx.proto:420:3: Expected "required", "optional", or "repeated".
onnx/onnx.proto:420:15: Missing field number.
make[2]: *** [external/onnx/onnx/onnx.pb.cc] Error 1
make[1]: *** [external/onnx/CMakeFiles/gen_onnx_proto.dir/all] Error 2
make: *** [all] Error 2

Yes, I know the md doc state protobuf 2.6.1, but should nt cmake simply assert/check the version of protoc/protobuf ?

I see in the main cmake :
find_package(Protobuf ${PROTOBUF_VERSION} REQUIRED)

but PROTOBUF_VERSION is not set so empty so not imposing any minimum version.
I could prepare a PR if you re ok to check protobuf version at cmake time.
Kind

build error when `LINK_STATIC_LIBPROTOBUF=ON` and `make -j` are used

$ cmake -DLINK_STATIC_LIBPROTOBUF=ON .. && make -j4
...
Scanning dependencies of target gen_onnx_proto
Scanning dependencies of target Protobuf
[  1%] Running gen_proto.py on onnx/onnx.in.proto
[  2%] Creating directories for 'Protobuf'
[  3%] Performing download step (download, verify and extract) for 'Protobuf'
-- Downloading...
   dst='/Users/sakai/Menoh/build/downloads/protobuf-cpp-3.6.1.tar.gz'
   timeout='none'
-- Using src='https://github.com/protocolbuffers/protobuf/releases/download/v3.6.1/protobuf-cpp-3.6.1.tar.gz'
Processing /Users/sakai/Menoh/external/onnx/onnx/onnx.in.proto
Writing /Users/sakai/Menoh/build/external/onnx/onnx/onnx.proto
Writing /Users/sakai/Menoh/build/external/onnx/onnx/onnx.proto3
generating /Users/sakai/Menoh/build/external/onnx/onnx/onnx_pb.py
[  4%] Running C++ protocol buffer compiler on /Users/sakai/Menoh/build/external/onnx/onnx/onnx.proto
/bin/sh: ../../protobuf-3.6.1/bin/protoc: No such file or directory
make[2]: *** [external/onnx/onnx/onnx.pb.cc] Error 127
make[1]: *** [external/onnx/CMakeFiles/gen_onnx_proto.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
-- [download 0% complete]
...
-- [download 100% complete]
-- verifying file...
       file='/Users/sakai/Menoh/build/downloads/protobuf-cpp-3.6.1.tar.gz'
-- Downloading... done
-- extracting...
     src='/Users/sakai/Menoh/build/downloads/protobuf-cpp-3.6.1.tar.gz'
     dst='/Users/sakai/Menoh/build/protobuf-3.6.1/src/Protobuf'
-- extracting... [tar xfz]
-- extracting... [analysis]
-- extracting... [rename]
-- extracting... [clean up]
-- extracting... done
[  6%] No patch step for 'Protobuf'
[  7%] No update step for 'Protobuf'
[  8%] Performing configure step for 'Protobuf'
...

full log

Tanh returns wrong value

The value returned by Tanh layer exceeds 1 although it should be clipped in (-1, 1).

In my environment, the following code shows 3.16228.

#include <iostream>
#include <menoh/menoh.hpp>

int main()
{
  auto model_data = menoh::make_model_data_from_onnx("tanh.onnx");

  menoh::variable_profile_table_builder vpt_builder;
  vpt_builder.add_input_profile("input", menoh::dtype_t::float_, {1, 1});
  vpt_builder.add_output_profile("output", menoh::dtype_t::float_);

  auto vpt = vpt_builder.build_variable_profile_table(model_data);
  menoh::model_builder model_builder(vpt);
  auto model = model_builder.build_model(model_data, "mkldnn");

  static_cast<float *>(model.get_variable("input").buffer_handle)[0] = 10;

  model.run();

  std::cout << static_cast<float *>(model.get_variable("output").buffer_handle)[0] << std::endl;
}

tanh.onnx

ONNX version is 3
opset_import_size is 1
4 
domain is 
model version is 0
producer name is Chainer
producer version is 4.3.0
parameter list

node list
node num is 1
0:Tanh
        input0: input
        output0: output

delete called on 'menoh_impl::mkldnn_with_generic_fallback_backend::context' that is abstract but has non-virtual destructor

I encountered the following warning. Is this benign?

[ 18%] Building CXX object menoh/CMakeFiles/menoh_objlib.dir/mkldnn_with_generic_fallback/model_core.cpp.o
In file included from /Users/sakai/Menoh/menoh/mkldnn_with_generic_fallback/model_core.cpp:1:
In file included from /Users/sakai/Menoh/menoh/mkldnn_with_generic_fallback/model_core.hpp:4:
In file included from /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/functional:487:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/memory:2285:5: warning: 
      delete called on
      'menoh_impl::mkldnn_with_generic_fallback_backend::context' that is
      abstract but has non-virtual destructor [-Wdelete-non-virtual-dtor]
    delete __ptr;
    ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/memory:2598:7: note: 
      in instantiation of member function
      'std::__1::default_delete<menoh_impl::mkldnn_with_generic_fallback_backend::context>::operator()'
      requested here
      __ptr_.second()(__tmp);
      ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/memory:2552:19: note: 
      in instantiation of member function
      'std::__1::unique_ptr<menoh_impl::mkldnn_with_generic_fallback_backend::context,
      std::__1::default_delete<menoh_impl::mkldnn_with_generic_fallback_backend::context>
      >::reset' requested here
  ~unique_ptr() { reset(); }
                  ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/utility:312:29: note: 
      in instantiation of member function
      'std::__1::unique_ptr<menoh_impl::mkldnn_with_generic_fallback_backend::context,
      std::__1::default_delete<menoh_impl::mkldnn_with_generic_fallback_backend::context>
      >::~unique_ptr' requested here
struct _LIBCPP_TEMPLATE_VIS pair
                            ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/memory:1727:18: note: 
      in instantiation of member function
      'std::__1::allocator<std::__1::pair<std::__1::basic_string<char>,
      std::__1::unique_ptr<menoh_impl::mkldnn_with_generic_fallback_backend::context,
      std::__1::default_delete<menoh_impl::mkldnn_with_generic_fallback_backend::context>
      > > >::destroy' requested here
            {__a.destroy(__p);}
                 ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/memory:1595:14: note: 
      in instantiation of function template specialization
      'std::__1::allocator_traits<std::__1::allocator<std::__1::pair<std::__1::basic_string<char>,
      std::__1::unique_ptr<menoh_impl::mkldnn_with_generic_fallback_backend::context,
      std::__1::default_delete<menoh_impl::mkldnn_with_generic_fallback_backend::context>
      > > > >::__destroy<std::__1::pair<std::__1::basic_string<char>,
      std::__1::unique_ptr<menoh_impl::mkldnn_with_generic_fallback_backend::context,
      std::__1::default_delete<menoh_impl::mkldnn_with_generic_fallback_backend::context>
      > > >' requested here
            {__destroy(__has_destroy<allocator_type, _Tp*>(), __a, __p);}
             ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/vector:413:25: note: 
      in instantiation of function template specialization
      'std::__1::allocator_traits<std::__1::allocator<std::__1::pair<std::__1::basic_string<char>,
      std::__1::unique_ptr<menoh_impl::mkldnn_with_generic_fallback_backend::context,
      std::__1::default_delete<menoh_impl::mkldnn_with_generic_fallback_backend::context>
      > > > >::destroy<std::__1::pair<std::__1::basic_string<char>,
      std::__1::unique_ptr<menoh_impl::mkldnn_with_generic_fallback_backend::context,
      std::__1::default_delete<menoh_impl::mkldnn_with_generic_fallback_backend::context>
      > > >' requested here
        __alloc_traits::destroy(__alloc(), _VSTD::__to_raw_pointer(--__s...
                        ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/vector:356:29: note: 
      in instantiation of member function
      'std::__1::__vector_base<std::__1::pair<std::__1::basic_string<char>,
      std::__1::unique_ptr<menoh_impl::mkldnn_with_generic_fallback_backend::context,
      std::__1::default_delete<menoh_impl::mkldnn_with_generic_fallback_backend::context>
      > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char>,
      std::__1::unique_ptr<menoh_impl::mkldnn_with_generic_fallback_backend::context,
      std::__1::default_delete<menoh_impl::mkldnn_with_generic_fallback_backend::context>
      > > > >::__destruct_at_end' requested here
    void clear() _NOEXCEPT {__destruct_at_end(__begin_);}
                            ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/vector:441:9: note: 
      in instantiation of member function
      'std::__1::__vector_base<std::__1::pair<std::__1::basic_string<char>,
      std::__1::unique_ptr<menoh_impl::mkldnn_with_generic_fallback_backend::context,
      std::__1::default_delete<menoh_impl::mkldnn_with_generic_fallback_backend::context>
      > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char>,
      std::__1::unique_ptr<menoh_impl::mkldnn_with_generic_fallback_backend::context,
      std::__1::default_delete<menoh_impl::mkldnn_with_generic_fallback_backend::context>
      > > > >::clear' requested here
        clear();
        ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/iterator:1425:74: note: 
      in instantiation of member function
      'std::__1::__vector_base<std::__1::pair<std::__1::basic_string<char>,
      std::__1::unique_ptr<menoh_impl::mkldnn_with_generic_fallback_backend::context,
      std::__1::default_delete<menoh_impl::mkldnn_with_generic_fallback_backend::context>
      > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char>,
      std::__1::unique_ptr<menoh_impl::mkldnn_with_generic_fallback_backend::context,
      std::__1::default_delete<menoh_impl::mkldnn_with_generic_fallback_backend::context>
      > > > >::~__vector_base' requested here
    template <class _Tp, class _Alloc> friend class _LIBCPP_TEMPLATE_VIS vector;
                                                                         ^
1 warning generated.

SIMD support in Menoh

Hi
I was trying Menoh vgg16 example.
Does Menoh utilizes SIMD IS (like sse4, avx2 ) to speed up the inference ?
If not is there an option to utilize SIMD for CPU in Menoh ?

Thanks

Use the build scripts for Travis CI on local machine

As introduced in #105, building Menoh on Travis CI will be dockerized (containerized) for Linux-based platforms. It is also useful for organizing development environment and running the tests on developer's local machine with minimum effort.

Unhelpful error message is shown when protobuf-compiler is not installed

The build process shows unhelpful error message when protobuf-compiler is not installed. Users will check the clone/download step since it says that external/onnx/onnx/onnx.pb.h is missing. The existence of external/onnx/onnx/onnx_pb.h is also confusing. Perhaps, users will think this is a typo.

...
[ 63%] Building CXX object menoh/CMakeFiles/menoh.dir/node.cpp.o
[ 66%] Building CXX object menoh/CMakeFiles/menoh.dir/onnx.cpp.o
/menoh/menoh/onnx.cpp:15:40: fatal error: external/onnx/onnx/onnx.pb.h: No such file or directory
compilation terminated.
menoh/CMakeFiles/menoh.dir/build.make:614: recipe for target 'menoh/CMakeFiles/menoh.dir/onnx.cpp.o' failed
CMakeFiles/Makefile2:328: recipe for target 'menoh/CMakeFiles/menoh.dir/all' failed
make[2]: *** [menoh/CMakeFiles/menoh.dir/onnx.cpp.o] Error 1
make[1]: *** [menoh/CMakeFiles/menoh.dir/all] Error 2
make: *** [all] Error 2
Makefile:127: recipe for target 'all' failed

Supporting dynamic shape of tensors

I want a feature that users can specify the shape of tensors after the model is built. Currently, the shape of tensors are determined when the model is built (more accurately, when the variable table profile is built). However, this is inconvenient in some cases.

• When processing a stream of data, the batchsize is not constant. Users need to wait enough data or add dummy data for padding.
• Some networks do not resize images into a constant size. For example, Faster R-CNN does not.

In order to allocate memory, I think we can assume the maximum shape of tensors are specified when building a model.

Menoh error loading model from onnx file

Hi
I'm trying to run Menoh with the ONNX file converted from the Keras/Tensorflow model.
The following error occurs when calling the menoh function "make_model_data_from_onnx".
menoh invalid attribute type error: attribute type TENSOR for "value"

This ONNX file was created using mmdnn.
$ mmdownload -f keras -n vgg16 -o ./
$ mmconvert -sf keras -iw imagenet_vgg16.h5 -df onnx -om imagenet_vgg16.onnx

OS: Windows 10
Keras 2.2.2
Tensorflow 1.8.0
onnx-tf 1.2.1
Menoh 1.2.0

For the following file It works.
wget('https://preferredjp.box.com/shared/static/o2xip23e3f0knwc5ve78oderuglkf2wt.onnx', './data/vgg16.onnx')

What should I check?

Kind

menoh_model_data_add_attribute_[int|ints]_to_current_node value type mismatch

menoh_model_data_add_attribute_int_to_current_node takes int32_t as integer value while menoh_model_data_add_attribute_ints_to_current_node takes ints values.

menoh/include/menoh/menoh.h

Lines 143 to 149 in cfb243b

	/*! \brief Add a new int attribute to latest added node in model_data
	*
	* \note Duplication of attribute_name is not allowed and it throws error.
	*/
	menoh_error_code MENOH_API menoh_model_data_add_attribute_int_to_current_node(
	menoh_model_data_handle model_data, const char* attribute_name,
	int32_t value);

menoh/include/menoh/menoh.h

Lines 156 to 162 in cfb243b

	/*! \brief Add a new int array attribute to latest added node in model_data
	*
	* \note Duplication of attribute_name is not allowed and it throws error.
	*/
	menoh_error_code MENOH_API menoh_model_data_add_attribute_ints_to_current_node(
	menoh_model_data_handle model_data, const char* attribute_name, int32_t size,
	const int* value);

This looks inconsistent.
Is this intended definition?

Ignoring unsupported operators in unused part of computation graph

I tried to run super resolution example of MXNet.

The model contains unsupported operator Transpose so I decided to specify the variable just before the Transpose (i.e. variable "25") as output. But Menoh still reports menoh_error_code_unsupported_operator error ("menoh unsupported operator error: Transpose") when I construct a variable profile table.

It would be nice if Menoh ignores unsupported operators in unused part of computation graph.

menoh_model_get_variable_dims()

In node-menoh, I wrote this helper function. It would be nice if there's a C-API function in menoh something similar (if possible) to the following:

int32_t size;
const int32_t *dims;
ec = menoh_model_get_variable_dims(model_handle, var_name, &size, &dims);

Where;

menoh_error_code  menoh_model_get_variable_dims(
    const menoh_model_handle model,
    const char* variable_name, 
    int32_t *size,
    const int32_t **dims);

It is sort of combining _get_variable_dims_size and _get_variable_dims_at all together without having to iterate over _dims_at function with its size...

Assumption: the dims values won't change unless the profile was altered by the app.

Utilize ONNX official features

The official libonnx contains some useful functions:

• dtype/shape inference
• Fusing optimization

We should utilize them.

Building time generation of `attribute_completion_and_shape_inference.hpp`

Currently attribute_completion_and_shape_inference.hpp is generated at coding time but it should be gnerated at building time.

Failure of MSVC build on Appveyor

MSVC build on Appveyor fails with the following error:
https://ci.appveyor.com/project/pfnet-research/menoh/builds/34608060/job/awhcqncaxowfypvm

  onnx_menoh_onnx.pb.cc
  onnx-operators_menoh_onnx.pb.cc
C:\protobuf-3.6.0-msvc\include\google/protobuf/stubs/logging.h(100): error C2220: warning treated as error - no 'object' file generated (compiling source file C:\projects\menoh\build\external\onnx\onnx\onnx-operators_menoh_onnx.pb.cc) [C:\projects\menoh\build\external\onnx\onnx_proto.vcxproj]
C:\protobuf-3.6.0-msvc\include\google/protobuf/stubs/logging.h(100): warning C4251: 'google::protobuf::internal::LogMessage::message_': class 'std::basic_string<char,std::char_traits<char>,std::allocator<char>>' needs to have dll-interface to be used by clients of class 'google::protobuf::internal::LogMessage' (compiling source file C:\projects\menoh\build\external\onnx\onnx\onnx-operators_menoh_onnx.pb.cc) [C:\projects\menoh\build\external\onnx\onnx_proto.vcxproj]
...

Although the same commit was built successfully before:
https://ci.appveyor.com/project/pfnet-research/menoh/builds/23908116/job/mr3hd0rq0m0vmu83

Failed to generate onnx.pb.h when LINK_STATIC_LIBPROTOBUF is enabled

external/onnx/onnx/onnx.pb.h is generated here in CMakeLists.txt:

menoh/CMakeLists.txt

Line 61 in a14ba33

execute_process(COMMAND ${PROTOBUF_PROTOC_EXECUTABLE} -I=${ONNX_DIR} --cpp_out=${ONNX_DIR} ${ONNX_DIR}/onnx/onnx.proto WORKING_DIRECTORY ${CMAKE_SOURCE_DIR})

But if LINK_STATIC_LIBPROTOBUF is enabled, protoc has not been built yet.

It seems that we need to run the command during make instead of during cmake.

Support opset version 9

As today, if trying to import an onnx file using opset 9 :

$ example/docclass_example
Document classifier example
terminate called after throwing an instance of 'menoh::error'
what(): menoh unsupported onnx opset version error: given onnx has opset version 9 > 8
Aborted

opset 9 chnagelog :
https://github.com/onnx/onnx/blob/master/docs/Changelog.md#version-9-of-the-default-onnx-operator-set