cedrickchee/pytorch-android

[EXPERIMENTAL] Demo of using PyTorch 1.0 inside an Android app. Test with your own deep neural network such as ResNet18/SqueezeNet/MobileNet v2 and a phone camera.

pytorch-android's Issues

Help please

In this project, it seems to be using resnet or squeezenet. However, in a project I'm doing of my own, I am currently using mobilenetv2. Currently, I have converted the mobilenetv2 model from pytorch to caffe2. If I use this caffe2 file, would it work seamlessly after replacing the _init_net.pb and _predict_net.pb or do I need to make other modifications? Thank you.

How to run this project using Android GPU

Hello,
Thanks to your project, I could successfully test Caffe2 models using Android CPU.

Now I am trying to test the same model on Android GPU,
But I couldn't find any sample code or documents that explains how to.

In Caffe2 official site, they said Caffe2 is optimized for ARM GPUs. (https://caffe2.ai/blog/2018/02/23/Caffe2_Now_Optimized_for_ARM_Mobile_GPUs.html)

Thank you in advance.

Error in building the code

I got the following error while building the code

Build command failed.
Error while executing process /home/yxchng/Android/Sdk/cmake/3.10.2.4988404/bin/cmake with arguments {--build /home/yxchng/git/new3/pytorch-android/app/.externalNativeBuild/cmake/debug/armeabi-v7a --target native-lib}
[1/4] Building C object CMakeFiles/cpufeatures.dir/home/yxchng/Android/Sdk/ndk-bundle/sources/android/cpufeatures/cpu-features.c.o
[2/4] Linking C static library libcpufeatures.a
[3/4] Building CXX object CMakeFiles/native-lib.dir/src/main/cpp/native-lib.cpp.o
FAILED: CMakeFiles/native-lib.dir/src/main/cpp/native-lib.cpp.o
/home/yxchng/Android/Sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ --target=armv7-none-linux-androideabi22 --gcc-toolchain=/home/yxchng/Android/Sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64 --sysroot=/home/yxchng/Android/Sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/sysroot -Dnative_lib_EXPORTS -I/home/yxchng/Android/Sdk/ndk-bundle/sources/android/cpufeatures -I../../../../src/main/cpp -g -DANDROID -fdata-sections -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -mfpu=vfpv3-d16 -fno-addrsig -march=armv7-a -mthumb -Wa,--noexecstack -Wformat -Werror=format-security -stdlib=libc++ -frtti -fexceptions -std=c++11 -O0 -fno-limit-debug-info -fPIC -MD -MT CMakeFiles/native-lib.dir/src/main/cpp/native-lib.cpp.o -MF CMakeFiles/native-lib.dir/src/main/cpp/native-lib.cpp.o.d -o CMakeFiles/native-lib.dir/src/main/cpp/native-lib.cpp.o -c ../../../../src/main/cpp/native-lib.cpp
In file included from ../../../../src/main/cpp/native-lib.cpp:8:
In file included from ../../../../src/main/cpp/caffe2/core/operator.h:18:
In file included from ../../../../src/main/cpp/caffe2/core/operator_gradient.h:7:
../../../../src/main/cpp/caffe2/utils/proto_utils.h:58:1: warning: control reaches end of non-void function [-Wreturn-type]
}
^
In file included from ../../../../src/main/cpp/native-lib.cpp:16:
In file included from ../../../../src/main/cpp/ATen/ATen.h:11:
../../../../src/main/cpp/ATen/Functions.h:3441:66: error: too few arguments to function call, expected 4, have 3
return detail::infer_type(self).as_strided(self, size, stride);
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^
../../../../src/main/cpp/ATen/core/Type.h:208:3: note: 'as_strided' declared here
virtual Tensor as_strided(const Tensor & self, IntList size, IntList stride, c10::optional<int64_t> storage_offset) const = 0;
^
In file included from ../../../../src/main/cpp/native-lib.cpp:16:
In file included from ../../../../src/main/cpp/ATen/ATen.h:11:
../../../../src/main/cpp/ATen/Functions.h:3444:67: error: too few arguments to function call, expected 4, have 3
return detail::infer_type(self).as_strided_(self, size, stride);
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^
../../../../src/main/cpp/ATen/core/Type.h:209:3: note: 'as_strided_' declared here
virtual Tensor & as_strided_(Tensor & self, IntList size, IntList stride, c10::optional<int64_t> storage_offset) const = 0;
^
In file included from ../../../../src/main/cpp/native-lib.cpp:16:
In file included from ../../../../src/main/cpp/ATen/ATen.h:11:
../../../../src/main/cpp/ATen/Functions.h:5243:37: error: no member named 'potrs' in 'at::TypeExtendedInterface'
return detail::infer_type(self).potrs(self, input2, upper);
~~~~~~~~~~~~~~~~~~~~~~~~ ^
1 warning and 3 errors generated.
ninja: build stopped: subcommand failed.

How to improve the performance on Android?

Thanks a lot for the overall android project the author posted, it has saved the time for me to compile from source.
I use the caffe2 MobilenetV1-ssd model for object detection and it works. However, the same model costs about 2.2s to infer an image in my android phone (although with a cpu that is not good enough--qualcomm snapdragon 616), whereas it costs 500ms to infer an image on Window 7 with CPU.
I wonder if the caffe2::predictor is running using multithreads or not？If no, how to configure it ? Is there any way to improve the performance on android ?

Thanks a lot if someone could help me about this.

Issues with SqueezeNet in the app (PyTorch master as of 09/04/2019)

Hi, I’ve been working with this AICamera example for PyTorch using Ji Lin's MobileNet V2 implementation. As Ji Lin’s implementation includes a dropout layer, I wasn’t able to export the model to Android in the stable PyTorch 1.0 release (execution with predictor failed in both Python and Android). So, I had to remove that layer. Also, I couldn’t export the pretrained SqueezeNet model from PyTorch as it uses ceil_mode=True and ONNX export failed with that.

This week I cloned the PyTorch master from GitHub, as I saw those issues were fixed. I built the library and installed it, then I was able to export the models and execute them with predictor in Python. Then I also built the Caffe 2 API for Android using the build_android.sh script from the repo and copied the files to the AICamera project. I then copied the MobileNet V2 with dropout init_net and predict_net to the Android project and it worked perfectly. Nevertheless, when I tried to use the pretrained SqueezeNet model that I had exported, the Android app crashed in the model execution with predictor (even though it executed without trouble in Python).

I want to know how to fix it as I’ve seen SqueezeNet has been used with AICamera.

Some details that might help:

• In stable PyTorch, the ONNX exported graph included the dropout and that was the reason why it couldn’t be executed with predictor. Now the dropout is not in the graph and I think that’s why it is executed correctly in both Python and PyTorch. What I mean is that it might have nothing to do with me building/updating the Caffe 2 API for Android.

• I tried with the SqueezeNet implementation from https://github.com/onnx/tutorials/blob/master/tutorials/PytorchCaffe2MobileSqueezeNet.ipynb which has ceil_mode=False and the execution in Android also fails.

• I tried with both the original SqueezeNet pretrained model and a transfer-learning-trained one but none of them worked (it must be something in the architecture).

• I tried with the SqueezeNet init_net.pb and predict_net.pb included in https://github.com/cedrickchee/pytorch-android (this repo) and it DID work in Android.

Thanks.

Have you tried using the quantized MobileNet from Caffe 2? Does it work?

https://github.com/caffe2/models/tree/master/mobilenet_v2_quantized#model

Help on preprocessing

Dear Cedric, thank you so much for this repo, it's been very very helpful. I am using a ResNet18 model for binary classification on my android app, I also made some changes on java code to capture a still image instead of a preview. I am stuck at pre-processing the images. In my original PyTorch code I had:

transforms.Resize(256)
transforms.CenterCrop(224)
transforms.ToTensor()
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])

I see in the cpp code some means b_mean = 104.00, g_mean = 116.66 and r_mean = 122.67. I'm not sure what these means are and what is the processing happening in the snippet below. I need help to preprocess the images, for now my output doesn't make sense at all. Thank you so much for help.

            float b_mean = 104.00698793f;
            float g_mean = 116.66876762f;
            float r_mean = 122.67891434f;

            auto b_i = 0 * IMG_H * IMG_W + j * IMG_W + i;
            auto g_i = 1 * IMG_H * IMG_W + j * IMG_W + i;
            auto r_i = 2 * IMG_H * IMG_W + j * IMG_W + i;

            if (infer_HWC) {
                b_i = (j * IMG_W + i) * IMG_C;
                g_i = (j * IMG_W + i) * IMG_C + 1;
                r_i = (j * IMG_W + i) * IMG_C + 2;
            }

            //R = Y + 1.402 (V-128)
            //G = Y - 0.34414 (U-128) - 0.71414 (V-128)
            // B = Y + 1.772 (U-V)
            input_data[r_i] = -r_mean + (float) ((float) min(255., max(0., (float) (y + 1.402 * (v - 128)))));
            input_data[g_i] = -g_mean + (float) ((float) min(255., max(0., (float) (y - 0.34414 * (u - 128) - 0.71414 * (v - 128)))));
            input_data[b_i] = -b_mean + (float) ((float) min(255., max(0., (float) (y + 1.772 * (u - v)))));

About libcaffe2.a

Missing libcaffe2.a file in this project. I build the ndk-r18 and get the libcaffe2.a with the size 1.95GB. Is your libcaffe2.a so big?