edmbernard/pybind11_opencv_numpy

Implementation of cv::Mat conversion to numpy.array for pybind11

pybind11_opencv_numpy

Binding between cv::Mat and np.array. And a small code example of how it work. The code work for OpenCV 3+ and OpenCV 4+

The code in this repository create a simple binding, function in c++ are implemented in tests/cpp/test.cpp file and python script that use them are in the tests folder like tests/test_binding.py.

/project folder
├── CMakeLists.txt
├── setup.py
├── ndarray_converter.cpp  # Effective implementation of the binding
├── ndarray_converter.h    # Header for binding functions
└── ...

There is 3 way to build the project, more information can be found on pybind11 website here :

1. build with cmake : the basic, compiled library will be generated in build folder
2. build with setup.py and cmake : install your module in python and dependency are managed by cmake/vcpkg (opencv and pybind11)
3. Not Supported Yet build with setup.py and setuptools : install your module in python and dependency are managed by python package (opencv and pybind11)

Build with cmake

Note : This method support opencv 3 and opencv 4.

Step 1 : Install dependencies

We use vcpkg to install pybind11 library

./vcpkg install pybind11 opencv4

Step 2 : Define the environnement variable to VCPKG Directory

on Linux

export VCPKG_DIR=/path/to/vcpkg/dir

on Windows

$Env:VCPKG_DIR="/path/to/vcpkg/dir"

Note : vcpkg dir should contain the following folder : scripts/buildsystems/vcpkg.cmake

Step 3 : Compile

mkdir build
cd build
# configure make with vcpkg toolchain
cmake .. -DCMAKE_TOOLCHAIN_FILE=${VCPKG_DIR}/scripts/buildsystems/vcpkg.cmake
# on Windows : cmake.exe .. -DCMAKE_TOOLCHAIN_FILE="$Env:VCPKG_DIR/scripts/buildsystems/vcpkg.cmake"
# generate your library, in this poject it's the test_module
cmake --build . --config Release

Numpy header

In case of error like: 'numpy/ndarrayobject.h' : No such file or directory

You have to be sure numpy is install on the computer either with python -m pip install numpy or sudo apt-get install python-numpy for linux. Even if numpy is installed, CMake was not able to find correctly numpy header during configuration. Probably because numpy is install on the user package. Its possible the explicitly set the directory with the following command :

cmake .. -DCMAKE_TOOLCHAIN_FILE=${VCPKG_DIR}/scripts/buildsystems/vcpkg.cmake -DNUMPY_INCLUDE_DIR="${PYTHON_USER_DIR}/LocalCache/local-packages/Python39/site-packages/numpy/core/include/"

Numpy header location can be found with the following command :

python -c "import numpy as np; print(np.get_include())"

Build with setup.py and cmake

Note : This method support opencv 3 and opencv 4.

You should first follow step 1 and 2 from the Build with cmake paragraph

Step 3 : Compile

OpenCV 3+ , OpenCV 4+

python3 -m pip install .

Execute Tests

Tests use the binding module generate by cmake.

python -m pytest

Limitation

Currently, the conversion between cv::Mat and Numpy.ndarray is done by copy.

pybind11_opencv_numpy's Issues

How can I convert other types like: cv2.KeyPoint to cv::KeyPoint ?

This repo is really helpful to me.

What prevents opencv4 from being supported via setup.py?

Could you please explain why opencv4 isn't compatible with the existing setup.py installation approach?

Very cool project

support opencv 4

Does project support opencv 4 ?
https://github.com/edmBernard/pybind11_opencv_numpy/blob/master/ndarray_converter.cpp#L105

binding for std::vector<cv::Mat>

I've tried but failed. Could you tell me how should I do? thank for your awesome work

undefined symbol when testing example.cpp

Hi,
Thanks for this project! I am kind of new to exposing C++ programs to Python. I tried to test to the example.cpp in this project with the the following compiling command:

g++ -Wall -g -fPIC -std=c++11 -c -I"./pybind11_opencv_numpy/" -I"/usr/include/python2.7" example.cpp

g++ -O3 -shared -fPIC -std=c++11 python-config --cflags --ldflags-L"./pybind11_opencv_numpy/"pkg-config --libs opencv example.o -o example.so

Gave me this error when trying to import example in python:
ImportError: ./example.so: undefined symbol: _ZN2cv3Mat10deallocateEv

Did I miss something when linking? What compiling commands were you guys using?

Regards,
Charlie Lam

Which Version Of Opencv is Supported?

the convertion between numpy and c++ cv::Mat. which opencv version does this project support? and aslo support with multithread operation (using the conversion in child thread)?

how to return a list of cvMat?

memory leak

I used your code to wrap a c++ camera library; see https://github.com/hstarmans/Arducampython.
In this code, I call get image a lot. This results in a memory leak.
I have now fixed this by handling the lifetime of the image pointer in C++.

cv::Mat *Camera::pythoncapture(uint32_t exptime)
{
    // data is sent over to python, you need to explicitly clear it
    // if capture is called directly; this results in a memory leak 
    if(pyimage != NULL){delete pyimage;}
    pyimage = capture(exptime);
    return  pyimage;
}

If I call this method from python i use;

.def("capture", &Camera::pythoncapture, py::return_value_policy::copy, "Capture image with given exposure time and return buffer", py::arg("exptime"))

See pybind documentation on functions. I tried other policies, they all lead to a memory leak. Using &Camera::capture directly leads to a memory leak as well.

ImportError: undefined symbol: _ZN2cv6imreadERKNSt7

Hi, why this error?

python test.py

  File "test.py", line 2, in <module>
    import opencv_numpy.example as eb
ImportError: **p/venv/lib/python3.7/site-packages/pybind11_opencv_numpy_example-0.1-py3.7-linux-x86_64.egg/opencv_numpy/example.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN2cv6imreadERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEi

python 3.7
opencv latest from github compiled (4)
numpy 1.18.2
ubuntu 19.04
gcc (Ubuntu 8.3.0-6ubuntu1) 8.3.0

build with setup

crash in NumpyAllocator::allocate (everything seems correct)

Hi, great piece of software. Without being able to isolate minimal example, I am seeing a crash (segmentation failut) when accessing a cv::Mat class member... Maybe the backtrace will make some sense to you? The offending line is

pybind11_opencv_numpy/ndarray_converter.cpp

Line 132 in 7502afd

PyObject* o = PyArray_SimpleNew(dims, _sizes, typenum);

These are parameters passed to PyArray_SimpleNew (dimensions are supposed to be 300x3190, type should be uint16_t / CV_16U; the same crash happens with a float / CV_32F array, anyway):

(gdb) print dims
$11 = 2
(gdb) print _sizes
$12 = {ptr = 0x7ffdd74bc700, sz = 3, buf = {300, 3190, 0, 0, 140728215521000, 140728215521072, 9570, 3, 9713958928, 0, 0, 0, 0, 0, 0, 0, 140728215521096, 140728215521168, 9570, 3, 9713958928, 0, 0, 0, 0, 0, 0, 
    0, 140728215521192, 140728215521264, 9570, 3, 120323965, 140728215521815, 140728215521344, 139896206255961, 8, 140728215521552, 140728215522016, 140728215522464, 140728215521408, 139896206249481, 
    4678710246891232643, 140728215521520, 140728216649878, 139896206189120, 0, 140728215521520, 140728215521440, 139896206189148, 140728215521472, 140728215521520, 140728215521472, 139896206189176, 0, 
    140728215521520, 0, 139896206233876, 140728215522160, 140728215521815, 140728215522016, 131010872, 130150320, 9, 130150496, 3190, 140728215521536, 140728215521520, 140728215521815, 131010872, 
    140728215521616, -149846995376473088, 140728215522016, 140728215522016, 140728215521664, 139896206223935, 140728215522160, 140728215521815, 140728215522016, -149846995376473088, 4631572622657737152, 0, 0, 0, 
    1, 139896210910640, 16890968, 139896205590472, 140728215521776, 131011088, 139896775679040, 0, 9713975296, 0, 0, 0, 0, 0, 0, 0, 140728215521768, 140728215521840, 3190, 1, 9713958912, 0, 0, 0, 0, 0, 0, 0, 
    140728215521864, 140728215521936, 3190, 1, 9713958912, 0, 0, 0, 0, 0, 0, 0, 140728215521960, 140728215522032, 3190, 1, 9713975298, 0, 0, 33829544, 140728215522112, 33829672, 140728215522128, 139896206066212}}
(gdb) print typenum
$13 = 4

while these are arguments of NumpyAllocator::allocate:

(gdb) print dims0
$18 = 2
(gdb) print sizes[0]
$19 = 300
(gdb) print sizes[1]
$20 = 3190
(gdb) print type
$21 = 2
(gdb) print data
$22 = (void *) 0x0
(gdb) print *step
$23 = 6380
(gdb) print flags
$24 = 0
(gdb) print usageFlags
$25 = cv::USAGE_DEFAULT

and this is the full stack trace

#0  0x00007f3c1fac895e in NumpyAllocator::allocate (this=0x7f3c1ff33070 <g_numpyAllocator>, dims0=2, sizes=0x7ffdd74bd038, type=2, data=0x0, step=0x7ffdd74bd080, flags=0, usageFlags=cv::USAGE_DEFAULT)
    at ../src/other/pybind11_opencv_numpy/ndarray_converter.cpp:132
#1  0x00007f3c1eb673c7 in cv::Mat::create(int, int const*, int) () from /usr/lib/x86_64-linux-gnu/libopencv_core.so.3.2
#2  0x00007f3c1eb856e5 in cv::_OutputArray::create(int, int, int, int, bool, int) const () from /usr/lib/x86_64-linux-gnu/libopencv_core.so.3.2
#3  0x00007f3c1eab7bd4 in cv::Mat::copyTo(cv::_OutputArray const&) const () from /usr/lib/x86_64-linux-gnu/libopencv_core.so.3.2
#4  0x00007f3c1fac7003 in NDArrayConverter::toNDArray (m=...) at ../src/other/pybind11_opencv_numpy/ndarray_converter.cpp:354
#5  0x00007f3c1fa4bc87 in pybind11::detail::type_caster<cv::Mat, void>::cast (m=..., defval=...) at ../src/other/pybind11_opencv_numpy/ndarray_converter.h:35

Is there anything obviously wrong?

How to use this with Mat pointer

I have the following C++ function to try returning a Mat pointer:

void generate_black_image(int x, int y, cv::Mat& dst) {
    cv::Mat image = cv::Mat::zeros(cv::Size(x,y),CV_8UC1);
    dst = image;
}

I tried using the following pybind:

PYBIND11_MODULE(core, m) {
  NDArrayConverter::init_numpy();
  m.def("generate_black_image", &I3DR::generate_black_image);
}

But I'm not sure how to use it in python as it doesn't deal with pointers.
I'm not sure how to run this in python but I tried passing the numpy array like this:

blank_image = np.array([])
generate_black_image(200, 200, blank_image)
print("Image created with size {}".format(blank_image.shape))
cv2.imshow("test", blank_image)
cv2.waitKey(0)

But that doesn't populate the numpy array with the data and remains empty.

I am doing it like this rather than a return variable as I am also trying to support C# with extern C so need to use standard C types.

Please let me know what I am doing wrong or if this is possible?

How to work with CV_32FC2 type?

How to work with CV_32FC2 type? i.e. how to pass numpy array to function that expect CV_32FC2 type?

Can you provide a license file?

I'm a little bit afraid of using software from github without license 😉 Could you please add one (like MIT or something else so that I'm able add your copyright hint)?

Thanks!

'numpy/ndarrayobject.h' file not found

When I execute python setup.py install, generate a error(ndarray_converter.cpp:7:10: fatal error: 'numpy/ndarrayobject.h' file not found).
How should I solve it?

AttributeError: type object 'numpy.flatiter' has no attribute 'converter'

Traceback (most recent call last):
  File "/home/kotbegemot/CLionProjects/testdrive/cmake-build-debug/main.py", line 1, in <module>
    import numpy as np
  File "/home/kotbegemot/CLionProjects/testdrive/venv/lib/python3.6/site-packages/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/home/kotbegemot/CLionProjects/testdrive/venv/lib/python3.6/site-packages/numpy/core/__init__.py", line 104, in <module>
    from . import _add_newdocs
  File "/home/kotbegemot/CLionProjects/testdrive/venv/lib/python3.6/site-packages/numpy/core/_add_newdocs.py", line 83, in <module>
    """))
  File "/home/kotbegemot/CLionProjects/testdrive/venv/lib/python3.6/site-packages/numpy/core/function_base.py", line 511, in add_newdoc
    _add_docstring(getattr(new, attr), docstring.strip(), warn_on_python)
AttributeError: type object 'numpy.flatiter' has no attribute 'converter'

Process finished with exit code 1

#include <pybind11/pybind11.h>

#include "ndarray_converter.hpp" /// https://github.com/edmBernard/pybind11_opencv_numpy

#include "test.hpp"

PYBIND11_MODULE(module, m) {

    NDArrayConverter::init_numpy();

    py::class_<test>(m, "test", py::buffer_protocol())
            .def(py::init<const std::string &, float, float, int>())
            .def("forward", &test::forward);

} 

import numpy as np
import cv2

from module import test
image = cv2.imread("frame.bmp")
d = test("test.db", 0,-1, 0)
out = d.forward(image)

print(out)

set(${PROJECT_NAME}_headers
        ndarray_converter.hpp
       test.hpp
)

set(${PROJECT_NAME}_sources
    ndarray_converter.cpp
     test.cpp
)

pybind11_add_module(module main.cpp ${${PROJECT_NAME}_sources} ${${PROJECT_NAME}_headers})
target_include_directories(
        module
        PRIVATE
        /usr/local/lib/python3.6/dist-packages/numpy/core/include/
)
target_link_directories(
        module
        PRIVATE
        /usr/local/lib/python3.6/dist-packages/numpy/core/lib/
)
target_link_libraries(module PRIVATE
        npymath
        ${CONAN_LIBS}
)

Struggling to build this

Im running Windows 10 and I've tried following both sets of set up instructions without success.

I've installed vcpkg, used that to install pybind11, and then ran:
pybind11_opencv_numpy-master\build>cmake .. -DCMAKE_TOOLCHAIN_FILE='.......\vcpkg.cmake'
and I get the following errors:
CMake Error at C:/Users/m.flemmer/Programs/vcpkg/installed/x86-windows/share/cmake/pybind11/FindPythonLibsNew.cmake:114 (list):
list index: 9 out of range (-8, 7)
Call Stack (most recent call first):
C:/Users/m.flemmer/Programs/vcpkg/scripts/buildsystems/vcpkg.cmake:266 (_find_package)
C:/Users/m.flemmer/Programs/vcpkg/installed/x86-windows/share/cmake/pybind11/pybind11Tools.cmake:16 (find_package)
C:/Users/m.flemmer/Programs/vcpkg/installed/x86-windows/share/cmake/pybind11/pybind11Config.cmake:100 (include)
C:/Users/m.flemmer/Programs/vcpkg/scripts/buildsystems/vcpkg.cmake:266 (_find_package)
CMakeLists.txt:19 (find_package)

CMake Error at C:/Users/m.flemmer/Programs/vcpkg/installed/x86-windows/share/cmake/pybind11/FindPythonLibsNew.cmake:122 (message):
Python config failure: Python is 64-bit, chosen compiler is 32-bit
Call Stack (most recent call first):
C:/Users/m.flemmer/Programs/vcpkg/scripts/buildsystems/vcpkg.cmake:266 (_find_package)
C:/Users/m.flemmer/Programs/vcpkg/installed/x86-windows/share/cmake/pybind11/pybind11Tools.cmake:16 (find_package)
C:/Users/m.flemmer/Programs/vcpkg/installed/x86-windows/share/cmake/pybind11/pybind11Config.cmake:100 (include)
C:/Users/m.flemmer/Programs/vcpkg/scripts/buildsystems/vcpkg.cmake:266 (_find_package)
CMakeLists.txt:19 (find_package)

If I use the above CMake command with -G"Visual Studio 15 2017 Win64" I get the error:
CMake Error at C:/Users/m.flemmer/Programs/vcpkg/scripts/buildsystems/vcpkg.cmake:266 (_find_package):
Could not find a configuration file for package "OpenCV" that is compatible
with requested version "3".

The following configuration files were considered but not accepted:

C:/Users/m.flemmer/Programs/OpenCV/build/install/OpenCVConfig.cmake, version: 4.1.0

I've also tried the python setup method:
pip install pybind11,
python setup.py build

and I get:
example.cpp
example.cpp(3): fatal error C1083: Cannot open include file: 'opencv2/core/core.hpp': No such file or directory
error: command 'C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\cl.exe' failed with exit status 2

Could someone advise me on how to proceed?

Cheers!

OpenCV 2.4 isn't supported

Hi,
Thanks for sharing this utility.
I just wanted to note that OpenCV 2.4 isn't supported anymore. e.g. UMatData is an OpenCV >= 3 type.