kingpeter2015/libovmatting

A C++ library for Background Matting

libovmatting

A C++ library for Background Matting using openvino and deep learning models.

Now support models such as BackgroundMattingV2, MODNet.

1 Application scenes

...

2 License

The code of libovmatting is released under the MIT License.

3 Development environment

CentOS 7
Ubuntu
Windows 10

4 Usage

4.1 Install openvino

Please refer https://software.intel.com/content/www/us/en/develop/tools/openvino-toolkit.html
Use version: openvino_2021.2.185

4.2 Build libovmatting library and test program

Clone the project
$git clone https://github.com/kingpeter2015/libovmatting.git

4.2.1 Ubuntu/CentOS

$cd libovmatting
$mkdir build
$cmake ..
$make

Run the test program
$cd ./build/
$LD_LIBRARY_PATH=./:$(LD_LIBRARY_PATH) ./sample

4.2.2 Windows 10

Install Visual Studio 2019
open Win64\ovmatting.sln
build and run in the Visual Studio IDE

5 About Source Code

5.1 Directory Structure

• Win64: Windows Solution and projects
  
• include: contains header files. ovmatter.h is api header file
  
• libovmatting: CMakefiles for compile libovmatting.so under linux
  
• samples: four samples using libovmatting library
  
• share: contains model files which are used by openvino inference engine.
  
• src: contains source code files
  

main.cpp: startup point of demos

5.2 Please Attention !!!

• 'include/ovmatter.h' is one only sdk header file which exposes api to applications.
• libovmatting library only supports two matting method: METHOD_BACKGROUND_MATTING_V2 and METHOD_MODNET. They are defined in include/ovmatter.h
• METHOD_BACKGROUND_MATTING_V2 method uses cnn model defined 'pytorch_mobilenetv2.bin/pytorch_mobilenetv2.xml'.
• METHOD_MODNET method uses cnn model defined in 'modnet.bin/modnet.xml'

if you want to use your own model, you can do following steps:

• Add a enum item in enum OV_MATTER_API MattingMethod which is defined in include/ovmatter.h
• declare and implement a new ovmatter class, and inherit 'MatterBaseImpl', then write a new cnn that uses your own models

That's all, have fun !

6 Reference

OpenVINO https://software.intel.com/content/www/us/en/develop/tools/openvino-toolkit.html
BackgroundMattingV2 https://github.com/PeterL1n/BackgroundMattingV2
MODNet https://github.com/ZHKKKe/MODNet

libovmatting's Issues

Get many errors like"error C4996: 'InferenceEngine::details::InferenceEngineException'" while build ovmatting.sln

Dear kingpeter2015
I'm now using visual studio 2019 to build ovmatting.sln ,but I get a lot of errors:

1>C:\Users\zjc83\source\repos\libovmatting\include\common.hpp(167,9): error C4996: 'InferenceEngine::details::InferenceEngineException': Use InferRequest::Exception

and at the end of the report ,there is an error goes

2>LINK : fatal error LNK1104: unable to open file“ovmatting.lib”

I have already installed openvino runtime ,so I suspect that whether I didn't configure the openvino environment in visual studio 2019
Could you please answer my confusion and give me some advices?

Thank you and hoping for your reply!

Azarxxx

run in cuda and args

params.effect = EFFECT_NONE;
params.device = "CPU";

params.cpu_threads_num = 0;
params.cpu_bind_thread = true;
params.cpu_throughput_streams = 1;

params.is_async = false;
params.interval = 1;
params.threshold_motion = -0.1f;

i run it in win10 cpu,
And,
can you tell me the args meaning?
when i use it to run HD video ，it is slow, and cuda is not used when params.device = "GPU";
only run it in intel gpu?

Cannot create MVN layer InstanceNormalization_205.

Hi,thanks your nice work. I meet some err.
I retrained the modnet model, I got a new onnx,and then replaced your onnx,
I meet the error:
MatterChannel::create():Cannot create MVN layer InstanceNormalization_205/Ins_Norm/MVN_id:500 from unsupported opset:opset6
Can not create Matte Channel.

Method to convert from `.onnx` to IR

I had a lot of trouble converting the official .onnx models to IR files.
Actually I'm a beginner at this. How did you do it? can you help me?

Specifically, the script mo.py is used to transform the model according to OpenVINO's documentation:

python .\mo.py --input_model <model_dir><model_file> --data_type <model_precision> --output_dir <ir_dir>

I only specified --input_model, --data_dype, --output_dir and --input_shape, but I got a lot of errors. what parameters did you use?

Thank you very much and your libovmatting.

can`t find opencv_core451.dll

I configured the environment as instructed, but when I ran the DemoMatting project, opencv_core451.dll was not found. Please tell me how to solve the problem.

EXE同一目录下找不到ovmatting.dl

请问作者遇到过这个问题吗（挠头）

Cannot create MVN layer InstanceNormalization_205/Ins_Norm/MVN_ id:475 from unsupported opset: opset6

my trained modnet.xml cannot be inferenced, but your .xml can be inferenced successfully. please help me!

Can I do portrait matting without inputting the video background

When load the IR model to inference,but I meet some error

I export the pytorch_mobilenetv2.pth to onnx , and convert it to IR (*.xml , *.bin), when the load the IR model I meet the error.

==============================================================================
C:\Users\10400\Desktop\MODNet-master\input-output>python infer_camera.py --xml onnx.xml

Traceback (most recent call last):
File "infer_camera.py", line 156, in
exec_net = ie.load_network(network=net, device_name='CPU')
File "ie_api.pyx", line 306, in openvino.inference_engine.ie_api.IECore.load_network
File "ie_api.pyx", line 315, in openvino.inference_engine.ie_api.IECore.load_network
RuntimeError:
Unsupported dynamic ops:
v3::NonZero NonZero_430 (Gather_4291168[0]:f32{1,270,480}) -> (i32{3,?})
v1::Transpose Transpose_431 (NonZero_430[0]:i32{3,?}, Constant_11970[0]:i32{2}) -> (i32{?,3})
v0::GatherIE GatherIE_20028 (Transpose_431[0]:i32{?,3}, Unsqueeze_20027[0]:i32{1}) -> (i32{?,1})
v0::Squeeze Gather_4331172 (GatherIE_20028[0]:i32{?,1}, Constant_22635[0]:i32{1}) -> (i32{?})
v1::Eltwise Mul_726/Fused_Mul_ (Gather_4331172[0]:i32{?}, Constant_11972[0]:i32{1}) -> (i32{?})
v0::GatherIE GatherIE_20035 (Transpose_431[0]:i32{?,3}, Unsqueeze_20034[0]:i32{1}) -> (i32{?,1})
v0::Squeeze Gather_4351192 (GatherIE_20035[0]:i32{?,1}, Constant_22636[0]:i32{1}) -> (i32{?})
v1::Eltwise Mul_728/Fused_Mul_ (Gather_4351192[0]:i32{?}, Constant_11974[0]:i32{1}) -> (i32{?})
v1::Eltwise Add_731 (Mul_726/Fused_Mul_[0]:i32{?}, Mul_728/Fused_Mul_[0]:i32{?}) -> (i32{?})
v0::GatherIE GatherIE_20042 (Transpose_431[0]:i32{?,3}, Unsqueeze_20041[0]:i32{1}) -> (i32{?,1})
v0::Squeeze Gather_4371154 (GatherIE_20042[0]:i32{?,1}, Constant_22637[0]:i32{1}) -> (i32{?})
v1::Eltwise Mul_733 (Gather_4371154[0]:i32{?}, Constant_11976[0]:i32{1}) -> (i32{?})
v1::Eltwise Add_734 (Add_731[0]:i32{?}, Mul_733[0]:i32{?}) -> (i32{?})
v1::Reshape Reshape_736 (Add_734[0]:i32{?}, Constant_11977[0]:i32{4}) -> (i32{?,1,1,1})
v0::ShapeOf ShapeOf_7572 (Transpose_431[0]:i32{?,3}) -> (i32{2})
v1::Broadcast Broadcast_9974 (Constant_11978[0]:i32{1}, Where_748[0]:i32{4}, Constant_9973[0]:u8{}) -> (i32{?,?,?,?})
v1::Eltwise Expand_749 (Reshape_736[0]:i32{?,1,1,1}, Broadcast_9974[0]:i32{?,?,?,?}) -> (i32{?,?,?,?})
v1::Broadcast Broadcast_9978 (Constant_11988[0]:i32{1}, Where_764[0]:i32{4}, Constant_9977[0]:u8{}) -> (i32{?,?,?,?})
v1::Eltwise Expand_765 (Constant_11987[0]:i32{1,4,4,4}, Broadcast_9978[0]:i32{?,?,?,?}) -> (i32{?,?,?,?})
v1::Eltwise Add_766 (Expand_749[0]:i32{?,?,?,?}, Expand_765[0]:i32{?,?,?,?}) -> (i32{?,?,?,?})
v1::Reshape Reshape_770 (Add_766[0]:i32{?,?,?,?}, Constant_11994[0]:i32{1}) -> (i32{?})
v0::Convert Cast_467 (Gather_4331172[0]:i32{?}) -> (f32{?})
v1::Reshape Unsqueeze_494/Unsqueeze (Cast_467[0]:f32{?}, Constant_12000[0]:i32{2}) -> (f32{?,1})
v0::Convert Cast_469 (Gather_4371154[0]:i32{?}) -> (f32{?})
v1::PowerIE Sub_473 (Cast_469[0]:f32{?}) -> (f32{?})
v1::Reshape Unsqueeze_495/Unsqueeze (Sub_473[0]:f32{?}, Constant_12001[0]:i32{2}) -> (f32{?,1})
v0::Convert Cast_468 (Gather_4351192[0]:i32{?}) -> (f32{?})
v1::PowerIE Sub_477 (Cast_468[0]:f32{?}) -> (f32{?})
v1::Reshape Unsqueeze_496/Unsqueeze (Sub_477[0]:f32{?}, Constant_12002[0]:i32{2}) -> (f32{?,1})
v1::PowerIE Add_481/Fused_Add_ (Cast_469[0]:f32{?}) -> (f32{?})
v1::Reshape Unsqueeze_497/Unsqueeze (Add_481/Fused_Add_[0]:f32{?}, Constant_12003[0]:i32{2}) -> (f32{?,1})
v1::PowerIE Add_489/Fused_Add_ (Cast_468[0]:f32{?}) -> (f32{?})
v1::Reshape Unsqueeze_498/Unsqueeze (Add_489/Fused_Add_[0]:f32{?}, Constant_12004[0]:i32{2}) -> (f32{?,1})
v0::Concat Concat_499 (Unsqueeze_494/Unsqueeze[0]:f32{?,1}, Unsqueeze_495/Unsqueeze[0]:f32{?,1}, Unsqueeze_496/Unsqueeze[0]:f32{?,1}, Unsqueeze_497/Unsqueeze[0]:f32{?,1}, Unsqueeze_498/Unsqueeze[0]:f32{?,1}) -> (f32{?,5})
v0::GatherIE Gather_5051212 (Concat_499[0]:f32{?,5}, Gather_5051212/Cast_141935_const[0]:i32{4}) -> (f32{?,4})
v0::GatherIE Gather_5011142 (Concat_499[0]:f32{?,5}, Gather_5011142/Cast_141827_const[0]:i32{1}) -> (f32{?,1})
v1::Reshape Squeeze_502 (Gather_5011142[0]:f32{?,1}, Constant_12007[0]:i32{1}) -> (f32{?})
v0::Convert Cast_503 (Squeeze_502[0]:f32{?}) -> (i32{?})
v3::ROIAlign RoiAlign_506 (Pad_466[0]:f32{1,36,546,966}, Gather_5051212[0]:f32{?,4}, Cast_503[0]:i32{?}) -> (f32{?,36,8,8})
v0::Convert Cast_535 (Gather_4331172[0]:i32{?}) -> (f32{?})
v1::Reshape Unsqueeze_562/Unsqueeze (Cast_535[0]:f32{?}, Constant_12010[0]:i32{2}) -> (f32{?,1})
v0::Convert Cast_537 (Gather_4371154[0]:i32{?}) -> (f32{?})
v1::PowerIE Sub_541 (Cast_537[0]:f32{?}) -> (f32{?})
v1::Reshape Unsqueeze_563/Unsqueeze (Sub_541[0]:f32{?}, Constant_12011[0]:i32{2}) -> (f32{?,1})
v0::Convert Cast_536 (Gather_4351192[0]:i32{?}) -> (f32{?})
v1::PowerIE Sub_545 (Cast_536[0]:f32{?}) -> (f32{?})
v1::Reshape Unsqueeze_564/Unsqueeze (Sub_545[0]:f32{?}, Constant_12012[0]:i32{2}) -> (f32{?,1})
v1::PowerIE Add_549/Fused_Add_ (Cast_537[0]:f32{?}) -> (f32{?})
v1::Reshape Unsqueeze_565/Unsqueeze (Add_549/Fused_Add_[0]:f32{?}, Constant_12013[0]:i32{2}) -> (f32{?,1})
v1::PowerIE Add_557/Fused_Add_ (Cast_536[0]:f32{?}) -> (f32{?})
v1::Reshape Unsqueeze_566/Unsqueeze (Add_557/Fused_Add_[0]:f32{?}, Constant_12014[0]:i32{2}) -> (f32{?,1})
v0::Concat Concat_567 (Unsqueeze_562/Unsqueeze[0]:f32{?,1}, Unsqueeze_563/Unsqueeze[0]:f32{?,1}, Unsqueeze_564/Unsqueeze[0]:f32{?,1}, Unsqueeze_565/Unsqueeze[0]:f32{?,1}, Unsqueeze_566/Unsqueeze[0]:f32{?,1}) -> (f32{?,5})
v0::GatherIE Gather_5731148 (Concat_567[0]:f32{?,5}, Gather_5731148/Cast_141837_const[0]:i32{4}) -> (f32{?,4})
v0::GatherIE Gather_5691150 (Concat_567[0]:f32{?,5}, Gather_5691150/Cast_141763_const[0]:i32{1}) -> (f32{?,1})
v1::Reshape Squeeze_570 (Gather_5691150[0]:f32{?,1}, Constant_12017[0]:i32{1}) -> (f32{?})
v0::Convert Cast_571 (Squeeze_570[0]:f32{?}) -> (i32{?})
v3::ROIAlign RoiAlign_574 (Pad_534[0]:f32{1,6,546,966}, Gather_5731148[0]:f32{?,4}, Cast_571[0]:i32{?}) -> (f32{?,6,8,8})
v0::Concat Concat_575 (RoiAlign_506[0]:f32{?,36,8,8}, RoiAlign_574[0]:f32{?,6,8,8}) -> (f32{?,42,8,8})
v1::ConvolutionIE Conv_576 (Concat_575[0]:f32{?,42,8,8}, 1524/Output_0/Data__const[0]:f32{24,42,3,3}, Reshape_18829[0]:f32{24}) -> (f32{?,24,6,6})
v0::Relu Relu_577 (Conv_576[0]:f32{?,24,6,6}) -> (f32{?,24,6,6})
v1::ConvolutionIE Conv_578 (Relu_577[0]:f32{?,24,6,6}, 1527/Output_0/Data__const[0]:f32{16,24,3,3}, Reshape_18832[0]:f32{16}) -> (f32{?,16,4,4})
v0::Relu Relu_579 (Conv_578[0]:f32{?,16,4,4}) -> (f32{?,16,4,4})
v0::ShapeOf ShapeOf_5939 (Relu_579[0]:f32{?,16,4,4}) -> (i32{4})
v0::ShapeOf ShapeOf_5947 (Relu_579[0]:f32{?,16,4,4}) -> (i32{4})
v4::Interpolate Resize_588 (Relu_579[0]:f32{?,16,4,4}, Resize_588/StridedSlice_sizes[0]:i32{2}, Resize_588/StridedSlice_scales[0]:f32{2}, Constant_12040[0]:i32{2}) -> (f32{?,16,?,?})
v0::Convert Cast_604 (Gather_4331172[0]:i32{?}) -> (f32{?})
v1::Reshape Unsqueeze_631/Unsqueeze (Cast_604[0]:f32{?}, Constant_12041[0]:i32{2}) -> (f32{?,1})
v0::Convert Cast_606 (Gather_4371154[0]:i32{?}) -> (f32{?})
v1::PowerIE Sub_610 (Cast_606[0]:f32{?}) -> (f32{?})
v1::Reshape Unsqueeze_632/Unsqueeze (Sub_610[0]:f32{?}, Constant_12042[0]:i32{2}) -> (f32{?,1})
v0::Convert Cast_605 (Gather_4351192[0]:i32{?}) -> (f32{?})
v1::PowerIE Sub_614 (Cast_605[0]:f32{?}) -> (f32{?})
v1::Reshape Unsqueeze_633/Unsqueeze (Sub_614[0]:f32{?}, Constant_12043[0]:i32{2}) -> (f32{?,1})
v1::PowerIE Add_618/Fused_Add_ (Cast_606[0]:f32{?}) -> (f32{?})
v1::Reshape Unsqueeze_634/Unsqueeze (Add_618/Fused_Add_[0]:f32{?}, Constant_12044[0]:i32{2}) -> (f32{?,1})
v1::PowerIE Add_626/Fused_Add_ (Cast_605[0]:f32{?}) -> (f32{?})
v1::Reshape Unsqueeze_635/Unsqueeze (Add_626/Fused_Add_[0]:f32{?}, Constant_12045[0]:i32{2}) -> (f32{?,1})
v0::Concat Concat_636 (Unsqueeze_631/Unsqueeze[0]:f32{?,1}, Unsqueeze_632/Unsqueeze[0]:f32{?,1}, Unsqueeze_633/Unsqueeze[0]:f32{?,1}, Unsqueeze_634/Unsqueeze[0]:f32{?,1}, Unsqueeze_635/Unsqueeze[0]:f32{?,1}) -> (f32{?,5})
v0::GatherIE Gather_6421190 (Concat_636[0]:f32{?,5}, Gather_6421190/Cast_141661_const[0]:i32{4}) -> (f32{?,4})
v0::GatherIE Gather_6381180 (Concat_636[0]:f32{?,5}, Gather_6381180/Cast_141955_const[0]:i32{1}) -> (f32{?,1})
v1::Reshape Squeeze_639 (Gather_6381180[0]:f32{?,1}, Constant_12048[0]:i32{1}) -> (f32{?})
v0::Convert Cast_640 (Squeeze_639[0]:f32{?}) -> (i32{?})
v3::ROIAlign RoiAlign_643 (Pad_603[0]:f32{1,6,1084,1924}, Gather_6421190[0]:f32{?,4}, Cast_640[0]:i32{?}) -> (f32{?,6,8,8})
v0::Concat Concat_644 (Resize_588[0]:f32{?,16,?,?}, RoiAlign_643[0]:f32{?,6,8,8}) -> (f32{?,22,8,8})
v1::ConvolutionIE Conv_645 (Concat_644[0]:f32{?,22,8,8}, 1530/Output_0/Data__const[0]:f32{12,22,3,3}, Reshape_18835[0]:f32{12}) -> (f32{?,12,6,6})
v0::Relu Relu_646 (Conv_645[0]:f32{?,12,6,6}) -> (f32{?,12,6,6})
v1::ConvolutionIE Conv_647 (Relu_646[0]:f32{?,12,6,6}, refiner.conv4.weight/Output_0/Data__const[0]:f32{4,12,3,3}, Reshape_18838[0]:f32{4}) -> (f32{?,4,4,4})
v1::Reshape Reshape_772 (Conv_647[0]:f32{?,4,4,4}, Constant_12049[0]:i32{1}) -> (f32{?})
v3::ScatterElementsUpdate ScatterElements_773 (Reshape_768[0]:f32{8294400}, Reshape_770[0]:i32{?}, Reshape_772[0]:f32{?}, Constant_12050[0]:i32{}) -> (f32{8294400})

Demo with modnet model does not work

c:\GitHubProjects\libovmatting\Win64\x64\Release>DemoMatting.exe
(model,bin,src,dst,bgr,bgr2):
c:\GitHubProjects\libovmatting\Win64\x64\Release\share\modnet.xml
c:\GitHubProjects\libovmatting\Win64\x64\Release\share\modnet.bin
c:\GitHubProjects\libovmatting\Win64\x64\Release\share\src.mp4
c:\GitHubProjects\libovmatting\Win64\x64\Release\share\dst.mp4
c:\GitHubProjects\libovmatting\Win64\x64\Release\share\src.png
c:\GitHubProjects\libovmatting\Win64\x64\Release\share\replace.jpg
Loading Inference Engine
Device info: CPU
        CPU
        openvino_intel_cpu_plugin version ......... 2.1
        Build ........... 2022.1.0-7019-cdb9bec7210-releases/2022/1

current working path:c:\GitHubProjects\libovmatting\Win64\x64\Release
xml path:c:\GitHubProjects\libovmatting\Win64\x64\Release\share\modnet.xml, bin path:c:\GitHubProjects\libovmatting\Win64\x64\Release\share\modnet.bin
MatterChannel::create():Check 'ov::PartialShape::merge_into(inputs_shape_scheme, this_input_shape)' failed at C:\j\workspace\private-ci\ie\build-windows-vs2019@3\b\repos\openvino\src\core\src\op\concat.cpp:74:
While validating node 'v0::Concat Concat_303 (Relu_300[0]:f32{1,64,45,80}, Resize_302[0]:f32{1,32,48,80}, Resize_230[0]:f32{1,3,45,80}) -> (f32{1,99,128,128})' with friendly_name 'Concat_303':
Argument shapes are inconsistent; they must have the same rank, and must have equal dimension everywhere except on the concatenation axis (axis 1).

Can not create Matter Channel.
This sample is an API example, for any performance measurements please use the dedicated benchmark_app tool

when i converter .onnx to .xml and .bin，instancenorm2d unsupport，have you solved it? please help me.