cheind/inpaint

:pencil2: Inpaint is a C++ library providing image inpainting algorithms

Hey!

I'm Chris. My research connects perception, robotics and machine learning.
To learn more, visit https://cheind.github.io/ and browse my repositories 👇

Latest Updates
_{2024/03/13      Just published a code demo on image stitching via analytic homographies https://github.com/cheind/image-stitch
2023/08/01      My findings on the 'Anoto micro-dot codec' got published alongside with code https://github.com/cheind/py-microdots
2022/12/01      Added pure PyTorch implementation of Instant NGP for accelerated learning of NeRFs https://github.com/cheind/pure-torch-ngp}

inpaint's Issues

Help in running

Hi, sorry to sound like a real noob.
But, any pointers on how to run this program would be extremely helpful!

Thanks

Problem on using the methods

I'm try to use your methods in another project, but feel hard to read the code. Do u have a entry that input an image and we can get the inpainted image. This can help me a lot!

What is the mean shift algorithm used for

It seem Mean Shift is not used in the algorithm

Build errors

I'm trying to build inpaint in Xcode 8 with C++11 and OpenCV 2.3.1.
Added the files from /src to the project and compiled. Getting many errors that seem related to OpenCV compatibility:

Linker Issue while building the source in Windows

Hi,
I am trying to build the source code with the below environment set-up:

1. Open CV version - opencv-2.4.11.exe
2. Visual studio 2013
3. The mode set to build the code is Debug x64.
4. Also, linked all open CV libraries [.dll files] present in the path "opencv\build\x64\vc12\bin".
5. I have also added "opencv\build\x64\vc12\lib" under the section "Linker->General->Additional Library Directories"
6. I have also added "opencv\build\include", "inc\inpaint", "tests" under the section "C/C++->General"
7. I have added all the files from "src" & "catch.hpp", "random_testdata.h" from "tests" to the visual studio project.

When i tried building the code, i obtained the following linker errors.

1. error LNK1120: 4 unresolved externals 20171219\windows\inpainting\x64\Debug\inpainting.exe inpainting
2. error LNK2019: unresolved external symbol "public: int __cdecl cv::_InputArray::cols(void)const " (?cols@_InputArray@cv@@QEBAHXZ) referenced in function "void __cdecl Inpaint::meanShift(class cv::_InputArray const &,class cv::_InputArray const &,class cv::_InputArray const &,class cv::_OutputArray const &,class cv::_OutputArray const &,class cv::_OutputArray const &,float,int,bool,bool,bool)" (?meanShift@Inpaint@@YAXAEBV_InputArray@cv@@00AEBV_OutputArray@3@11MH_N22@Z) 20171219\windows\inpainting\mean_shift.obj inpainting
3. error LNK2019: unresolved external symbol "public: int __cdecl cv::_InputArray::rows(void)const " (?rows@_InputArray@cv@@QEBAHXZ) referenced in function "void __cdecl Inpaint::meanShift(class cv::_InputArray const &,class cv::_InputArray const &,class cv::_InputArray const &,class cv::_OutputArray const &,class cv::_OutputArray const &,class cv::_OutputArray const &,float,int,bool,bool,bool)" (?meanShift@Inpaint@@YAXAEBV_InputArray@cv@@00AEBV_OutputArray@3@11MH_N22@Z) 20171219\windows\inpainting\mean_shift.obj inpainting
4. error LNK2019: unresolved external symbol "public: void * __cdecl cv::_InputArray::getObj(void)const " (?getObj@_InputArray@cv@@QEBAPEAXXZ) referenced in function "void __cdecl Inpaint::imagePyramid(class cv::InputArray const &,class cv::OutputArray const &,class cv::Size,int)" (?imagePyramid@Inpaint@@YAXAEBV_InputArray@cv@@AEBV_OutputArray@3@V?$Size@h@3@H@Z) 20171219\windows\inpainting\pyramid.obj inpainting
5. error LNK2019: unresolved external symbol main referenced in function __tmainCRTStartup 20171219\windows\inpainting\MSVCRTD.lib(crtexe.obj) inpainting

Could you please help me resolve this error?
We need additional information.
Could you please let us know if this code can take care about object overlapping scenario?

Is there a python version of this project

Is any python version of this project available ?

Thanks
Akash

Maximum value for patchSize? (Exception: -215:Assertion failed in function 'copyTo')

Thank you for the amazing library. It works brilliantly.
If I use a value greater than 30 for patchSize, then the following exception is thrown:

(cv::Exception &) e = 0x0000000283732080 (msg = "OpenCV(4.7.0-dev) /source/opencv/modules/core/src/copy.cpp:442: error: (-215:Assertion failed) size() == mask.size() in function 'copyTo'\n", code = -215, err = "size() == mask.size()", func = "copyTo", file = "/source/opencv/modules/core/src/copy.cpp", line = 442)

It happens in the following function:

    void CriminisiInpainter::propagatePatch(cv::Point target, cv::Point source)
    {
        cv::Mat_<uchar> copyMask = centeredPatch<PATCHFLAGS>(_targetRegion, target.y, target.x, _halfPatchSize);

	    centeredPatch<PATCHFLAGS>(_image, source.y, source.x, _halfPatchSize).copyTo(
		    centeredPatch<PATCHFLAGS>(_image, target.y, target.x, _halfPatchSize), 
		    copyMask);

	    centeredPatch<PATCHFLAGS>(_isophoteX, source.y, source.x, _halfPatchSize).copyTo(
		    centeredPatch<PATCHFLAGS>(_isophoteX, target.y, target.x, _halfPatchSize), 
		    copyMask);		

	    centeredPatch<PATCHFLAGS>(_isophoteY, source.y, source.x, _halfPatchSize).copyTo(
		    centeredPatch<PATCHFLAGS>(_isophoteY, target.y, target.x, _halfPatchSize), 
		    copyMask);

	    float cPatch = _confidence.at<float>(target);
	    centeredPatch<PATCHFLAGS>(_confidence, target.y, target.x, _halfPatchSize).setTo(cPatch, copyMask);
	
	    copyMask.setTo(0);
    }

Is there a maximum value for the patchSize? What could be the reason for the incorrect calculation of the values?

How I can run this app?

Hi @cheind
I downloaded your code to my computer(Windows 10, Visual Studio 2015, OpenCV 4.1), and compiled successfully.
When i try to run inpaint_image_criminisi.exe, I couldn't see the final image.

After building you might want to try out inpaint_image_criminisi

Launch inpaint_image_criminisi <image>.
Use the sliders on top of the UI to adjust patch and stencil size. Patch size refers to the size of texels that will be used during inpainting. It should be roughly the size of minimum texture feature size you intend to inpaint. Stencil size refers to radius of the interactive drawing tools explained in the following 2 steps.
While holding your left mouse button pressed, move your mouse to mark the target area to be inpainted.
While holding your right mouse button pressed, move your mouse to mark the source area from which samples can be copied to target area. If you skip this step, the entire image without source region is used as source area.
Press e to toggle between editing / inpaint mode.
Press x to quit.

My steps:
1). Open command prompt and move to the folder which contain inpaint_image_criminisi.exe
2). Run inpaint_image_criminisi.exe test1.jpg
3). It showing an window. I select some region with mouse left click. Also select some region with mouse right click.
4). Press e key. But there is no any result or change.

What I am wrong?
Thank you

Use mask image

Is there any way to run this providing a mask image instead of using a GUI to create the mask?

How to build it for android

i want to build it for android

time consuming even though the pathsize is set to maximum?

hello, this code has a good performance. But I want to process video, so it is very slow even though I set set the pathsize to maximum.
Anyone has any advises? @cheind

Build fails on Ubuntu 14.04 (g++ 4.8.4 opencv 3.0)

I haven't got this to build yet, but have encountered the following warnings at least

Build failure fixed with the following

diff --git a/src/criminisi_inpainter.cpp b/src/criminisi_inpainter.cpp
index c23e912..55415fe 100644
--- a/src/criminisi_inpainter.cpp
+++ b/src/criminisi_inpainter.cpp
@@ -17,6 +17,8 @@
    along with Inpaint.  If not, see <http://www.gnu.org/licenses/>.
 */
 
+#include <limits>
+
 #include <inpaint/criminisi_inpainter.h>
 #include <inpaint/patch.h>
 #include <inpaint/timer.h>
@@ -315,4 +317,4 @@ namespace Inpaint {
 
         ci.image().copyTo(image.getMat());
     }
-}
\ No newline at end of file
+}

Build warning - maybe specify std c++11 in CMake?

/tmp/inpaint/src/template_match_candidates.cpp: In member function ‘void Inpaint::TemplateMatchCandidates::removeInvalidBlocks(const cv::Mat&, std::vector<cv::Rect_<int> >&)’:
/tmp/inpaint/src/template_match_candidates.cpp:220:13: warning: lambda expressions only available with -std=c++11 or -std=gnu++11 [enabled by default]
             }), rects.end())

And ultimately build fails with

/tmp/inpaint/src/template_match_candidates.cpp:220:14: error: no matching function for call to ‘remove_if(std::vector<cv::Rect_<int> >::iterator, std::vector<cv::Rect_<int> >::iterator, Inpaint::TemplateMatchCandidates::removeInvalidBlocks(const cv::Mat&, std::vector<cv::Rect_<int> >&)::__lambda0)’
             }), rects.end());
              ^
/tmp/inpaint/src/template_match_candidates.cpp:220:14: note: candidate is:
In file included from /usr/include/c++/4.8/algorithm:62:0,
                 from /tmp/inpaint/src/template_match_candidates.cpp:20:
/usr/include/c++/4.8/bits/stl_algo.h:1134:5: note: template<class _FIter, class _Predicate> _FIter std::remove_if(_FIter, _FIter, _Predicate)
     remove_if(_ForwardIterator __first, _ForwardIterator __last,
     ^
/usr/include/c++/4.8/bits/stl_algo.h:1134:5: note:   template argument deduction/substitution failed:
/tmp/inpaint/src/template_match_candidates.cpp: In substitution of ‘template<class _FIter, class _Predicate> _FIter std::remove_if(_FIter, _FIter, _Predicate) [with _FIter = __gnu_cxx::__normal_iterator<cv::Rect_<int>*, std::vector<cv::Rect_<int> > >; _Predicate = Inpaint::TemplateMatchCandidates::removeInvalidBlocks(const cv::Mat&, std::vector<cv::Rect_<int> >&)::__lambda0]’:
/tmp/inpaint/src/template_match_candidates.cpp:220:14:   required from here
/tmp/inpaint/src/template_match_candidates.cpp:220:14: error: template argument for ‘template<class _FIter, class _Predicate> _FIter std::remove_if(_FIter, _FIter, _Predicate)’ uses local type ‘Inpaint::TemplateMatchCandidates::removeInvalidBlocks(const cv::Mat&, std::vector<cv::Rect_<int> >&)::__lambda0’
             }), rects.end());
              ^
/tmp/inpaint/src/template_match_candidates.cpp:220:14: error:   trying to instantiate ‘template<class _FIter, class _Predicate> _FIter std::remove_if(_FIter, _FIter, _Predicate)’
/tmp/inpaint/src/template_match_candidates.cpp: In function ‘void Inpaint::findTemplateMatchCandidates(cv::InputArray, cv::InputArray, cv::InputArray, cv::OutputArray, cv::Size, int, float)’:
/tmp/inpaint/src/template_match_candidates.cpp:245:117: error: no matching function for call to ‘Inpaint::TemplateMatchCandidates::findCandidates(cv::Mat, cv::Mat, cv::Mat, int&, float&)’
         tmc.findCandidates(templ.getMat(), templMask.getMat(), candidates.getMat(), maxWeakErrors, maxMeanDifference);
                                                                                                                     ^
/tmp/inpaint/src/template_match_candidates.cpp:245:117: note: candidate is:
/tmp/inpaint/src/template_match_candidates.cpp:66:10: note: void Inpaint::TemplateMatchCandidates::findCandidates(const cv::Mat&, const cv::Mat&, cv::Mat&, int, float)
     void TemplateMatchCandidates::findCandidates(
          ^
/tmp/inpaint/src/template_match_candidates.cpp:66:10: note:   no known conversion for argument 3 from ‘cv::Mat’ to ‘cv::Mat&’
make[2]: *** [CMakeFiles/inpaint.dir/src/template_match_candidates.cpp.o] Error 1
make[1]: *** [CMakeFiles/inpaint.dir/all] Error 2
make: *** [all] Error 2

process works on 1 cpu

Hello congratz its a good project
but its works only 1 cpu at the time

About inpaint in 3D space

Hello, I had use your 2D inpainting and I'm think about 3D inpainting.

Can I use your 2D inpaint to 3D ? Do you think it is possible? Can you give me some advice?

Thank you so much!

There is no difference between the original image and the edited image

I try to use your library created in code
` Mat &ori_image = *((Mat *) origin_address);
Mat &target_mat = *((Mat *) target_address);
Mat &source_mat = *((Mat *) source_address);
Mat &result_mat = *((Mat *) result_address);

cvtColor(ori_image, ori_image, COLOR_RGBA2RGB);
cvtColor(result_mat, result_mat, COLOR_RGBA2RGB);
cvtColor(target_mat, target_mat, COLOR_RGBA2GRAY);
cvtColor(source_mat, source_mat, COLOR_RGBA2GRAY);

Inpaint::CriminisiInpainter inpainter;

inpainter.setSourceImage(ori_image);
//
inpainter.setTargetMask(target_mat);
inpainter.setSourceMask(source_mat);
inpainter.setPatchSize(20.0f);
inpainter.initialize();


inpainter.image().copyTo(result_mat);
LOGD("write success");`

but the image of the result and the original is the same, the targetMat is a white image on a black background, and so is the originMat but in a different location, I wonder if there is any need to edit or add anything to your code I don't let it work

Can't build

Please check out

attached file

potential bug

I think there might be a bug in updatefillfront. The confidence should be an average of confidence of the last step. However, some new confidence values are updated and used when computing the neighbor's confidence.

I am not sure if you intentionally make it this way, or it is a bug.

Help: Any Reverse Function of Inpaint?

Inpaint is fantastic! However, I came across a problem where the reverse function of Inpaint is required. Unlike Inpaint which removes an object without trace, Is there any algorithm that you know of could add an object to the original image without visible trace? Although I could add an foreground image directly, but it obviously doesn't blend well with the background image. Much appreciation if you could offer some hint!

crash in TemplateMatchCandidates::findCandidates

I observe crash on this assert:

    CV_Assert(
        templ.type() == CV_MAKETYPE(CV_8U, _integrals.size()) &&
        templ.size() == _templateSize && 
        (templMask.empty() || templMask.size() == _templateSize));

the problem is that templ.size() == _templateSize is FALSE. This is because _templateSize=[19,19] and templ is just [9,9]. Templ is [9,9] since centeredPatch returns half size of patch (because targetPatchLocation=(0,0) so it had to cut patch in half).

CMake on Windows 10

Hello, I'm using Windows 10 with Visual Studio 2015 x64. However when I use CMake to configure, It can't find OpenCV_DIR. I tried to install OpenCV 2.4.13 and Opencv 3.1, and tried to manually select path for OpenCV_DIR too, but it doesn't work...
I'm wandering that can I cmake on Windows 10? or maybe I did it in a wrong way?
Thank you.

Compling error

Ubuntu 16.04,OpenCV3.3
Maybe caused by the version of my OpenCV?
Which version you used?
[ 23%] Built target inpaint [ 35%] Built target patch_match [ 47%] Built target inpaint_benchmarks [ 58%] Built target inpaint_image_criminisi [ 64%] Building CXX object CMakeFiles/inpaint_tests.dir/tests/patch_match.cpp.o /home/quantumliu/cv/inpaint/tests/patch_match.cpp: In function ‘void ____C_A_T_C_H____T_E_S_T____48()’: /home/quantumliu/cv/inpaint/tests/patch_match.cpp:55:43: error: ‘FILLED’ is not a member of ‘cv’ cv::rectangle(mask, r, cv::Scalar(0), cv::FILLED); ^ CMakeFiles/inpaint_tests.dir/build.make:182: recipe for target 'CMakeFiles/inpaint_tests.dir/tests/patch_match.cpp.o' failed make[2]: *** [CMakeFiles/inpaint_tests.dir/tests/patch_match.cpp.o] Error 1 CMakeFiles/Makefile2:215: recipe for target 'CMakeFiles/inpaint_tests.dir/all' failed make[1]: *** [CMakeFiles/inpaint_tests.dir/all] Error 2 Makefile:83: recipe for target 'all' failed make: *** [all] Error 2

problem when running the program