Using Resynthesizer as a Library

[bootchk]

Using Resynthesizer as a Library

[cabreraalex]

In case someone stumbles on this, this is what I ended up writing to do the inpainting: https://github.com/poloclub/interactive-classification/tree/master/inpaint

[bootchk]

I structured the project to have a library with a documented API, but I have only used the library with apps: GIMP and one other. And never tried to extract the library as a separate project with a simpler development environment (without the GIMP and Python baggage.) So yes, might be more convenient to keep the current project build, just reference (link to) the library from another project. As | recall, the project does build an .a archive, if not it should.

Your solution #1 would carry a lot of baggage ( libgimp, but not Python interpreter). If you are already using libgimp in your app, then that solution is feasible.

What the Python "heal selection" plugin does is nothing sophisticated: just clone an image and rearrange the selection bit masks on each. I assume since you are working on an image app, you are using image tools that would easily do that.

The problem is an API adaptor: to convert the image data structures in your tools (what you call "backing core functions"? which is terminology I don't deeply understand) to the image data structures required by the Resynthesizer library. That should not be hard. And there are many functions in the Resynthesizer project named like "adapt..." which do adaption from GIMP data structures to the internal ones used by the Resynthesizer. (Which as I recall, are optimized for data locality.) (There are two different adaptions?)

Also, the library depends on libg so your project also might, but there is code in the project to eliminate that dependence also (using substitutes for a few libg functions e.g. g_assert)

All the above off the top of my head, and might be misleading. If you provide more details, I might explore further and might be able to answer more questions and even change the project.

[cabreraalex]

Thank you so much! This is exactly what I was looking for. The overhead should not be too big of an issue, the main functionality is a website where a mask and an image are provided and then a call is made to a server which will return the inpainted image. So I don't think running GIMP in the backend should be too much of an issue.

The one thing that would be an improvement is to potentially compile the C code to JavaScript and be able to run the inpainting from resynthesizer in-browser, but I think I will first attempt the GIMP batch-mode as it seems the simplest and then attempt to use the C library and potentially port it.

Thanks again for the information! I will let you know how it goes and if I reach any roadblocks.

[bootchk]

Cool.

Now I am wondering why you chose Resynthesizer over the other choices. Probably I have not explored the choices as deeply as you. I would guess that the Resynthesizer is not screaming fast compared to others but gives better results in general. Just guessing that for example the boat masts in your example image on your web site might not be handled well by the liquid rescale algorithm.

I recall that there are inpainting algorithms in OpenCV (?) . I haven't explored what image libraries a deep learning system for images might be using. Which leaves me wondering whether I should try to put the Resynthesizer algorithm in OpenCV (or some other.)

[cabreraalex]

The OpenCV Algorithms are not even close to PatchMatch/Resynthesizer. Telea is the primary one, and it is not meant for taking out large sections of an image, just removing small lines and issues (Like their examples in their paper: https://pdfs.semanticscholar.org/622d/5f432e515da69f8f220fb92b17c8426d0427.pdf). Anything more complicated just blends the colors in.

It would be amazing to have Resynthesizer in OpenCV, it is definitely on a whole different level of quality than the current algorithms, and is comparably fast: for small images it usually only takes a few seconds. So thank you again for making a fantastic open system :).

[JohnVV]

as MUCH as i like Resynthesizer ( been using it for many many years ) and recommending it to others

it was and basically still is a PHD paper bit of code
( that is where i cam across it first many years back )
http://www.logarithmic.net/pfh/thesis

in Gimp i need to run it many times on smaller parts of a big area ,and even then use the gimp heal tool on the bad areas

for line removal i use gmic and not the openCV install on my system , it works better

imho for Resynthesizer to be added into OpenCV and / or even to NASA's Vision workbench tool set , it still needs work

for other inpainting tools have a look here
http://www.ipol.im/#index.topic-inpainting

[bootchk]

I appreciate your thoughts. My conclusion is that no one algorithm does all things well. Until someone figures that out, I suggest several in-painting algorithms might be in OpenCV.

I've read the thesis. I know the code inside and out, and the original code. It's all high quality. If not for changes to GIMP, the original code would still work fine. And the same core algorithm does other amazing stuff. But that's true of the other algorithms like liquid rescale.

[JohnVV]

bootchk the plugin dose need to be ported from 8 bit per chan to 32 bit float for 2.10 and 3.0

this is /was/ still is / maybe something i will look into

[cabreraalex]

Those are valid points, but i spent hours looking for a good, open source implementation of an inpainting algorithm that works with large semantic regions, and I could not find anything like Resynthesizer despite some of its issues. Deep learning seems to be finally taking the space, but until those models are public and can run on normal non-GPU hardware, this was the best I found.

NVIDIA inpainting: https://www.youtube.com/watch?v=gg0F5JjKmhA

[bootchk]

JohnVV: resynthesizer still works in 2.10 (although there is an installation issue I need to resolve.) I don't know what the plans are for GIMP 3.0, I am hoping that if they break libgimp-2.0 API (which is the API that resynthesizer depends on) that they will implement a compatibility mode so that resynthesizer will still work with minimal changes, albeit with some loss of colors (? I don't know the correct terminology.)

cabreraalex: Very interesting video. I see it too has limitations, e.g. it gave Ernest Borgnine female eyes. Let's suppose they fix that, and have a model that keeps gender constant. How is it to infer what the user's intent is? E.g. to inpaint to eliminate warts while keeping constant age and gender, or to feminize or to fountain of youth/make elderly? I suppose you would need to train the algorithm for all those use cases, and the user would need to choose. Which is what we have concluded in this thread: the user still needs to choose from among alternatives. I wouldn't say "taking" the space, only "entering" the space.

I have also played around with other algorithms, mainly bidirectional-similarity (which I think is in Photoshop) and a saliency algorithm.

My experiment with the bidirectional similarity algorithm came to naught, I couldn't get it to work. It illustrates the difficulty of reducing an academic description of an algorithm to practice. Also, they have software patented that algorithm.

My experiment with a saliency algorithm was also enlightening. The author open sourced an implementation. In my examination and testing, it didn't accurately implement what the paper described. Never underestimate the difficulty of software.