john-wl/sd-webui-inpaint-difference

A1111 extension to find the inpaint mask to use based on the difference between two images.

sd-webui-inpaint-difference

Overview

An extension to find the inpaint mask to use based on the difference between two images. It adds an operation mode in the img2img tab.

Installation

1. Go to Extensions > Available
2. Click the Load from: button
3. Enter "inpaint difference" in the search bar
4. Click the Install button of the "inpaint difference" Tab cell
5. Restart the webui

Usage

Here is the suggested workflow for this extension:

Additional parameters are added under the img2img operation mode to further edit the mask.
You can also look into the extension's settings for the brush color and other options.

sd-webui-inpaint-difference's Issues

Extension does not load in latest A1111 version (dev branch)

I updated my webui and now the extension does not load.

This is the error:

    Traceback (most recent call last):
      File "D:\AI\stable-diffusion-webui\modules\script_callbacks.py", line 248, in after_component_callback
        c.callback(component, **kwargs)
      File "D:\AI\stable-diffusion-webui\extensions\sd-webui-inpaint-difference\lib_inpaint_difference\img2img_tab_extender.py", line 45, in on_after_component
        cls.register_default_amount_of_tabs()
      File "D:\AI\stable-diffusion-webui\extensions\sd-webui-inpaint-difference\lib_inpaint_difference\img2img_tab_extender.py", line 67, in register_default_amount_of_tabs
        for child in cls.img2img_tabs_block.children
    AttributeError: 'NoneType' object has no attribute 'children'

---```

SDNEXT: Error only on server restart: "ValueError: zip() argument 2 is longer than argument 1"

15:10:55-506926 INFO Hacking i2i-batch is already done.
╭────────────────────────── Traceback (most recent call last) ───────────────────────────╮
│ C:\AI\SDNEXT\automatic\launch.py:246 in │
│ │
│ 245 installer.log.info('Server restarting...') │
│ ❱ 246 uv, instance = start_server(immediate=False, server=instance) │
│ 247 else: │
│ │
│ C:\AI\SDNEXT\automatic\launch.py:171 in start_server │
│ │
│ 170 else: │
│ ❱ 171 uvicorn = server.webui(restart=not immediate) │
│ 172 if args.profile: │
│ │
│ C:\AI\SDNEXT\automatic\webui.py:311 in webui │
│ │
│ 310 start_common() │
│ ❱ 311 start_ui() │
│ 312 modules.sd_models.write_metadata() │
│ │
│ C:\AI\SDNEXT\automatic\webui.py:229 in start_ui │
│ │
│ 228 timer.startup.record("before-ui") │
│ ❱ 229 shared.demo = modules.ui.create_ui(timer.startup) │
│ 230 timer.startup.record("ui") │
│ │
│ C:\AI\SDNEXT\automatic\modules\ui.py:994 in create_ui │
│ │
│ 993 │
│ ❱ 994 loadsave = ui_loadsave.UiLoadsave(cmd_opts.ui_config) │
│ 995 components = [] │
│ │
│ C:\AI\SDNEXT\automatic\extensions\sd-webui-inpaint-difference\lib_inpaint_difference\w │
│ ebui_nasty_hijacks.py:49 in hijack__init__ │
│ │
│ 48 def hijack__init__(*args, **kwargs): │
│ ❱ 49 Img2imgTabExtender.create_custom_tabs() │
│ 50 return original_ui_settings__init__(*args, **kwargs) │
│ │
│ C:\AI\SDNEXT\automatic\extensions\sd-webui-inpaint-difference\lib_inpaint_difference\i │
│ mg2img_tab_extender.py:80 in create_custom_tabs │
│ │
│ 79 img2img_tabs = cls._get_img2img_tabs() │
│ ❱ 80 cls.setup_navigation_events(img2img_tabs) │
│ 81 for tab_data in cls.tab_data_list: │
│ │
│ C:\AI\SDNEXT\automatic\extensions\sd-webui-inpaint-difference\lib_inpaint_difference\i │
│ mg2img_tab_extender.py:95 in setup_navigation_events │
│ │
│ 94 block_data_iterator = zip(img2img_tabs[cls.amount_of_default_tabs:], cls │
│ ❱ 95 for tab_block, custom_tab in block_data_iterator: │
│ 96 def update_func(custom_tab): │
╰───────────────────────────────────────────────────────────────────────────╯
ValueError: zip() argument 2 is longer than argument 1
Appuyez sur une touche pour continuer...

[A1111 v1.8.0] Dosen't Work ? ... Oh no it works ! just restart 3 time

Hello John-WL,

For my part, Sd-webui-inpaint-difference is my favorit tool to create quickly a mask, but this morning I have pass on the A1111 v1.8.0 but the Inpaint-difference panel has disappeared. It's normal ? Or I have miss the update ? Just I whant know if I am the only one in the truble ^.^'

Remove 1 line

In gradio_helpers.py

Remove global state

Remove this state:
https://github.com/John-WL/sd-webui-inpaint-difference/blob/b57f45c6adb598b85b7b12e77657cb7135d6d616/lib_inpaint_difference/globals.py

Settings are global by nature so they can stay there.

AttributeError: 'list' object has no attribute 'is_negative_prompt'

19:06:20-656180 ERROR gradio call: AttributeError
[2023-11-21 19:06:20,656] sd: [ERROR] gradio call: AttributeError
[2023-11-21 19:06:20,656] sd: [ERROR] gradio call: AttributeError
[2023-11-21 19:06:20,656] sd: [ERROR] gradio call: AttributeError
[2023-11-21 19:06:20,656] sd: [ERROR] gradio call: AttributeError
[2023-11-21 19:06:20,656] sd: [ERROR] gradio call: AttributeError
[2023-11-21 19:06:20,656] sd: [ERROR] gradio call: AttributeError
[2023-11-21 19:06:20,656] sd: [ERROR] gradio call: AttributeError
╭────────────────────────── Traceback (most recent call last) ──────────────────────────╮
│ C:\AI\SDNEXT\automatic\modules\call_queue.py:34 in f │
│ │
│ 33 │ │ │ try: │
│ ❱ 34 │ │ │ │ res = func(*args, **kwargs) │
│ 35 │ │ │ │ progress.record_results(id_task, res) │
│ │
│ C:\AI\SDNEXT\automatic\extensions\sd-webui-inpaint-difference\lib_inpaint_difference\ │
│ webui_nasty_hijacks.py:35 in hijack_func │
│ │
│ 34 │ │ │
│ ❱ 35 │ │ return original_img2img_processing(id_task, mode, prompt, negative_promp │
│ 36 │ │ │ init_img_with_mask, inpaint_color_sketch, inpaint_color_sketch_orig, │
│ │
│ C:\AI\SDNEXT\automatic\modules\img2img.py:226 in img2img │
│ │
│ 225 │ │ if processed is None: │
│ ❱ 226 │ │ │ processed = processing.process_images(p) │
│ 227 │ p.close() │
│ │
│ C:\AI\SDNEXT\automatic\modules\processing.py:738 in process_images │
│ │
│ 737 │ │ │ with context_hypertile_vae(p), context_hypertile_unet(p): │
│ ❱ 738 │ │ │ │ res = process_images_inner(p) │
│ 739 │ finally: │
│ │
│ C:\AI\SDNEXT\automatic\extensions-builtin\sd-webui-controlnet\scripts\batch_hijack.py │
│ :42 in processing_process_images_hijack │
│ │
│ 41 │ │ │ # we are not in batch mode, fallback to original function │
│ ❱ 42 │ │ │ return getattr(processing, '_controlnet_original_process_images_in │
│ 43 │
│ │
│ C:\AI\SDNEXT\automatic\modules\processing.py:859 in process_images_inner │
│ │
│ 858 │ │ │ if shared.backend == shared.Backend.ORIGINAL: │
│ ❱ 859 │ │ │ │ uc = get_conds_with_caching(modules.prompt_parser.get_learned │
│ 860 │ │ │ │ c = get_conds_with_caching(modules.prompt_parser.get_multicond │
│ │
│ C:\AI\SDNEXT\automatic\modules\processing.py:817 in get_conds_with_caching │
│ │
│ 816 │ │ with devices.autocast(): │
│ ❱ 817 │ │ │ cache[1] = function(shared.sd_model, required_prompts, steps) │
│ 818 │ │ cache[0] = (required_prompts, steps) │
│ │
│ C:\AI\SDNEXT\automatic\venv\lib\site-packages\sdwss\hijacker.py:12 in wrapper │
│ │
│ 11 │ │ │ def wrapper(*args, **kwargs): │
│ ❱ 12 │ │ │ │ return function(*args, **kwargs, original_function=self._origin │
│ 13 │
│ │
│ C:\AI\SDNEXT\automatic\venv\lib\site-packages\sdwss\sd_webui\prompt_parser.py:22 in │
│ get_learned_conditioning │
│ │
│ 21 │ hires_steps, use_old_scheduling, * = args if args else (None, True) │
│ ❱ 22 │ encoder = PromptScheduleEncoder(model, prompts, steps, hires_steps, not pro │
│ 23 │ conds = [encoder.process_prompt(prompt) for prompt in prompts] │
╰───────────────────────────────────────────────────────────────────────────────────────╯
AttributeError: 'list' object has no attribute 'is_negative_prompt'

Dict can be refactored into function named params directly

https://github.com/John-WL/sd-webui-inpaint-difference/blob/8051da3e5ef4cc378d5001944819d583fd05d9d2/lib_inpaint_difference/ui.py#L60

Img2Img is broken with the latest version

Traceback (most recent call last):
      File "D:\AI\stable-diffusion-webui\modules\call_queue.py", line 74, in f
        res = list(func(*args, **kwargs))
                   ^^^^^^^^^^^^^^^^^^^^^
      File "D:\AI\stable-diffusion-webui\modules\call_queue.py", line 53, in f
        res = func(*args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^
      File "D:\AI\stable-diffusion-webui\modules\call_queue.py", line 37, in f
        res = func(*args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^
      File "D:\AI\stable-diffusion-webui\venv\Lib\site-packages\sdwi2iextender\lib\img2img_component_injector.py", line 131, in hijack_img2img
        script_instance.resolve_image_and_mask(*script_args)
      File "D:\AI\stable-diffusion-webui\venv\Lib\site-packages\sdwi2iextender\lib\img2img_component_injector.py", line 93, in resolve_image_and_mask
        self.image = image_components[init_images_index]
                     ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
    IndexError: tuple index out of range

Inpaint difference commit 34498ad
A1111 webui v1.10.1-4-g48239090

Gradio patch should be dimension agnostic

Error message from #4 (comment):

Traceback (most recent call last):
  File "F:\stablediffusion\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 488, in run_predict
    output = await app.get_blocks().process_api(
  File "F:\stablediffusion\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1434, in process_api
    data = self.postprocess_data(fn_index, result["prediction"], state)
  File "F:\stablediffusion\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1335, in postprocess_data
    prediction_value = block.postprocess(prediction_value)
  File "F:\stablediffusion\stable-diffusion-webui\venv\lib\site-packages\gradio\components\image.py", line 316, in postprocess        
    return processing_utils.encode_pil_to_base64(y)
  File "F:\stablediffusion\stable-diffusion-webui-1-8-0-rc\extensions\sd-webui-inpaint-difference\lib_inpaint_difference\gradio_hijack
s.py", line 13, in encode_pil_to_base64_new
    image_arr = np.asarray(pil_image)[:, :, ::-1]
IndexError: too many indices for array: array is 2-dimensional, but 3 were indexed

The issue seems to be apparent when inpaint-diff and the mosaic extension are used at the same time, though I believe the problem is more general than this conflict, and it should be fixed asap.

SDNEXT: Expected 3D (unbatched) or 4D (batched) input to conv2d, but got input of size: [0]

01:28:47-304120 ERROR gradio call: RuntimeError
╭─────────────────────── Traceback (most recent call last) ───────────────────────╮
│ C:\AI\SDNEXT\automatic\modules\call_queue.py:31 in f │
│ │
│ 30 │ │ │ try: │
│ ❱ 31 │ │ │ │ res = func(*args, **kwargs) │
│ 32 │ │ │ │ progress.record_results(id_task, res) │
│ in img2img:264 │
│ │
│ C:\AI\SDNEXT\automatic\modules\processing.py:193 in process_images │
│ │
│ 192 │ │ │ with context_hypertile_vae(p), context_hypertile_unet(p): │
│ ❱ 193 │ │ │ │ processed = process_images_inner(p) │
│ 194 │
│ │
│ C:\AI\SDNEXT\automatic\extensions-builtin\sd-webui-controlnet\scripts\batch_hij │
│ │
│ 41 │ │ │ # we are not in batch mode, fallback to original function │
│ ❱ 42 │ │ │ return getattr(processing, '__controlnet_original_process_ima │
│ 43 │
│ │
│ C:\AI\SDNEXT\automatic\modules\processing.py:264 in process_images_inner │
│ │
│ 263 │ │ │ with devices.autocast(): │
│ ❱ 264 │ │ │ │ p.init(p.all_prompts, p.all_seeds, p.all_subseeds) │
│ 265 │ │ extra_network_data = None │
│ │
│ ... 9 frames hidden ... │
│ │
│ C:\AI\SDNEXT\automatic\repositories\ldm\modules\diffusionmodules\model.py:523 i │
│ │
│ 522 │ │ # downsampling │
│ ❱ 523 │ │ hs = [self.conv_in(x)] │
│ 524 │ │ for i_level in range(self.num_resolutions): │
│ │
│ C:\AI\SDNEXT\automatic\venv\lib\site-packages\torch\nn\modules\module.py:1511 i │
│ │
│ 1510 │ │ else: │
│ ❱ 1511 │ │ │ return self._call_impl(*args, **kwargs) │
│ 1512 │
│ │
│ C:\AI\SDNEXT\automatic\venv\lib\site-packages\torch\nn\modules\module.py:1520 i │
│ │
│ 1519 │ │ │ │ or _global_forward_hooks or _global_forward_pre_hooks): │
│ ❱ 1520 │ │ │ return forward_call(*args, **kwargs) │
│ 1521 │
│ │
│ C:\AI\SDNEXT\automatic\venv\lib\site-packages\torch\nn\modules\conv.py:460 in f │
│ │
│ 459 │ def forward(self, input: Tensor) -> Tensor: │
│ ❱ 460 │ │ return self._conv_forward(input, self.weight, self.bias) │
│ 461 │
│ │
│ C:\AI\SDNEXT\automatic\venv\lib\site-packages\torch\nn\modules\conv.py:456 in _ │
│ │
│ 455 │ │ │ │ │ │ │ _pair(0), self.dilation, self.groups) │
│ ❱ 456 │ │ return F.conv2d(input, weight, bias, self.stride, │
│ 457 │ │ │ │ │ │ self.padding, self.dilation, self.groups) │
╰────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Expected 3D (unbatched) or 4D (batched) input to conv2d, but got input of size: [0]

SDNEXT: AttributeError: 'Options' object has no attribute 'img2img_editor_height'

AttributeError: 'Options' object has no attribute 'img2img_editor_height'

15:55:08-422685 ERROR executing callback:
C:\AI\SDNEXT_MASTER\automatic\extensions\sd-webui-inpaint-difference\lib_inpaint_difference\webui_callbacks.py
after_component_callback: AttributeError
╭─────────────────────────────────────────────────── Traceback (most recent call last) ────────────────────────────────────────────────────╮
│ C:\AI\SDNEXT_MASTER\automatic\modules\script_callbacks.py:285 in after_component_callback │
│ │
│ 284 │ │ │ t0 = time.time() │
│ ❱ 285 │ │ │ c.callback(component, **kwargs) │
│ 286 │ │ │ timer(t0, c.script, 'after_component') │
│ │
│ C:\AI\SDNEXT_MASTER\automatic\extensions\sd-webui-inpaint-difference\lib_inpaint_difference\webui_callbacks.py:35 in on_after_component │
│ │
│ 34 │ │ │ DifferenceGlobals.ui_params = ui_params # for hiding/showing when clicking │
│ ❱ 35 │ │ create_inpaint_difference_ui( │
│ 36 │ │ │ BlockManager(tab), │
│ │
│ C:\AI\SDNEXT_MASTER\automatic\extensions\sd-webui-inpaint-difference\lib_inpaint_difference\ui.py:17 in create_inpaint_difference_ui │
│ │
│ 16 │ │ │
│ ❱ 17 │ │ DifferenceGlobals.inpaint_mask_component = gr.Image(label="Difference mask", int │
│ 18 │
│ │
│ C:\AI\SDNEXT_MASTER\automatic\modules\shared.py:621 in getattr │
│ │
│ 620 │ │ │ return self.data_labels[item].default │
│ ❱ 621 │ │ return super(Options, self).getattribute(item) # pylint: disable=super-with- │
│ 622 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
AttributeError: 'Options' object has no attribute 'img2img_editor_height'

Improve mask generation speed

So I found that for large images, gradio takes a significant amount of time JUST to convert images to a format readable by the browser (base64).
The maintainers of gradio won't implement any improvement to this before gradio v4, but they mention a patch here:
gradio-app/gradio#2635 (comment)

I did some tests.
I am getting ~2x faster mask generation for large images (7.2 seconds went down to 3.4 seconds for 2K x 2K images).

I will add this hijack by default in the extension, but I'll add it as a setting thing instead if it creates conflicts with other extensions somehow.
I don't think conflicts are very likely though, so for now the patch is imposed.

Using scale_to slider throws error

The reason is that there doesn't seem to be a reliable hijackable function to modify the behavior of the img2img.img2img function.

The solution I have found is to make a nasty patch in-place by modifying the file and re-compiling it at run time. This should enable the injection of all the components we need.

I think it should be implemented in sdwi2iextender lib, though.

I'll remove the patch when a1111 Webui 1.9 is out.

[bug] img2img is broken for dev branch

This commit broke img2img tab if this extension is enabled
AUTOMATIC1111/stable-diffusion-webui@bf2f7b3#diff-759dea4948b054f717bd2230f7ed804fa24b5ca64d46b3e79aad8ff3f6346f99L149

Error:

modules/img2img.py", line 217, in img2img
p.user = request.username
                ^^^^^^^^^^^^^^^^
AttributeError: 'int' object has no attribute 'username

UI operation mode will conflict with other extensions' operation modes

Hijacking the UI to add an operation mode is not very clean, and it will conflict with other extensions that try to do the same.
Adding operation modes in the UI is not very common among extensions in general though, so I don't think this is a big deal for now.

Also, the next release of the Webui will potentially add a new load order feature, and this may help to solve the problem.

The plan is to leverage the load order mechanism if possible on release.
If it's still not enough, I'll have to take things into my own hands by either making a PR to the Webui, or publishing a small pip package to hijack uniformly the UI across extensions.

Unique elem_id is a problem

Hello, I've made an extension, which changes "Masked content" option in img2img tab. And I've noticed it doesn't apply on your extension tab, because your "Masked content" has different elem_id. inpaint_difference_inpainting_fill instend of img2img_inpainting_fill The same with other options which you share with regular inpaint

It's not problem for my extension, I can add your elem_id in my list. But it's not logical for my opinion, because it is the same img2img tab, and these options are used in the same img2img pipline. Maybe someone else want to change something in inpaint, and it wont apply to your tab

P.s. I want to say, your extension is amazing :) I use it in pair with "upscaler for img2img" from extra options, to simulate hires fix to avoid blurred inpainted images