thereforegames/txt2mask

Automatically create masks for Stable Diffusion inpainting using natural language.

txt2mask's Issues

NameError: name 'plug' is not defined

After the most recent update for SD2.0, I get this error while loading scripts.

Non CUDA error?

I am using Mac M1 machine, probably non CUDA, and getting error when using txt2mask script:

RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.
Time taken: 1m 18.08s

Issues when trying to run the replacer

img2img: A robot's arm
Error completing request
Arguments: ("A robot's arm", '', 'None', 'None', <PIL.Image.Image image mode=RGB size=944x958 at 0x1FDA7385690>, {'image': <PIL.Image.Image image mode=RGBA size=944x958 at 0x1FDA7387310>, 'mask': <PIL.PngImagePlugin.PngImageFile image mode=RGBA size=944x958 at 0x1FDA7384250>}, None, 0, 20, 0, 4, 0, False, False, 1, 1, 1, 7, 0.75, -1.0, -1.0, 0, 0, 0, 512, 512, 0, 0, 64, False, 0, 8, '', '', '', '', 1, 50, 0, 4, 1, '

Recommended settings: Sampling Steps: 80-100, Sampler: Euler a, Denoising strength: 0.8

', 128, 4, ['left', 'right', 'up', 'down'], 1, 0.05, 128, 4, 0, ['left', 'right', 'up', 'down'], False, None, '', False, 'robot', 96, '

If you like my work, please consider showing your support on Patreon. Thank you! ❤

', 1, '', 4, '', True) {}
Traceback (most recent call last):
File "C:\Users\USER\Documents\StableDiffusion\modules\ui.py", line 134, in f
res = list(func(*args, **kwargs))
File "C:\Users\USER\Documents\StableDiffusion\webui.py", line 55, in f
res = func(*args, **kwargs)
File "C:\Users\USER\Documents\StableDiffusion\modules\img2img.py", line 131, in img2img
processed = modules.scripts.scripts_img2img.run(p, *args)
File "C:\Users\USER\Documents\StableDiffusion\modules\scripts.py", line 159, in run
processed = script.run(p, *script_args)
File "C:\Users\USER\Documents\StableDiffusion\scripts\txt2mask.py", line 102, in run
p.image_mask = get_mask()
File "C:\Users\USER\Documents\StableDiffusion\scripts\txt2mask.py", line 62, in get_mask
model.load_state_dict(torch.load(d64_file, map_location=torch.device('cuda')), strict=False);
File "C:\Users\USER\Documents\StableDiffusion\venv\lib\site-packages\torch\serialization.py", line 713, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "C:\Users\USER\Documents\StableDiffusion\venv\lib\site-packages\torch\serialization.py", line 920, in _legacy_load
magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, 'v'.

Any fixes known yet or am I missing something?

[Feature Request] Output the mask image along with the generated image in the UI

Output the mask image along with the generated image on the Output UI, that way you can preview the mask, and use the generated mask with another script by simply dragging the outputted mask to the Input Mask area

Weights won't download as repo is over data quota.

Searching for a fix for the "_pickle.UnpicklingError: invalid load key, 'v'." i wiped the 1KiB files only to have them replace with the same files. Doing a manual git pull of the clipseg repo resulted in the following error:
Error downloading object: weights/rd16-uni.pth (61545cd): Smudge error: Error downloading weights/rd16-uni.pth (61545cdb3a28f99d33d457c64a9721ade835a9dfbda604c459de6831c504167a): batch response: This repository is over its data quota. Account responsible for LFS bandwidth should purchase more data packs to restore access.

Is there another host for these weights?

UnpicklingError: invalid load key, 'v'.

When I try to use this script, this is all i get, and the prompt fails.

Not showing up in img2img scripts dropdown

I tried downloading a zip directly and cloning it directly to my notebook on Paperspace, but the script doesn't show up in the Scripts dropdown in either txt2img or img2img

The script isn't showing

I installed as you said, clone or download this repo and place the files in the base directory of Automatic's web UI and restarted the server, but the script is not showing the script's dropdown.

Not really useable, does not mask anything properly

I tested this quite a bit and while I appreciate the work and the idea behind it, this is not useful for real world usage.
I've tested it on a variety of images, tried to mask hands, faces, flags, umbrellas, sheets on a table, arms or just people.
It did not work one single time regardless of "precision" settings.

What's the error returned if it can't find/select anything from term?

I tried to get it to select "leaves, leaf" even tried "leafs" is there a way/place to visualize it's selection criteria before attempting to render?

``img2img: animal skulls, realistic
100%|███████████████████████████████████████| 335M/335M [00:07<00:00, 44.6MiB/s]
Error completing request
Arguments: ('animal skulls, realistic', '', 'None', 'None', <PIL.Image.Image image mode=RGB size=1024x1024 at 0x25CE318ADD0>, None, None, 0, 20, 0, 4, 0, False, False, 0, 1, 1, 7, 0.75, -1.0, -1.0, 0, 0, 0, 1024, 1024, 0, 0, 64, False, 0, 7, '', '', '', '', 1, 50, 0, 4, 1, 128, 4, 0, ['left', 'right', 'up', 'down'], None, False, 'leaves, leafs', 96, '

If you like my work, please consider showing your support on Patreon. Thank you! ❤

', 1, '', 4, '', True) {}
Traceback (most recent call last):
File "E:\stable-diffusion-webui\modules\ui.py", line 125, in f
res = list(func(*args, **kwargs))
File "E:\stable-diffusion-webui\webui.py", line 71, in f
res = func(*args, **kwargs)
File "E:\stable-diffusion-webui\modules\img2img.py", line 131, in img2img
processed = modules.scripts.scripts_img2img.run(p, *args)
File "E:\stable-diffusion-webui\modules\scripts.py", line 140, in run
processed = script.run(p, *script_args)
File "E:\stable-diffusion-webui\scripts\txt2mask.py", line 86, in run
p.image_mask = get_mask()
File "E:\stable-diffusion-webui\scripts\txt2mask.py", line 46, in get_mask
model.load_state_dict(torch.load('./repositories/clipseg/weights/rd64-uni.pth', map_location=torch.device('cuda')), strict=False);
File "E:\stable-diffusion-webui\venv\lib\site-packages\torch\serialization.py", line 713, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "E:\stable-diffusion-webui\venv\lib\site-packages\torch\serialization.py", line 920, in _legacy_load
magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, 'v'.

img2img: animal skulls, realistic
Error completing request
Arguments: ('animal skulls, realistic', '', 'None', 'None', <PIL.Image.Image image mode=RGB size=1024x1024 at 0x25CE31F3B80>, None, None, 0, 20, 0, 4, 0, False, False, 0, 1, 1, 7, 0.75, -1.0, -1.0, 0, 0, 0, 1024, 1024, 0, 0, 64, False, 0, 7, '', '', '', '', 1, 50, 0, 4, 1, 128, 4, 0, ['left', 'right', 'up', 'down'], None, False, 'leaf', 96, '<div class="gr-block gr-box relative w-full overflow-hidden border-solid border border-gray-200 gr-panel"><p>If you like my work, please consider showing your support on <strong><a href="https://patreon.com/thereforegames" target="_blank">Patreon</a></strong>. Thank you! &#10084;</p></div>', 1, '', 4, '', True) {}
Traceback (most recent call last):
  File "E:\stable-diffusion-webui\modules\ui.py", line 125, in f
    res = list(func(*args, **kwargs))
  File "E:\stable-diffusion-webui\webui.py", line 71, in f
    res = func(*args, **kwargs)
  File "E:\stable-diffusion-webui\modules\img2img.py", line 131, in img2img
    processed = modules.scripts.scripts_img2img.run(p, *args)
  File "E:\stable-diffusion-webui\modules\scripts.py", line 140, in run
    processed = script.run(p, *script_args)
  File "E:\stable-diffusion-webui\scripts\txt2mask.py", line 86, in run
    p.image_mask = get_mask()
  File "E:\stable-diffusion-webui\scripts\txt2mask.py", line 46, in get_mask
    model.load_state_dict(torch.load('./repositories/clipseg/weights/rd64-uni.pth', map_location=torch.device('cuda')), strict=False);
  File "E:\stable-diffusion-webui\venv\lib\site-packages\torch\serialization.py", line 713, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "E:\stable-diffusion-webui\venv\lib\site-packages\torch\serialization.py", line 920, in _legacy_load
    magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, 'v'.`

Inpainitng with full resolution option on have some errors

Images tell all.but In case yoyu need to full error as written.

img2img: Emma Watson face close up smiling
Error completing request
Arguments: ('Emma Watson face close up smiling', '', 'None', 'None', None, {'image': <PIL.Image.Image image mode=RGBA size=671x1000 at 0x19A9D5816F0>, 'mask': <PIL.PngImagePlugin.PngImageFile image mode=RGBA size=671x1000 at 0x19A9D582A40>}, None, 0, 72, 0, 13, 1, False, False, 1, 1, 1, 7.5, 0.4, -1.0, -1.0, 0, 0, 0, 512, 512, 0, 0, 64, True, 0, 8, '', '', '', '', 1, 50, 0, 4, 1, '

Recommended settings: Sampling Steps: 80-100, Sampler: Euler a, Denoising strength: 0.8

', 128, 4, ['left', 'right', 'up', 'down'], 1, 0.05, 128, 4, 0, ['left', 'right', 'up', 'down'], False, None, '', False, 'face', 96, '

If you like my work, please consider showing your support on Patreon. Thank you! ❤

', 1, '', 4, '', True) {}
Traceback (most recent call last):
File "C:\stable-diffusion-webui\modules\ui.py", line 134, in f
res = list(func(*args, **kwargs))
File "C:\stable-diffusion-webui\webui.py", line 55, in f
res = func(*args, **kwargs)
File "C:\stable-diffusion-webui\modules\img2img.py", line 131, in img2img
processed = modules.scripts.scripts_img2img.run(p, *args)
File "C:\stable-diffusion-webui\modules\scripts.py", line 159, in run
processed = script.run(p, *script_args)
File "C:\stable-diffusion-webui\scripts\txt2mask.py", line 104, in run
processed = processing.process_images(p)
File "C:\stable-diffusion-webui\modules\processing.py", line 263, in process_images
p.init(seed=all_seeds[0])
File "C:\stable-diffusion-webui\modules\processing.py", line 440, in init
image_masked.paste(image.convert("RGBA").convert("RGBa"), mask=ImageOps.invert(self.mask_for_overlay.convert('L')))
File "C:\stable-diffusion-webui\venv\lib\site-packages\PIL\Image.py", line 1628, in paste
self.im.paste(im, box, mask.im)
ValueError: images do not match

error at first step of img2mask

Hi, I get the following error :
0it [00:00, ?it/s]
Error completing request
Arguments: (1, 'photo of a man with beard and big eyes looking straight at camera, space stars and galaxies background', '', 'None', 'None', <PIL.Image.Image image mode=RGB size=1920x1786 at 0x22136802710>, {'image': <PIL.Image.Image image mode=RGBA size=512x512 at 0x22136802B00>, 'mask': <PIL.PngImagePlugin.PngImageFile image mode=RGBA size=512x512 at 0x22136801C00>}, None, None, 0, 100, 1, 4, 0, False, False, 1, 1, 12, 0, -1.0, -1.0, 0, 0, 0, False, 512, 512, 1, False, 32, 0, '', '', 9, 0, 0.1, 10, 7, 19.9, 0.1, 0.001, 'photo of a man with beard and eyes looking straight at camera, white wall background', '', -1, 50, 0, True, 4, 1, '

Recommended settings: Sampling Steps: 80-100, Sampler: Euler a, Denoising strength: 0.8

', 128, 8, ['left', 'right', 'up', 'down'], 1, 0.05, 128, 4, 0, ['left', 'right', 'up', 'down'], False, False, None, '', '

Will upscale the image to twice the dimensions; use width and height sliders to set tile size

', 64, 0, 'background', '', 100, 0, 1, True, '

If you like my work, please consider showing your support on Patreon. Thank you! ❤

', 1, '', 0, '', True, False) {}
Traceback (most recent call last):
File "C:\AI\SD\AUTOMATIC1111\stable-diffusion-webui\modules\ui.py", line 184, in f
res = list(func(*args, **kwargs))
File "C:\AI\SD\AUTOMATIC1111\stable-diffusion-webui\webui.py", line 64, in f
res = func(*args, **kwargs)
File "C:\AI\SD\AUTOMATIC1111\stable-diffusion-webui\modules\img2img.py", line 124, in img2img
processed = modules.scripts.scripts_img2img.run(p, *args)
File "C:\AI\SD\AUTOMATIC1111\stable-diffusion-webui\modules\scripts.py", line 159, in run
processed = script.run(p, *script_args)
File "C:\AI\SD\AUTOMATIC1111\stable-diffusion-webui\scripts\txt2mask.py", line 175, in run
p.image_mask = get_mask().resize((p.init_images[0].width,p.init_images[0].height))
File "C:\AI\SD\AUTOMATIC1111\stable-diffusion-webui\scripts\txt2mask.py", line 111, in get_mask
model = CLIPDensePredT(version='ViT-B/16', reduce_dim=64, complex_trans_conv=True)
TypeError: CLIPDensePredT.init() got an unexpected keyword argument 'complex_trans_conv'

thank you in advance.

I have a request but...It can be challenging!

I don't know if this is possible or not but for example I want to change the cloth of multiple people across multiple images,or I want to change all the faces to certain someone's face with txt2image masking.Like deepfakes so to say. I may have sequence of images or random images doesn't matter.

So what I have in mind is as in the picture,I will mask out the faces,I will write a prompt and do a batch inpainting by utilizing the automatic masking of txt2mask.

I am not sure if you can do it alone or maybe automatic himself needs to be involved,but this is a great idea imo and maybe a new harder task or challenge for you :)

Please tell me it can be done.

In short what I am asking is a batch inpainting feature powered by clip masking with txt2mask.

Can't replace object masked

I'm testing it, and I can't seem to get SD to replace the object masked. Perhaps that is just one of the current limitations of img2img. For example, trying to replace the astronaut riding a horse with riding an elephant, and I get strange output like this:

The mask prompt is "a horse" and the prompt is "an elephant." I can see the script is masking the horse well, but then the elephant is drawn all strange in the masked area. How do you get the astronaut riding an elephant? Again, this might not be a problem with txt2mask specifically, but with the img2img method. Should the top prompt contain a more general description of the entire scene, or only what is going in the masked area?

Feature Request Negative Masking

Hi there,thank you for fixing the full resolution problem.It works perfect.

I have changed the default delimeter from "|" to "," cos it makes more sense and easy to write.I would like to see it as a default.

Anyway,as title says it all how about negative masking like negative prompt.
For example I tried to mask chest and shoulders it worked but the blending of the neck area didn't go as good as I expected cos it was a very stylized picture.( I also tried many styles with prompt to blend better)
So I have added chest,shoulders,neck this time it included the whole face area. I used mask blur,mask precision and mask padding but yet I failed.

This gave me the idea to exclude with negative masking.As it is easy to mask out faces why not discard them from the total combined masked area? If this can be done we can achieve far more greater results with much precise control.

I am hoping that there is a way and I think it is possible.Because your delimeter adds masks together,it can subract from it.Ofc it requires another input field as negative prompt and better nagative prompts also shall support combining with a delimeter.

In addition to these a checkbox for inverting the mask can be a cool feature I believe.

txt2img->txt2mask (to fix small faces)

Since SD faces get worse the smaller they get, it would be useful if some of the algorithms used for txt2mask could be used to automate the following process in one

1. txt2img eg photo of jmp909 person wearing a red hat etc
2. send the image to img2img
3. apply txt2mask on eg head or face
4. inpaint photo of jmp909 person at full resolution for just the masked part eg head or face (without the extra part of the prompt so would need a separate input as well for the simplified prompt)

Hope that makes sense

thanks

Will this work with the api as well?

Or is this for the UI editor only?

Error when using Batch IMG2IMG

Hello, this used to work fine but after a clean install of windows I get this error when I use batch img2img combined with TXT2Mask

*** Error completing request███████████████▎ | 9/27 [00:19<00:38, 2.17s/it]
*** Arguments: ('task(9v0vv1nqvnugvp9)', 5, 'lora:more_details:1 face of a person', '', [], None, None, None, None, None, None, None, 20, 16, 4, 0, 1, False, False, 1, 1, 7, 1.5, 0.4, -1.0, -1.0, 0, 0, 0, False, 0, 1280, 960, 1, 2, 0, 32, 0, 'C:\test', '', '', [], False, [], '', <gradio.routes.Request object at 0x000001EF6C8B1660>, 9, False, 'img2img', False, '', '', False, 'Euler a', False, 'v1-5-pruned-emaonly.safetensors [6ce0161689]', True, 0.5, True, 4, True, 32, True, False, 30, False, 6, False, 512, 512, '', False, 1, 'Both ▦', False, '', False, True, True, False, False, False, False, 100, 100, False, '', '', '', 'generateMasksTab', 4, 4, 2.5, 30, 1.03, 1, 1, 5, 0.5, 5, False, True, False, 20, '

\n
• CFG Scale should be 2 or lower.
\n

\n', True, True, '', '', True, 50, True, 1, 0, False, 4, 0.5, 'Linear', 'None', '

Recommended settings: Sampling Steps: 80-100, Sampler: Euler a, Denoising strength: 0.8

', 128, 8, ['left', 'right', 'up', 'down'], 1, 0.05, 128, 4, 0, ['left', 'right', 'up', 'down'], False, False, 'positive', 'comma', 0, False, False, '', '

Will upscale the image by the selected scale factor; use width and height sliders to set tile size

', 64, 0, 2, 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, 'face lips eyes forehead eyebrows ears cheeks chin head mouth tongue ', '', 100, 0, 0, True, '

If you like my work, please consider showing your support on Patreon. Thank you! ❤

') {}
Traceback (most recent call last):
File "C:\Users\David\stable-diffusion-webui\modules\call_queue.py", line 58, in f
res = list(func(*args, **kwargs))
File "C:\Users\David\stable-diffusion-webui\modules\call_queue.py", line 37, in f
res = func(*args, **kwargs)
File "C:\Users\David\stable-diffusion-webui\modules\img2img.py", line 226, in img2img
process_batch(p, img2img_batch_input_dir, img2img_batch_output_dir, img2img_batch_inpaint_mask_dir, args, to_scale=selected_scale_tab == 1, scale_by=scale_by, use_png_info=img2img_batch_use_png_info, png_info_props=img2img_batch_png_info_props, png_info_dir=img2img_batch_png_info_dir)
File "C:\Users\David\stable-diffusion-webui\modules\img2img.py", line 118, in process_batch
infotext = proc.infotext(p, n)
File "C:\Users\David\stable-diffusion-webui\modules\processing.py", line 457, in infotext
return create_infotext(p, self.all_prompts, self.all_seeds, self.all_subseeds, comments=[], position_in_batch=index % self.batch_size, iteration=index // self.batch_size)
File "C:\Users\David\stable-diffusion-webui\modules\processing.py", line 624, in create_infotext
"Seed": p.all_seeds[0] if use_main_prompt else all_seeds[index],
IndexError: list index out of range

EOFError: Ran out of input

Okay so I was getting the Unpickling error and did the fix to delete the weights in clipseg and restarted but now I'm getting a new error.

Traceback (most recent call last):
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/ui.py", line 134, in f
res = list(func(*args, **kwargs))
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/webui.py", line 58, in f
res = func(*args, **kwargs)
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/img2img.py", line 115, in img2img
processed = modules.scripts.scripts_img2img.run(p, *args)
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/scripts.py", line 158, in run
processed = script.run(p, *script_args)
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/scripts/txt2mask.py", line 175, in run
p.image_mask = get_mask().resize((p.init_images[0].width,p.init_images[0].height))
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/scripts/txt2mask.py", line 121, in get_mask
model.load_state_dict(torch.load(d64_file, map_location=torch.device('cuda')), strict=False);
File "/usr/local/lib/python3.7/dist-packages/torch/serialization.py", line 713, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/usr/local/lib/python3.7/dist-packages/torch/serialization.py", line 920, in _legacy_load
magic_number = pickle_module.load(f, **pickle_load_args)
EOFError: Ran out of input

What objects has this been trained on or does it understand?

Tried leaves/leafs/leaf & got error.

readme overwrites AUTOMATICS

this is an awesome script, but its readme overwrites AUTOMATICS

Keep getting this error

Way to change maximum input tokens?

Is there a config to change the max accepted prompts? Thanks.