haofanwang/controlnet-for-diffusers

Transfer the ControlNet with any basemodel in diffusers🔥

controlnet-for-diffusers's Issues

[Discussion] Diffuser Framework vs WebUI Extension

Hello @haofanwang,

I am trying to replicate Mikubill Transfer Control. So, initially I followed this guide and after comparing the AnyV3_Canny_Model.pth created to the WebUI (using same prompt, seed etc.), I realized that they are not the same... is it normal? Please, check the differences below:

In addition, I am trying to save the merged files when using Mikubill repo, tried to add the following line in cldm.py in the end of the PlugableControlModel init(): torch.save(final_state_dict, './control_any3_openpose.pth')

But i don't think it is correct... do you have any thoughts on this?

TimeoutError: [Errno 110] Connection timed out when convert model into diffusers

hi,

I'm following the README instruction strictly to convert control into diffusers-model.
when i do
python ./scripts/convert_controlnet_to_diffusers.py --checkpoint_path control_any3_openpose.pth --dump_path control_any3_openpose --device cpu

i got

global_step key not found in model
Traceback (most recent call last):
  File "/data/gpu_env_common/env/anaconda3/envs/controlnet_diffusers/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/data/gpu_env_common/env/anaconda3/envs/controlnet_diffusers/lib/python3.8/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/data/gpu_env_common/env/anaconda3/envs/controlnet_diffusers/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/data/gpu_env_common/env/anaconda3/envs/controlnet_diffusers/lib/python3.8/site-packages/urllib3/connection.py", line 414, in connect
    self.sock = ssl_wrap_socket(
  File "/data/gpu_env_common/env/anaconda3/envs/controlnet_diffusers/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/data/gpu_env_common/env/anaconda3/envs/controlnet_diffusers/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/data/gpu_env_common/env/anaconda3/envs/controlnet_diffusers/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/data/gpu_env_common/env/anaconda3/envs/controlnet_diffusers/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/data/gpu_env_common/env/anaconda3/envs/controlnet_diffusers/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/gpu_env_common/env/anaconda3/envs/controlnet_diffusers/lib/python3.8/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/data/gpu_env_common/env/anaconda3/envs/controlnet_diffusers/lib/python3.8/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/data/gpu_env_common/env/anaconda3/envs/controlnet_diffusers/lib/python3.8/site-packages/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/data/gpu_env_common/env/anaconda3/envs/controlnet_diffusers/lib/python3.8/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/data/gpu_env_common/env/anaconda3/envs/controlnet_diffusers/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/data/gpu_env_common/env/anaconda3/envs/controlnet_diffusers/lib/python3.8/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/data/gpu_env_common/env/anaconda3/envs/controlnet_diffusers/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/data/gpu_env_common/env/anaconda3/envs/controlnet_diffusers/lib/python3.8/site-packages/urllib3/connection.py", line 414, in connect
    self.sock = ssl_wrap_socket(
  File "/data/gpu_env_common/env/anaconda3/envs/controlnet_diffusers/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/data/gpu_env_common/env/anaconda3/envs/controlnet_diffusers/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/data/gpu_env_common/env/anaconda3/envs/controlnet_diffusers/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/data/gpu_env_common/env/anaconda3/envs/controlnet_diffusers/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/data/gpu_env_common/env/anaconda3/envs/controlnet_diffusers/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
urllib3.exceptions.ProtocolError: ('Connection aborted.', TimeoutError(110, 'Connection timed out'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./scripts/convert_controlnet_to_diffusers.py", line 105, in <module>
    pipe = load_pipeline_from_control_net_ckpt(
  File "/data/gpu_env_common/env/anaconda3/envs/controlnet_diffusers/lib/python3.8/site-packages/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py", line 1469, in load_pipeline_from_control_net_ckpt
    r = requests.get(
  File "/data/gpu_env_common/env/anaconda3/envs/controlnet_diffusers/lib/python3.8/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/data/gpu_env_common/env/anaconda3/envs/controlnet_diffusers/lib/python3.8/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/data/gpu_env_common/env/anaconda3/envs/controlnet_diffusers/lib/python3.8/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/data/gpu_env_common/env/anaconda3/envs/controlnet_diffusers/lib/python3.8/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/data/gpu_env_common/env/anaconda3/envs/controlnet_diffusers/lib/python3.8/site-packages/requests/adapters.py", line 547, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', TimeoutError(110, 'Connection timed out'))

Thanks for any help.

How to use LORA with ControlNet-for-Diffusers？

how can i get LORA module in the ControlNet-for-Diffusers?

Requires Update to work with Controlnet v1.1

Hi @haofanwang ,

This repo would require significant update to work with the latest version of diffusers, given the considerable development of the controlnet pipeline in the month of March. Request you to update this repo. I tried running with my code which I made before but it throws up errors, specifically on model keys, not being able to set pipe.unet.in_channels=4, and so on.

The inpainting pipeline doesn't work with the latest diffusers release.

Can't load state_dict of control model in Google Colab

Hi!
As I follow your Colab notebook. There seems to have an error while loading control model for conversion in tool_transfer_control.py. Detailed log here:

/content/ControlNet
2023-02-23 03:43:39.735813: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-02-23 03:43:45.107427: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2023-02-23 03:43:45.108480: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2023-02-23 03:43:45.108525: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
logging improved.
Loaded state_dict from [./models/v1-5-pruned.ckpt]
^C

It automatically terminated the command while loading load_state_dict(path_sd15_with_control).
Do you have the same problem while running in Colab?

Implementation with custom base models

Hi, here is an implementation of this minus diffusers running on Replicate: https://replicate.com/okaris/controlnet I built this to get to know the model. Might help someone else

finetuning model using diffuser

does it possible to fine-tune this model using diffuser in this stage?

Acceptable image size of `controlnet_hint` is (512, 512) but is (1024, 512)

Hi,
I am using the StableDiffusionControlNetInpaintPipeline that you provided in this repo to inpaint an image of 1024x512.
However, I encountered such an error:

File ~/.local/lib/python3.8/site-packages/diffusers-0.13.0.dev0-py3.8.egg/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_controlnet_inpaint.py:550, in StableDiffusionControlNetInpaintPipeline.controlnet_hint_conversion(self, controlnet_hint, height, width, num_images_per_prompt)
    548         return self.controlnet_hint_conversion(controlnet_hint, height, width, num_images_per_prompt)
    549     else:
--> 550         raise ValueError(
    551             f"Acceptable image size of `controlnet_hint` is ({width}, {height}) but is {controlnet_hint.size}"
    552         )
    553 else:
    554     raise ValueError(
    555         f"Acceptable type of `controlnet_hint` are any of torch.Tensor, np.ndarray, PIL.Image.Image but is {type(controlnet_hint)}"
    556     )

ValueError: Acceptable image size of `controlnet_hint` is (512, 512) but is (1024, 512)

pls. help!

Please help!

python ./scripts/convert_controlnet_to_diffusers.py --checkpoint_path control_any3_openpose.pth --dump_path control_any3_openpose --device cpu

Did not find file convert_controlnet_to_diffusers.py

More information regarding creation of models to use for inpainting

I didnt understand this comment which took me sometime to solve for, adding what I found so others can use it:

We have downloaded models locally, you can also load from huggingface control_sd15_seg is converted from control_sd15_seg.safetensors using instructions above and

To use any control model already present in control net models , then the way to do it is :

1. Download the models and annotators from the controlnet huggingface repo and place it under models folder: https://huggingface.co/lllyasviel/ControlNet

2. Use the takuma repo and download the diffusers

pip install "git+https://github.com/takuma104/diffusers.git@controlnet"
git clone https://github.com/takuma104/diffusers.git
cd diffusers
git checkout controlnet

3. Run this script to create the model which can be used with the pipeline, for which you will need to install transformers:
   conda install -c huggingface transformers
python ./scripts/convert_controlnet_to_diffusers.py --checkpoint_path [./models/control_sd15_canny.pth] --dump_path [./models/control_sd15_canny]--device cpu

4. check path of installed diffusers :
   assume you already know the absolute path of installed diffusers
   cp pipeline_stable_diffusion_controlnet_inpaint.py PATH/pipelines/stable_diffusion
   The PATH of the installed diffusers is not the same as the diffusers folder present in the repo. This installed diffusers will be present in the anaconda3/envs/control/lib/python3.8/site-packages/diffusers

5. Importing in files:

In addition to importing it in the following

PATH/pipelines/__init__.py
PATH/__init__.py

add the following line to the pipelines/stable_diffusion/init.py as well
from .pipeline_stable_diffusion_controlnet_inpaint import StableDiffusionControlNetInpaintPipeline

After this the code should work.

performance worst

I convert canny model to diffuser using the ./scripts/convert_controlnet_to_diffusers.py, and inference with own image, but the result is very bad.

Support multiple-image inference

Hi,

it seems like only single-image inference is supported in StableDiffusionControlNetPipeline.

For multi-image inference, one could go to the location of the diffusers package and modify the "# 2. pre-process" code block (around line 600).

original:

# 2. pre-process
sample = self.conv_in(sample)
if controlnet_hint is not None:
    sample += self.controlnet_input_hint_block(controlnet_hint)

modified:

# 2. pre-process
sample = self.conv_in(sample)
if controlnet_hint is not None:
    hint = self.controlnet_input_hint_block(controlnet_hint)
    if hint.shape[0] > 1:
        hint = hint.repeat(2, 1, 1, 1)
    sample += hint

Hope it helps.

controlnet inpaint with multiple conditions

img2img pipeline

any chance you can add an example of using control net with img2img to the Collab doc (without inpainting)?

I followed the instructions and tried adding the StableDiffusionControlNetInpaintImg2ImgPipeline class without any luck :

from diffusers.utils import load_image
from diffusers import StableDiffusionInpaintPipeline, StableDiffusionControlNetInpaintImg2ImgPipeline,  ControlNetModel

# we have downloaded models locally, you can also load from huggingface
# control_sd15_seg is converted from control_sd15_seg.safetensors using instructions above
controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-depth")
pipe_control = StableDiffusionControlNetInpaintImg2ImgPipeline.from_pretrained("C:/Users/User/Desktop/cnet_img2img/diffusers/control_openjourney-v2_depth",controlnet=controlnet,torch_dtype=torch.float16).to('cuda')
pipe_inpaint = StableDiffusionInpaintPipeline.from_pretrained("prompthero/openjourney-v2",torch_dtype=torch.float16).to('cuda')

# yes, we can directly replace the UNet
pipe_control.unet = pipe_inpaint.unet
pipe_control.unet.in_channels = 4

# we also the same example as stable-diffusion-inpainting
image = load_image("https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png")
mask = load_image("https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png")

# the segmentation result is generated from https://huggingface.co/spaces/hysts/ControlNet
control_image = load_image("https://raw.githubusercontent.com/haofanwang/ControlNet-for-Diffusers/main/images/desk_depth.png")

image = pipe_control(prompt="Face of a yellow cat, high resolution, sitting on a park bench", 
                     negative_prompt="lowres, bad anatomy, worst quality, low quality",
                     controlnet_hint=control_image, 
                     image=image,
                     mask_image=mask,
                     width=448,
                     height=640,
                     num_inference_steps=100).images[0]

image.save("inpaint_seg.jpg")

# complete this scentence```


gives me the error:

```Incorrect configuration settings! The config of `pipeline.unet`: FrozenDict([('sample_size', 64), ('in_channels', 4), ('out_channels', 4), ('center_input_sample', False), ('flip_sin_to_cos', True), ('freq_shift', 0), ('down_block_types', ['CrossAttnDownBlock2D', 'CrossAttnDownBlock2D', 'CrossAttnDownBlock2D', 'DownBlock2D']), ('mid_block_type', 'UNetMidBlock2DCrossAttn'), ('up_block_types', ['UpBlock2D', 'CrossAttnUpBlock2D', 'CrossAttnUpBlock2D', 'CrossAttnUpBlock2D']), ('only_cross_attention', False), ('block_out_channels', [320, 640, 1280, 1280]), ('layers_per_block', 2), ('downsample_padding', 1), ('mid_block_scale_factor', 1), ('act_fn', 'silu'), ('norm_num_groups', 32), ('norm_eps', 1e-05), ('cross_attention_dim', 768), ('attention_head_dim', 8), ('dual_cross_attention', False), ('use_linear_projection', False), ('class_embed_type', None), ('num_class_embeds', None), ('upcast_attention', False), ('resnet_time_scale_shift', 'default'), ('time_embedding_type', 'positional'), ('timestep_post_act', None), ('time_cond_proj_dim', None), ('conv_in_kernel', 3), ('conv_out_kernel', 3), ('projection_class_embeddings_input_dim', None), ('_class_name', 'UNet2DConditionModel'), ('_diffusers_version', '0.11.0.dev0'), ('_name_or_path', 'C:\\Users\\User\\.cache\\huggingface\\hub\\models--prompthero--openjourney-v2\\snapshots\\32e0aa8629c1d5ed82ff19f1543017bceb5f84d6\\unet')]) expects 4 but received `num_channels_latents`: 4 + `num_channels_mask`: 1 + `num_channels_masked_image`: 4 = 9. Please verify the config of `pipeline.unet` or your `mask_image` or `image` input.```

'Segmentation fault (core dumped)'

Hi mate, when I was using tool_transfer_control.py, I got the segmentation fault error, all the models were downloaded from official link.
Do you have any idea to solve it?

Do we need to do all those conversions for inpaining?

Hello
Do we need to do all those conversions mentioned under "ControlNet + Anything-v3" for inpainting?

Also, in the inpainting guide, there are these lines

# we have downloaded models locally, you can also load from huggingface
# control_sd15_seg is converted from control_sd15_seg.safetensors using instructions above
pipe_control = StableDiffusionControlNetInpaintPipeline.from_pretrained("./diffusers/control_sd15_seg",torch_dtype=torch.float16).to('cuda')
pipe_inpaint = StableDiffusionInpaintPipeline.from_pretrained("./diffusers/stable-diffusion-inpainting",torch_dtype=torch.float16).to('cuda')

Can anyone help understand what does the 2nd line means?
Also, for pipe_inpaint , do we pass the stable diffusion diffuser model path?

convert script missing ?

Thanks for your greate job ! I have some problems to convert the model to diffusers. When i finished cloning the diffusers, I switched into the branch "controlnet", but couldnot find a script convert_controlnet_to_diffusers.py under ~/diffusers/scripts. Could you help me to figure it out ?

TypeError: forward() got an unexpected keyword argument

Continuing from an earlier issue, but has now spread to every argument.

Have been getting this error every single time, after a great deal of debugging in an effort to get this code to run.
I can not figure out why this error persists, as from what Ive seen, this method is used for every single example of building an inference pipeline with Diffusers.
Some help here would be greatly appreciated because something could be bugged.

What is the specific difference between controlnet_inpainting and sd_inpainting?

According to the effect of sd_inpaint, mask_image is used to control the area that needs to be regenerated, so the difficulty is actually the generation of mask image. From the input of controlnet_inpainting and the final effect, it requires an additional seg_map, and the seg_map image does not show its role in the final effect.

The top image is the effect of stable diffusion inpainting and the bottom image is the effect of controlnet_inpainting.

Error no file named model_index.json found in directory:xxx

hi, after I convert the pretrained .pth file (from original controlnet) to diffusers like file(get "diffusion_pytorch_model.bin" and "config.json"), and run controlnet inference, it returns error"Error no file named model_index.json found in directory:xxx".

Use scripts/convert_original_stable_diffusion_to_diffusers.py to convert model.

It seems that diffusers provide official implement to convert the controlnet model into a module called ControlNetModel. Please refer to huggingface/diffusers#2550 to see related command. This is compatible to the latest official package.

Unexpected key(s) in state_dict: "text_model.embeddings.position_ids".

Sorry to bother you.When i try to python ./scripts/convert_controlnet_to_diffusers.py --checkpoint_path ../models/control_any3_openpose.pth --dump_path control_any3_openpose --device cpu. Something went wrong.
global_step key not found in model
Traceback (most recent call last):
File "./scripts/convert_controlnet_to_diffusers.py", line 105, in
pipe = load_pipeline_from_control_net_ckpt(
File "/home/jjc/anaconda3/envs/m3/lib/python3.8/site-packages/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py", line 1566, in load_pipeline_from_control_net_ckpt
text_model = convert_ldm_clip_checkpoint(checkpoint)
File "/home/jjc/anaconda3/envs/m3/lib/python3.8/site-packages/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py", line 847, in convert_ldm_clip_checkpoint
text_model.load_state_dict(text_model_dict)
File "/home/jjc/anaconda3/envs/m3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1482, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for CLIPTextModel:
Unexpected key(s) in state_dict: "text_model.embeddings.position_ids".

Install transformers while running the code

Hi! I had an issue while running the code, I was facing an error on transformer utils not being present. It got resolved when I downloaded transformers. I do not know whether it was for me alone, but wanted to mention it incase anyone else is facing similar issue

How to generate the pose image from an input image?

Hi~, thanks for your great work! I was wondering how to generate the pose image from an input image. thanks for your help.

"tmp not found" when I'm using pth conversion

Hello, I have an urgent issue that I need to resolve. Why am I getting an error that says "tmp not found" when I'm using pth conversion? Please, if you could take a moment to answer, I would greatly appreciate it. It's extremely urgent. Thank you.

python ./scripts/convert_controlnet_to_diffusers.py --checkpoint_path /data//ControlNet-for-Diffusers-main/control_sd15_scribble/control_sd15_scribble.pth --dump_path control_sd15_scribble --device cpu
global_step key not found in model
╭────────────────────────────── Traceback (most recent call last) ───────────────────────────────╮
│ /data//ControlNet-for-Diffusers-main/diffusers/./scripts/convert_controlnet_to_diffusers.py: │
│ 105 in │
│ │
│ 102 │ parser.add_argument("--device", type=str, help="Device to use (e.g. cpu, cuda:0, cud │
│ 103 │ args = parser.parse_args() │
│ 104 │ │
│ ❱ 105 │ pipe = load_pipeline_from_control_net_ckpt( │
│ 106 │ │ checkpoint_path=args.checkpoint_path, │
│ 107 │ │ original_config_file=args.original_config_file, │
│ 108 │ │ image_size=args.image_size, │
│ │
│ /data//anaconda3/envs/controlnet/lib/python3.8/site-packages/diffusers/pipelines/stable_diff │
│ usion/convert_from_ckpt.py:1475 in load_pipeline_from_control_net_ckpt │
│ │
│ 1472 │ │ │ │ │ ) │
│ 1473 │ │ │ │ │ open(original_config_file, "wb").write(r.content) │
│ 1474 │ │ │
│ ❱ 1475 │ │ original_config = OmegaConf.load(original_config_file) │
│ 1476 │ │
│ 1477 │ if num_in_channels is not None: │
│ 1478 │ │ original_config["model"]["params"]["unet_config"]["params"]["in_channels"] = nu │
│ │
│ /data//anaconda3/envs/controlnet/lib/python3.8/site-packages/omegaconf/omegaconf.py:183 in │
│ load │
│ │
│ 180 │ │ from .utils import get_yaml_loader │
│ 181 │ │ │
│ 182 │ │ if isinstance(file, (str, pathlib.Path)): │
│ ❱ 183 │ │ │ with io.open(os.path.abspath(file_), "r", encoding="utf-8") as f: │
│ 184 │ │ │ │ obj = yaml.load(f, Loader=get_yaml_loader()) │
│ 185 │ │ elif getattr(file_, "read", None): │
│ 186 │ │ │ obj = yaml.load(file_, Loader=get_yaml_loader()) │
╰────────────────────────────────────────────────────────────────────────────────────────────────╯
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp8594uj16/inference.yaml'

Weird colors when using Pytorch 2.0

Hi. I recently ControlNet for diffusers, and found that colors were being affected. After some search, I discovered that the culprit was that I updated Pytorch to 2.0.0. Inference is much faster, but there are some compatibility issues between ControlNet for Diffusers and the latest Pytorch release.

Please advise,

Alejandro.

ImportError: cannot import name 'KarrasDiffusionSchedulers' from 'diffusers.schedulers'

Thank you for your work. When I tried to reproduce your second part of image repair work, an error occurred. Your code included KarrasDiffusionSchedulers, but this class does not exist in the diffusers library. Is there a problem with my operation? How should I fix it?

Traceback (most recent call last):
File "/root/autodl-tmp/ControlNet-for-Diffusers/test.py", line 2, in
from diffusers.utils import load_image
File "/root/miniconda3/envs/lora/lib/python3.10/site-packages/diffusers/init.py", line 39, in
from .pipelines import (
File "/root/miniconda3/envs/lora/lib/python3.10/site-packages/diffusers/pipelines/init.py", line 42, in
from .alt_diffusion import AltDiffusionImg2ImgPipeline, AltDiffusionPipeline
File "/root/miniconda3/envs/lora/lib/python3.10/site-packages/diffusers/pipelines/alt_diffusion/init.py", line 33, in
from .pipeline_alt_diffusion import AltDiffusionPipeline
File "/root/miniconda3/envs/lora/lib/python3.10/site-packages/diffusers/pipelines/alt_diffusion/pipeline_alt_diffusion.py", line 36, in
from ..stable_diffusion.safety_checker import StableDiffusionSafetyChecker
File "/root/miniconda3/envs/lora/lib/python3.10/site-packages/diffusers/pipelines/stable_diffusion/init.py", line 46, in
from .pipeline_stable_diffusion_controlnet_inpaint import StableDiffusionControlNetInpaintPipeline
File "/root/miniconda3/envs/lora/lib/python3.10/site-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_controlnet_inpaint.py", line 25, in
from ...schedulers import KarrasDiffusionSchedulers
ImportError: cannot import name 'KarrasDiffusionSchedulers' from 'diffusers.schedulers' (/root/miniconda3/envs/lora/lib/python3.10/site-packages/diffusers/schedulers/init.py)

Crashed running on m1 Mac

Is it possible to run control net on Mac? Thanks!

Got the fellowing error:

output = pipe(

File "/Users/kelong/Library/Python/3.9/lib/python/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/Users/kelong/Library/Python/3.9/lib/python/site-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_controlnet.py", line 779, in call
latents = self.scheduler.step(noise_pred, t, latents, **extra_step_kwargs).prev_sample
File "/Users/kelong/Library/Python/3.9/lib/python/site-packages/diffusers/schedulers/scheduling_unipc_multistep.py", line 528, in step
sample = self.multistep_uni_c_bh_update(
File "/Users/kelong/Library/Python/3.9/lib/python/site-packages/diffusers/schedulers/scheduling_unipc_multistep.py", line 465, in multistep_uni_c_bh_update
rhos_c = torch.linalg.solve(R, b)
NotImplementedError: The operator 'aten::_linalg_solve_ex.result' is not currently implemented for the MPS device. If you want this op to be added in priority during the prototype phase of this feature, please comment on pytorch/pytorch#77764. As a temporary fix, you can set the environment variable PYTORCH_ENABLE_MPS_FALLBACK=1 to use the CPU as a fallback for this op. WARNING: this will be slower than running natively on MPS.

Question Regarding Multi Control

@haofanwang
There are two pipelines, one for depth and one for openpose, lets call them pipe_control_depth, pipe_control_openpose.

Which pipeline should be used for generating the output? What does the input for the pipeline look like? Should the pipeline load both the models? Not sure how it works.

Runtime Error due to Shape Mismatching when Replacing Base Model with SD1.5 Inpainting

Dear @haofanwang,

I am writing to report an issue that I encountered while following the steps to replace the base model with SD1.5 inpainting in your repository. Specifically, I edited the path as instructed and ran the script python tool_transfer_control.py.

path_sd15 = './models/v1-5-pruned.ckpt'
path_sd15_with_control = './models/control_sd15_scribble.pth'
path_input = './models/sd-v1-5-inpainting.safetensors'
path_output = './models/control_sd-v1-5-inpainting_scribble.pth'

However, I faced a runtime error related to shape mismatching. I believe that the error is caused by the fact that the base model (Stable Diffusion 1.5) only has 4 input channels, while the inpainting model (Stable Diffusion 1.5 inpainting) has 9 input channels.

Therefore, I would greatly appreciate it if you could advise me on how to resolve this issue. Thank you in advance for your assistance.

Best regards,

What is the specific difference between controlnet_inpainting and controlnet_inpainting_img2img?

I tried to test the two algorithms but I feel that the effect is similar. Can I understand that the img2img version takes into account the image input allowed by the controlnet_inpainting , but controlnet_inpainting also support img .

Do you plan to design a multi-contronet in diffuser?

Issue with diffusers models

Thanks for putting this manual together!
Unfortunately, I cant find the script convert_controlnet_to_diffusers anywhere, not in the PR.
Could you help me with this? Thanks!

I tried ControlNet + new base model with segmentation but got error

Hey
I followed the guide for ControlNet + new base model with seg version of ControlNet but for this error

In the transfer file, I have this

path_sd15 = './models/v1-5-pruned.ckpt'
path_sd15_with_control = './models/control_sd15_seg.pth'
path_input = './models/converted_to_ckpt_st.safetensors'
path_output = './models/control_ayush_seg.pth'

Can anyone please help?

the image not matching text prompt for depth control inpainting

Thanks for your great work. I test your code for your demo img, the result is good. however, I test it for depth control inpainting, the output image is not match with text prompt. Can you give me some advices.
first, I test the depth controlnet in original controlnet, the prompt is "a firefighter wearing red uniform clothes". the result is good.

the above image is input for depth control net.

the above image is depth prediction map by control net.

the above image is output by control net.
second, I add a mask image to control net, and use the same input image and text prompt, depth map from control net. However, the output image is not matched with text prompt. the clothes color is not red.

the above is mask img.

the above is final output by your proposed code.

TypeError: forward() got an unexpected keyword argument 'controlnet_hint'

It looks like something has changed with diffusers and it doesn't have this param:

Require more information regarding inpainting

Hi!

@haofanwang

I am trying to understand how to perform inpaintiing with control net which you had mentioned in the third part of the Read me. I have gotten extremely poor results in comparison to normal inpainting with the same model, and therefore feel something is not quite right in the code. I tried with the canny edge detection annotator

The way my process is structured is that I get an input image and a mask, which is what is required for typical inpainting pipeline.
Then using the annotators, I create a canny image of a reference image, which I pass as the control hint.

However, the output in no way ensures the fidelity of the original input and also doesn't follow the canny edge properly.

Code is below:

import torch
from diffusers.utils import load_image
from diffusers import StableDiffusionInpaintPipeline, StableDiffusionControlNetInpaintPipeline
from annotator.util import resize_image, HWC3
from annotator.canny import CannyDetector
import PIL
from PIL import Image
import cv2
import einops
import gradio as gr
import numpy as np
import torch
import random

def getCannyImage(input_image,image_resolution, low_threshold, high_threshold):
    input_image=np.array(input_image)
    apply_canny = CannyDetector()
    with torch.no_grad():
        img = resize_image(HWC3(input_image), image_resolution)
        H, W, C = img.shape
        detected_map = apply_canny(img, low_threshold, high_threshold)
        detected_map = HWC3(detected_map)
    cannyImageNumpy =[255-detected_map]
    cannyImage=Image.fromarray(cannyImageNumpy[0])
    return cannyImage 

pipe_control = StableDiffusionControlNetInpaintPipeline.from_pretrained("models/control_sd15_canny",torch_dtype=torch.float16).to('cuda')
pipe_inpaint = StableDiffusionInpaintPipeline.from_pretrained("models/stable-diffusion-inpainting",torch_dtype=torch.float16).to('cuda')
pipe_control.unet = pipe_inpaint.unet
pipe_control.unet.in_channels = 4



image = load_image("./inputImages/input.png")
mask = load_image("./inputImages/mask.png")
canny_input=load_image("./inputImages/reference.png")

# Canny edge detection parameters
image_resolution, low_threshold, high_threshold=512,100,200
cannyImage=getCannyImage(canny_input,image_resolution, low_threshold, high_threshold)
control_image = cannyImage
control_image = control_image

image = pipe_control(prompt=" Woman performing Yoga on a tennis court", 
                     negative_prompt="lowres, bad anatomy, worst quality, low quality",
                     controlnet_hint=control_image, 
                     image=image,
                     mask_image=mask,
                     num_inference_steps=100).images[0]

image.save("inpaint_canny.jpg")

Reference and Input Images are also below, mask is essentially mask of the input image:

Please let me know if I am missing something.

ControlNetInpaintPipeline: Errors for loading Controlnet Diffusers folder

Thanks for your work, After I converted the Controlnet.pth to diffusers, an error still occurred while loading. Did my conversion fail?How can I solve this problem？The error log is as follows：

The config attributes {'controlnet_hint_channels': 3} were passed to UNet2DConditionModel, but are not expected and will be ignored. Please verify your config.json configuration file.
Traceback (most recent call last):
File "/home/sd/Harddisk/zxp/ControlNet-for-Diffusers/Test.py", line 8, in
pipe_control = StableDiffusionControlNetInpaintPipeline.from_pretrained("./control_sd15_seg",torch_dtype=torch.float16).to('cuda')
File "/home/sd/miniconda3/envs/Controlnet/lib/python3.9/site-packages/diffusers/pipelines/pipeline_utils.py", line 865, in from_pretrained
loaded_sub_model = load_method(os.path.join(cached_folder, name), **loading_kwargs)
File "/home/sd/miniconda3/envs/Controlnet/lib/python3.9/site-packages/diffusers/models/modeling_utils.py", line 567, in from_pretrained
raise ValueError(
ValueError: Cannot load <class 'diffusers.models.unet_2d_condition.UNet2DConditionModel'> from ./control_sd15_seg/controlnet because the following keys are missing:
up_blocks.1.resnets.2.conv_shortcut.weight, ... ....
up_blocks.2.resnets.2.norm2.bias, up_blocks.3.resnets.0.norm2.weight, up_blocks.2.resnets.0.conv1.bias, up_blocks.1.resnets.1.norm1.bias.
Please make sure to pass low_cpu_mem_usage=False and device_map=None if you want to randomely initialize those weights or else make sure your checkpoint file is correct.

Cant convert to diffusers

Hi! Thanks for the awesome work!
Unfortunatelly I am stuck on the last step.

It throws the following error when trying to convert to diffusers:

Transferred model saved at E:\sd\output_gui2/models//control_anything-v3.0_canny/control_anything-v3.0_canny.pth global_step key not found in model In this conversion only the non-EMA weights are extracted. If you want to instead extract the EMA weights (usually better for inference), please make sure to add the --extract_ema flag. Traceback (most recent call last): File "E:\sd\output_gui2\sdiff_GUI.py", line 434, in <module> convertControl(['--checkpoint_path', (dir_path + "/models/" + '/control_' + values["mod"] + "_canny/" + 'control_' + values["mod"] + "_canny.pth"), '--dump_path', (dir_path + "/models/" + 'control_' + values["mod"] + "_canny" ),'--device', 'cpu']) File "E:\sd\output_gui2\convert_controlnet_to_diffusers.py", line 105, in main pipe = load_pipeline_from_control_net_ckpt( File "C:\Users\brurp\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\diffusers\pipelines\stable_diffusion\convert_from_ckpt.py", line 1549, in load_pipeline_from_control_net_ckpt converted_ctrl_checkpoint = convert_controlnet_checkpoint(checkpoint, unet_config) File "C:\Users\brurp\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\diffusers\pipelines\stable_diffusion\convert_from_ckpt.py", line 584, in convert_controlnet_checkpoint new_checkpoint["time_embedding.linear_1.weight"] = unet_state_dict["time_embed.0.weight"] KeyError: 'time_embed.0.weight'

confused about only replace unet mode for inpainting

Thanks for your great work. the inpaiting function is amazing. but I am confused why the inpaint works by only replacing unet weight.

Why don't you import from `diffusers` instead of using relative imports?

if you replace imports like from ...models import AutoencoderKL, UNet2DConditionModel with from diffusers.models import AutoencoderKL, UNet2DConditionModel you can copy paste the pipeline files in your code and simply run it without patching diffusers

Should i open a PR to do this?

Also, why not add these pipelines to the diffusers package with a PR instead of keeping them in a separate repository?

Reduce Memory Consumption

Hi @haofanwang Is it possible to directly save the sdv 1.5 unet + depth control model into another new model called SD_V1.5_depth and load that model in our pipe control and not call the pipe_inpaint pipeline at all, there by saving space in VRAM? Is it possible? Or am I doing some mistake in my assumption?

Can we use an reference image as control image ?

Hi, First of all, thank you for the amazing work done in Control Net, I am new to this area. I need to know that, without using any segmentation image or pose image, can we use a real image as the control image input

Thanks in advance

A little poor about the inpainting quality controlled by canny

support controlnet use sd1.5 img2img

if I want to contronet support sd img2img , in difusser I only make pipe_sd.unet = pipe_control.unet ? is right ?

diffusers community pipeline based on your repo

Hi @haofanwang

Thanks so much for making this repo!

We just added community pipelines to the diffuser based on your repo - let us know if you have any feedback! 🤗🤗🤗
huggingface/diffusers#2561

YiYi