Add segmentation painting and MLSD support.

[houseofsecrets]

Add segmentation painting and MLSD support.

[Danamir]

It's all good when it comes to generating images but can't trace them. I guess that's something to solve.

The problem was with the default threshold values used. MLSD detector is now working after #47 .

[Danamir]

It seems you are using MLSD as a detector, ie. the modules (preprocessors) part of ControlNet, not the models (rendering). The detectors/preprocessors are used to detect features in an image, or do some computations before the image can be used by the models for rendering.

Check your http://127.0.0.1:7860/controlnet/model_list URL to find the correct model to add to config.json controlnet_models list. For example on my system it's control_v11p_sd15_mlsd_fp16.

Once added to the configuration, restart the application and use the shortcut shift + m to select the MLSD model. Check your webui log, it should display something like ControlNet model control_v11p_sd15_mlsd_fp16 [77b5ad24] loaded before rendering.

There seems to be a bug with the mlsd detector at the moment. I can detect features in an image with the webui, but calling the same detector with ctrl + d gets a white image. It may be due to the image format sent by the API.

[CodeMonkey80]

No, no, you misread. MLSD model loaded and worked for me and the detector doesn't seem to do anything, just as you replied. It's all good when it comes to generating images but can't trace them. I guess that's something to solve.

Any comment on segmentation painting? Is it doable?

[Danamir]

Ok I was not sure you were using MLSD for the rendering part also.

In theory segmentation painting should work, but would necessitate at least a color picker and a paint filler of some kind. It's not something I expect to code anytime soon, but maybe someone will. 😅

Another approach could be to modify the img2img mode to send the image data to ControlNet instead, and use an external editor with the correct editing capabilities.