[bug]: Inpainting results in value error v4.0.0rc5 Cannot convert inpainting models to diffusers

[Fubu4u2]

[bug]: Inpainting results in value error v4.0.0rc5 Cannot convert inpainting models to diffusers

[psychedelicious]

Please link us to the specific inpainting model that is causing the problem

[Fubu4u2]

It is not any particular inpainting model. I am receiving the same error with any inpainting model I try and use. Even runway ml's sd15 from huggingface.

https://huggingface.co/runwayml/stable-diffusion-inpainting/blob/main/sd-v1-5-inpainting.ckpt

[Fubu4u2]

here is log when importing and attempting to convert sd1-5 inpainting ckpt

https://huggingface.co/runwayml/stable-diffusion-inpainting/blob/main/sd-v1-5-inpainting.ckpt

[2024-03-27 06:43:28,838]::[ModelInstallService]::INFO --> Registered sd-v1-5-inpainting.ckpt with id b4b10089-92db-4855-ac8d-f89385cb16ef
[2024-03-27 06:43:28,868]::[InvokeAI]::WARNING --> 'int' object has no attribute 'startswith'
[2024-03-27 06:43:29,107]::[ModelInstallService]::INFO --> 1 new models registered; 0 unregistered
[2024-03-27 06:43:29,107]::[ModelInstallService]::INFO --> Scanning autoimport directory for new models
[2024-03-27 06:43:29,118]::[ModelInstallService]::INFO --> 0 new models registered
[2024-03-27 06:43:29,118]::[ModelInstallService]::INFO --> Model installer (re)initialized
[2024-03-27 06:43:29,118]::[uvicorn.access]::INFO --> 172.71.166.253:0 - "PATCH /api/v2/models/sync HTTP/1.0" 204
[2024-03-27 06:43:29,305]::[uvicorn.access]::INFO --> 172.71.166.252:0 - "GET /api/v2/models/ HTTP/1.0" 200
[2024-03-27 06:43:37,721]::[ModelLoadService]::INFO --> Converting /workspace/invokeai/models/sd-1/main/sd-v1-5-inpainting.ckpt to diffusers format
[2024-03-27 06:43:39,389]::[uvicorn.access]::INFO --> 172.71.166.253:0 - "PUT /api/v2/models/convert/b4b10089-92db-4855-ac8d-f89385cb16ef HTTP/1.0" 500
[2024-03-27 06:43:39,389]::[uvicorn.error]::ERROR --> Exception in ASGI application

Traceback (most recent call last):
File "/workspace/invokeai/venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 412, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/workspace/invokeai/venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in call
return await self.app(scope, receive, send)
File "/workspace/invokeai/venv/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/workspace/invokeai/venv/lib/python3.10/site-packages/starlette/applications.py", line 123, in call
await self.middleware_stack(scope, receive, send)
File "/workspace/invokeai/venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in call
raise exc
File "/workspace/invokeai/venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in call
await self.app(scope, receive, _send)
File "/workspace/invokeai/venv/lib/python3.10/site-packages/starlette/middleware/gzip.py", line 24, in call
await responder(scope, receive, send)
File "/workspace/invokeai/venv/lib/python3.10/site-packages/starlette/middleware/gzip.py", line 44, in call
await self.app(scope, receive, self.send_with_gzip)
File "/workspace/invokeai/venv/lib/python3.10/site-packages/starlette/middleware/cors.py", line 91, in call
await self.simple_response(scope, receive, send, request_headers=headers)
File "/workspace/invokeai/venv/lib/python3.10/site-packages/starlette/middleware/cors.py", line 146, in simple_response
await self.app(scope, receive, send)
File "/workspace/invokeai/venv/lib/python3.10/site-packages/fastapi_events/middleware.py", line 43, in call
await self.app(scope, receive, send)
File "/workspace/invokeai/venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/workspace/invokeai/venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/workspace/invokeai/venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/workspace/invokeai/venv/lib/python3.10/site-packages/starlette/routing.py", line 758, in call
await self.middleware_stack(scope, receive, send)
File "/workspace/invokeai/venv/lib/python3.10/site-packages/starlette/routing.py", line 778, in app
await route.handle(scope, receive, send)
File "/workspace/invokeai/venv/lib/python3.10/site-packages/starlette/routing.py", line 299, in handle
await self.app(scope, receive, send)
File "/workspace/invokeai/venv/lib/python3.10/site-packages/starlette/routing.py", line 79, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/workspace/invokeai/venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/workspace/invokeai/venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/workspace/invokeai/venv/lib/python3.10/site-packages/starlette/routing.py", line 74, in app
response = await func(request)
File "/workspace/invokeai/venv/lib/python3.10/site-packages/fastapi/routing.py", line 278, in app
raw_response = await run_endpoint_function(
File "/workspace/invokeai/venv/lib/python3.10/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
return await dependant.call(**values)
File "/workspace/invokeai/venv/lib/python3.10/site-packages/invokeai/app/api/routers/model_manager.py", line 691, in convert_model
model_manager.load.load_model(model_config, submodel_type=SubModelType.Scheduler)
File "/workspace/invokeai/venv/lib/python3.10/site-packages/invokeai/app/services/model_load/model_load_default.py", line 80, in load_model
).load_model(model_config, submodel_type)
File "/workspace/invokeai/venv/lib/python3.10/site-packages/invokeai/backend/model_manager/load/load_default.py", line 62, in load_model
model_path = self._convert_if_needed(model_config, model_path, submodel_type)
File "/workspace/invokeai/venv/lib/python3.10/site-packages/invokeai/backend/model_manager/load/load_default.py", line 82, in _convert_if_needed
return self._convert_model(config, model_path, cache_path)
File "/workspace/invokeai/venv/lib/python3.10/site-packages/invokeai/backend/model_manager/load/model_loaders/stable_diffusion.py", line 90, in _convert_model
convert_ckpt_to_diffusers(
File "/workspace/invokeai/venv/lib/python3.10/site-packages/invokeai/backend/model_manager/convert_ckpt_to_diffusers.py", line 46, in convert_ckpt_to_diffusers
pipe = download_from_original_stable_diffusion_ckpt(Path(checkpoint_path).as_posix(), **kwargs)
File "/workspace/invokeai/venv/lib/python3.10/site-packages/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py", line 1472, in download_from_original_stable_diffusion_ckpt
set_module_tensor_to_device(unet, param_name, "cpu", value=param)
File "/workspace/invokeai/venv/lib/python3.10/site-packages/accelerate/utils/modeling.py", line 348, in set_module_tensor_to_device
raise ValueError(
ValueError: Trying to set a tensor of shape torch.Size([320, 9, 3, 3]) in "weight" (which has shape torch.Size([320, 4, 3, 3])), this look incorrect.
[2024-03-27 06:43:39,668]::[uvicorn.access]::INFO --> 172.71.166.252:0 - "GET /api/v2/models/ HTTP/1.0" 200
[2024-03-27 06:43:49,499]::[uvicorn.access]::INFO --> 172.71.166.252:0 - "GET /ws/socket.io/?EIO=4&transport=polling&t=Ov-fZFp&sid=hwH2oPBiIL6ulaIFAAAQ HTTP/1.0" 200
[2024-03-27 06:43:49,637]::[uvicorn.access]::INFO --> 172.71.166.252:0 - "POST /ws/socket.io/?EIO=4&transport=polling&t=Ov-ffJN&sid=hwH2oPBiIL6ulaIFAAAQ HTTP/1.0" 200

[Fubu4u2]

My guess is that model needs to be converted to diffusers format in order to inpaint. Thus inpainting is failing because models are still in ckpt format. If that is the case then the issue is really that inpainting models are failing to convert to diffussers.

https://huggingface.co/runwayml/stable-diffusion-inpainting/blob/main/sd-v1-5-inpainting.ckpt

attempts to convert inpaint models to diffusers results in

[2024-03-27 06:43:39,389]::[uvicorn.error]::ERROR --> Exception in ASGI application

and

ValueError: Trying to set a tensor of shape torch.Size([320, 9, 3, 3]) in "weight" (which has shape torch.Size([320, 4, 3, 3])), this look incorrect.

[psychedelicious]

Thanks, there was a problem with the diffusers conversion logic, resolved in #6064.

[Fubu4u2]

happy to help