[Feature Request]: SD XL support

[leejet]

[Feature Request]: SD XL support

[leejet]

I'm willing to implement SDXL once I've improved the support for SD 1.x and added support for SD 2.x.

[Green-Sky]

Beside LCM being available for XL models, stability.ai released SDXL-turbo a destil(finetune?) that can generate good images with a single step.

https://huggingface.co/stabilityai/sdxl-turbo

[leejet]

@leejet this can be closed

Generally, I don't proactively close issues unless they've been resolved for an extended period without any response from the person who opened the issue. I prefer the individuals who opened the issue to confirm its resolution and close it themselves.

[leejet]

@ranjithum The VAE in SDXL encounters NaN issues under FP16, but unfortunately, the ggml_conv_2d only operates under FP16. Hence, a parameter is needed to specify the VAE that has fixed the FP16 NaN issue. You can find it here: SDXL VAE FP16 Fix.

./bin/sd -m ../models/sd_xl_base_1.0.safetensors --vae ../models/sdxl_vae-fp16-fix.safetensors -H 1024 -W 1024 -p "a lovely cat" -v

[shaharhi]

is is compatible with this repo?

[FSSRepo]

@leejet this can be closed

[leejet]

@zhongwei Support for SDXL has been added. You can try pulling the latest code from the master branch.

[ranjithum]

Did anyone try running sd_xl ? For some reason its generating a empty image (its pitch black) .. Following is the command i used and its output

$ ./bin/sd  -m ~/downloaded_models/sdxl-turbo/sd_xl_turbo_1.0_fp16.safetensors -p "a lovely cat"
[INFO]  stable-diffusion.cpp:5386 - loading model from '~/downloaded_models/sdxl-turbo/sd_xl_turbo_1.0_fp16.safetensors'
[INFO]  model.cpp:638  - load ~/downloaded_models/sdxl-turbo/sd_xl_turbo_1.0_fp16.safetensors using safetensors format
[INFO]  stable-diffusion.cpp:5412 - Stable Diffusion XL
[INFO]  stable-diffusion.cpp:5418 - Stable Diffusion weight type: f16
[INFO]  stable-diffusion.cpp:5573 - total memory buffer size = 6570.56MB (clip 1565.66MB, unet 4909.43MB, vae 95.47MB)
[INFO]  stable-diffusion.cpp:5579 - loading model from '~/downloaded_models/sdxl-turbo/sd_xl_turbo_1.0_fp16.safetensors' completed, taking 1.78s
[INFO]  stable-diffusion.cpp:5593 - running in eps-prediction mode
[INFO]  stable-diffusion.cpp:6486 - apply_loras completed, taking 0.00s
[INFO]  stable-diffusion.cpp:6525 - get_learned_condition completed, taking 1547 ms
[INFO]  stable-diffusion.cpp:6535 - sampling using Euler A method
[INFO]  stable-diffusion.cpp:6539 - generating image: 1/1 - seed 42
  |==================================================| 20/20 - 18.15s/it
[INFO]  stable-diffusion.cpp:6551 - sampling completed, taking 353.73s
[INFO]  stable-diffusion.cpp:6559 - generating 1 latent images completed, taking 353.89s
[INFO]  stable-diffusion.cpp:6561 - decoding 1 latents
[INFO]  stable-diffusion.cpp:6571 - latent 1 decoded, taking 17.36s
[INFO]  stable-diffusion.cpp:6575 - decode_first_stage completed, taking 17.36s
[INFO]  stable-diffusion.cpp:6590 - txt2img completed in 372.80s
[INFO]  main.cpp:538  - save result image to 'output.png'

i also tried downloading the unets/vae etc.. and passing the same as argument (along with some minor code changes to load f16.safetensor instead of just .safetensor - std::string unet_path = path_join(file_path, "unet/diffusion_pytorch_model.safetensors");)

$ ./bin/sd  -m ~/downloaded_models/sdxl-turbo/sd_xl_turbo_1.0_fp16.safetensors --vae ~/downloaded_models/sdxl-turbo/ -p "a lovely cat"
[INFO]  stable-diffusion.cpp:5386 - loading model from '~/downloaded_models/sdxl-turbo/sd_xl_turbo_1.0_fp16.safetensors'
[INFO]  model.cpp:638  - load ~/downloaded_models/sdxl-turbo/sd_xl_turbo_1.0_fp16.safetensors using safetensors format
[INFO]  stable-diffusion.cpp:5395 - loading vae from '~/downloaded_models/sdxl-turbo/'
[INFO]  model.cpp:632  - load ~/downloaded_models/sdxl-turbo/ using diffusers format
[INFO]  stable-diffusion.cpp:5412 - Stable Diffusion XL
[INFO]  stable-diffusion.cpp:5418 - Stable Diffusion weight type: f16
[WARN]  stable-diffusion.cpp:5503 - unknown tensor 'unet.add_embedding.linear_1.bias' in model file
[WARN]  stable-diffusion.cpp:5503 - unknown tensor 'unet.add_embedding.linear_1.weight' in model file
[WARN]  stable-diffusion.cpp:5503 - unknown tensor 'unet.add_embedding.linear_2.bias' in model file
[WARN]  stable-diffusion.cpp:5503 - unknown tensor 'unet.add_embedding.linear_2.weight' in model file
[WARN]  stable-diffusion.cpp:5503 - unknown tensor 'model.diffusion_model.output_blocks.2.1.conv.bias' in model file
[WARN]  stable-diffusion.cpp:5503 - unknown tensor 'model.diffusion_model.output_blocks.2.1.conv.weight' in model file
[INFO]  stable-diffusion.cpp:5573 - total memory buffer size = 6570.56MB (clip 1565.66MB, unet 4909.43MB, vae 95.47MB)
[INFO]  stable-diffusion.cpp:5579 - loading model from '~/downloaded_models/sdxl-turbo/sd_xl_turbo_1.0_fp16.safetensors' completed, taking 2.61s
[INFO]  stable-diffusion.cpp:5593 - running in eps-prediction mode
[INFO]  stable-diffusion.cpp:6486 - apply_loras completed, taking 0.00s
[INFO]  stable-diffusion.cpp:6525 - get_learned_condition completed, taking 1592 ms
[INFO]  stable-diffusion.cpp:6535 - sampling using Euler A method
[INFO]  stable-diffusion.cpp:6539 - generating image: 1/1 - seed 42
  |==================================================| 20/20 - 18.09s/it
[INFO]  stable-diffusion.cpp:6551 - sampling completed, taking 353.85s
[INFO]  stable-diffusion.cpp:6559 - generating 1 latent images completed, taking 353.85s
[INFO]  stable-diffusion.cpp:6561 - decoding 1 latents
[INFO]  stable-diffusion.cpp:6571 - latent 1 decoded, taking 17.08s
[INFO]  stable-diffusion.cpp:6575 - decode_first_stage completed, taking 17.08s
[INFO]  stable-diffusion.cpp:6590 - txt2img completed in 372.51s
[INFO]  main.cpp:538  - save result image to 'output.png'

But its the same result.. i have tried the older stable diffusion - stable-diffusion-2-1/v2-1_768-nonema-pruned.safetensors it works..
I m running on ubuntu 22.03.

[ranjithum]

@leejet - Perfect thanks.. It worked..

[Green-Sky]

@leejet we should probably put up a warning in the program, when f32 vae is used. (until its fixed).