eps696/sdfu

Stable Diffusers for studies

sdfu's Issues

What settings/steps have you implemented for your examples?

i made to work your repo, but i dont get the smooth transitions you have in your examples on twitter, which steps did you use or which .py file are you executing for such transitions?

SDXL.py throws an error when generating more than one image with -n argument

When using this repo SDXL.py will generate an error if the -n flag is set to anything other than one. When I run this command: python \src\sdxl.py -t "blue cyber dog" -n 3 -s 34 I get the following output:

File "D:\dev\python\sdfu\SDfu\src\sdxl.py", line 122, in genmix
image = pipe(_, _, _, _, a.steps, , a.cfg_scale, , , a.num, 0, g, z, c, uc, pool_c, pool_uc).images[0]
File "C:\Users\pictu\Envs\SDFu\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "C:\Users\pictu\Envs\SDFu\lib\site-packages\diffusers\pipelines\stable_diffusion_xl\pipeline_stable_diffusion_xl.py", line 1039, in call
noise_pred = self.unet(
File "C:\Users\pictu\Envs\SDFu\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "C:\Users\pictu\Envs\SDFu\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\pictu\Envs\SDFu\lib\site-packages\diffusers\models\unet_2d_condition.py", line 1001, in forward
emb = emb + aug_emb if aug_emb is not None else emb
RuntimeError: The size of tensor a (6) must match the size of tensor b (18) at non-singleton dimension 0`

Can't get longanimatediff to work

I tried to use the longanimatediff model from here,
https://huggingface.co/Lightricks/LongAnimateDiff

but an error says the config.json isn't the right format

SDXL not working

I am using the SDXL mode and it is stuck - does not process the image, see terminal window response:

python src/sdxl.py -v -t prompts.txt --size 1280-768 -fs 5 -lb 0.8
Loading pipeline components...: 100%|████████████████████████████████████████████████████| 7/7 [00:01<00:00,  4.95it/s]
 sd xl .. 50 .. 7.5 .. 1.0 .. 22559
[                                                  ] 0/3
Start...

FYI, the image is processed in SD1.5 mode as normal

AnimateDiff Error

Trying to run the animatediff example but I get this error. I tried less frames, but no difference.
FYI, AnimateDiff with SD and ComfyUI works fine in my computer, not sure why this is crashing.
Thanks

E:\SDFU>python src/anima.py -t "fiery dragon in a China shop" -m 15drm --frames 10 --loop
[ ] 0/23
Start...
Traceback (most recent call last):
File "E:\SDFU\src\anima.py", line 128, in
main()
File "C:\Users\sd163\Anaconda3\envs\SD\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "E:\SDFU\src\anima.py", line 118, in main
video = sd.generate(z_, cs_frames, uc_frames, **gendict).squeeze(0) # [c,f,h,w]
File "E:\SDFU\src\core\sdsetup.py", line 489, in generate
noise_pred = calc_noise(lat_in, t, conds, ukwargs)
File "E:\SDFU\src\core\sdsetup.py", line 427, in calc_noise
noises = self.unet(x, t, conds, **ukwargs).sample.chunk(bs) # pred noise residual at step t
File "C:\Users\sd163\Anaconda3\envs\SD\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "C:\Users\sd163\Anaconda3\envs\SD\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "E:\SDFU\src\core\unet_motion_model.py", line 58, in animdiff_forward
sample, res_samples = downsample_block(hidden_states=sample, temb=emb, encoder_hidden_states=encoder_hidden_states, attention_mask=attention_mask,
File "C:\Users\sd163\Anaconda3\envs\SD\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "C:\Users\sd163\Anaconda3\envs\SD\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\sd163\Anaconda3\envs\SD\lib\site-packages\diffusers\models\unets\unet_3d_blocks.py", line 1228, in forward
hidden_states = motion_module(
File "C:\Users\sd163\Anaconda3\envs\SD\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "C:\Users\sd163\Anaconda3\envs\SD\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\sd163\Anaconda3\envs\SD\lib\site-packages\diffusers\models\transformers\transformer_temporal.py", line 176, in forward
hidden_states = block(
File "C:\Users\sd163\Anaconda3\envs\SD\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "C:\Users\sd163\Anaconda3\envs\SD\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\sd163\Anaconda3\envs\SD\lib\site-packages\diffusers\models\attention.py", line 329, in forward
attn_output = self.attn1(
File "C:\Users\sd163\Anaconda3\envs\SD\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "C:\Users\sd163\Anaconda3\envs\SD\lib\site-packages\torch\nn\modules\module.py", line 1520, in call_impl
return forward_call(*args, **kwargs)
File "C:\Users\sd163\Anaconda3\envs\SD\lib\site-packages\diffusers\models\attention_processor.py", line 522, in forward
return self.processor(
File "C:\Users\sd163\Anaconda3\envs\SD\lib\site-packages\diffusers\models\attention_processor.py", line 1191, in call
hidden_states = xformers.ops.memory_efficient_attention(
File "C:\Users\sd163\Anaconda3\envs\SD\lib\site-packages\xformers\ops\fmha_init.py", line 247, in memory_efficient_attention
return memory_efficient_attention(
File "C:\Users\sd163\Anaconda3\envs\SD\lib\site-packages\xformers\ops\fmha_init.py", line 365, in _memory_efficient_attention
return memory_efficient_attention_forward(
File "C:\Users\sd163\Anaconda3\envs\SD\lib\site-packages\xformers\ops\fmha_init.py", line 385, in memory_efficient_attention_forward
out, * = op.apply(inp, needs_gradient=False)
File "C:\Users\sd163\Anaconda3\envs\SD\lib\site-packages\xformers\ops\fmha\cutlass.py", line 202, in apply
return cls.apply_bmhk(inp, needs_gradient=needs_gradient)
File "C:\Users\sd163\Anaconda3\envs\SD\lib\site-packages\xformers\ops\fmha\cutlass.py", line 266, in apply_bmhk
out, lse, rng_seed, rng_offset = cls.OPERATOR(
File "C:\Users\sd163\Anaconda3\envs\SD\lib\site-packages\torch_ops.py", line 755, in call
return self._op(*args, **(kwargs or {}))
RuntimeError: CUDA error: invalid configuration argument
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

LatentBlending - No Attribute "use_lcm"

I try to run the example: >python src/latwalk.py --img_ref _in/pix --latblend 0.8 --size 1024-576
But I get this error:

Traceback (most recent call last):
  File "E:\SDFU\src\latwalk.py", line 200, in <module>
    main()
  File "C:\Users\sd163\Anaconda3\envs\SD\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "E:\SDFU\src\latwalk.py", line 180, in main
    lb.run_transition(W, H, 1.- a.latblend, a.fstep, reuse = i>0)
  File "E:\SDFU\src\core\latblend.py", line 137, in run_transition
    lats1 = self.compute_latents1() if not reuse or len(self.tree_lats[-1]) != self.steps else self.tree_lats[-1]
  File "E:\SDFU\src\core\latblend.py", line 170, in compute_latents1
    lats1 = self.run_diffusion(cond, pool_c, im_cond, lat_start)
  File "C:\Users\sd163\Anaconda3\envs\SD\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "E:\SDFU\src\core\latblend.py", line 371, in run_diffusion
    if self.sd.use_kdiff or self.isxl or self.use_lcm: # trailing (lcm) or k- schedulers require reset on every generation
AttributeError: 'LatentBlending' object has no attribute 'use_lcm'

Minor comments...

Looks like examples refer to inpainting models which are not downloaded by the download.py script (commented out). I removed the commenting and they seem to download (and work) ok.

Installing xformers as instructed did not work in practice (resulted in cpu only version) while pip install xformers works ok.
(A bit unsure whether "Install CUDA11.8" means install from NVIDIA into /usr/local/, I did not as I prefer a conda-only install and it appears to work anyway).

'img_list' is not defined, and more errors when trying text to video

running this..
python src/anima.py -t text.txt -imr _in/pix -m 15drm --frames 200

results in this error,

File "C:\Users\ga_ma\Desktop\SDfu\SDfu\SDfu\src\anima.py", line 128, in
main()
File "C:\Users\ga_ma\pinokio\bin\miniconda\envs\SD\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "C:\Users\ga_ma\Desktop\SDfu\SDfu\SDfu\src\anima.py", line 61, in main
img_conds = sd.img_cus(a.img_ref, isset(a, 'allref')) # list of [2,1,1024]
File "C:\Users\ga_ma\Desktop\SDfu\SDfu\SDfu\src\core\sdsetup.py", line 284, in img_cus
img_conds = [self.img_cu(load_img(im, 224, tensor=False)[0]) for im in img_list(img_path)] # list N of [2,1,..]
NameError: name 'img_list' is not defined. Did you mean: 'file_list'?

but I only got this far by not installing xformers.. when I did install xformers, I got this error,

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torchaudio 2.2.1+cu118 requires torch==2.2.1+cu118, but you have torch 2.2.0 which is incompatible.
torchvision 0.17.1+cu118 requires torch==2.2.1+cu118, but you have torch 2.2.0 which is incompatible.
Successfully installed torch-2.2.0 xformers-0.0.24

(SD) C:\Users\ga_ma\Desktop\SDfu\SDfu\SDfu>python src/anima.py -t text.txt -imr in/pix -m 15drm --frames 200
WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
PyTorch 2.2.0+cu121 with CUDA 1201 (you have 2.2.0+cpu)
Python 3.10.11 (you have 3.10.13)
Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
Memory-efficient attention, SwiGLU, sparse and more won't be available.
Set XFORMERS_MORE_DETAILS=1 for more details
Traceback (most recent call last):
File "C:\Users\ga_ma\Desktop\SDfu\SDfu\SDfu\src\anima.py", line 14, in
from core.sdsetup import SDfu, device
File "C:\Users\ga_ma\Desktop\SDfu\SDfu\SDfu\src\core\sdsetup.py", line 19, in
from .utils import load_img, makemask, isok, isset, progbar, file_list
File "C:\Users\ga_ma\Desktop\SDfu\SDfu\SDfu\src\core\utils.py", line 16, in
from .txt2mask import Txt2Mask
File "C:\Users\ga_ma\Desktop\SDfu\SDfu\SDfu\src\core\txt2mask.py", line 36, in
from torchvision import transforms
File "C:\Users\ga_ma\pinokio\bin\miniconda\envs\SD\lib\site-packages\torchvision_init.py", line 6, in
from torchvision import _meta_registrations, datasets, io, models, ops, transforms, utils
File "C:\Users\ga_ma\pinokio\bin\miniconda\envs\SD\lib\site-packages\torchvision_meta_registrations.py", line 164, in
def meta_nms(dets, scores, iou_threshold):
File "C:\Users\ga_ma\pinokio\bin\miniconda\envs\SD\lib\site-packages\torch\library.py", line 440, in inner
handle = entry.abstract_impl.register(func_to_register, source)
File "C:\Users\ga_ma\pinokio\bin\miniconda\envs\SD\lib\site-packages\torch_library\abstract_impl.py", line 30, in register
if torch._C._dispatch_has_kernel_for_dispatch_key(self.qualname, "Meta"):
RuntimeError: operator torchvision::nms does not exist