irisrainbowneko/dreamartist-sd-webui-extension

DreamArtist for Stable-Diffusion-webui extension

DreamArtist++

DreamArtist++ for training lora with just one image has been released, try it now:

HCP-Diffusion

All future updates of the DreamArtist series will be released in this new framework.

DreamArtist (webui Eextension)

Paper:

This repo is the official *Stable-Diffusion-webui extension version implementation of "DreamArtist: Towards Controllable One-Shot Text-to-Image Generation via Contrastive Prompt-Tuning" with Stable-Diffusion-webui.

Standalone version: DreamArtist

Everyone is an artist. Rome wasn't built in a day, but your artist dreams can be!

With just one training image DreamArtist learns the content and style in it, generating diverse high-quality images with high controllability. Embeddings of DreamArtist can be easily combined with additional descriptions, as well as two learned embeddings.

Setup and Running

Clone this repo to extension folder.

git clone https://github.com/7eu7d7/DreamArtist-sd-webui-extension.git extensions/DreamArtist

Training and Usage

First create the positive and negative embeddings in DreamArtist Create Embedding Tab.

Preview Setting

After that, the names of the positive and negative embedding ({name} and {name}-neg) should be filled into the txt2img Tab with some common descriptions. This will ensure a correct preview image.

Train

Then, select positive embedding and set the parameters and image folder path in the DreamArtist Train Tab to start training. The corresponding negative embedding is loaded automatically. If your VRAM is low or you want save time, you can uncheck the reconstruction.

Recommended parameters

better to train without filewords

Remember to check the option below, otherwise the preview is wrong.

Inference

Fill the trained positive and negative embedding into txt2img to generate with DreamArtist prompt.

Attention Mask

Attention Mask can strengthen or weaken the learning intensity of some local areas. Attention Mask is a grayscale image whose grayscale values are related to the learning intensity show in the following table.

grayscale	0%	25%	50%	75%	100%
intensity	0%	50%	100%	300%	500%

The Attention Mask is in the same folder as the training image and its name is the name of the training image + "_att". You can choose whether to enable Attention Mask for training.

Since there is a self-attention operation in VAE, it may change the distribution of features. In the Process Att-Map tab, it can superimpose the attention map of self-attention on the original Att-Map.

Dynamic CFG

Dynamic CFG can improve the performance, especially when the data set is large (>20). For example, linearly from 1.5 to 3.0 (1.5-3.0), or with a 0-π/2 cycle of cosine (1.5-3.0:cos), or with a -π/2-0 cycle of cosine (1.5-3.0:cos2). Or you can also customize non-linear functions, such as 2.5-3.5:torch.sqrt(rate), where rate is a variable from 0-1.

Tested models (need ema version):

• Stable Diffusion v1.4
• Stable Diffusion v1.5
• animefull-latest
• Anything v3.0
• momoko-e

Embeddings can be transferred between different models of the same dataset.

Pre-trained embeddings:

Download

Name	Model	Image	embedding length (Positive, Negative)	iter	lr	cfg scale
ani-nahida	animefull-latest		3, 6	8000	0.0025	3
ani-cocomi	animefull-latest		3, 6	8000	0.0025	3
ani-gura	animefull-latest		3, 6	12000	0.0025	3
ani-g	animefull-latest		3, 10	1500	0.003	5
asty-bk	animefull-latest		3, 6	5000	0.003	3
asty-gc	animefull-latest		3, 10	1000	0.005	5
real-dog	sd v1.4		3, 3	1000	0.005	5
real-sship	sd v1.4		3, 3	3000	0.003	5
sty-cyber	sd v1.4		3, 5	15000	0.0025	5
sty-shuimo	sd v1.4		3, 5	15000	0.0025	5

Style Clone

Prompt Compositions

Comparison on One-Shot Learning

Other Results

dreamartist-sd-webui-extension's Issues

Simplified Chinese localization 简体中文翻译问题

你好，我是webui简体中文语言包的协作者之一。

目前我们对DreamArtist的翻译选择存在困惑，想问问你对此有没有什么意见？

梦作家？梦想艺术家？梦想画家？

can't multiply sequence by non-int of type 'float'

Not too sure what's going on.
`Error completing request
Arguments: (...)

Traceback (most recent call last):
File "C:\Git\SD15\modules\call_queue.py", line 45, in f
res = list(func(*args, **kwargs))
File "C:\Git\SD15\modules\call_queue.py", line 28, in f
res = func(*args, **kwargs)
File "C:\Git\SD15\extensions\DreamArtist-sd-webui-extension\scripts\dream_artist\ui.py", line 30, in train_embedding
embedding, filename = dream_artist.cptuning.train_embedding(*args)
File "C:\Git\SD15\extensions\DreamArtist-sd-webui-extension\scripts\dream_artist\cptuning.py", line 626, in train_embedding
processed = processing.process_images(p)
File "C:\Git\SD15\modules\processing.py", line 470, in process_images
res = process_images_inner(p)
File "C:\Git\SD15\modules\processing.py", line 575, in process_images_inner
samples_ddim = p.sample(conditioning=c, unconditional_conditioning=uc, seeds=seeds, subseeds=subseeds, subseed_strength=p.subseed_strength, prompts=prompts)
File "C:\Git\SD15\modules\processing.py", line 707, in sample
samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
File "C:\Git\SD15\modules\sd_samplers.py", line 527, in sample
samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={
File "C:\Git\SD15\modules\sd_samplers.py", line 439, in launch_sampling
return func()
File "C:\Git\SD15\modules\sd_samplers.py", line 527, in
samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={
File "C:\Git\SD15\venv\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "C:\Git\SD15\repositories\k-diffusion\k_diffusion\sampling.py", line 145, in sample_euler_ancestral
denoised = model(x, sigmas[i] * s_in, **extra_args)
File "C:\Git\SD15\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "C:\Git\SD15\modules\sd_samplers.py", line 354, in forward
denoised = self.combine_denoised(x_out, conds_list, uncond, cond_scale)
File "C:\Git\SD15\modules\sd_samplers.py", line 309, in combine_denoised
denoised[i] += (x_out[cond_index] - denoised_uncond[i]) * (weight * cond_scale)
TypeError: can't multiply sequence by non-int of type 'float'`

I have followed instructions and made a positive prompt and a negative prompt including the embeddings in the txt2img tab.
Files are captioned.

Removing DreamArtists requires you to reinstall stable diffusion

basically the title but the act of disabling dreamArtist causes the entire stable diffusion to become unusable unless you delete it all and reinstall it.

web-ui cannot start without extension present

Due to the changes made in ui.py, disabling the extension means that the web-ui will no longer start.

It may be a bug on Automatic's part, but the hooks seem to get added back in even if the extension is disabled (though I haven't re-tested that yet).

So currently it requires replacing ui.py and deleting the extension before opening the web ui, to allow it to work again.

Any successful result replication?

Hey guys, I am just wondering if anyone has successfully replicated the 1 image embedding and recreated similar results from 7eu7d7?
Right now I have no luck testing it myself.

Training time for the embedding takes around 2.5 hours on my 3090 GPU for 8000 steps. Some results only resemble a bit of that 1 training image.

AttributeError: module 'modules.script_callbacks' has no attribute 'on_ui_train_tabs'

I get the following when launching the webui:

Error loading script: dream_artist_main.py
Traceback (most recent call last):
  File "/home/username/Stab-Diff/AUTOMATIC1111/stable-diffusion-webui/modules/scripts.py", line 170, in load_scripts
    exec(compiled, module.__dict__)
  File "/home/username/Stab-Diff/AUTOMATIC1111/stable-diffusion-webui/extensions/DreamArtist-sd-webui-extension/scripts/dream_artist_main.py", line 156, in <module>
    script_callbacks.on_ui_train_tabs(on_ui_train_tabs)
AttributeError: module 'modules.script_callbacks' has no attribute 'on_ui_train_tabs'

And there seems to be no DreamArtist tab.

Embeddings not loading

Hi @7eu7d7 , thanks for this awesome model!

I created an embed:

But when I try to load, it doesn't show up.

Did I do anything wrong?

Cheers from Stockholm

Can we get a say HuggingFace online demo?

Please?

Recommended parameters

It would be nice to have a "guide" on configuring the training params to get the same results as the previews in this repository.

For example:

• Number of vectors per token
• Number of negative vectors per token
• Initialization text (assuming I'm trying to recreate the nahida example, would I just use "girl"? Or "girl, white hair, dress, side ponytail" or something similar?)
• Learning rate
• CFG Scale
• Classifier (I'm using this https://github.com/7eu7d7/pixiv_AI_crawler/releases/tag/v2, is this correct?)
• Prompt template
• Max steps (how long do we have to train to see a decent result?)
• Any other options in the webui like "Stop At last layers of CLIP model"

Thanks for the work on this extension!

Can't get tab to show in GUI

I followed the instructions and git clone https://github.com/7eu7d7/DreamArtist-sd-webui-extension.git extensions/DreamArtist load it up and nothing. Also, get an error when I try and update Webui extensions but I'm assuming that's because it's not in the WebUI list of extensions. Any help would be welcomed though thank you.

File "C:\AiArt\NewSD\stable-diffusion-webui\modules\ui_extensions.py", line 59, in check_updates
ext.check_updates()
File "C:\AiArt\NewSD\stable-diffusion-webui\modules\extensions.py", line 59, in check_updates
for fetch in repo.remote().fetch("--dry-run"):
File "C:\AiArt\NewSD\stable-diffusion-webui\venv\lib\site-packages\git\remote.py", line 886, in fetch
res = self._get_fetch_info_from_stderr(proc, progress,
File "C:\AiArt\NewSD\stable-diffusion-webui\venv\lib\site-packages\git\remote.py", line 763, in _get_fetch_info_from_stderr
with open(fetch_head.abspath, 'rb') as fp:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\AiArt\NewSD\stable-diffusion-webui\extensions\DreamArtist\.git\FETCH_HEAD'

Error checking updates for stable-diffusion-webui-prompt-utilities:
Traceback (most recent call last):
File "C:\AiArt\NewSD\stable-diffusion-webui\modules\ui_extensions.py", line 59, in check_updates
ext.check_updates()
File "C:\AiArt\NewSD\stable-diffusion-webui\modules\extensions.py", line 59, in check_updates
for fetch in repo.remote().fetch("--dry-run"):
File "C:\AiArt\NewSD\stable-diffusion-webui\venv\lib\site-packages\git\remote.py", line 886, in fetch
res = self._get_fetch_info_from_stderr(proc, progress,
File "C:\AiArt\NewSD\stable-diffusion-webui\venv\lib\site-packages\git\remote.py", line 763, in _get_fetch_info_from_stderr
with open(fetch_head.abspath, 'rb') as fp:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\AiArt\NewSD\stable-diffusion-webui\extensions\stable-diffusion-webui-prompt-utilities\.git\FETCH_HEAD'

RuntimeError: Input type (torch.cuda.HalfTensor) and weight type (torch.FloatTensor) should be the same

Tried running a training session following the example on the site but I keep getting this error.

Tried launching webui with --no-half option but does not change anything.

Any idea?

Training at rate of 0.003 until step 100000
Preparing dataset...
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:02<00:00,  3.80it/s]
  0%|                                                                                                                                   | 0/100000 [00:00<?, ?it/s]
Applying xformers cross attention optimization.
Error completing request
Arguments: ('test-jungho_lee', '0.003', 1, 'D:\\dreambooth\\train_jungho_lee\\portrait-pp', 'textual_inversion', 512, 640, 100000, 500, 500, 'D:\\stable-diffusion-webui\\textual_inversion_templates\\style.txt', True, False, '', '', 20, 1, 7, -1.0, 448, 640, 5.0, '', True, True, 1, 1) {}
Traceback (most recent call last):
  File "D:\stable-diffusion-webui\modules\ui.py", line 185, in f
    res = list(func(*args, **kwargs))
  File "D:\stable-diffusion-webui\webui.py", line 54, in f
    res = func(*args, **kwargs)
  File "D:\stable-diffusion-webui\extensions\DreamArtist\scripts\dream_artist\ui.py", line 30, in train_embedding
    embedding, filename = dream_artist.cptuning.train_embedding(*args)
  File "D:\stable-diffusion-webui\extensions\DreamArtist\scripts\dream_artist\cptuning.py", line 413, in train_embedding
    x_samples_ddim = shared.sd_model.decode_first_stage.__wrapped__(shared.sd_model, output[2])  # forward with grad
  File "D:\stable-diffusion-webui\repositories\stable-diffusion\ldm\models\diffusion\ddpm.py", line 763, in decode_first_stage
    return self.first_stage_model.decode(z)
  File "D:\stable-diffusion-webui\repositories\stable-diffusion\ldm\models\autoencoder.py", line 331, in decode
    z = self.post_quant_conv(z)
  File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\conv.py", line 457, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\conv.py", line 453, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Input type (torch.cuda.HalfTensor) and weight type (torch.FloatTensor) should be the same

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument weight in method wrapper__convolution_backward)

Trying to train locally,but got error
Preparing dataset...
100%|██████████████████████████████████████████████████████████████████████████████████| 14/14 [00:02<00:00, 5.10it/s]
use convnext discriminator
0%| | 0/100000 [00:02<?, ?it/s]
Error completing request
Arguments: ('mmkex', '0.005', 1, 'D:\Grabber\mmkapt', 'textual_inversion', 128, 128, 100000, 500, 500, 'D:\stable-diffusion-webui\textual_inversion_templates\style_filewords.txt', True, False, '', '', 20, 0, 7, -1.0, 512, 512, 5.0, 'D:\stable-diffusion-webui\models\convnext\checkpoint-best_t5.pth', True, False, 1, 1) {}
Traceback (most recent call last):
File "D:\stable-diffusion-webui\modules\ui.py", line 185, in f
res = list(func(*args, **kwargs))
File "D:\stable-diffusion-webui\webui.py", line 54, in f
res = func(*args, **kwargs)
File "D:\stable-diffusion-webui\extensions\DreamArtist-sd-webui-extension\scripts\dream_artist\ui.py", line 30, in train_embedding
embedding, filename = dream_artist.cptuning.train_embedding(*args)
File "D:\stable-diffusion-webui\extensions\DreamArtist-sd-webui-extension\scripts\dream_artist\cptuning.py", line 430, in train_embedding
loss.backward()
File "D:\stable-diffusion-webui\venv\lib\site-packages\torch_tensor.py", line 396, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\autograd_init_.py", line 173, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument weight in method wrapper__convolution_backward)

Crash during image generation when checking Read parameters from txt2img tab

When checking the Read parameters (prompt, etc...) from txt2img tab when making previews box it will crash during image generation.
Full Crash:

Error completing request
Arguments: ('testv2', '0.005', 1, '/home/bunny/stable-diffusion-webui/dataset/', 'textual_inversion', 512, 512, 100000, 5, 500, '/home/bunny/stable-diffusion-webui/textual_inversion_templates/none.txt', True, True, '', '', 27, 0, 7, 1337.0, 512, 512, 5.0, '', True, True, 1, 1) {}
Traceback (most recent call last):
  File "/home/bunny/stable-diffusion-webui/modules/ui.py", line 185, in f
    res = list(func(*args, **kwargs))
  File "/home/bunny/stable-diffusion-webui/webui.py", line 54, in f
    res = func(*args, **kwargs)
  File "/home/bunny/stable-diffusion-webui/extensions/DreamArtist/scripts/dream_artist/ui.py", line 30, in train_embedding
    embedding, filename = dream_artist.cptuning.train_embedding(*args)
  File "/home/bunny/stable-diffusion-webui/extensions/DreamArtist/scripts/dream_artist/cptuning.py", line 489, in train_embedding
    processed = processing.process_images(p)
  File "/home/bunny/stable-diffusion-webui/modules/processing.py", line 423, in process_images
    res = process_images_inner(p)
  File "/home/bunny/stable-diffusion-webui/modules/processing.py", line 442, in process_images_inner
    file.write(processed.infotext(p, 0))
  File "/home/bunny/stable-diffusion-webui/modules/processing.py", line 281, in infotext
    return create_infotext(p, self.all_prompts, self.all_seeds, self.all_subseeds, comments=[], position_in_batch=index % self.batch_size, iteration=index // self.batch_size)
  File "/home/bunny/stable-diffusion-webui/modules/processing.py", line 411, in create_infotext
    negative_prompt_text = "\nNegative prompt: " + p.negative_prompt if p.negative_prompt else ""
TypeError: can only concatenate str (not "tuple") to str

indices should be either on cpu or on the same device as the indexed tensor (cpu)

Every extension works fine except this one, its just the default webui code i didnt change anything

/notebooks/stable-diffusion-webui
Patching transformers to fix kwargs errors.
Dreambooth API layer loaded
Aesthetic Image Scorer: Unable to load Windows tagging script from tools directory
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
Loading weights [81761151] from /notebooks/stable-diffusion-webui/models/Stable-diffusion/v1-5-pruned-emaonly.ckpt
Global Step: 840000
Applying xformers cross attention optimization.
Model loaded.
no display name and no $DISPLAY environment variable
Loaded a total of 6 textual inversion embeddings.
Embeddings: bad_prompt_version2, gadget, testInversion, bad_prompt, testest-neg, testest
Running on local URL:  http://127.0.0.1:7860/
Running on public URL: $$$$$$$

This share link expires in 72 hours. For free permanent hosting and GPU upgrades (NEW!), check out Spaces: https://huggingface.co/spaces
Loading weights [e02601f3] from /notebooks/stable-diffusion-webui/models/Stable-diffusion/sd_v1-5_vae.ckpt
Applying xformers cross attention optimization.
Weights loaded.
Training at rate of 0.005 until step 8000
Preparing dataset...
100%|█████████████████████████████████████████████| 5/5 [00:01<00:00,  4.97it/s]
  0%|                                                  | 0/8000 [00:00<?, ?it/s]
Applying xformers cross attention optimization.
Error completing request
Arguments: ('testest', '0.005', 1, '/notebooks/kohya_ss/train_data/conceptart/', 'dream_artist', 512, 512, 8000, 500, 500, '/notebooks/stable-diffusion-webui/textual_inversion_templates/style.txt', True, False, '', '', 20, 0, 7, -1.0, 512, 512, 3.0, '', True, True, 1, 1, 1.0, 25.0, 1.0, 25.0, 0.9, 0.999, False, 1) {}
Traceback (most recent call last):
  File "/notebooks/stable-diffusion-webui/modules/ui.py", line 185, in f
    res = list(func(*args, **kwargs))
  File "/notebooks/stable-diffusion-webui/webui.py", line 57, in f
    res = func(*args, **kwargs)
  File "/notebooks/stable-diffusion-webui/extensions/DreamArtist-sd-webui-extension/scripts/dream_artist/ui.py", line 30, in train_embedding
    embedding, filename = dream_artist.cptuning.train_embedding(*args)
  File "/notebooks/stable-diffusion-webui/extensions/DreamArtist-sd-webui-extension/scripts/dream_artist/cptuning.py", line 440, in train_embedding
    output = shared.sd_model(x, c_in, scale=cfg_scale)
  File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/notebooks/stable-diffusion-webui/repositories/stable-diffusion/ldm/models/diffusion/ddpm.py", line 879, in forward
    return self.p_losses(x, c, t, *args, **kwargs)
  File "/notebooks/stable-diffusion-webui/extensions/DreamArtist-sd-webui-extension/scripts/dream_artist/cptuning.py", line 287, in p_losses_hook
    logvar_t = self.logvar[t_raw].to(self.device)
RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)

Batch size doesn't work

Batch size doesn't work:
RuntimeError: einsum(): operands do not broadcast with remapped shapes [original->remapped]: [16, 4096, 40]->[16, 4096, 1, 40] [32, 77, 40]->[32, 1, 77, 40]

Embedding changes visuals and styles when token is above 75.

So I was working on a new embed, at first it worked flawlessly under 75 tokens. When I was testing for more details and hit 89 tokens, the style and everything changed drastically across all sampling methods.

When I removed the embedding (both prompt and negative), it returns back to normal (except the trained parts).

I has been happening to quite a lot of my friends too.

Cannot load embeddings on M1 Pro Macbook

I created an embedding named keqing, initialization text is girl, initialization text(negative) is *, on my M1 Pro macbook.
When restart the web-ui to loading the embeddings, the error was:

Warning: caught exception 'Torch not compiled with CUDA enabled', memory monitor disabled
No module 'xformers'. Proceeding without it.
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Loading weights [7d13c797] from /Users/lxy/PycharmProjects/stable-diffusion-webui/models/Stable-diffusion/nsfw_pro.ckpt
Applying cross attention optimization (InvokeAI).
Model loaded.
Error verifying pickled file from /Users/lxy/PycharmProjects/stable-diffusion-webui/embeddings/keqing.pt:
Traceback (most recent call last):
  File "/Users/lxy/PycharmProjects/stable-diffusion-webui/modules/safe.py", line 131, in load_with_extra
    check_pt(filename, extra_handler)
  File "/Users/lxy/PycharmProjects/stable-diffusion-webui/modules/safe.py", line 89, in check_pt
    unpickler.load()
  File "/Users/lxy/PycharmProjects/stable-diffusion-webui/modules/safe.py", line 62, in find_class
    raise Exception(f"global '{module}/{name}' is forbidden")
Exception: global 'torch._utils/_rebuild_device_tensor_from_numpy' is forbidden


The file may be malicious, so the program is not going to read it.
You can skip this check with --disable-safe-unpickle commandline argument.


Error loading emedding keqing.pt:
Traceback (most recent call last):
  File "/Users/lxy/PycharmProjects/stable-diffusion-webui/modules/textual_inversion/textual_inversion.py", line 134, in load_textual_inversion_embeddings
    process_file(fullfn, fn)
  File "/Users/lxy/PycharmProjects/stable-diffusion-webui/modules/textual_inversion/textual_inversion.py", line 104, in process_file
    if 'string_to_param' in data:
TypeError: argument of type 'NoneType' is not iterable

Error verifying pickled file from /Users/lxy/PycharmProjects/stable-diffusion-webui/embeddings/keqing-neg.pt:
Traceback (most recent call last):
  File "/Users/lxy/PycharmProjects/stable-diffusion-webui/modules/safe.py", line 131, in load_with_extra
    check_pt(filename, extra_handler)
  File "/Users/lxy/PycharmProjects/stable-diffusion-webui/modules/safe.py", line 89, in check_pt
    unpickler.load()
  File "/Users/lxy/PycharmProjects/stable-diffusion-webui/modules/safe.py", line 62, in find_class
    raise Exception(f"global '{module}/{name}' is forbidden")
Exception: global 'torch._utils/_rebuild_device_tensor_from_numpy' is forbidden


The file may be malicious, so the program is not going to read it.
You can skip this check with --disable-safe-unpickle commandline argument.


Error loading emedding keqing-neg.pt:
Traceback (most recent call last):
  File "/Users/lxy/PycharmProjects/stable-diffusion-webui/modules/textual_inversion/textual_inversion.py", line 134, in load_textual_inversion_embeddings
    process_file(fullfn, fn)
  File "/Users/lxy/PycharmProjects/stable-diffusion-webui/modules/textual_inversion/textual_inversion.py", line 104, in process_file
    if 'string_to_param' in data:
TypeError: argument of type 'NoneType' is not iterable

Loaded a total of 0 textual inversion embeddings.
Embeddings: 
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.

My pytorch version is as follows:
open-clip-torch 2.7.0
pytorch-lightning 1.7.6
torch 1.12.1
torchdiffeq 0.2.3
torchmetrics 0.10.3
torchsde 0.2.5
torchvision 0.13.1

Here are some related issues I found:
AUTOMATIC1111/stable-diffusion-webui#5634
https://discuss.pytorch.org/t/cant-get-attribute-rebuild-parameter-on-module-torch-utils/32603
AUTOMATIC1111/stable-diffusion-webui#5431

请问训练时是需要使用启用EMA权重的模型吗？

比如说，ND的final（925997e9）模型是latest（e68e1fc）仅剩EMA权重的修剪版，那么是否意味着我可以直接使用final模型进行训练而不必加载latest模型的config文件来启用EMA权重？

还有一个问题就是any3.0的原版模型是全量模型吗？该怎么确定一个模型是否有启用ema权重呢？

How many Steps do you reckon?

he is at 3000, asking for a friend?

Can't train at all in Colab, said there are tensors on two devices, happens in "Train with reconstruction"

I can't seem to run DreamArtist in a fresh install because of this error, and I can't seem to find out where did it went wrong. Got any ideas on how to fix this? Thanks in advance.

Traceback (most recent call last):
  File "/content/stable-diffusion-webui/modules/call_queue.py", line 45, in f
    res = list(func(*args, **kwargs))
  File "/content/stable-diffusion-webui/modules/call_queue.py", line 28, in f
    res = func(*args, **kwargs)
  File "/content/stable-diffusion-webui/extensions/DreamArtist-sd-webui-extension/scripts/dream_artist/ui.py", line 30, in train_embedding
    embedding, filename = dream_artist.cptuning.train_embedding(*args)
  File "/content/stable-diffusion-webui/extensions/DreamArtist-sd-webui-extension/scripts/dream_artist/cptuning.py", line 542, in train_embedding
    loss.backward()
  File "/usr/local/envs/automatic/lib/python3.10/site-packages/torch/_tensor.py", line 396, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/usr/local/envs/automatic/lib/python3.10/site-packages/torch/autograd/__init__.py", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument weight in method wrapper__convolution_backward)

Can you feed many images when training an embedding?

I saw this text while training an embedding for the shuimo embedding.
I named the embedding test1 and give it access to images folder with only one image inside it (shuimo.jpg).

So I speculated that you can feed more than one image to train an embedding.
Is it true that I can feed more images to training process and would that theoretically improve performance?

Error loading script: dream_artist_main.py

Error loading script: dream_artist_main.py
Traceback (most recent call last):
File "D:\stable-diffusion-webui\modules\scripts.py", line 170, in load_scripts
exec(compiled, module.dict)
File "D:\stable-diffusion-webui\scripts\dream_artist_main.py", line 135, in
script_callbacks.on_ui_train_tabs(on_ui_train_tabs)
AttributeError: module 'modules.script_callbacks' has no attribute 'on_ui_train_tabs'

Traceback (most recent call last):
File "D:\stable-diffusion-webui\launch.py", line 252, in
start()
File "D:\stable-diffusion-webui\launch.py", line 247, in start
webui.webui()
File "D:\stable-diffusion-webui\webui.py", line 149, in webui
demo = modules.ui.create_ui(wrap_gradio_gpu_call=wrap_gradio_gpu_call)
File "D:\stable-diffusion-webui\modules\ui.py", line 1285, in create_ui
if hasattr(params, "dream_artist_trigger"): params.dream_artist_trigger()
NameError: name 'params' is not defined

No module called launch - Colab Error

Hi, went to run this via colab, normally I don't have any issues with the colab running.
Installing the extension gives the following error:

Connected Error running install.py for extension /content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/DreamArtist-sd-webui-extension. Command: "/usr/bin/python3" "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/DreamArtist-sd-webui-extension/install.py" Error code: 1 stdout: <empty> stderr: Traceback (most recent call last): File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/DreamArtist-sd-webui-extension/install.py", line 1, in <module> import launch ModuleNotFoundError: No module named 'launch'

If I restart the webui, it does look like it loads, but I'm unable to train as it throws an error or runs out of memory.
The colab has 16GB of vRAM.

Thanks!

About subject textual inversion

你好~
当我使用dreamart提出的正负embedding训练方法对风格图片进行训练的时候，能够正常得到对应的风格embedding。
但是当我使用具体实物图片进行subject训练的时候，无法得到对应的正确embedding，想咨询下是哪里出了问题~
我的训练参数：
'train_args': ['real-sship', '0.003', 1, '/path/to/stable-diffusion-webui/textual_inversion_traindata',
'dream_artist', 512, 512, 3000, 100, 100, '/path/to/stable-diffusion-webui/textual_inversion_templates/subject.txt',
True, False, '', '', 20, 0, 7, -1.0, 512, 512,'5.0','',True, True,1,1,0.97,25.0,0.97,25.0,0.9,0.999,False,1,False,'0.000005']
}
正常使用stable diffusion代码训练embedding能够得到正常结果。

Preview generation does not use selected sampler due to recent WebUI commit changing sampler_index to sampler_name

Currently when using Read parameters (prompt, etc...) from txt2img tab when making previews, the sampler selected is ignored and instead the default ("Euler a") sampler is used. This is due to this recent commit to WebUI that changed sampler_index to sampler_name.
AUTOMATIC1111/stable-diffusion-webui@cdc8020#diff-d3503031ef91fb35651a650f994dd8c94d405fe8e690c41817b1d095d66b1c69R348
The one line in this repo affected is this.
https://github.com/7eu7d7/DreamArtist-sd-webui-extension/blob/069ce4124580c69dc2d373a2a9e856b08afa715f/scripts/dream_artist/cptuning.py#L516
I believe this issue can be solved by matching textual_inversion.py by importing sd_samplers from modules and changing p.sampler_index = preview_sampler_index to p.sampler_name = sd_samplers.samplers[preview_sampler_index].name. I will test to see if this fixes it and the correct sampler is respected when generating previews. I would make a PR, but I'm not sure how you'll want to handle backwards compatibility if people aren't on the latest WebUI. E.g. should p.sampler_index still be left along side the new fixed p.sampler_name entry.
Edit: Adding sd_samplers after the images import and changing the line at 516 in cptuning.py to reflect the above change resolves the preview sampler issue and the correct one is used.

RuntimeError: CUDA out of memory

I'm using RTX3080 10GB.
tried DreamArtist but the console says OOM error message...
so how much vram do I need?

tried uncheck 'Train with reconstruction'

the error message is like below
Preparing dataset... 100%|████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:03<00:00, 3.79s/it] 0%| | 0/100000 [00:01<?, ?it/s] Applying xformers cross attention optimization. Error completing request Arguments: ('test1', '0.005', 1, 'D:\\stable-diffusion-webui\\image_source\\girl', 'textual_inversion', 512, 512, 100000, 500, 500, 'D:\\stable-diffusion-webui\\textual_inversion_templates\\style_filewords.txt', True, False, '', '', 20, 0, 7, -1.0, 512, 512, 5.0, '', True, False, 1, 1) {} Traceback (most recent call last): File "D:\stable-diffusion-webui\modules\ui.py", line 185, in f res = list(func(*args, **kwargs)) File "D:\stable-diffusion-webui\webui.py", line 54, in f res = func(*args, **kwargs) File "D:\stable-diffusion-webui\extensions\DreamArtist\scripts\dream_artist\ui.py", line 30, in train_embedding embedding, filename = dream_artist.cptuning.train_embedding(*args) File "D:\stable-diffusion-webui\extensions\DreamArtist\scripts\dream_artist\cptuning.py", line 430, in train_embedding loss.backward() File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\_tensor.py", line 396, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\__init__.py", line 173, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\function.py", line 253, in apply return user_fn(self, *args) File "D:\stable-diffusion-webui\repositories\stable-diffusion\ldm\modules\diffusionmodules\util.py", line 139, in backward input_grads = torch.autograd.grad( File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\__init__.py", line 276, in grad return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass RuntimeError: CUDA out of memory. Tried to allocate 1024.00 MiB (GPU 0; 10.00 GiB total capacity; 6.31 GiB already allocated; 0 bytes free; 7.84 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

(btw, thanks for this wonderful project)

Prompt File Template / Image Captions / Text2Img Tokens

Dreamartist may be working better than regular TI but with the lack of setup details, i can't seem to get it to work. That's why i have a few questions for the authors of it or whoever can help :

1. what should we use for the prompt file template? style_filewords.txt, or style.txt or subject.txt? I'm trying to train some new poses and also some new concepts (like a cyclops - man with one eye) - what should i be using for each of them?
2. do we need image captions for the training images or are they helpful?
3. what sort of prompt we need to use (and negative prompt also)? just the positive or negative embedding name or should we add more details to each prompt?

hyper_params.json not found for embedding training

I cannot run train embedding after created it because of missing hyper_param.json.
How do I create it? or I have to prepare it before training?

Error completing request
Arguments: ('my_keyword', '0.005', 1, '/workspace/my_keyword', 'textual_inversion', 576, 1024, 100, 0, 0, '/workspace/my_keyword/prompt-template.txt', True, True, '', '', 20, 0, 7, -1.0, 512, 512, 10.0, '', True, True, 1, 1) {}
Traceback (most recent call last):
  File "/stable-diffusion-webui/modules/ui.py", line 185, in f
    res = list(func(*args, **kwargs))
  File "/stable-diffusion-webui/webui.py", line 54, in f
    res = func(*args, **kwargs)
  File "/stable-diffusion-webui/extensions/DreamArtist/scripts/dream_artist/ui.py", line 30, in train_embedding
    embedding, filename = dream_artist.cptuning.train_embedding(*args)
  File "/stable-diffusion-webui/extensions/DreamArtist/scripts/dream_artist/cptuning.py", line 367, in train_embedding
    with open(os.path.join(log_directory, 'hyper_param.json'), 'w') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'textual_inversion/2022-11-16/my_keyword/hyper_param.json'

Training does not work AttributeError: 'NoneType' object has no attribute 'detach

The following error message appears and training cannot be started. Is this due to a misconfiguration?
Note: This log is from standalone DreamArtist, but the exact same error was output in Extension.

Training at rate of 0.005 until step 30000
Preparing dataset...
100%|██████████████████████████████████████| 1/1 [00:05<00:00,  5.43s/it]
  0%|                                          | 0/30000 [00:01<?, ?it/s]
Error completing request
Arguments: ('emma', '0.005', 1, 'D:\\StableDiffusion\\work\\APT\\emma', 'dream_artist', 512, 512, 30000, 1000, 1000, 'D:\\SDWU\\DreamArtist-stable-diffusion\\textual_inversion_templates\\default.txt', True, True, '', '', 15, 0, 7, -1.0, 512, 512, 5.0, '', True, False, 1, 1) {}
Traceback (most recent call last):
  File "D:\SDWU\DreamArtist-stable-diffusion\modules\ui.py", line 185, in f
    res = list(func(*args, **kwargs))
  File "D:\SDWU\DreamArtist-stable-diffusion\webui.py", line 54, in f
    res = func(*args, **kwargs)
  File "D:\SDWU\DreamArtist-stable-diffusion\modules\dream_artist\ui.py", line 36, in train_embedding
    embedding, filename = modules.dream_artist.cptuning.train_embedding(*args)
  File "D:\SDWU\DreamArtist-stable-diffusion\modules\dream_artist\cptuning.py", line 436, in train_embedding
    f"grad:{embedding.vec.grad.detach().cpu().abs().mean().item():.7f}, "
AttributeError: 'NoneType' object has no attribute 'detach'

Error after latest update

AttributeError: 'UiTrainTabParams' object has no attribute 'dream_artist_trigger'

Hello!
Since the pull of the latest update of your repo the webui-user.bat is crashing when i am trying to start it with the error copied to the end of this text.
Sadly i was not able to train anything successful with your extension yet. Everything seems to be worse then without it.

Could you include the black and white dog´s example files to your downloadable embeddings, please? I would like to try with him and compare. Indeed, it is my dog and his name is Timmy :)
Thank you very much in advance!!

Here´s the error i get when launching the .bat file:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ D:\AI-Painting\01_stable-diffusion-webui\launch.py:256 in │
│ │
│ 253 │
│ 254 if name == "main": │
│ 255 │ prepare_enviroment() │
│ ❱ 256 │ start() │
│ 257 │
│ │
│ D:\AI-Painting\01_stable-diffusion-webui\launch.py:251 in start │
│ │
│ 248 │ if '--nowebui' in sys.argv: │
│ 249 │ │ webui.api_only() │
│ 250 │ else: │
│ ❱ 251 │ │ webui.webui() │
│ 252 │
│ 253 │
│ 254 if name == "main": │
│ │
│ D:\AI-Painting\01_stable-diffusion-webui\webui.py:149 in webui │
│ │
│ 146 │ initialize() │
│ 147 │ │
│ 148 │ while 1: │
│ ❱ 149 │ │ demo = modules.ui.create_ui(wrap_gradio_gpu_call=wrap_gradio_gpu_call) │
│ 150 │ │ │
│ 151 │ │ app, local_url, share_url = demo.launch( │
│ 152 │ │ │ share=cmd_opts.share, │
│ │
│ D:\AI-Painting\01_stable-diffusion-webui\modules\ui.py:1311 in create_ui │
│ │
│ 1308 │ │ │ │ shared.ti_outcome = ti_outcome │
│ 1309 │ │ │ │ setup_progressbar(progressbar, ti_preview, 'ti', textinfo=ti_progress) │
│ 1310 │ │ │
│ ❱ 1311 │ │ params.dream_artist_trigger() │
│ 1312 │ │ create_embedding.click( │
│ 1313 │ │ │ fn=modules.textual_inversion.ui.create_embedding, │
│ 1314 │ │ │ inputs=[ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
AttributeError: 'UiTrainTabParams' object has no attribute 'dream_artist_trigger'

RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)

Weights from pretrained model not used in ConvNeXt: ['epoch', 'global_step', 'pytorch-lightning_version', 'state_dict', 'callbacks', 'lr_schedulers']
use convnext discriminator
0%| | 0/100000 [00:00<?, ?it/s]
Applying xformers cross attention optimization.
Error completing request
Arguments: ('dreamArtist', '0.003', 1, '/notebooks/testing_Dreamartist', 'textual_inversion', 512, 512, 100000, 500, 500, '/notebooks/stable-diffusion-webui/textual_inversion_templates/style.txt', True, False, '', '', 20, 0, 7, -1.0, 512, 512, 5.0, '/storage/models/v1-5-pruned-emaonly.ckpt', True, True, 1, 1) {}
Traceback (most recent call last):
File "/notebooks/stable-diffusion-webui/modules/ui.py", line 185, in f
res = list(func(*args, **kwargs))
File "/notebooks/stable-diffusion-webui/webui.py", line 54, in f
res = func(*args, **kwargs)
File "/notebooks/stable-diffusion-webui/extensions/DreamArtist-sd-webui-extension/scripts/dream_artist/ui.py", line 30, in train_embedding
embedding, filename = dream_artist.cptuning.train_embedding(*args)
File "/notebooks/stable-diffusion-webui/extensions/DreamArtist-sd-webui-extension/scripts/dream_artist/cptuning.py", line 413, in train_embedding
output = shared.sd_model(x, c_in, scale=cfg_scale)
File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1190, in _call_impl
return forward_call(*input, **kwargs)
File "/notebooks/stable-diffusion-webui/repositories/stable-diffusion/ldm/models/diffusion/ddpm.py", line 879, in forward
return self.p_losses(x, c, t, *args, **kwargs)
File "/notebooks/stable-diffusion-webui/extensions/DreamArtist-sd-webui-extension/scripts/dream_artist/cptuning.py", line 269, in p_losses_hook
logvar_t = self.logvar[t].to(self.device)
RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)

Issue on colab: logvar_t = self.logvar[t_raw].to(self.device) RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)

Training at rate of 0.0025 until step 8000
Preparing dataset...
100% 1/1 [00:00<00:00, 4.46it/s]
0% 0/8000 [00:00<?, ?it/s]
Applying cross attention optimization (Doggettx).
Error completing request
Arguments: ('NaughtV1', '0.0025', 1, '/content/1', 'dream_artist', 512, 512, 8000, 1000, 1000, '/content/stable-diffusion-webui/textual_inversion_templates/style.txt', True, True, 'blue skin, demon tail, demon wings, tail, colored skin, horns, wings, penis, demon boy, demon horns, black sclera, twintails, NaughtV1,1boy, colored sclera, demon girl, long hair, smile, solo, grey background, nude, navel, very long hair, bat wings, nail polish, erection, black nails, otoko no ko, looking at viewer, pointy ears, standing, purple nails', 'NaughtV1-neg,lowres, bad anatomy, bad hands, text, censored,error, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts,signature, watermark, username, blurry, artist name', 20, 0, 7, -1.0, 512, 512, '3', '', True, False, 1, 1, 1.0, 25.0, 1.0, 25.0, 0.9, 0.999, False, 1, False, '0.000005') {}
Traceback (most recent call last):
File "/content/stable-diffusion-webui/modules/call_queue.py", line 45, in f
res = list(func(*args, **kwargs))
File "/content/stable-diffusion-webui/modules/call_queue.py", line 28, in f
res = func(*args, **kwargs)
File "/content/stable-diffusion-webui/extensions/DreamArtist-sd-webui-extension/scripts/dream_artist/ui.py", line 30, in train_embedding
embedding, filename = dream_artist.cptuning.train_embedding(*args)
File "/content/stable-diffusion-webui/extensions/DreamArtist-sd-webui-extension/scripts/dream_artist/cptuning.py", line 520, in train_embedding
output = shared.sd_model(x, c_in, scale=(cfg_l, cfg_h), att_mask=att_mask, dy_cfg_f=dy_cfg_f)
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1190, in _call_impl
return forward_call(*input, **kwargs)
File "/content/stable-diffusion-webui/repositories/stable-diffusion-stability-ai/ldm/models/diffusion/ddpm.py", line 846, in forward
return self.p_losses(x, c, t, *args, **kwargs)
File "/content/stable-diffusion-webui/extensions/DreamArtist-sd-webui-extension/scripts/dream_artist/cptuning.py", line 305, in p_losses_hook
logvar_t = self.logvar[t_raw].to(self.device)
RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)

和dreambooth的区别

想请教一下，相比现有的dreambooth，这版主要的改进是什么呢？为什么只需要一张图就够了呢？

Seems to need --disable-safe-unpickle on embeds DA created?

Its detecting the embeds as bad, dunno why?

`venv "E:\DreamArtist-stable-diffusion\venv\Scripts\Python.exe"
Python 3.10.9 (tags/v3.10.9:1dd9be6, Dec 6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)]
Commit hash:
Installing requirements for Web UI
Launching Web UI with arguments: --xformers
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
Loading weights [925997e9] from E:\DreamArtist-stable-diffusion\models\Stable-diffusion\animefull-final-pruned.ckpt
Applying xformers cross attention optimization.
Model loaded.
Loaded a total of 0 words embeddings.
Embeddings:
Running on local URL: http://127.0.0.1:7860

To create a public link, set share=True in launch().
Error verifying pickled file from E:\DreamArtist-stable-diffusion\embeddings\Elins-neg.pt:
Traceback (most recent call last):
File "E:\DreamArtist-stable-diffusion\modules\safe.py", line 131, in load_with_extra
check_pt(filename, extra_handler)
File "E:\DreamArtist-stable-diffusion\modules\safe.py", line 84, in check_pt
check_zip_filenames(filename, z.namelist())
File "E:\DreamArtist-stable-diffusion\modules\safe.py", line 76, in check_zip_filenames
raise Exception(f"bad file inside {filename}: {name}")
Exception: bad file inside E:\DreamArtist-stable-diffusion\embeddings\Elins-neg.pt: Elins-neg/data.pkl

The file may be malicious, so the program is not going to read it.
You can skip this check with --disable-safe-unpickle commandline argument.

Error loading emedding Elins-neg.pt:
Traceback (most recent call last):
File "E:\DreamArtist-stable-diffusion\modules\dream_artist\cptuning.py", line 137, in load_words_embeddings
process_file(fullfn, fn)
File "E:\DreamArtist-stable-diffusion\modules\dream_artist\cptuning.py", line 107, in process_file
if 'string_to_param' in data:
TypeError: argument of type 'NoneType' is not iterable

Error verifying pickled file from E:\DreamArtist-stable-diffusion\embeddings\Elins.pt:
Traceback (most recent call last):
File "E:\DreamArtist-stable-diffusion\modules\safe.py", line 131, in load_with_extra
check_pt(filename, extra_handler)
File "E:\DreamArtist-stable-diffusion\modules\safe.py", line 84, in check_pt
check_zip_filenames(filename, z.namelist())
File "E:\DreamArtist-stable-diffusion\modules\safe.py", line 76, in check_zip_filenames
raise Exception(f"bad file inside {filename}: {name}")
Exception: bad file inside E:\DreamArtist-stable-diffusion\embeddings\Elins.pt: Elins/data.pkl

The file may be malicious, so the program is not going to read it.
You can skip this check with --disable-safe-unpickle commandline argument.

Error loading emedding Elins.pt:
Traceback (most recent call last):
File "E:\DreamArtist-stable-diffusion\modules\dream_artist\cptuning.py", line 137, in load_words_embeddings
process_file(fullfn, fn)
File "E:\DreamArtist-stable-diffusion\modules\dream_artist\cptuning.py", line 107, in process_file
if 'string_to_param' in data:
TypeError: argument of type 'NoneType' is not iterable

Loaded a total of 0 words embeddings.
Embeddings:`

error while training

I got that when it was training

When i click train i receive this error, i use AUTOMATIC1111 extension

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument weight in method wrapper__native_layer_norm_backward)

[Feature request] Add option to use different textual inversion templates for negative prompts

Context

I have been experimenting using filewords in prompts for DreamArtist subject training, in order to exclude certain unwanted features from the source image, the results seem to be pretty sound.

My tests are conducted with the nahida sample, without the filewords I would get results like:

It can be seen that though the results do look good, there are some weird features popping out of the images, like the mushroom from the left lower corner of the input picture; Furthermore, all generated picture are in a forest/nature setting.

Then I used deepbooru to generate a series of filewords, and deleted those that I do want in the final trained embedding, leaving the fileswords to be:

1girl, bush, fireflies, forest, grass, leaf, light_rays, log, looking_at_viewer, mushroom, nature, on_grass, outdoors, plant, smile, star_\(sky\), tree, tree_stump

Then with these filewords I trained a new embedding, producing results like:

It seems that the filewords help us completely get rid of the mushrooms and the forest setting.

Side Note: It seems that some character features are excluded as well. Like breast size, green eyes and the green theme of the clothing. My guess is that because we excluded nature and forest via filewords, it is likely that all green portions of the image are correlated with these tags and thus excluded as well. Since original picture's character breast is painted with green, excluding that the model fell back to average breast sizes of the model.

Feature request

The above example shows that filewords tuning can be a very flexible tool of manipulating DreamArtist training process (or textual inversion training in general). I am wondering if we can do something specifically for negative prompt as well?

Since we can exclude "unwanted features" from the positive embedding, can we exclude "wanted features" from the negative embedding so that they can be reinforced?

We already had Positive "filewords" only to let negative prompt to use the template without the _filewords postfix, but ideally we would want another field to specify a completely different template for the negative prompts, and then we can specify the wanted features for that particular training in a completely separate file.

This may not be a big change after all, I may submit a PR myself for it if I had time.

ValueError: some parameters appear in more than one parameter group

Training error on M1 Mac: failed assertion `Source and weight input channels mismatch'

Full output below(错误信息如下):
Training at rate of 0.003 until step 100000
Preparing dataset...
100%|████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 2.32it/s]
0%| | 0/100000 [00:00<?, ?it/s]
/AppleInternal/Library/BuildRoots/810eba08-405a-11ed-86e9-6af958a02716/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Runtimes/MPSRuntime/Operations/GPUConv2DOps.mm:214: failed assertion Source and weight input channels mismatch
./run_webui_mac.sh: line 13: 74318 Abort trap: 6 python webui.py --precision full --no-half --use-cpu Interrogate GFPGAN CodeFormer $@ --disable-safe-unpickle
/Users/szy/miniconda/envs/web-ui/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '

Additional Info:
I'm running on MacBook Pro with M1 Pro and 32 GB RAM (macOS Ventura). When I initially tried DreamArtist extension, the error I got is not the one above, instead it is something similar to #1 , output something like "Input Type of MPSTensor and weight type should be the same". After that my regular textual inversion training seemed broke, and I tried to completely reinstall webui at a different location. The above error message is what I have now. Other than that txt2img and regular training works fine.(简单来说我一开始得到的错误讯息像是 #1 , 不过Input type 是MPS Tensor。重装webui后得到的是如上的错误信息。不知道两者之间有没有关联)

This is a bug

When I import modules.textual_inversion, import error occurs. I think it should be " import modules.dream_artist".
https://github.com/7eu7d7/DreamArtist-sd-webui-extension/blob/fbc85703a182a32bf1db169d0fd88e5c2e3fb5a6/scripts/dream_artist/cptuning.py#L20

Input and model on different devices error when training with reconstruction

I got the following error on google colab when trying to train with reconstruction turned on.

  File "/content/stable-diffusion-webui/modules/ui.py", line 185, in f
    res = list(func(*args, **kwargs))
  File "/content/stable-diffusion-webui/webui.py", line 54, in f
    res = func(*args, **kwargs)
  File "/content/stable-diffusion-webui/modules/dream_artist/ui.py", line 36, in train_embedding
    embedding, filename = modules.dream_artist.cptuning.train_embedding(*args)
  File "/content/stable-diffusion-webui/modules/dream_artist/cptuning.py", line 413, in train_embedding
    x_samples_ddim = shared.sd_model.decode_first_stage.__wrapped__(shared.sd_model, output[2])  # forward with grad
  File "/content/stable-diffusion-webui/repositories/stable-diffusion/ldm/models/diffusion/ddpm.py", line 763, in decode_first_stage
    return self.first_stage_model.decode(z)
  File "/content/stable-diffusion-webui/repositories/stable-diffusion/ldm/models/autoencoder.py", line 331, in decode
    z = self.post_quant_conv(z)
  File "/usr/local/envs/automatic/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/envs/automatic/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 457, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/usr/local/envs/automatic/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 453, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Input type (torch.cuda.HalfTensor) and weight type (torch.HalfTensor) should be the same

I also got

Traceback (most recent call last):
  File "/content/gdrive/.shortcut-targets-by-id/1VNhtnoJC8AkKUr4-kCbT3hxMSzfiOGCd/sd/stable-diffusion-webui/modules/ui.py", line 185, in f
    res = list(func(*args, **kwargs))
  File "/content/gdrive/.shortcut-targets-by-id/1VNhtnoJC8AkKUr4-kCbT3hxMSzfiOGCd/sd/stable-diffusion-webui/webui.py", line 54, in f
    res = func(*args, **kwargs)
  File "/content/gdrive/.shortcut-targets-by-id/1VNhtnoJC8AkKUr4-kCbT3hxMSzfiOGCd/sd/stable-diffusion-webui/extensions/DreamArtist/scripts/dream_artist/ui.py", line 30, in train_embedding
    embedding, filename = dream_artist.cptuning.train_embedding(*args)
  File "/content/gdrive/.shortcut-targets-by-id/1VNhtnoJC8AkKUr4-kCbT3hxMSzfiOGCd/sd/stable-diffusion-webui/extensions/DreamArtist/scripts/dream_artist/cptuning.py", line 486, in train_embedding
    processed = processing.process_images(p)
  File "/content/gdrive/.shortcut-targets-by-id/1VNhtnoJC8AkKUr4-kCbT3hxMSzfiOGCd/sd/stable-diffusion-webui/modules/processing.py", line 423, in process_images
    res = process_images_inner(p)
  File "/content/gdrive/.shortcut-targets-by-id/1VNhtnoJC8AkKUr4-kCbT3hxMSzfiOGCd/sd/stable-diffusion-webui/modules/processing.py", line 442, in process_images_inner
    file.write(processed.infotext(p, 0))
  File "/content/gdrive/.shortcut-targets-by-id/1VNhtnoJC8AkKUr4-kCbT3hxMSzfiOGCd/sd/stable-diffusion-webui/modules/processing.py", line 281, in infotext
    return create_infotext(p, self.all_prompts, self.all_seeds, self.all_subseeds, comments=[], position_in_batch=index % self.batch_size, iteration=index // self.batch_size)
  File "/content/gdrive/.shortcut-targets-by-id/1VNhtnoJC8AkKUr4-kCbT3hxMSzfiOGCd/sd/stable-diffusion-webui/modules/processing.py", line 411, in create_infotext
    negative_prompt_text = "\nNegative prompt: " + p.negative_prompt if p.negative_prompt else ""
TypeError: can only concatenate str (not "tuple") to str

When trying to read from txt2img for making previews. But I managed to bypass the problem by replacing line 470

p.negative_prompt = preview_prompt.replace(ds.placeholder_token, ds.placeholder_token + '-neg') if use_negative else preview_negative_prompt,

of cptuning.py with

p.negative_prompt = preview_negative_prompt

which makes more sense to me.

How long does it usually take for 15,000 iterations to finish if you have access to a capable GPU?

Basically I just want to know whether this method is viable for making an app that users can upload pictures to train an embedding and it would not cost a lot and do not take a lot of time. It can be quite good if it's less costly and less time consuming than Dreambooth or other fine-tuning schemes.
So the question is essentially how long it typically requires to train an embedding on A100 GPUs compared to Dreambooth?

请问dreamartist是需要16gb vram才能运作吗？

Error completing request
Arguments: ('zxcweasd', 114514, '0.003', 1, 'F:\ai\asdggg', 'dream_artist', 512, 512, 8000, 100, 500, 'F:\ai\stable-diffusion-webui\textual_inversion_templates\subject.txt', False, False, '', '', 20, 0, 7, -1.0, 512, 512, '3.0', '', True, True, False, 0.74, 1, 1.0, 25.0, 1.0, 25.0, 0.9, 0.999, False, 1, False, '0.000005') {}
Traceback (most recent call last):
File "F:\ai\stable-diffusion-webui\modules\call_queue.py", line 45, in f
res = list(func(*args, **kwargs))
File "F:\ai\stable-diffusion-webui\modules\call_queue.py", line 28, in f
res = func(*args, **kwargs)
File "F:\ai\stable-diffusion-webui\extensions\DreamArtist-sd-webui-extension\scripts\dream_artist\ui.py", line 32, in train_embedding
embedding, filename = dream_artist.cptuning.train_embedding(*args)
File "F:\ai\stable-diffusion-webui\extensions\DreamArtist-sd-webui-extension\scripts\dream_artist\cptuning.py", line 557, in train_embedding
loss.backward()
File "F:\ai\stable-diffusion-webui\venv\lib\site-packages\torch_tensor.py", line 396, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "F:\ai\stable-diffusion-webui\venv\lib\site-packages\torch\autograd_init_.py", line 173, in backward
Variable.execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
File "F:\ai\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\function.py", line 253, in apply
return user_fn(self, *args)
File "F:\ai\stable-diffusion-webui\venv\lib\site-packages\torch\utils\checkpoint.py", line 146, in backward
torch.autograd.backward(outputs_with_grad, args_with_grad)
File "F:\ai\stable-diffusion-webui\venv\lib\site-packages\torch\autograd_init.py", line 173, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: CUDA out of memory. Tried to allocate 1024.00 MiB (GPU 0; 8.00 GiB total capacity; 5.31 GiB already allocated; 0 bytes free; 6.85 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

每次尝试跑的时候都会弹出这个问题还是有什么方法可以解决吗？

AttributeError: 'FrozenOpenCLIPEmbedder' object has no attribute 'transformer'

Traceback (most recent call last):
File "X:\AI\StableDiffusion\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 284, in run_predict
output = await app.blocks.process_api(
File "X:\AI\StableDiffusion\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 982, in process_api
result = await self.call_function(fn_index, inputs, iterator)
File "X:\AI\StableDiffusion\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 824, in call_function
prediction = await anyio.to_thread.run_sync(
File "X:\AI\StableDiffusion\stable-diffusion-webui\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "X:\AI\StableDiffusion\stable-diffusion-webui\venv\lib\site-packages\anyio_backends_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "X:\AI\StableDiffusion\stable-diffusion-webui\venv\lib\site-packages\anyio_backends_asyncio.py", line 867, in run
result = context.run(func, *args)
File "X:\AI\StableDiffusion\stable-diffusion-webui\extensions\DreamArtist-sd-webui-extension\scripts\dream_artist\ui.py", line 11, in create_embedding
filename = dream_artist.cptuning.create_embedding(name, nvpt, overwrite_old, init_text=initialization_text)
File "X:\AI\StableDiffusion\stable-diffusion-webui\extensions\DreamArtist-sd-webui-extension\scripts\dream_artist\cptuning.py", line 181, in create_embedding
embedding_layer = cond_model.wrapped.transformer.text_model.embeddings
File "X:\AI\StableDiffusion\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1207, in getattr
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'FrozenOpenCLIPEmbedder' object has no attribute 'transformer'
Traceback (most recent call last):
File "X:\AI\StableDiffusion\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 284, in run_predict
output = await app.blocks.process_api(
File "X:\AI\StableDiffusion\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 982, in process_api
result = await self.call_function(fn_index, inputs, iterator)
File "X:\AI\StableDiffusion\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 824, in call_function
prediction = await anyio.to_thread.run_sync(
File "X:\AI\StableDiffusion\stable-diffusion-webui\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "X:\AI\StableDiffusion\stable-diffusion-webui\venv\lib\site-packages\anyio_backends_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "X:\AI\StableDiffusion\stable-diffusion-webui\venv\lib\site-packages\anyio_backends_asyncio.py", line 867, in run
result = context.run(func, *args)
File "X:\AI\StableDiffusion\stable-diffusion-webui\extensions\DreamArtist-sd-webui-extension\scripts\dream_artist\ui.py", line 11, in create_embedding
filename = dream_artist.cptuning.create_embedding(name, nvpt, overwrite_old, init_text=initialization_text)
File "X:\AI\StableDiffusion\stable-diffusion-webui\extensions\DreamArtist-sd-webui-extension\scripts\dream_artist\cptuning.py", line 181, in create_embedding
embedding_layer = cond_model.wrapped.transformer.text_model.embeddings
File "X:\AI\StableDiffusion\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1207, in getattr
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'FrozenOpenCLIPEmbedder' object has no attribute 'transformer'

Question: How do I know if the training is working or if I should stop and change things?

At which point should I give up on the current training if it's not producing the type of images I want and try changing parameters and starting over, instead of waiting for more training steps on the current training? What should I look for as indication it is or isn't working?

RuntimeError: Sizes of tensors must match except in dimension 0. Expected size 231 but got size 154 for tensor number 1 in the list.

Tried training a nahida embedding with 5 pictures, these are my settings:

Can this method support deepdanbooru's caption?

I just want to know, did this method can support deepdanbooru's txt of tags, I can not determine, but it seems not really worked?

Shouldn't this be `scikit-learn`?

https://github.com/7eu7d7/DreamArtist-sd-webui-extension/blob/fbc85703a182a32bf1db169d0fd88e5c2e3fb5a6/install.py#L3-L4

In the end it'll install the right package, but PEP 503 strongly recommends using the registered name, scikit-learn in this case.

VRAM usage?

Hello, I was just wondering how much VRAM this takes to train?
I got it working, but I quickly ran out of VRAM on my RTX 2070S (only 8gb).