chendaryen/key-locked-rank-one-editing-for-text-to-image-personalization

An Pytorch implementation of the paper Key-Locked Rank One Editing for Text-to-Image Personalization

key-locked-rank-one-editing-for-text-to-image-personalization's Issues

shape mismatch: value tensor of shape [1024] cannot be broadcast to indexing result of shape [0, 768]

Hi Team,
I am able to run the sd v2 related code using command

python main.py --name actor --base ./configs/perfusion_custom_sd_v2.yaml --basedir ./ckpt -t True --gpus 0,

the training runs fine but when I try to create the immages I get the following errors related to shape mismatch: value tensor of shape [1024] cannot be broadcast to indexing result of shape [0, 768]

I am using below command to generate images

python scripts/perfusion_txt2img.py --ddim_eta 0.0 --steps 50 --scale 6.0 --beta 0.7 --tau 0.15 --n_samples 4 --n_iter 1 --personalized_ckpt ./ckpt/2023-11-27T09-04-38_actor/models/last.ckpt --prompt "photo of actor"

Loading model from ./ckpt/v1-5-pruned-emaonly.ckpt and ./ckpt/2023-11-27T09-39-45_teddy/models/last.ckpt
Global Step: 840000
data: 0%| | 0/1 [00:00<?, ?it/s]
Sampling: 0%| | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/home/ubuntu/code/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/scripts/perfusion_txt2img.py", line 357, in
main()
File "/home/ubuntu/code/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/scripts/perfusion_txt2img.py", line 317, in main
encoding = model.cond_stage_model.encode(prompts, embedding_manager=model.embedding_manager)
File "/home/ubuntu/code/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/ldm/modules/encoders/modules.py", line 285, in encode
return self(text, **kwargs)
File "/home/ubuntu/miniconda3/envs/aiml2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ubuntu/code/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/ldm/modules/encoders/modules.py", line 280, in forward
z = self.transformer(input_ids=tokens, **kwargs)
File "/home/ubuntu/miniconda3/envs/aiml2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ubuntu/code/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/ldm/modules/encoders/modules.py", line 259, in transformer_forward
return self.text_model(
File "/home/ubuntu/miniconda3/envs/aiml2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ubuntu/code/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/ldm/modules/encoders/modules.py", line 219, in text_encoder_forward
hidden_states = self.embeddings(input_ids=input_ids, position_ids=position_ids,
File "/home/ubuntu/miniconda3/envs/aiml2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ubuntu/code/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/ldm/modules/encoders/modules.py", line 143, in embedding_forward
inputs_embeds = embedding_manager(input_ids, inputs_embeds)
File "/home/ubuntu/miniconda3/envs/aiml2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ubuntu/code/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/perfusion/embedding_manager.py", line 117, in forward
embedded_text[tokenized_text == placeholder_token] = placeholder_embedding
RuntimeError: shape mismatch: value tensor of shape [1024] cannot be broadcast to indexing result of shape [0, 768]

Questions about the reason for applying EMA to the target input i*

Firstly, I would like to extend my congratulations on your acceptance into SIGGRAPH. I find your work quite interesting and appreciate your contribution to the field.

And I got a question in Section 4.4 in your paper, I am curious to know the rationale behind using the Exponential Moving Average (EMA) for the target input (i.e., $i_* = 0.99i_* + 0.01 e_\text{concept}$), instead of directly using $i_* = e_\text{concept}$. In EMA, the $e_\text{concept}$ just accounts for such a small part. I am keen to understand the intuition underlying the application of EMA in this context.

Thank you for your time and consideration. I look forward to hearing back from you.

Device error

Hi bro, I tried the code with teddy, using the command line you provide in readme, but got the device error below:

Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument index in method wrapper_CUDA__index_select)

Where should I put the downloaded pre-trained model?

Traceback (most recent call last):
File "/home/wjy/miniconda3/envs/perfusion/lib/python3.10/site-packages/urllib3/connection.py", line 198, in _new_conn
sock = connection.create_connection(
File "/home/wjy/miniconda3/envs/perfusion/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
raise err
File "/home/wjy/miniconda3/envs/perfusion/lib/python3.10/site-packages/urllib3/util/connection.py", line 73, in create_connection
sock.connect(sa)
OSError: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/wjy/miniconda3/envs/perfusion/lib/python3.10/site-packages/urllib3/connectionpool.py", line 793, in urlopen
response = self._make_request(
File "/home/wjy/miniconda3/envs/perfusion/lib/python3.10/site-packages/urllib3/connectionpool.py", line 491, in _make_request
raise new_e
File "/home/wjy/miniconda3/envs/perfusion/lib/python3.10/site-packages/urllib3/connectionpool.py", line 467, in _make_request
self._validate_conn(conn)
File "/home/wjy/miniconda3/envs/perfusion/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
conn.connect()
File "/home/wjy/miniconda3/envs/perfusion/lib/python3.10/site-packages/urllib3/connection.py", line 616, in connect
self.sock = sock = self._new_conn()
File "/home/wjy/miniconda3/envs/perfusion/lib/python3.10/site-packages/urllib3/connection.py", line 213, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f050d0470d0>: Failed to establish a new connection: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/wjy/miniconda3/envs/perfusion/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/home/wjy/miniconda3/envs/perfusion/lib/python3.10/site-packages/urllib3/connectionpool.py", line 847, in urlopen
retries = retries.increment(
File "/home/wjy/miniconda3/envs/perfusion/lib/python3.10/site-packages/urllib3/util/retry.py", line 515, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /laion/CLIP-ViT-H-14-laion2B-s32B-b79K/resolve/main/open_clip_pytorch_model.bin (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f050d0470d0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/wjy/miniconda3/envs/perfusion/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1261, in hf_hub_download
metadata = get_hf_file_metadata(
File "/home/wjy/miniconda3/envs/perfusion/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
return fn(*args, **kwargs)
File "/home/wjy/miniconda3/envs/perfusion/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1667, in get_hf_file_metadata
r = _request_wrapper(
File "/home/wjy/miniconda3/envs/perfusion/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 385, in _request_wrapper
response = _request_wrapper(
File "/home/wjy/miniconda3/envs/perfusion/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 408, in _request_wrapper
response = get_session().request(method=method, url=url, **params)
File "/home/wjy/miniconda3/envs/perfusion/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/home/wjy/miniconda3/envs/perfusion/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/home/wjy/miniconda3/envs/perfusion/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 67, in send
return super().send(request, *args, **kwargs)
File "/home/wjy/miniconda3/envs/perfusion/lib/python3.10/site-packages/requests/adapters.py", line 519, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /laion/CLIP-ViT-H-14-laion2B-s32B-b79K/resolve/main/open_clip_pytorch_model.bin (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f050d0470d0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))"), '(Request ID: 6b7a84f5-757b-4262-aa52-e489e568858f)')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/wjy/perfusion/main.py", line 461, in
model = instantiate_from_config(config.model)
File "/home/wjy/perfusion/main.py", line 137, in instantiate_from_config
return get_obj_from_str(config["target"])(**config.get("params", dict()))
File "/home/wjy/perfusion/perfusion/perfusion.py", line 54, in init
super().init(*args, **kwargs)
File "/home/wjy/perfusion/ldm/models/diffusion/ddpm.py", line 565, in init
self.instantiate_cond_stage(cond_stage_config)
File "/home/wjy/perfusion/ldm/models/diffusion/ddpm.py", line 640, in instantiate_cond_stage
model = instantiate_from_config(config)
File "/home/wjy/perfusion/ldm/util.py", line 175, in instantiate_from_config
return get_obj_from_str(config["target"])(**config.get("params", dict()), **kwargs)
File "/home/wjy/perfusion/ldm/modules/encoders/modules.py", line 340, in init
model, _, _ = open_clip.create_model_and_transforms(arch, device=torch.device('cpu'), pretrained=version)
File "/home/wjy/miniconda3/envs/perfusion/lib/python3.10/site-packages/open_clip/factory.py", line 383, in create_model_and_transforms
model = create_model(
File "/home/wjy/miniconda3/envs/perfusion/lib/python3.10/site-packages/open_clip/factory.py", line 281, in create_model
checkpoint_path = download_pretrained(pretrained_cfg, cache_dir=cache_dir)
File "/home/wjy/miniconda3/envs/perfusion/lib/python3.10/site-packages/open_clip/pretrained.py", line 552, in download_pretrained
target = download_pretrained_from_hf(model_id, cache_dir=cache_dir)
File "/home/wjy/miniconda3/envs/perfusion/lib/python3.10/site-packages/open_clip/pretrained.py", line 522, in download_pretrained_from_hf
cached_file = hf_hub_download(model_id, filename, revision=revision, cache_dir=cache_dir)
File "/home/wjy/miniconda3/envs/perfusion/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
return fn(*args, **kwargs)
File "/home/wjy/miniconda3/envs/perfusion/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1406, in hf_hub_download
raise LocalEntryNotFoundError(
huggingface_hub.utils._errors.LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.

I manually downloaded the open_clip_pytorch_model.bin model, where should I put it?please tell me ,thanks!

Any plan for supporting Support SDXL-1.0?

Hi author,
Thanks for this excellent work. I would like to ask any plan for supporting SDXL-1.0? It will be very useful. Thanks

training image folder option not present in the configs/perfusion_inference_sd_v2.yaml

Hi Team,
Great work! I am trying your code. I was able to run your code and try on some customization. v 1 but with v2 I am facing issues.
In the configs/perfusion_inference_sd_v2.yaml I don't see the property to mention the data key which is present in the configs/perfusion_custom.yaml.
I tried to add this below data key in the yaml but it throws error. Please let me know why data key is not present in the perfusion_inference_sd_v2.yaml. Please help me on this.

`data:
target: main.DataModuleFromConfig
params:
batch_size: 4
num_workers: 10
wrap: true
train:
target: perfusion.dataset.PersonalizedBase
params:
data_root: '/path/to/your/images/'
flip_p: 0.5 # 0. when asymmetric
prompt_template:

lightning:
callbacks:
image_logger:
target: main.ImageLogger
params:
frequency_base: 5
batch_frequency: 25
modelcheckpoint_step:
params:
every_n_train_steps: 50
trainer:
max_steps: 400
log_every_n_steps: 1
accumulate_grad_batches: 4
precision: 16`

How does this excellent work apply to the webui?

Hi author,
Thanks for this excellent work. I would like to ask any plan for supporting webui extension? It will be very useful. Thanks

Using CLIP similarity to automatically select a balanced weight is necessary.

Hi Author,
Thanks for your excellent work. Would you explain a little bit about how to use CLIP similarity to automatically select a balanced weight? I can help with that. Thanks

Load Model Problem TypeError: expected str, bytes or os.PathLike object, not NoneType

Hi,
I tried the command

     python main.py \
        --name teddy \
        --base ./configs/perfusion_teddy.yaml \
        --basedir ./ckpt \
        -t True \
        --gpus 0,

but got the error below

Traceback (most recent call last):
File "/data_heat/rjt_project/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/main.py", line 461, in
model = instantiate_from_config(config.model)
File "/data_heat/rjt_project/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/main.py", line 137, in instantiate_from_config
return get_obj_from_str(config["target"])(**config.get("params", dict()))
File "/data_heat/rjt_project/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/perfusion/perfusion.py", line 54, in init
super().init(*args, **kwargs)
File "/data_heat/rjt_project/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/ldm/models/diffusion/ddpm.py", line 565, in init
self.instantiate_cond_stage(cond_stage_config)
File "/data_heat/rjt_project/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/ldm/models/diffusion/ddpm.py", line 640, in instantiate_cond_stage
model = instantiate_from_config(config)
File "/data_heat/rjt_project/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/ldm/util.py", line 175, in instantiate_from_config
return get_obj_from_str(config["target"])(**config.get("params", dict()), **kwargs)
File "/data_heat/rjt_project/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/ldm/modules/encoders/modules.py", line 120, in init
self.tokenizer = CLIPTokenizer.from_pretrained(version)
File "/home/user/miniconda3/envs/perfusion_v1/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1841, in from_pretrained
return cls._from_pretrained(
File "/home/user/miniconda3/envs/perfusion_v1/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2004, in _from_pretrained
tokenizer = cls(*init_inputs, **init_kwargs)
File "/home/user/miniconda3/envs/perfusion_v1/lib/python3.10/site-packages/transformers/models/clip/tokenization_clip.py", line 334, in init
with open(vocab_file, encoding="utf-8") as vocab_handle:
TypeError: expected str, bytes or os.PathLike object, not NoneType

I creat the env
by
conda create --name perfusion_v1 python=3.10
conda install pytorch torchvision pytorch-cuda=11.7 -c pytorch -c nvidia
conda install xformers -c xformers
And pip follow page
- albumentations==1.3.1
- pytorch-lightning==1.6.0
- omegaconf==2.1.1
- einops==0.6.1
- transformers==4.31.0
- kornia==0.7.0
- open_clip_torch==2.20.0
- git+https://github.com/openai/CLIP.git
I also get SD1.5 ckpt from
https://huggingface.co/runwayml/stable-diffusion-v1-5/resolve/main/v1-5-pruned-emaonly.ckpt

Thanks

Problem when running training with num_vectors_per_token>1

Hi!
After modifying the config yaml file by increasing the num_vectors_per_token parameter, when I try to run the training I get an error:

Traceback (most recent call last):
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/main.py", line 596, in
trainer.fit(model, data)
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 771, in fit
self._call_and_handle_interrupt(
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 724, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 812, in _fit_impl
results = self._run(model, ckpt_path=self.ckpt_path)
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1237, in _run
results = self._run_stage()
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1324, in _run_stage
return self._run_train()
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1354, in _run_train
self.fit_loop.run()
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/venv/lib/python3.10/site-packages/pytorch_lightning/loops/base.py", line 204, in run
self.advance(*args, **kwargs)
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/venv/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 269, in advance
self._outputs = self.epoch_loop.run(self._data_fetcher)
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/venv/lib/python3.10/site-packages/pytorch_lightning/loops/base.py", line 204, in run
self.advance(*args, **kwargs)
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/venv/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 208, in advance
batch_output = self.batch_loop.run(batch, batch_idx)
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/venv/lib/python3.10/site-packages/pytorch_lightning/loops/base.py", line 204, in run
self.advance(*args, **kwargs)
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/venv/lib/python3.10/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 88, in advance
outputs = self.optimizer_loop.run(split_batch, optimizers, batch_idx)
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/venv/lib/python3.10/site-packages/pytorch_lightning/loops/base.py", line 204, in run
self.advance(*args, **kwargs)
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/venv/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 203, in advance
result = self._run_optimization(
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/venv/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 249, in _run_optimization
closure()
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/venv/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 148, in call
self._result = self.closure(*args, **kwargs)
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/venv/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 134, in closure
step_output = self._step_fn()
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/venv/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 427, in _training_step
training_step_output = self.trainer._call_strategy_hook("training_step", *step_kwargs.values())
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1766, in _call_strategy_hook
output = fn(*args, **kwargs)
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/venv/lib/python3.10/site-packages/pytorch_lightning/strategies/strategy.py", line 333, in training_step
return self.model.training_step(*args, **kwargs)
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/ldm/models/diffusion/ddpm.py", line 442, in training_step
loss, loss_dict = self.shared_step(batch)
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/perfusion/perfusion.py", line 125, in shared_step
loss = self(x, c, mask=mask)
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/perfusion/perfusion.py", line 129, in forward
encoding = self.cond_stage_model.encode(c['c_crossattn'], embedding_manager=self.embedding_manager)
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/ldm/modules/encoders/modules.py", line 285, in encode
return self(text, **kwargs)
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/ldm/modules/encoders/modules.py", line 280, in forward
z = self.transformer(input_ids=tokens, **kwargs)
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/ldm/modules/encoders/modules.py", line 259, in transformer_forward
return self.text_model(
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/ldm/modules/encoders/modules.py", line 219, in text_encoder_forward
hidden_states = self.embeddings(input_ids=input_ids, position_ids=position_ids,
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/ldm/modules/encoders/modules.py", line 143, in embedding_forward
inputs_embeds = embedding_manager(input_ids, inputs_embeds)
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/azureuser/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/perfusion/embedding_manager.py", line 139, in forward
[tokenized_text[row][:col], placeholder_token.repeat(num_vectors_for_token).to(device),
AttributeError: 'int' object has no attribute 'repeat'

I figured this is probably because placeholder_token is supposed to be a tensor, therefore I fixed it by adding placeholder_token = torch.tensor(placeholder_token).to(device) line to perfusion/embedding_manager.py in the else clause after checking if self.max_vectors_per_token == 1 (line 116).

It seems to be working after that :)

multiconcept for woman and teddy

Hi Author,
I found that if I used teddy.ckpt and herburn.ckpt to get multiconcept results. The image results don't include any teddy.

Very long training time

I'm not super experienced and don't know if something is actually going wrong..

I managed to get the training going and am seeing results that go somewhat in the right direction.
But the training is taking quite long (especially in comparison to the speed promised in the paper), and the quality of the trained concepts is very lacking.

After about 1h of training it has not finished the first epoch. It says something like "Epoch 0: (2276/6250)" - do those numbers refer to the training timesteps? Because in the config file the max number of steps is set to 400, and i do reach that number appearently, because it outputs the step=400.ckpt model when i let it run all the way through.

Is something going wrong here?

I'm using a NVIDIA GeForce RTX 3060 with around 12 GB ram, CUDA is working fine.

missing 38 keys and having 2 unexpected keys

Restored from ./ckpt/v2-1_512-ema-pruned.ckpt with 38 missing and 2 unexpected keys
Missing Keys:
['logvar', 'C_inv', 'target_input', 'model.diffusion_model.input_blocks.1.1.transformer_blocks.0.attn2.to_k.target_output', 'model.diffusion_model.input_blocks.1.1.transformer_blocks.0.attn2.to_v.target_output', 'model.diffusion_model.input_blocks.2.1.transformer_blocks.0.attn2.to_k.target_output', 'model.diffusion_model.input_blocks.2.1.transformer_blocks.0.attn2.to_v.target_output', 'model.diffusion_model.input_blocks.4.1.transformer_blocks.0.attn2.to_k.target_output', 'model.diffusion_model.input_blocks.4.1.transformer_blocks.0.attn2.to_v.target_output', 'model.diffusion_model.input_blocks.5.1.transformer_blocks.0.attn2.to_k.target_output', 'model.diffusion_model.input_blocks.5.1.transformer_blocks.0.attn2.to_v.target_output', 'model.diffusion_model.input_blocks.7.1.transformer_blocks.0.attn2.to_k.target_output', 'model.diffusion_model.input_blocks.7.1.transformer_blocks.0.attn2.to_v.target_output', 'model.diffusion_model.input_blocks.8.1.transformer_blocks.0.attn2.to_k.target_output', 'model.diffusion_model.input_blocks.8.1.transformer_blocks.0.attn2.to_v.target_output', 'model.diffusion_model.middle_block.1.transformer_blocks.0.attn2.to_k.target_output', 'model.diffusion_model.middle_block.1.transformer_blocks.0.attn2.to_v.target_output', 'model.diffusion_model.output_blocks.3.1.transformer_blocks.0.attn2.to_k.target_output', 'model.diffusion_model.output_blocks.3.1.transformer_blocks.0.attn2.to_v.target_output', 'model.diffusion_model.output_blocks.4.1.transformer_blocks.0.attn2.to_k.target_output', 'model.diffusion_model.output_blocks.4.1.transformer_blocks.0.attn2.to_v.target_output', 'model.diffusion_model.output_blocks.5.1.transformer_blocks.0.attn2.to_k.target_output', 'model.diffusion_model.output_blocks.5.1.transformer_blocks.0.attn2.to_v.target_output', 'model.diffusion_model.output_blocks.6.1.transformer_blocks.0.attn2.to_k.target_output', 'model.diffusion_model.output_blocks.6.1.transformer_blocks.0.attn2.to_v.target_output', 'model.diffusion_model.output_blocks.7.1.transformer_blocks.0.attn2.to_k.target_output', 'model.diffusion_model.output_blocks.7.1.transformer_blocks.0.attn2.to_v.target_output', 'model.diffusion_model.output_blocks.8.1.transformer_blocks.0.attn2.to_k.target_output', 'model.diffusion_model.output_blocks.8.1.transformer_blocks.0.attn2.to_v.target_output', 'model.diffusion_model.output_blocks.9.1.transformer_blocks.0.attn2.to_k.target_output', 'model.diffusion_model.output_blocks.9.1.transformer_blocks.0.attn2.to_v.target_output', 'model.diffusion_model.output_blocks.10.1.transformer_blocks.0.attn2.to_k.target_output', 'model.diffusion_model.output_blocks.10.1.transformer_blocks.0.attn2.to_v.target_output', 'model.diffusion_model.output_blocks.11.1.transformer_blocks.0.attn2.to_k.target_output', 'model.diffusion_model.output_blocks.11.1.transformer_blocks.0.attn2.to_v.target_output', 'embedding_manager.string_to_param_dict.', 'embedding_manager.initial_embeddings.', 'embedding_manager.get_embedding_for_tkn.weight']

Unexpected Keys:
['model_ema.decay', 'model_ema.num_updates']

DO I have to worry about it?

AssertionError: String 'birdbath' maps to more than a single token.

  File "/home/code/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/main.py", line 461, in <module>
    model = instantiate_from_config(config.model)
  File "/home/code/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/main.py", line 137, in instantiate_from_config
    return get_obj_from_str(config["target"])(**config.get("params", dict()))
  File "/home/code/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/perfusion/perfusion.py", line 65, in __init__
    self.embedding_manager = instantiate_from_config(personalization_config, embedder=self.cond_stage_model)
  File "/home/code/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/ldm/util.py", line 175, in instantiate_from_config
    return get_obj_from_str(config["target"])(**config.get("params", dict()), **kwargs)
  File "/home/code/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/perfusion/embedding_manager.py", line 87, in __init__
    init_word_token = get_token_for_string(initializer_words[idx])
  File "/home/code/Key-Locked-Rank-One-Editing-for-Text-to-Image-Personalization/perfusion/embedding_manager.py", line 19, in get_clip_token_for_string
    assert torch.count_nonzero(
AssertionError: String 'birdbath' maps to more than a single token. Please use another string

So how to solve this bug?

No font file font "data/DejaVuSans.ttf"

Hi Author,
I meet the following error while I run the command based on SD1.5.
No sources: "data/DejaVuSans.ttf"
Thank you!