kwonminki/asyrp_official

official repo for Asyrp : Diffusion Models already have a Semantic Latent Space (ICLR2023)

asyrp_official's Issues

Custom Dataset Diffusion model

Thanks for your great work!

I want to train Asyrp on some new domain, and I saw #4 that I need to train my own diffusion model first in order to train Asyrp.

May I ask are there any limitation on the Diffusion Model? For example I want to use guided Diffusion, it is matter to train it to be class conditional or not?

Problem installing requirements

(.venv) user@DESKTOP:~/Asyrp_official$ pip install -r requirements.txt
Collecting blobfile==1.3.1
Using cached blobfile-1.3.1-py3-none-any.whl (70 kB)
ERROR: Could not find a version that satisfies the requirement clip==1.0 (from versions: 0.0.1, 0.1.0, 0.2.0)
ERROR: No matching distribution found for clip==1.0

How can I get clip==1.0?

do ./checkpoint/* work?

Hi! I've tried several checkpoints saved in ./checkpoint, like smiling, angry, young, it seems that they couldn't change corresponding attributes. Are there any problems of these checkpoints you have uploaded?

Need more detailed training & inference settings

Thanks for your work!
I am trying to train Asyrp for animal face editing on my own device. For attribute 'Happ Dog' , the training setting is as below:

sh_file_name="script_train.sh"
gpu="7"

config="afhq.yml" # if you use other dataset, config/path_config.py should be matched
guid="dog_happy"



CUDA_VISIBLE_DEVICES=$gpu python main.py --run_train                        \
                        --config $config                                    \
                        --exp ./runs/$guid                                  \
                        --edit_attr $guid                                   \
                        --do_train 1                                        \
                        --do_test 1                                         \
                        --n_train_img 1000                                   \
                        --n_test_img 32                                     \
                        --n_iter 1                                          \
                        --bs_train 1                                        \
                        --t_0 999                                           \
                        --n_inv_step 40                                     \
                        --n_train_step 40                                   \
                        --n_test_step 1000                                   \
                        --get_h_num 1                                       \
                        --train_delta_block                                 \
                        --sh_file_name $sh_file_name                        \
                        --save_x0                                           \
                        --use_x0_tensor                                     \
                        --hs_coeff_delta_h 1.0                              \
                        --lr_training 0.5                                   \
                        --clip_loss_w 1.5                                   \
                        --l1_loss_w 3.0                                     \
                        --retrain 1                                         \
                        --sh_file_name "script_train.sh"                    \
                        --load_random_noise
                        --lpips_addnoise_th 1.2                           \ # if you compute lpips, use it.
                        --lpips_edit_th 0.33                              \
                        --add_noise_from_xt                               \ # if you compute lpips, use it.

However, the editing results are not so good. The changes are undistinguishable, some examples are shown below.

I wonder could you privode a more detailed training & inference setting for 'Happ Dog'. ?
Looking forward for your reply, Thank you!

Add requirements.txt file

Hi, could you add a requirements.txt file of the versions of the python packages used for this paper?

Bad performance

Thanks for you work!

I tried to reproduce the results for "happy dog" using released pretrained models but the performance was bad.

Here are the settings of inference scripts.

sh_file_name="script_inference.sh"
gpu="0"
config="afhq.yml"
guid="dog_happy"
test_step=40    # if large, it takes long time.
dt_lambda=1.0   # hyperparameter for dt_lambda. This is the method that will appear in the next paper.

CUDA_VISIBLE_DEVICES=$gpu python main.py --run_test                         \
                        --config $config                                    \
                        --exp ./runs/${guid}                                \
                        --edit_attr $guid                                   \
                        --do_train 0                                       \
                        --do_test 1                                        \
                        --n_train_img 0                                   \
                        --n_test_img 32                                     \
                        --n_iter 5                                          \
                        --bs_train 1                                        \
                        --t_0 999                                           \
                        --n_inv_step 40                                     \
                        --n_train_step 40                                   \
                        --n_test_step $test_step                            \
                        --get_h_num 1                                       \
                        --train_delta_block                                 \
                        --sh_file_name $sh_file_name                        \
                        --save_x0                                           \
                        --use_x0_tensor                                     \
                        --hs_coeff_delta_h 1.0                              \
                        --dt_lambda $dt_lambda                              \
                        --add_noise_from_xt                                 \
                        --lpips_addnoise_th 1.2                             \
                        --lpips_edit_th 0.33                                \
                        --sh_file_name "script_inference.sh"                \
                        --manual_checkpoint_name "dog_happy_LC_dog_t999_ninv40_ngen40_0.pth" \

The pretrained model is "afhqdog_p2.pt"

Some examples of results are

Do you have any suggestions on this?

Transfer Guided Diffusion U-Net to your Diffusion Object and back

Hi,

first thank you for your amazing work!

However, I have some questions in part related to the other issues here.

1. How can I do standard inference with the pre-trained models? E.g. for celebahq_p2.pt the model was supposedly trained with the code of P2 weighting. But the code in the original P2 weighting repository has a totally different structure than your code base. If you load the checkpoint you get a models.ddpm.diffusion.Diffusion object, but this is totally different from what other forks of guided diffusion like P2 weighting understand under a Diffusion object and rather has the variables and the interface of the U-Net only, but without information about the actual diffusion process anymore. If one wraps around a models.guided_diffusion.gaussian_diffusion.GaussianDiffusion then one can execute p_sample_loop successfully, but the results of the reverse diffusion process is random noise. How can I test the pretrained models without directly executing Asyrp only doing normal reverse diffusion?

2. The same problem also exists in the other direction as already mentioned in #10 - you can train a models.guided_diffusion.unet.UNetModel with the common guided diffusion code (e.g. also the P2 variant), but how are you using this U-Net now in your code?

3. I guess the meta-question is, how am I doing the transfer between your models.ddpm.diffusion.Diffusion object and the models.guided_diffusion.unet.UNetModel in either direction?

Best,
Sidney

Distorted image color

Thanks for the excellent work!
All my generated images had distorted color whether I used self-trained checkpoints or released pretrained models.
test_19_4_ngen50.png

test_7_4_ngen50.png

train_0_0_ngen50.png

train_0_1_ngen50.png

train_0_2_ngen50.png

I only modified some command options of the scripts:

sh_file_name="script_precompute.sh"
gpu="6"

config="custom.yml" # if you use other dataset, config/path_config.py should be matched
guid="smiling"  # guid should be in utils/text_dic.py


CUDA_VISIBLE_DEVICES=$gpu python main.py --run_train                        \
                        --config $config                                    \
                        --exp ./runs/$guid                                  \
                        --edit_attr $guid                                   \
                        --do_train 1                                        \
                        --do_test 1                                         \
                        --n_train_img 100                                   \
                        --n_test_img 32                                     \
                        --bs_train 1                                        \
                        --get_h_num 1                                       \
                        --train_delta_block                                 \
                        --t_0 999                                           \
                        --n_inv_step 50                                     \
                        --n_train_step 50                                   \
                        --n_test_step 50                                    \
                        --just_precompute                                   \
                        --custom_train_dataset_dir "/data/dataset/CelebA-HQ-1024_0-999"       \
                        --custom_test_dataset_dir "/data/dataset/CelebA-HQ-1024_0-999"         \
                        --sh_file_name "script_precompute.sh"       \
                        --model_path      "/mnt/Asyrp_official/pretrained/celebahq_p2.pt"

#!/bin/bash

sh_file_name="script_train.sh"
gpu="6"

config="custom.yml" # if you use other dataset, config/path_config.py should be matched
guid="smiling" # guid should be in utils/text_dic.py


CUDA_VISIBLE_DEVICES=$gpu python main.py --run_train                        \
                        --config $config                                    \
                        --exp ./runs/$guid                                  \
                        --edit_attr $guid                                   \
                        --do_train 1                                        \
                        --do_test 1                                         \
                        --n_train_img 100                                   \
                        --n_test_img 32                                     \
                        --n_iter 5                                          \
                        --bs_train 4                                        \
                        --t_0 999                                           \
                        --n_inv_step 1000                                     \
                        --n_train_step 50                                   \
                        --n_test_step 1000                                   \
                        --get_h_num 1                                       \
                        --train_delta_block                                 \
                        --sh_file_name $sh_file_name                        \
                        --save_x0                                           \
                        --use_x0_tensor                                     \
                        --hs_coeff_delta_h 1.0                              \
                        --lr_training 0.5                                   \
                        --clip_loss_w 1.0                                   \
                        --l1_loss_w 3.0                                     \
                        --retrain 1                                         \
                        --sh_file_name "script_train.sh"  \
                        --model_path      "/mnt/Asyrp_official/pretrained/celebahq_p2.pt" \
                        --custom_train_dataset_dir "/data/dataset/CelebA-HQ-1024_0-999"       \
                        --custom_test_dataset_dir "/data/dataset/CelebA-HQ-1024_0-999"         \

#!/bin/bash

sh_file_name="script_inference.sh"
gpu="6"
config="custom.yml"
guid="smiling"
test_step=1000    # if large, it takes long time.
dt_lambda=1.0   # hyperparameter for dt_lambda. This is the method that will appear in the next paper.

CUDA_VISIBLE_DEVICES=$gpu python main.py --run_test                         \
                        --config $config                                    \
                        --exp ./runs/${guid}                                \
                        --edit_attr $guid                                   \
                        --do_train 1                                    \
                        --do_test 1                                         \
                        --n_train_img 100                                  \
                        --n_test_img 32                                     \
                        --n_iter 5                                          \
                        --bs_train 1                                        \
                        --t_0 999                                           \
                        --n_inv_step 50                                     \
                        --n_train_step 50                                   \
                        --n_test_step $test_step                            \
                        --get_h_num 1                                       \
                        --train_delta_block                                 \
                        --sh_file_name $sh_file_name                        \
                        --save_x0                                           \
                        --use_x0_tensor                                     \
                        --hs_coeff_delta_h 1.0                              \
                        --dt_lambda $dt_lambda                              \
                        --add_noise_from_xt                                 \
                        --lpips_addnoise_th 1.2                             \
                        --lpips_edit_th 0.33                                \
                        --sh_file_name "script_inference.sh"              \
                        --model_path      "/mnt/Asyrp_official/pretrained/celebahq_p2.pt" \
                        --custom_train_dataset_dir "/data/dataset/CelebA-HQ-1024_0-999"                \
                        --custom_test_dataset_dir "/data/dataset/CelebA-HQ-1024_0-999"     \
                        --manual_checkpoint_name "smiling_LC_CelebA_HQ_t999_ninv40_ngen40_0.pth" \

README.md spelling error

In the dataset portion you have the following:

 --custom_train_dataset_dir "your/costom/datset/dir/train" \
 --custom_test_dataset_dir "your/costom/dataset/dir/test" \

and just wanted to say it should be "custom" (small nitpick : ))

There is no option for Celeba_HQ_P2 despite recommendation

Hi,

when trying to apply your script with the config https://github.com/kwonminki/Asyrp_official/blob/main/configs/celeba_p2.yml you get the following error:

"sh script_get_lpips.sh
INFO: underlay of /usr/bin/nvidia-smi required more than 50 (274) bind mounts
INFO - main.py - 2023-12-07 17:58:17,991 - Using device: cuda
Get lpips distance...
ERROR - main.py - 2023-12-07 17:58:28,427 - Traceback (most recent call last):
File "/home/sidney/workspace/Asyrp_official/main.py", line 337, in main
runner.compute_lpips_distance()
File "/opt/conda/envs/peal/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/sidney/workspace/Asyrp_official/diffusion_latent.py", line 1198, in compute_lpips_distance
model = self.load_pretrained_model()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sidney/workspace/Asyrp_official/diffusion_latent.py", line 99, in load_pretrained_model
raise ValueError
"

because "CelebA_HQ_P2" is none of the options.

Best,
Sidney

Colab demo

Hello,

Thanks for the fantastic work, would you plan to give any kind of demo (preferably Google Colab) so that it could smoothly test out the idea? Perhaps, could make the work more popular as a broad range of people can try it and reduce setting up time.

About CUSTOM dataset

It's an excellent job. I really appreciate it.
I observed when getting LPIPS distance, it has a denoising step which makes use of pre-trained diffusion models.
If my data domain makes a great difference from the pre-trained dataset, should I pretrain the diffusion model on my dataset first?

Extension to text-conditioned models

Hi,

Great work, do you have any idea how we could extend your model to text-to-image conditioned diffusion models?

Access denied to fetch pretrained model

Hi, I want ton replicate the results on google colab using this notebook using the default settings in script_inference.sh but this is url is throwing 403 error

Asyrp_official/diffusion_latent.py

Lines 84 to 85 in a6c82dc

	elif self.config.data.dataset in ["CelebA_HQ", "CUSTOM", "CelebA_HQ_Dialog"]:
	url = "https://image-editing-test-12345.s3-us-west-2.amazonaws.com/checkpoints/celeba_hq.ckpt"

Custom Dataset training

Hello,

Thanks for your awesome work!

I have a question to ask the author, is it possible to learn by including attribute values and images that do not exist in the pretrained model in custom data? (ex. Can I put new Food image and Food attribute values in CelebHQ pretrained model?)

Looking forward for your reply, Thank you!

Custom dataset training (P2 weighting)

Hello,

I have trained a diffusion model with a custom dataset using P2 weighting (the training code in the guided_diffusion folder), however, the generated samples are not correct when I use the functions (denoisin_step, generalized_steps) you have used in the main script. However, when I use the sampling codes in guided_diffusion/gaussian_diffusion.py module, it correctly generates samples. So could you please provied the details about how you trained CelebA_P2 model so that I can train my own dataset using P2 weighting with the same settings you trained in order to make it run with the sampling codes in your code?

Thanks