arpitbansal297/cold-diffusion-models

Official implementation of Cold-Diffusion for different transformations in pytorch.

cold-diffusion-models's Issues

Please don't link directly to Arxiv pdfs

As a general rule, please link to the abstract, not the pdf file. It helps with "save to zotero" and such, but also reduces load on the arxiv servers from people who just casually click the link to see what's behind it.

Nice work though :)

Animorphosis

Thanks for the good work! I am reproducing the Animorphosis part, but get confused with the settings.
What images shall I put in the following two folders? I should put the original images in "--data_path_start", and what images should I put into the "--data_path_start"? The Animorphosis images?
parser.add_argument('--data_path_start', default='../deblurring-diffusion-pytorch/AFHQ/afhq/train/', type=str)
parser.add_argument('--data_path_end', default='../deblurring-diffusion-pytorch/root_celebA_128_train_new/', type=str)

Kernel size mismatch

I''ve been trying testing blurring using weights on the drive,

! python ./AFHQ_128_test.py --load_path ./AFHQ_blur_generation.pt --gmm_cluster 1 --noise 0.002 --discrete --time_steps 10 --blur_size 15 --blur_std 0.01 --blur_routine Individual_Incremental --sampling_routine x0_step_down --data_path ./test_data --save_folder ./save_folder --test_type test_data

but I always get this error

size mismatch for module.gaussian_kernels.0.weight: copying a param with shape torch.Size([3, 1, 27, 27]) from checkpoint, the shape in current model is torch.Size([3, 1, 1, 1])
size mismatch for module.gaussian_kernels.1.weight: copying a param with shape torch.Size([3, 1, 27, 27]) from checkpoint, the shape in current model is torch.Size([3, 1, 3, 3]).
size mismatch for module.gaussian_kernels.2.weight: copying a param with shape torch.Size([3, 1, 27, 27]) from checkpoint, the shape in current model is torch.Size([3, 1, 5, 5]).
size mismatch for module.gaussian_kernels.3.weight: copying a param with shape torch.Size([3, 1, 27, 27]) from checkpoint, the shape in current model is torch.Size([3, 1, 7, 7]).
size mismatch for module.gaussian_kernels.4.weight: copying a param with shape torch.Size([3, 1, 27, 27]) from checkpoint, the shape in current model is torch.Size([3, 1, 9, 9]).
size mismatch for module.gaussian_kernels.5.weight: copying a param with shape torch.Size([3, 1, 27, 27]) from checkpoint, the shape in current model is torch.Size([3, 1, 11, 11]).
size mismatch for module.gaussian_kernels.6.weight: copying a param with shape torch.Size([3, 1, 27, 27]) from checkpoint, the shape in current model is torch.Size([3, 1, 13, 13]).
size mismatch for module.gaussian_kernels.7.weight: copying a param with shape torch.Size([3, 1, 27, 27]) from checkpoint, the shape in current model is torch.Size([3, 1, 15, 15]).
size mismatch for module.gaussian_kernels.8.weight: copying a param with shape torch.Size([3, 1, 27, 27]) from checkpoint, the shape in current model is torch.Size([3, 1, 17, 17]).
size mismatch for module.gaussian_kern....

doubts about the whole concept

why does the network have to train the way you intended?

In every training step, the target is maximum quality non-degraded sample.

if the network is good enough, it learns maximum quality in 1 pass, what are 49 more steps for?

or are you assuming that the network is bad enough, and therefore instead of the target it learns some improvement to the target, e.g. from a blurred image, a less blurred image? But then applying 49 degradations to such an image, we will not get an improvement, because the blurriness of the image after the first pass is more than will be obtained with 49 degradations in proposed sampling method.

At the moment I am experimenting with increasing the detail of the image after autoencoder using ColdDiffusion.
My algorithm is to take the output image from frozen autoencoder, calculate the difference between target and pred, and apply degradation to blur this difference by applying the difference to pred.

example degradations in dynamic:

2023-04-26_09-37-07.mp4

Then I trained the model with 50 steps and got this. Background was not trained, only the face.

2023-04-26_09-40-28.mp4

Details are increased.

But then I checked the output from 1st and further passes and I got the same image all time !

and by the way it has more details if not to use sampling degradations.

I would love to hear any comments or thoughts from you on this.

Error!!

Cold-Diffusion-Models/deblurring-diffusion-pytorch/deblurring_diffusion_pytorch/deblurring_diffusion_pytorch.py

Line 1250 in f8b1379

for i in range(len(X_0s)):

len(X_ts) < len(X_0s)

Hence, this line should be changed to for i in range(len(X_ts)):

Please check!

Add LICENSE file

Thank you for this great work! I wonder if you'd be able to add a LICENSE file to this repo? Currently, the setup.py file says that it's under the MIT license, which is the license used for the original lucidrains repo, but most of the other files don't have any license specified. Adding a LICENSE file would make it clear under what terms people are allowed to use your code, which would be really helpful!

Lack of files

In "Cold-Diffusion-Models/deblurring-diffusion-pytorch/celebA_128_test.py" Line 12:

"from gmm_pycave import GaussianMixture as GaussianMixture"

where is the "gmm_pycave"?

No Inference function

There is no implemention of function trainer.test_from_data('train', s_times=args.sample_steps) from /resolution-diffusion-pytorch/celebA_128_test.py
How can I write the inference function for super-resolution?

Pretrained models

Thanks for the creative work of Cold Diffusion. The results are very impressive.

I want to use Cold Diffusion as backbone for downstream tasks. Are there any pretrained models or checkpoint files you would like to share?

Configuration file Problem

Can you provide the detailed configuration file and hardware information? Thx a lot.

Individual_Incremental vs else in the all_sample function

In Line 610 of deblurring_diffusion_pytorch.py, the all_sample function is defined to use in test_from_data function during testing.

In here there are two options -

if self.blur_routine == 'Individual_Incremental': img = self.gaussian_kernels[t - 1](img) else: for i in range(t): with torch.no_grad(): img = self.gaussian_kernels[i](img)

One applies the blur kernel of strength at the t-1 -th step to the original image , while the other (which is the "final") applies the kernel of increasing strength to the previous step starting from original image until the last step.

My question is What is the point of this ? Is this just to compare the "power" of the cold diffusion model when the degradation is applied in this schedule ?

AttributeError: 'GaussianDiffusion' object has no attribute 'module'

Cold-Diffusion-Models/deblurring-diffusion-pytorch/deblurring_diffusion_pytorch/deblurring_diffusion_pytorch.py

Line 1241 in f8b1379

X_0s, X_ts = self.ema_model.module.all_sample(batch_size=batches, img=og_img, times=s_times) # change for DP

Modified this to:
X_0s, X_ts = self.ema_model.all_sample(batch_size=batches, img=og_img, times=s_times) # change for DP

seems to solve the error.

Quantitative comparison between the hot diffusion and the cold diffusion

This is amazing work. Thanks for your contribution.

In the Sec.3 of your paper, you present the quantitative results of using the deblurring model, the inpainting model, the SR model, and the snowfication model. I am wondering what is the hot diffusion(denoising model)'s quantitative result under the same setting as your cold diffusion experiments. Would you like to show it?

Unconditional generation using super resolution

Hi,
the instruction of unconditional generation using super resolution seems to be wrong in the README file, could you please update it? Thanks a lot!