maclory/deep-iterative-collaboration

Pytorch implementation of Deep Face Super-Resolution with Iterative Collaboration between Attentive Recovery and Landmark Estimation (CVPR 2020)

deep-iterative-collaboration's Issues

RuntimeError: CUDA out of memory. Tried to allocate 3.00 GiB (GPU 0; 14.76 GiB total capacity; 12.32 GiB already allocated; 1.39 GiB free; 12.34 GiB reserved in total by PyTorch)

Hello, I would like to inquire. When I use an image to test, I solver it in line 69 of test.py The test() video memory is out. I use the testla t4 video card with 16G video memory. I want to ask why. Thank you!

LR size 32 to SR size 128

你好，感谢作者，我以成功使用模型。但是我想要更改LR的大小，改成3232放大4倍图像，我在dic_arch.py中已经更改了
elif self.upscale_factor == 4: stride = 2 padding = 1 kernel_size = 4
在train_**.json options文件中也已经更改了scale和size大小，但是当我运行的时候，报错了
Traceback (most recent call last): File "train.py", line 207, in <module> main() File "train.py", line 114, in main iter_loss = solver.train_step() File "/content/code/solvers/SRLandmarkSolver.py", line 122, in train_step SR_list, heatmap_list = self.model(self.LR) File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "/usr/local/lib/python3.7/dist-packages/torch/nn/parallel/data_parallel.py", line 159, in forward return self.module(*inputs[0], **kwargs[0]) File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "/content/code/networks/dic_arch.py", line 102, in forward FB_out = self.block(x, merge_heatmap_5(heatmap, self.detach_attention)) File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "/content/code/networks/srfbn_hg_arch.py", line 74, in forward x = self.fusion_block(x, heatmap) File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "/content/code/networks/blocks.py", line 153, in forward feature = feature.view(batch_size, self.num_heatmap, -1, w, h) * attention.unsqueeze(2) RuntimeError: The size of tensor a (64) must match the size of tensor b (32) at non-singleton dimension 4
你能指导我怎么做吗

How to training own custom dataset?

Hi, buddy, thank your wonderful demo. I want to train own images instead of CelebA and Helen dataset. How to modify the cfg file? Could you tell me some suggest? Please.

Problem loading the model

Greetings.

Thank you uploading this code and providing access to the pre-trained models too.
There seems to be some issue when I'm trying to load the pre-trained models. I'm trying to load the DICGAN CelebA model and test it on my own dataset. I tried loading the DICGAN Helen model too but the error is same. Can you kindly guide what I'm doing wrong here?

`===> Loading model from [../models/DICGAN_CelebA.pth]...

Traceback (most recent call last):

File "test.py", line 128, in
main()

File "test.py", line 43, in main
solver = create_solver(opt)

File "/home/zeeshan/codes/super_resolution_algorithms/Deep-Iterative-Collaboration/code/solvers/init.py", line 8, in create_solver
solver = SRLandmarkGANSolver(opt)

File "/home/zeeshan/codes/super_resolution_algorithms/Deep-Iterative-Collaboration/code/solvers/SRLandmarkGANSolver.py", line 113, in init
self.load()

File "/home/zeeshan/codes/super_resolution_algorithms/Deep-Iterative-Collaboration/code/solvers/SRLandmarkGANSolver.py", line 361, in load
res_str = _load_func(self.model, checkpoint)

File "/home/zeeshan/codes/super_resolution_algorithms/Deep-Iterative-Collaboration/code/solvers/SRLandmarkGANSolver.py", line 293, in _load_func
if len(res.missing_keys) != 0:

AttributeError: 'NoneType' object has no attribute 'missing_keys'
`

Apply for LICENSE

Apply for a reference modification license.

Question about evaluation

Hi, thank you for your sharing the code! Why do you crop the image border when calculating PSNR and SSIM?
Looking forward to your reply.

training set

Thank you very much for the code you provided. I tried to create my own training set to train the network, but the effect was not very good. Can you provide the training set of celebA and Helen? We look forward to your reply!

Training set

Your article is very enlightening to me. Can you tell me how you can crop the un-aligned face image. By choosing the center of the key or what?

Can not reproduce the result using default option and model

I changed the pathes of model and images in the test_DICGAN_Helen.json and test_DIC_Helen.json, then ran test.py use these options as told in README. But the results are really terrrible. Did I miss something? The data used to test the program is Helen that downloaded as asked and preprocessed as said in #1. The models I used were DIC_Helen.pth and DICGAN_Helen.pth respectively.

Vram requirements

Hi there. I wanted to try this project with its amazing examples of face upscaling. I'm on a P5000 with roughly 13.3GB of usable VRAM in Windows 10 but I've gotten out of memory errors in trying to upscale my own image. I even cropped in to the face as far as I could but it always fails.

I'm on Python 3.8 with the latest versions of all of the requirements and cudatoolkit 10,1 (from conda)

These are some of the messages I've gotten back

RuntimeError: CUDA out of memory. Tried to allocate 5.13 GiB (GPU 0; 16.00 GiB total capacity; 5.34 GiB already allocated; 4.59 GiB free; 8.47 GiB reserved in total by PyTorch)

RuntimeError: CUDA out of memory. Tried to allocate 2.95 GiB (GPU 0; 16.00 GiB total capacity; 12.10 GiB already allocated; 962.25 MiB free; 12.12 GiB reserved in total by PyTorch)

question about evaluation and test?

Hi, During the test and train of Helen dataset, I find that the optimal model is chosen from the evaluation results, but I found that the evaluation and test used the same set of data, which made me a little confused. Is this reasonable?

About the pretrained HG model

Is it possible to change the number of key points for training DIC without adding any extra works?
I want to run some cross-domain experiments with the network you proposed, however, the number of key points(landmark) of my dataset is less than 68 which is 20 and therefore I can't use the HG pre-train model which is pretrained in 68 key points. Can I just not use the HG pre-train model to train DIC or I need to train the HG pre-train model from scratch with my own key points? Thanks for your reply.

About FSRNet

Can you tell me how to generate FSRNet images?

Question about validation

It seems that in the validating phase, it's time-consuming. Is there any way to deal with that?

How to make Low Resolution image?

Hi, Thank you for your research.

I wonder how LR image made it.

When i reproduce your code, down-sample cv2.resize(img, cv2.INTERCUBIC).
Then, result is weird.

Please, answer me.
Thank you.

作者您好，我在跑训练模型代码的时候，代码里info path 这个路径下的.pkl文件，我用的是您提供的，但是报了以下错误，请问这个文件需要自己去生成吗？希望您能解答，万分感谢。

===> Training Epoch: [1/8]... Learning Rate: 0.000100
Traceback (most recent call last):
File "train.py", line 196, in
main()
File "train.py", line 98, in main
for _, batch in enumerate(train_loader):
File "/usr/local/anaconda3/envs/DIC/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 819, in next
return self._process_data(data)
File "/usr/local/anaconda3/envs/DIC/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 846, in _process_data
data.reraise()
File "/usr/local/anaconda3/envs/DIC/lib/python3.7/site-packages/torch/_utils.py", line 369, in reraise
raise self.exc_type(msg)
TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/usr/local/anaconda3/envs/DIC/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
data = fetcher.fetch(index)
File "/usr/local/anaconda3/envs/DIC/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/usr/local/anaconda3/envs/DIC/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/wang107552002794/Deep-Iterative-Collaboration/code/data/HRLandmarkDataset.py", line 45, in getitem
hr, hr_path = self._load_file(idx) # Numpy float32, HWC, RGB, [0,255]
File "/home/wang107552002794/Deep-Iterative-Collaboration/code/data/HRLandmarkDataset.py", line 85, in _load_file
hr = util_dataset.read_img(hr_path, self.opt['data_type'])
File "/home/wang107552002794/Deep-Iterative-Collaboration/code/data/util_dataset.py", line 18, in read_img
img = cv2.imread(path)[:, :, ::-1]
TypeError: 'NoneType' object is not subscriptable

Question about the train.json

Hi,thank you very much for your sharing the code!
I have some question about the parameter setting.
In train_CelebA, there is no "distort", while in the train_helen, the "distort" is [0.66, 1.33]. Why the difference occur here. In the paper, the LR and HR from both CelebA and Helen are 16X16 and 128x128.
I am looking forward to you answer. Thanks.

The annotation Helen_test.pkl doesn't have keys of 'landmark' and 'bbox'.

Hi authors, thanks for your excellent work. I am a newer for face super-resolution.
Now I am trying to train DIC model by using Helen dataset, but for the Helen_train.json, the 'info_path' in the 'Helen_train.json', if I set it as 'Helen_test.pkl' file, it shows an error 'IndexError: only integers, slices (:), ellipsis (...), numpy.newaxis (None) and integer or boolean arrays are valid indices'. Any advice will be appreciated, thank you!

The data format in the Helen_test.pkl file in annotations is incorrect.

The data format in the Helen_test.pkl file in annotations is inconsistent with other pkl files, resulting in the inability to use the get_info function to read the image information. Could you please update this file? Thank you very much. If you can, can you tell me how these files are generated?

Questions about dataset

Hi,thank you very much for your sharing the code!
I have some question about why Face SR is dataset depedent.

I tried using the Helen model on the CelebA dataset, but the results were not good.
Also, I tried using Helen model and CelebA model to some real-world small faces, but results were bad.

Is this model need some fine-tuning to use in other dataset?

RuntimeError: Invalid magic number; corrupt file?

Hi! When test.py read the pre-trained model 'CelebA_test.pkl', RuntimeError raised.

It seems that something went wrong to read pkl file with torch.load() function.

train FeedbackHourGlass to detect landmarks

hi, buddy,
I want to know the way you train your FeedbackHourGlass model, the file FB_HG_68_CelebA.pth you used as a pretrained landmark detect model for DIC or DIC-GAN. When I train this model alone, the loss is very large. Can you tell me how you train it? Thanks a a lot.

Has anyone tested the effect on other datasets?

I first sampled the LFW dataset, then used the pre-training model of the original author to perform super-resolution on the LFW dataset, and serious artifacts appeared in the face image.

Has anyone tested the effect on other face data sets? Would you like to know what caused it? Thank you!

更改scale的一些问题

你好作者。我想尝试使用32*32的LR图片放大4x到SR图片的时候，我遇到问题了。我已经将Option/train.json的scale=8, lr_size=32,
在dic_arch.py里也添加了

self.upscale_factor == 4:
    # with PixelShuffle at start, need to upscale 2x only
    stride = 2
    padding = 1
    kernel_size = 4

但是我尝试运行的时候, 发现attentive fusion块表示维度不对应，我在论文观察结构中知道Face Super-Resolution分支中，LR图片经过
Pixel Shuffle后相比于之前的1616size，输入进当前分支的Conv 11的时候，因为image的size已经成了6464的大小，他在输入
Attentive Fusion时候与featmap的3232维度不匹配了，于是乎我定位到StackedHourGlass.py文件，尝试更改pre_conv_block块，
尝试让其生成一个64*64的feat map和feature；在尝试去掉MaxPool2D层，但是失败了，报错信息显示预加载的HG模型权重不匹配。
我要重新训练一个HG模型权重吗?? 但是预加载模型不影响Face Alignment分支的A1部分块(即在StackedHourGlass.py/self.pre_conv_block)吧?
请问一下作者，给我一些指导意见吗？ 感谢万分

关于数据集人脸关键点的问题

您好作者，能分享下您的人脸关键点数据集制作代码吗？（pkl）
请问您选择的是哪种模型得到标准人脸关键点数据的？是模型中的HG吗？
如果使用预先设置好的数据集（lr+hr+人脸关键点）三种数据作为模型的输入，是否能加快训练呢？

Upscaling Factor of 4

Greetings.

This is an interesting work. Congratulations to the authors on achieving this. I have been able to test the algorithm for up scaling images 8 times. I was wondering, has the algorithm been trained for 4 times up scaling too? If so, can you provide the pre-trained weights for it?

Thank you.

About pretrained HourGlass model

How to pretrain the HourGlass model? Can you provide more details? Thank you!

Questions about inference

Hi,

First of all confratulations!!

I ran the test.py with the celeb database images that have the HR and LR images successfully. But I would like to know if it is possible to run any other image LR only with face landmarks.

Thank you for your help!
Kevin

About test eval_landmark.py

Hello, author, thanks for your sharing.

I found that when I use your code eval_landmark.py to test the real CelebA image, the results is not good, like follows.

The model I used is HG_68_CelebA.pth that I download in your shared link, the test image is CelebA-HQ Image 0.jpg, and I resize it to 256.

Lower upscale factor

Hello, I was wondering if there's a way to train the model but use x4 upscale factor instead of x8, thank you!

How to generate SR image with my own LR image

Hi @Maclory and @Steve-Tod , thanks for your excellent work! I can successfully test the model using provided test datasets (CelebA|Helen).
However, I want to test my own LR image without anymore information (no according SR image or landmark annotation), how could I use your model to generate according SR image? Which file should I run or modify?
Thank you in advance!

Can I run the code with amd gpu? Or only nvidia gpu are usable?

如何生成pkl格式文件

作者你好，我正在尝试使用自己的数据集复现您的代码，我已经使用openface生成了CSV格式的数据（其中包含2D的Facia landmark），但是我不知道如何生成此代码用的pkl文件作为GT。希望作者能够提供支持。谢谢！

How to make Visual Result in the paper.

Hi,

I'm graduate student studying about FSR.

May i ask you how to make Visual Result(visual comparison with SOTA FSRs) in the paper and this repository?

Because in my case, it shows different(weird) results depending on how i crop face in the image.
I tried cropping while maintaining face aspect ratio. But still the result is weird.

The result here in CelebA dataset tightly cropped by RetinaFace ..
Second figure is padding image to maintain aspect ratio.

And this is result by using dataset presented by this repository.

Can you explained that?

while running test.py(RuntimeError: CUDA out of memory)

Help Please

0%| | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
File "test.py", line 128, in
main()
File "test.py", line 69, in main
solver.test()
File "/home/govind/audio-video/image-enhance/Deep-Iterative-Collaboration/code/solvers/SRLandmarkGANSolver.py", line 220, in test
SR_list, heatmap_list = forward_func(self.LR)
File "/home/govind/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
return self.module(*inputs[0], **kwargs[0])
File "/home/govind/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/govind/audio-video/image-enhance/Deep-Iterative-Collaboration/code/networks/dic_arch.py", line 89, in forward
FB_out_first = self.first_block(x)
File "/home/govind/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/govind/audio-video/image-enhance/Deep-Iterative-Collaboration/code/networks/srfbn_hg_arch.py", line 130, in forward
LD_H = self.downtranBlocksidx-1
File "/home/govind/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/govind/anaconda3/lib/python3.7/site-packages/torch/nn/modules/container.py", line 100, in forward
input = module(input)
File "/home/govind/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/govind/anaconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 345, in forward
return self.conv2d_forward(input, self.weight)
File "/home/govind/anaconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 342, in conv2d_forward
self.padding, self.dilation, self.groups)
RuntimeError: CUDA out of memory. Tried to allocate 3.00 GiB (GPU 0; 15.90 GiB total capacity; 12.32 GiB already allocated; 2.86 GiB free; 12.34 GiB reserved in total by PyTorch)

and current configurations:
{
"name": "CelebA_GAN",
"mode": "sr_align_gan",
"degradation": "BI",
"gpu_ids": [0],
"use_tb_logger": false,
"scale": 8,
"is_train": false,
"rgb_range": 1,
"save_image": true,
"datasets": {
"test_CelebA": {
"mode": "LRHR",
"name": "CelebA",
"dataroot_HR": "/home/govind/audio-video/image-enhance/Deep-Iterative-Collaboration/image/",
"dataroot_LR": "/home/govind/audio-video/image-enhance/Deep-Iterative-Collaboration/image/", single image inside
"data_type": "img",
"LR_size": 16,
"HR_size": 128
}
},
"networks": {
"which_model": "DIC",
"num_features": 48,
"in_channels": 3,
"out_channels": 3,
"num_steps": 4,
"num_groups": 6,
"detach_attention": false,
"hg_num_feature": 256,
"hg_num_keypoints": 68,
"num_fusion_block": 7
},
"solver": {
"pretrained_path": "/home/govind/audio-video/image-enhance/Deep-Iterative-Collaboration/models/DICGAN_CelebA.pth"
},
"path": {
"root": "../"
}
}

The loss value very huge

Hi, buddy, your demo is wonderful. Here is my training question of train.py demo:
I used a CelebA face dataset training, and I got loss value:
[CelebALandmarkVal] PSNR: 9.29 SSIM: 0.0175 Loss: 1433107.623884 Best PSNR: 9.40 in Step: [10000]
===> Saving last checkpoint to [../experiments/DIC_in3f48_x8_myself/epochs/step_0014000_ckp.pth] ...]
Fri Sep 25 11:16:13 2020 <epoch: 1, iter: 14,250, lr:1.250e-05> loss_g_pix: 4.3555e+06 loss_g_align: 9.2217e+18 loss_g_feature: 4.9574e+06 loss_g_GAN: 5.4255e+08 loss_g: 9.2217e+17 loss_d: 0.0000e+00 loss_total: 9.2217e+17 pred_d_real: 3.3765e+01 pred_d_fake: -5.4255e+08
Fri Sep 25 11:18:13 2020 <epoch: 1, iter: 14,500, lr:1.250e-05> loss_g_pix: 4.7608e+06 loss_g_align: 1.0164e+19 loss_g_feature: 5.4015e+06 loss_g_GAN: 5.9279e+08 loss_g: 1.0164e+18 loss_d: 0.0000e+00 loss_total: 1.0164e+18 pred_d_real: 3.4847e+01 pred_d_fake: -5.9279e+08
Fri Sep 25 11:20:13 2020 <epoch: 1, iter: 14,750, lr:1.250e-05> loss_g_pix: 4.3234e+06 loss_g_align: 2.2932e+19 loss_g_feature: 4.8757e+06 loss_g_GAN: 5.3533e+08 loss_g: 2.2932e+18 loss_d: 0.0000e+00 loss_total: 2.2932e+18 pred_d_real: 2.9394e+01 pred_d_fake: -5.3533e+08
Something wrong in my dataset? Or train.json file? Hope you let me know what's the detail. Thank you.

Creating Annotations for Training on Other Datasets

I am attempting to try and train a new model using a higher resolution HR and LR data sets. The CelebA-HQ and FFHQ are both at 1024 resolution. I am wondering what type of annotation is expected for these datasets and if a custom annotation was used for the CelebA set provided here.

I notice on the CelebA homepage they talk about using 5 landmarks and 40 binary attributes in their annotation but it appears this repo is using 68 point dlib landmarking? Is detecting and generating the expected landmark annotations for an image set already something in this code I can leverage to create new annotations for new data sets? I see there are some pretrained detection models and am wondering if I should use those or should I use a tool like OpenFace or ImgLab to generate 68 point landmark annotations with dlib in pkl format?

I hope these aren't stupid questions and greatly appreciate any tips you can give on preparing a custom data sets for training

Can I run the test code with cpu?

Can I run the test code with cpu?

The processing of image in HG model

Hi, when you train the HG model, what's the image processing? In my view, I think dividing 255 is necessaray, but do you substract the mean of images afterwards? If so, the mean of images is the same?

The train result is 24.10dB that much lower than paper's

I changed the train two parameter of defualt:
gpu_idx: [0,1] -> [0]
n_workers: 8 -> 0

did it influence the result?

questions about testing

During testing, in my opinion, it will produce an 8x scale SR image w/o annotations (pkl files)?.
Can this model produce a face deblurred image without resolution changing?
Due to this model based on 8x scale, how much memory is needed to upscale a 128x128 image?

Size of tensor A must match size of tensor B

Dear Author,
Reference to my previous opened issue #16, can you kindly guide me further over what exactly is causing this error? The previous issue was closed before my problem being solved. I am still stuck at the same error. I am working with pre trained weights of CelebA dataset, however the issue exists on with pre trained weights on Helen dataset as well. I am working with the DICGAN model. Some specific resolutions that the code successfully works on are 24x24, 32x32, 40x40 and 56x56. The code fails with this error on almost all resolutions between these, as trial and error has showed.

Runtime error: Size of tensor A must match size of tensor B

Greetings.

I am trying to run the code on separate images. My first question is that, is it necessary for the input to this algorithm to have the same width and height (a square image)?
If so, I have tried resizing the image to convert it into an image with the same number of pixels in its width and height. When I have given this resized image as input, I get the error, "RuntimeError: The size of tensor a (17) must match the size of tensor b (16) at non-singleton dimension 3". The input image is a small one below 64x64 pixels always. Interestingly, the algorithm works for some of these images and gives the 8x upscaled output, but fails with this error for most of the images.
Can you kindly guide me over what is causing this error?

How to define the training/testing set

Thanks for your excellent work. If I want to train the model on Helen from scratch. Should I take all of the 5 parts from All images on Helen website as the training set? Or whether the testing set is selected from these images, so that this part needs to be excluded in advance? How can I get the list of training images.

img changed to HD

Hello, I have a picture that needs to be changed to HD. What should I do?

can the model work for pictures with multiple faces?

good work. Can the model work for pictures with multiple faces?

Some questions about crop Helen & CelebA datasets.

How do your crop face region in image? by hand? Thank you.

Annotation

Hi,thank you very much for your sharing the code, I have some confusion about how the .pkl in the annotation file is generated Looking forward to your reply.

tensorflow version

Thanks for your excellent work.Do you have tensorflow verision about this code?

Input question

Can't the input be a jpg file?