megvii-model/hinet

hinet's Issues

Boundary artifacts in output images

Hi,

I was able to reproduce your results on PSNR/SSIM. However, I observed that the output images have border artifacts even though I turned off the grids attribute:

I guessed during inference the image was paritioned, inferenced, and then stitched back. I am not very familiar with basicsr lib, so I simply tried turning off girds attribute, but it had no effect. Any ideas on what happened? Thank you!

Why half normalization?

Hi, thinks for this great work.
I wonder how did you come up with this half normalization idea, is there are some reference paper with similar and detailed explain?
Best regards.

关于HIN的params和flops？

因为HIN block这个块是自带卷积的，所以在测试中发现HIN带来的params和flops的增长是比较大的，所以想问一下paper里面的HIN消融实验的结果，params和flops是否进行过对齐？

Image Deblur - Custom dataset Error

Thanks for writing a good paper.
I have data[input,target] that I have.
Now

./datasets/
./datasets/GoPro/
./datasets/GoPro/train/
./datasets/GoPro/train/input/
./datasets/GoPro/train/target/
./datasets/GoPro/test/
./datasets/GoPro/test/input/
./datasets/GoPro/test/target/

Create a folder in the same way as

python scripts/data_preparation/gopro.py
Through preprocessing, blur_crops, blur_crops.imdb / sharp_crops, sharp_crops.imdb datasets were created.

Finally

python -m torch.distributed.launch --nproc_per_node=8 --master_port=4321 basicsr/train.py -opt options/train/GoPro/HINet.yml --launcher pytorch

I tried to learn through the above command.
But I got the following error:

ValueError: Keys in lq_folder and gt_folder are different.
...
...
subprocess.CalledProcessError: Command '['/home/ubuntu/anaconda3/envs/HINet/bin/python', '-u', 'basicsr/train.py', '--local_rank=7', '-opt', 'options/train/GoPro/HINet.yml', '--launcher', 'pytorch']' returned non-zero exit status 1.

How can I fix the error?

Thanks.

About the model

We don't seem to have found the structural code for the specific details of the model and in which directory it is located.

Training Cost

Hey @mayorx ,

Thanks for your impressive work! Since we want to do some similar work based on your project, we are in short of computation resources.

Hence, may I know the training cost of your work?

What kind of GPUs were you using? and How many GPUs? May I know how long you cost to train a model for a draining task?

Best Regards and many thanks,

About training

Hi.
Thank you very much for open source your code. But I used your network model and hyperparameters to train on the GoPro dataset (the training process was written by myself), and the gradient explosion problem occurred. I tried for a long time, but I didn’t find the problem. Do you have any suggestions?

数据集的问题

我按照您的方式组织了数据集的结构，但是报错：
FileNotFoundError: [Errno 2] No such file or directory: './datasets/GoPro/train/blur_crops.lmdb/meta_info.txt'
请问’blur_crops.lmdb/meta_info.txt‘是什么呢 我该如何解决

About datasets

dear author:
I have a question that training on gopro, have 2 blur folder, that blur and blur_gamma, I want to know you used is blur or blur_gamma? thankyou

Inference code for a single image

Great job creators! Very impressive work!
I wonder what book influenced your python skills the most.

Is this project adopting the MPRNets' license?

Any plans for realising inference code for a single image. E.g demo code. If not maybe I can submit a pull request, any preferences for a way it should be formalized?

Keep it up!

关于损失函数和hin block

您好，感谢您做出的杰出的贡献！
想请教您是否在损失函数上可以结合一些其他的函数以便更能发挥网络的优点

About train

Hi, this is a nice job. However, Do you use the default parameters in "train/REDS/HINet.yml" to train HINet? I can not reproduce the results of the pre-trained model you provided.

What is the operation of "python setup.py develop --no_cuda_ext" used for?

maybe you have forgotten: "python setup.py develop --no_cuda_ext"

Originally posted by @mexthecat in #38 (comment)

What is the operation of "python setup.py develop --no_cuda_ext" used for?

What is the lincense of the Denoise project？

ImportError

Your network does not run on my research machine:

Traceback (most recent call last):
  File "basicsr/demo.py", line 10, in <module>
    from basicsr.models import create_model
ImportError: cannot import name 'create_model'

EDIT: Just tried on my local computer (RTX 3060) different error:

Traceback (most recent call last):
  File "/home/X/HINet/basicsr/demo.py", line 10, in <module>
    from basicsr.models import create_model
ModuleNotFoundError: No module named 'basicsr'

关于预训练模型

Hi,您好，非常感激您所做的工作，
请问您的预训练模型是如何得到的呢，我看到好像是用HINet在不同的小数据集（比如Gopro/SSID等）上预训练得到的（？），而不是用类似vgg这样的通用模型在ImageNet上预训练得到的。
如果我想做一些调整比如更改您的结构，是不是也要用和您同样的方式进行预训练呢

About the real rain dataset

I would like to compare the performance of these two real rain images, can you send me the result image?

Size miss matched

When running the SIDD-1x pre trained model this error generated.

\basicsr\models\archs\hinet_arch.py", line 179, in forward
out = torch.cat([up, bridge], 1)
RuntimeError: torch.cat(): Sizes of tensors must match except in dimension 1. Got 10 and 11 in dimension 2 (The offen
ding index is 1)

wrong display: Validation SIDD_val, # psnr: 23.2228

hi, author ,thanks for your enjoy, it's a nice work! But when i do reproduce of HINet in sidd dataset, during training(train.py) i can get psnr39.9124, in(test.py)i used the trained model, got the wrong display: Validation SIDD_val, # psnr: 23.2228, i don't know what went wrong?

psnrloss

训练自己的数据集的时候发现Loss变成nan了，后来发现torch.log((pred - target) ** 2)，当两张图的差异很小时，此处变成了负数。现在换成L1loss重新训练看看

Request for denoising demo

Hi,

I really appreciate placing a demo for deblurring on GoPro images. Can you also provide a demo for denoising with SIDD models? It will be really helpful if you can provide some description of how the .yml parameters impact the inference on images with different pre-trained models.

Thank you very much!

Yusuf

请教您关于PSNR损失

感谢和赞叹您的成就！
关于PSNRLOSS，它看起来是类似MSELOSS的。在文章中的PSNRLOSS公式中，我有些疑问：请问为什么是（Ri+Xi），Y 呢，我个人以为是仅使用每阶段的重建图像（Xi）和ground truth（Y）计算即可。另外，Ri作为输入的模糊图像，我以为两阶段的Ri是相同的，但是实际上是不同的吗？（因为您用了下标i）。这两个问题我比较费解，能否请您帮忙讲解。
期待您的回复！再次感谢您的耐心！

System Requirements?

Dear Author:

You have done a great job. I am planning to work on your research work. I have few questions for you.

1. What are the system requirements?
2. Is it possible to implement it on Google Colab? If yes can you share how have you done it?

Thank you,

Did it work on Image Super-resolution (SR) task？

Interesting work!
Have you tried the Image Super-resolution (SR) task?

Running demo

I get the following shape mismatch error whenever I try to run the demo:

Disable distributed.
load net keys <built-in method keys of collections.OrderedDict object at 0x7f04c6da2540>
Traceback (most recent call last):
File "basicsr/demo.py", line 46, in
main()
File "basicsr/demo.py", line 40, in main
model = create_model(opt)
File "/media/spoodermun/YoData/FISH/noise_mitigation/HINet/basicsr/models/init.py", line 44, in create_model
model = model_cls(opt)
File "/media/spoodermun/YoData/FISH/noise_mitigation/HINet/basicsr/models/image_restoration_model.py", line 36, in init
self.load_network(self.net_g, load_path,
File "/media/spoodermun/YoData/FISH/noise_mitigation/HINet/basicsr/models/base_model.py", line 287, in load_network
net.load_state_dict(load_net, strict=strict)
File "/media/spoodermun/YoData/morph/morphology_detection/YOLO/pytorch-env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1223, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for HINet:
size mismatch for down_path_1.0.identity.weight: copying a param with shape torch.Size([64, 64, 1, 1]) from checkpoint, the shape in current model is torch.Size([32, 32, 1, 1]).
size mismatch for down_path_1.0.identity.bias: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([32]).
size mismatch for down_path_1.0.conv_1.weight: copying a param with shape torch.Size([64, 64, 3, 3]) from checkpoint, the shape in current model is torch.Size([32, 32, 3, 3]).

The demo config is as follows:

------------------------------------------------------------------------

Copyright (c) 2021 megvii-model. All Rights Reserved.

------------------------------------------------------------------------

Modified from BasicSR (https://github.com/xinntao/BasicSR)

Copyright 2018-2020 BasicSR Authors

------------------------------------------------------------------------

general settings

name: demo
model_type: ImageRestorationModel
scale: 1
num_gpu: 1 # set num_gpu: 0 for cpu mode
manual_seed: 10

single image inference and save image

img_path:
input_img: ./noise_mitigation/test_images/enhanced/1p36_0_0.png
output_img: ./demo/demo1.png

network structures

network_g:
type: HINet
wf: 32
hin_position_left: 0
hin_position_right: 4

path

path:
pretrain_network_g: ./noise_mitigation/HINet/HINet-SIDD-1x.pth
strict_load_g: true
resume_state: ~

validation settings

val:
grids: true
crop_size: 256

dist training settings

dist_params:
backend: nccl
port: 29500

Unable to use other pretrained weights

Hi,

Thank you for this wonderful project. I am able to successfully run the demo with the GOPRO weights on multiple images and it works flawlessly. However, if I just tweak the demo.yml file and change the pre-trained weights to HINet-SIDD-1x.pth, I get an error with the state_dit. I am new to PyTorch, so this may be a trivial issue. Your help and guidance are much appreciated. I am attaching the demo.yml file to this post and the output:

# ------------------------------------------------------------------------
# Copyright (c) 2021 megvii-model. All Rights Reserved.
# ------------------------------------------------------------------------
# Modified from BasicSR (https://github.com/xinntao/BasicSR)
# Copyright 2018-2020 BasicSR Authors
# ------------------------------------------------------------------------
# general settings
name: demo
model_type: ImageRestorationModel
scale: 1
num_gpu: 1 # set num_gpu: 0 for cpu mode
manual_seed: 10

# single image inference and save image
img_path:
  input_img: ./demo/GOPR0384_11_00-000001.png
  output_img: ./demo/demo1.png

# network structures
network_g:
  type: HINet
  wf: 64
  hin_position_left: 3
  hin_position_right: 4

# path
path:
  pretrain_network_g: ./experiments/pretrained_models/HINet-SIDD-1x.pth
  strict_load_g: true
  resume_state: ~

# validation settings
val:
  grids: true
  crop_size: 256

# dist training settings
dist_params:
  backend: nccl
  port: 29500

The error log is as follows:

(HINet) C:\Users\Yusuf\Desktop\HMC Research\HINet>python basicsr/demo.py -opt options/demo/demo.yml
Disable distributed.
 load net keys <built-in method keys of collections.OrderedDict object at 0x000002284651FEA0>
Current net - loaded net:
Loaded net - current net:
  down_path_1.0.norm.bias
  down_path_1.0.norm.weight
  down_path_1.1.norm.bias
  down_path_1.1.norm.weight
  down_path_1.2.norm.bias
  down_path_1.2.norm.weight
  down_path_2.0.norm.bias
  down_path_2.0.norm.weight
  down_path_2.1.norm.bias
  down_path_2.1.norm.weight
  down_path_2.2.norm.bias
  down_path_2.2.norm.weight
Traceback (most recent call last):
  File "basicsr/demo.py", line 46, in <module>
    main()
  File "basicsr/demo.py", line 40, in main
    model = create_model(opt)
  File "c:\users\yusuf\desktop\hmc research\hinet\basicsr\models\__init__.py", line 44, in create_model
    model = model_cls(opt)
  File "c:\users\yusuf\desktop\hmc research\hinet\basicsr\models\image_restoration_model.py", line 37, in __init__
    self.opt['path'].get('strict_load_g', True), param_key=self.opt['path'].get('param_key', 'params'))
  File "c:\users\yusuf\desktop\hmc research\hinet\basicsr\models\base_model.py", line 287, in load_network
    net.load_state_dict(load_net, strict=strict)
  File "C:\Users\Yusuf\anaconda3\envs\HINet\lib\site-packages\torch\nn\modules\module.py", line 847, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for HINet:
        Unexpected key(s) in state_dict: "down_path_1.0.norm.weight", "down_path_1.0.norm.bias", "down_path_1.1.norm.weight", "down_path_1.1.norm.bias", "down_path_1.2.norm.weight", "down_path_1.2.norm.bias", "down_path_2.0.norm.weight", "down_path_2.0.norm.bias", "down_path_2.1.norm.weight", "down_path_2.1.norm.bias", "down_path_2.2.norm.weight", "down_path_2.2.norm.bias".

训练设置

你好，请问在只有两张卡的情况下，该如何设置初始学习率和训练的迭代次数呢？

Pretrained model settings in SIDD model

Hi,

Which settings should be used when using SIDD pretrained model and to run demo.py? Especially the lines in the following file are having mismatches when using different pretrained model than the deblurring example.

https://github.com/megvii-model/HINet/blob/main/options/demo/demo.yml#L20-L24

在训练中的一些疑问

你好，感谢你开源如此棒的源码。
我在训练HINet时，遇到几个问题，想和你探讨一下。

1. 在推理时，你的做法是分块进行推理。但分块推理会造成图片的不对齐，有明显的裂缝。请问这个该如何解决呢？使用全图进行推理吗？
2. 在预测时，对于同一张图，有些区域的效果会好，但有些区域的效果会变更差（例如变得更加模糊），请问有遇到过这个问题吗？可能是那里出现了问题呢？
3. 在实际应用中，图片可能是清晰的，是否有一个判断准则，来判断图片清晰而不进行denoise或者deblur呢？
   谢谢。

No module named ‘basicsr’

想要训练，但是所有的import处都在报错： No module named ‘basicsr’,请问是怎么回事呢

Did it work on image dehazing task？

▲
Is this wonderful HINet useful for image dehazing

a question about the validation

hello, thanks for your excellent work, but i have a question about the validation set, the validation set of MPRNet is Rain100H, but your validation sets is all the five test sets, is it fair?

NameError: name 'random_augmentation' is not defined

Hi

Could you please provide function for random_augmentation

Thanks

Train period

Thanks for ur great job! When i run the train.py with "python3 basicsr/train.py -opt options/train/GoPro/HINet.yml", i encountered the message "No such file or directory: '../datasets/GoPro/train/blur_crops.lmdb/meta_info.txt'" i want to know what's the content of this file and how can i make this file.
Thank u in advance!

About the test results

How can I show the image after the test?

Question about the number of parameters and the amount of computation

Hello! Thank you for your work.
I measured the number of parameters of the HINet model for deblurring.
Is it correct that the number of parameters of HINet used for deblurring is 88.67M (Million)?

The training GOPRO dataset seems to be stuck

The training GOPRO dataset seems to be stuck, and it keeps showing 2021-11-09 12:28:47,364 INFO: Start training from epoch: 0, iter: 0

About Inference

Hi, how can I Inference multiple images in the folder not just a single image?
Thank

关于训练的问题

您好！
感谢您的工作，我有一个疑问，如果我不使用多进程进行训练的话，我应该如何设置这条命令呢“python -m torch.distributed.launch --nproc_per_node=8 --master_port=4321 basicsr/t.py -opt options/train/GoPro/HINet.yml --launcher pytorch”？
现在我设置的一直在报错，非常期待能够得到您的回复！

Question of chunk function in the HIN Block

Hello,

I'm a student learning the image restoration lately.
I recently study the architecture of HINet, and I really appreciate you to provide the complete code.

And I have a question about the HIN Block.
I want to know why you use chunk function to split feature's channels to two parts in the beginning of HIN Block,
rather than using the whole channel feature after 3x3 convolution.
Then, use aggregation or concatenation to fuse two feature maps.
Or you had done some experiments to summarize the final architecture. I want to realize your idea~!

Thank you!

Output image problems

Hello.

I really appreciate you to propose the excellent paper.
I am trying HINet and have a question.

We are facing the problem that when we run the program, some of the images are output with black fill.
At this time, there are no errors or other problems.
If I should show you the image, I will paste it here or send it to you by email.

Thank you!

FileNotFoundError: [Errno 2] No such file or directory: './datasets/SIDD/val/input_crops.lmdb/meta_info.txt'

This is nice work and thanks for your share. however, when I train the model there have some issues, in the val dataset. I found you utilize the VAlidationNoisyBlocksSrgb.mat to validation the model. but there have some mistakes when I run the program. can you help me to solve this problem? thank you very much.

About test time ensemble.

Hi,

Thanks for sharing this work. About the SIDD results in your paper, is that reported under test time ensemble or single inference?

Best,

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [3, 64, 3, 3]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

When I used your model and wrote my own training code, I found a bug.

def main(opt):

loader_train = getDataLoader(opt)
model = getModel(opt)
model_saver = utils.modelSaver(opt['save_path'], opt['save_freq'])
criterion = nn.L1Loss()
model, criterion = updateGPU(model, criterion, opt)
optimizer, scheduler = getOptimizer(model, opt)

last_avg_psnr = 0
last_avg_loss = 0
for epoch in range(0, opt['epochs']):
    
    scheduler.step(epoch)
    avg_psnr = 0
    avg_loss = 0

    num_iterations = len(loader_train)
    num_iter_counts = 0
    l_total = 0
    loss = 0.
    for i, (input_train, target_train) in enumerate(loader_train):

        if(random.random() > opt['sampling_rate']):
            continue

        num_iter_counts += 1

        model.train()
        model.zero_grad()
        optimizer.zero_grad()

        input_train = Variable(input_train)
        target_train = Variable(target_train)
        
        if opt['use_gpu']:
            input_train = input_train.cuda()
            target_train = target_train.cuda()
        
        out_train = model(input_train)
        if not isinstance(out_train, list):
            out_train = [out_train]
        for pred in out_train:

            loss += criterion(target_train, pred)
        torch.autograd.set_detect_anomaly(True)
        loss.backward(retain_graph=True)
        optimizer.step()

        model.eval()
        out_train= model(input_train)
        for pp in out_train:
            pp += torch.clamp(pp, 0., 1.)
        psnr_train = utils.batch_PSNR(pp, target_train, 1.)
        avg_psnr += psnr_train 
        avg_loss += loss.item() 
            
        print("[epoch %d/%d][%d/%d], last avg loss: %.4f, avg PSNR: %.4f" % (epoch+1, opt['epochs'], i+1, num_iterations, last_avg_loss, last_avg_psnr), end='\r')

    # macs, params = get_model_complexity_info(model, (3, 80, 80), as_strings=True,
    #                                          print_per_layer_stat=True, verbose=True)
    # print('{:<30}  {:<8}'.format('Computational complexity: ', macs))
    # print('{:<30}  {:<8}'.format('Number of parameters: ', params))
    last_avg_psnr = avg_psnr / (num_iter_counts * 1.0)
    last_avg_loss = avg_loss / (num_iter_counts * 1.0)
    model_saver.saveModel(model, epoch, last_avg_psnr)

File "/home/szp/src/train.py", line 158, in
main(opt)
File "/home/szp/src/train.py", line 105, in main
loss.backward(retain_graph=True)
File "/home/szp/.local/lib/python3.6/site-packages/torch/tensor.py", line 245, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "/home/szp/.local/lib/python3.6/site-packages/torch/autograd/init.py", line 147, in backward
allow_unreachable=True, accumulate_grad=True) # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [3, 64, 3, 3]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

Process finished with exit code 1

测试预训练模型

我使用您提供的去模糊预训练模型在GoPro测试psnr32.77和论文给出的32.71不符，是重训的模型吗？另外我发现大量图片中存在很亮的竖条纹，是本来就这样，还是我这边测试的问题，望回复，感谢！

训练问题

1.大量的import报错
2.输入的数据需要是lmdb格式的？ValueError: lq folder and gt folder should both in lmdb formats. But received lq: ./datasets/de_white/train/input; gt: ./datasets/de_white/train/target

Google Colab demo

Thank you for sharing this impresive results.
Do you plan to implement the Google Colab demo?

Question about denoising task

Hi thank you for the great work
I have a question about denoising task (training SIDD)

In the options, it seems like training SIDD only use L1 loss not PSNR loss
Is this option same with your paper experiment setting?
Then use only L1 loss is sufficient for denoising task?

Thank you!

Num of parameters in Table 4.

Hi, thanks for sharing this great works! It seems MPRNet uses different parameters for each task, (like channels=40, 64, 80, etc). Could you please add an extra column in Table 4 showing the exact parameters for each model? I really appreciate if you can do so!

Import error

I have tried many ways, how should i deal with this ??