xinntao/esrgan

ECCV18 Workshops - Enhanced SRGAN. Champion PIRM Challenge on Perceptual Super-Resolution. The training codes are in BasicSR.

Hi there 👋

Homepage: https://xinntao.github.io/

GoogleScholar: https://scholar.google.com.hk/citations?user=FQgZpQoAAAAJ

esrgan's Issues

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8d in position 9: invalid start byte

Then I want to lauch test.py I have this error:
''Traceback (most recent call last):
File "test.py", line 21, in
model = model.to(device)
File "C:\Users\Никита_Владимир-ПК\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 381, in to
return self._apply(convert)
File "C:\Users\Никита_Владимир-ПК\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 187, in _apply
module._apply(fn)
File "C:\Users\Никита_Владимир-ПК\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 187, in _apply
module._apply(fn)
File "C:\Users\Никита_Владимир-ПК\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 193, in apply
param.data = fn(param.data)
File "C:\Users\Никита_Владимир-ПК\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 379, in convert
return t.to(device, dtype if t.is_floating_point() else None, non_blocking)
File "C:\Users\Никита_Владимир-ПК\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\cuda_init.py", line 163, in _lazy_init
_cudart = load_cudart()
File "C:\Users\Никита_Владимир-ПК\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\cuda_init.py", line 60, in load_cudart
lib = find_cuda_windows_lib()
File "C:\Users\Никита_Владимир-ПК\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\cuda_init.py", line 33, in find_cuda_windows_lib
out = out.decode().strip()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8d in position 9: invalid start byte''
Why this is happening ? Tell me please. My PC spec`s^
Windows 10
AMD Ryzen 2400g 3.89 Ghz
Geforce 1060 4 GB
8 GB RAM

keep going if some file processing fail ?

Hello.
When i use the test.py all is working great, but if i have some picture bigger than my computer can process in a whole set of picture, the processing stops then it can't process one file.
Is there any way to keep the processing going to the nexts file in case of processing error on one file ?

thanks

PermissionError: [WinError 5] 拒绝访问。

模型训练了一段时间后就出现拒绝访问的错误，重新启动又能训练一阵，但是训练了一阵之后又出现这种错误。

RuntimeError: cuda runtime error (30) : unknown error at ..\aten\src\THC\THCGeneral.cpp:51

Hi, I installed CUDAv10 on Windows 10, but I'm not able to run the test script:

x:\GIT\ESRGAN>python test.py models/RRDB_ESRGAN_x4.pth
THCudaCheck FAIL file=..\aten\src\THC\THCGeneral.cpp line=51 error=30 : unknown error
Traceback (most recent call last):
  File "test.py", line 21, in <module>
    model = model.to(device)
  File "c:\DEV\Utils\Python37\lib\site-packages\torch\nn\modules\module.py", line 381, in to
    return self._apply(convert)
  File "c:\DEV\Utils\Python37\lib\site-packages\torch\nn\modules\module.py", line 187, in _apply
    module._apply(fn)
  File "c:\DEV\Utils\Python37\lib\site-packages\torch\nn\modules\module.py", line 187, in _apply
    module._apply(fn)
  File "c:\DEV\Utils\Python37\lib\site-packages\torch\nn\modules\module.py", line 193, in _apply
    param.data = fn(param.data)
  File "c:\DEV\Utils\Python37\lib\site-packages\torch\nn\modules\module.py", line 379, in convert
    return t.to(device, dtype if t.is_floating_point() else None, non_blocking)
  File "c:\DEV\Utils\Python37\lib\site-packages\torch\cuda\__init__.py", line 162, in _lazy_init
    torch._C._cuda_init()
RuntimeError: cuda runtime error (30) : unknown error at ..\aten\src\THC\THCGeneral.cpp:51

Any idea why? (maybe using LSD instead? ;-)

ImportError: No module named cv2

Hello,
I followed the various steps and installed all of the dependancies. When I run the test code I get: ImportError: No module named cv2

cv2 is located at: /usr/local/lib/python3.7/site-packages/cv2

I have a file called .bash_profile located at /Users/MyUserName/.bash_profile which I added this line to: export PATH="/usr/local/opt/python/libexec/bin:$PATH"

Also I get an error after installing PsyTorch and running the first line of their test code to verify correct install: from future import print_function

That test fails with error: from: can't read /var/mail/future

I have no idea what's going on with this as it's my first time trying to run ESRGAN

I used Homebrew to install Python 3 then used the pip3 command to install the required dependencies. I did not use Anaconda as I've never used that before but have used Homebrew many times in the past on other projects.

Any help is much appreciated:-)

UserWarning: nn.Upsample is deprecated. Use nn.functional.interpolate instead

I have the problem:
Model path models/RRDB_ESRGAN_x4.pth.
Testing...
1 baboon
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/upsampling.py:129: UserWarning: nn.Upsample is deprecated. Use nn.functional.interpolate instead.
warnings.warn("nn.{} is deprecated. Use nn.functional.interpolate instead.".format(self.name))
2 comic

Can this model test with gray image?

there is a gray image in the set14, i want to test it. but i failed. so i want to know whether this model can test gray image.

why the code doesnot release memory after Validation?

@xinntao Hi, when the code do its first Validation, it will allocate about 2GB GPU memory, but do not release them after Validation.

the SR image of SRGAN BSD100

Thank you for your wonderful work. I am also very interested in GAN, how do you get the SR image of SRGAN's BSD100?

Image with alpha channel fails

Hi, you can work around this by moving the alpha to a separate grayscale image and process it separate and then stitch it together again. But it would be nicer if was a automated process.

Training a new model with different scaling factor

Hi there,

If I want to train a new ESRGAN model, a 2.5x upsampling/downsampling factor, for example, how do I get the corresponding pre-trained PSNR model?

Gpu usage low

Im getting low gpu usage from my GTX 1080ti

Anything i can do to fix that?

something about β

ESRGAN/block.py

Line 208 in ce16ee8

return x5.mul(0.2) + x

it seems different between your code and paper.which is correct?
thx!

How to reduce fringe interference in results?

I got some fringe interference (条纹干涉) in results, just like SRGAN results in the above picture. And I don't know how to call about such phenomenon officially which is often happened in GAN training.
It is worse in visual perception, and is there any way to reduce it? Or, which part of optimizations in ESRGAN reduce such phenomenon?
THX!

the pix loss

Thank you very much for you job firstly
When I try to exchange l_g_pix = self.l_pix_w * self.cri_pix(self.fake_H, self.var_H) with l_g_pix = self.l_pix_w * self.cri_pix(self.var_H,self.fake_H), I found the value of l_g_pix is differend ,How to explain this?

Hope for reply

why not subtract the mean of the image.?

@xinntao Hi, I didn't find the operation about subtracting the mean of the image. I am confused that if the image should be normalized or not. I think it may be useful and easy to train the network with the zero mean of the input images.
Looking forward to your reply.
best wishes.

Runtime error: Torch: Not enough memory: you tried to allocate 2GB - on a 32gb machine?

Why is this happening? I am also getting a similar error when trying to run SFTGAN, only then it's trying to allocate 4gb. Also, I am not seeing the ram fill up in the task manager when I get this error. It also seems that the program is capable of reporting correct attempted memory allocation sizes for other errors. For instance, when I try running SFTGAN segmentation test on a 6144x6144 image, it reports attempting to commit 40gb and failing at it, and this time I am seeing the memory fill up completely via the task manager. Not with the current error though. Image size is 4096x4096 by the way. Any suggestions on how to get around this are much appreciated.

Thanks

Add running at CPU

yolkis@YOLKIS:~/ESRGAN$ python3 test.py models/RRDB_ESRGAN_x4.pth
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=74 error=35 : CUDA driver version is insufficient for CUDA runtime version
Traceback (most recent call last):
File "test.py", line 19, in
model = model.cuda()
File "/home/yolkis/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 258, in cuda
return self._apply(lambda t: t.cuda(device))
File "/home/yolkis/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 185, in _apply
module._apply(fn)
File "/home/yolkis/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 185, in _apply
module._apply(fn)
File "/home/yolkis/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 191, in _apply
param.data = fn(param.data)
File "/home/yolkis/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 258, in
return self._apply(lambda t: t.cuda(device))
RuntimeError: cuda runtime error (35) : CUDA driver version is insufficient for CUDA runtime version at /pytorch/aten/src/THC/THCGeneral.cpp:74

测试结果图片的质量

您好，当我直接用您训练好的模型拿来测试时，生成的高分图像质量比较差，有几个问题想请教，
1、LR图片的分辨率是128×64，格式是jpg格式的，不是等宽高的，这样对结果影响大吗？
2、如果想生成的分辨率是原来的两倍而不是四倍，需要从改变训练图片上入手吗？

Not enough Vram

Hello,

So I've tried ESRGAN with higher resolution images, such as 1280x720, but it says that my GPU (970gtx) doesn't have enough VRAM to process it. Is there a way to make the higher resolution images work with my GPU? Maybe someway to cache the data?
I tried the CPU option, but even with 16GB of RAM, it still says not enough memory.

differenct about ESRGAN and PSNR

hello, I check the 2 pretrained models that you provided, what is difference about 2 models. different training dataset or different loss function.

cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

Fails to run on CUDA 9.0.
My current environment works fine on CUDA with TensorFlow-gpu and CNTK.
Rig: 8700K with 32 GB and 2080 RTX with 8GB.
Latest drivers.

File "G:/Projects/ESRGAN-master/test.py", line 39, in
output = model(img_LR).data.squeeze().float().cpu().clamp_(0, 1).numpy()
...........
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

Maybe a VRAM issue ?

预训练的模型没法直接用来测试

您好,首先感谢下您的代码,对我有很大的帮助,但现在有一个问题,想请教您一下,使用您提供的训练好的模型进行跑您的测试代码时,会出现如下问题
root@c07bd16360cb:/home/qiuyj/ESRGAN# python test.py models/RRDB_ESRGAN_x4.pth
Traceback (most recent call last):
File "test.py", line 17, in
model.load_state_dict(torch.load(model_path), strict=True)
File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 721, in load_state_dict
self.class.name, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for RRDB_Net:
Missing key(s) in state_dict: "model.9.bias", "model.9.weight".
Unexpected key(s) in state_dict: "model.1.sub.2.RDB1.conv1.0.weight", "model.1.sub.2.RDB1.conv1.0.bias", "model.1.sub.2.RDB1.conv2.0.weight", "model.1.sub.2.RDB1.conv2.0.bias", "model.1.sub.2.RDB1.conv3.0.weight", "model.1.sub.2.RDB1.conv3.0.bias", "model.1.sub.2.RDB1.conv4.0.weight", "model.1.sub.2.RDB1.conv4.0.bias", "model.1.sub.2.RDB1.conv5.0.weight", "model.1.sub.2.RDB1.conv5.0.bias", "model.1.sub.2.RDB2.conv1.0.weight", "model.1.sub.2.RDB2.conv1.0.bias", "model.1.sub.2.RDB2.conv2.0.weight", "model.1.sub.2.RDB2.conv2.0.bias", "

About the definition of the loss function

Hi！In the code, I don't see the definition of the loss function and the use of RelativisticGAN.Is this right?
Thanks！

CPU Device Segfaults

Using 'cpu' instead of 'cuda' causes a segfault. CUDA 10.0.130, PyTorch 1.0.0 (latest versions)

Not sure where any other information regarding the crash would be located.

Runtime Error

I'm getting this runtime error during processing: RuntimeError: CUDA out of memory. Tried to allocate 289.50 MiB (GPU 0; 4.00 GiB total capacity; 2.92 GiB already allocated; 0 bytes free; 2.18 MiB cached)

My GPU should have enough memory for the image, but ESRGAN is just not using it for some reason. I'm sure that I do something wrong here...

My System:
Windows 10
AMD Ryzen 1600 3.5 Ghz
Geforce 970 4 GB
16 GB RAM

生成lmdb文件时出现了除0的错误

问题出现在progress_bar.py文件中的update函数fps = self.completed / elapsed这里elapsed变成零了。我把elapsed = time.time() - self.start_time中的两个值打印出来发现是一样的：
Read images...
[ ] 0/36352, elapsed: 0s, ETA:
Start...
1554711715.5890906
1554711715.5890906
0.0
Traceback (most recent call last):
File "create_lmdb.py", line 22, in
pbar.update('Read {}'.format(v))
File "D:\code\BasicSR\codes\utils\progress_bar.py", line 43, in update
fps = self.completed / elapsed
ZeroDivisionError: float division by zero
其中我的路径设置是这样的：
img_folder = 'D:\code\BasicSR\codes\data\HR\*.png'
lmdb_save_path = 'D:\code\BasicSR\codes\data\HR.lmdb'
需要您的帮助！谢谢

Enlarging images with alpha channel

Hello, is there any way to configure ESRGAN to preserve alpha (transparency) channel in png format images? It could be very useful when converting various textures with transparency for games. Best regards

How to calculate Perceptual index?

Hi, thanks for your sharing.
Could you send me a script for calculating Perceptual index? It is my email, [email protected]

something about discriminator

hello，the discriminator ,in pix2pix uses patchgan but srgan do not use the structure and pix2pix use the
connection of fake image and real image while srgan only use fake image as input，how to choose two differnet structure ？ thanks

CUDA version

I can not train the code ( ESRGAN model), and I get this error

I think it's related to CUDA version, I am using CUDA 10.0

Changing `upscale` factor?

Is there any easy way to tune the upscale=4 factor? would that need a specifically trained network?

Your pre-trained model on lossy JPEG images

Thanks for sharing your code and approach.

But testing your approach using everyday images is a hard task since the images are mostly JPEGs and have noisy artifact blocks and in 99% of cases we don't have access to the RAW noiseless version of the images.

Can you provide a pre-trained model on noisy images? I'm curious to see if the SRGAN-based methods only do nicely on the noiseless raw images or if they can do both DeNoising + SuperResolution too?

Thanks again.

Will you release the model training code?

I want to see how you guys implement the loss function. It'll be great if you release the training code. Thank you.

issues about smaller initialization and instance norm

hello, smaller initialization can help to train a very deep network in your paper, if I use instance norm，can I need smaller initialization? thanks

import architecture as arch error

Hi! when I run test I get this errors

Traceback (most recent call last):
File "test.py", line 7, in
import architecture as arch
File "/User/1/ESRGAN-master/architecture.py", line 33
self.model = B.sequential(fea_conv, B.ShortcutBlock(B.sequential(*rb_blocks, LR_conv)),
SyntaxError: only named arguments may follow *expression

i tried this, but didn't help
here

Killed: 9

After running the ESRGAN test on a 12" 240dpi 6000 pixel file the process fails with:

Killed: 9

Does anyone know what this means and how I can get the process to complete successfully?

What is the minimum requirements to run ESRGAN in general?

I was testing ESRGAN out on a workstation when I realized CUDA 2.1 is not supported for PyTorch.

I'm assuming that also means ESRGAN is not supported because of CUDA 2.1.

But to be exact, I don't see any minimum requirements to run ESRGAN. Thus, I have to come to this conclusion the hard way.

To prevent such things from happening, I would like to request on the README.md to list out the minimum requirements to properly run ESRGAN.

• What Nvidia CUDA version is required?
• What Nvidia architecture is needed? (Fermi, Volta, Pascal, Turing, etc.?)
• What Nvidia card type and up can be used? (Quadro, NVS, GeForce?)

Thanks.

Out of memor for .jpg images

When I use a .jpg image to test, system tell me that it`s out of memory.(I run it in Google Colab with 11.7G memory)

a question about residual in your paper

In paper of edsr,rcan and so on, before residual there should be a conv.
For example, in edsr , (conv-act-conv)*n_blocks-conv , and there is a residual before and after this.
In rcan, (block1-block2-...-block20-conv) is a group , there is a residual before and after a group, which is called short residual, and the residual before and after (group1-group2-...-group10-conv) is called a long residual.
In your paper (P5, figure4 right, RRDB), there is a conv for short residual in the dense block, but no another conv for long residual, do you think it should be a conv after the third dense block?

about the results you provided

When I compute the psnr between the HR images and the SR images you provided in Google Drive. The psnr of your SR baboon image in Google drive is 18.57dB, but in your paper the value is 20.35dB. Can you provide the psnr.py? or tell me what happened about my code? and I use skimage.measure.compare_psnr( )

vgg19.features.children() &vgg19.features.()

hello,thank you for you good job, I have some questions, what is the different between vgg19.features.children() &vgg19.features.()

inference with jpg files

Hello, As I know GAN algorithm has problem on jpg images.
do you try your algorithm for jpg files? is there abnormal noise texture exist in output?

How to finetune a PI oriented SR model from the one you submitted in PIRM2018?

I try to build a PI oriented SR model based on your excellent work. Would you please give some guide on how to fine tune your model or elaborate some training details for PI oriented training ..

White balance is affected

Hi, I did a quiick test for our game. The white balance of the image is affected by the upscale. Here I switch between the two. The upscale is 4 maps, one base color, one specular/glossness (alpha chanel processed seperate), normalmap and AO map.

https://www.youtube.com/watch?v=79Cy8jUOjRA

As you can see the white balance of the texture is affected

About result images in Google drive

Hello.

Thank you very much for sharing the model, code, and results!

I have a quick question:

Are result images in the Google drive (PIRM-test) are different from your PIRM 2018 submission? I re-calculated RMSE and PI, but results were slightly different from the report.

If so, I would like to know the difference.

Thank you!

The training datasets of the trained PSNR-oriented model

@xinntao Hello, I want to know that the training datasets of the trained PSNR-oriented model. Is it three datasets DIV2K, Flickr2K, and OutdoorSceneTraining dataset?

run error

Hello, i try to run test.py with python 2.7 below errors occurred:

weiyang.su@deeplearning:~/Super_Resolution/ESRGAN-master$ python test.py models/RRDB_PSNR_x4.pth
models/RRDB_PSNR_x4.pth
Traceback (most recent call last):
File "test.py", line 18, in
model.load_state_dict(torch.load(model_path), strict=True)
File "/home/weiyang.su/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 721, in load_state_dict
self.class.name, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for RRDB_Net:
Missing key(s) in state_dict: "model.1.sub.1.bias", "model.1.sub.1.weight", "model.5.bias", "model.5.weight", "model.7.bias", "model.7.weight".
Unexpected key(s) in state_dict: "model.8.weight", "model.8.bias", "model.10.weight", "model.10.bias", "model.1.sub.2.RDB1.conv1.0.weight", "model.1.sub.2.RDB1.conv1.0.bias", "model.1.sub.2.RDB1.conv2.0.weight", "model.1.sub.2.RDB1.conv2.0.bias", "model.1.sub.2.RDB1.conv3.0.weight", "model.1.sub.2.RDB1.conv3.0.bias", "model.1.sub.2.RDB1.conv4.0.weight", "model.1.sub.2.RDB1.conv4.0.bias", "model.1.sub.2.RDB1.conv5.0.weight", "model.1.sub.2.RDB1.conv5.0.bias", "model.1.sub.2.RDB2.conv1.0.weight", "model.1.sub.2.RDB2.conv1.0.bias", "model.1.sub.2.RDB2.conv2.0.weight", "model.1.sub.2.RDB2.conv2.0.bias", "model.1.sub.2.RDB2.conv3.0.weight", "model.1.sub.2.RDB2.conv3.0.bias", "model.1.sub.2.RDB2.conv4.0.weight", "model.1.sub.2.RDB2.conv4.0.bias", "model.1.sub.2.RDB2.conv5.0.weight", "model.1.sub.2.RDB2.conv5.0.bias", "model.1.sub.2.RDB3.conv1.0.weight", "model.1.sub.2.RDB3.conv1.0.bias", "model.1.sub.2.RDB3.conv2.0.weight", "model.1.sub.2.RDB3.conv2.0.bias", "model.1.sub.2

do you face similar problem?

hidden grid on the result image

Grate work!! ,

but after zooming result images(JPG format) have some hidden grid noise that makes it ugly >
wil changing any parameter will solve this,

evaluate D

I have some doubts about GAN.
D and G are alternately trained during the training.
How to evaluate that the D can make a correct judgment?Thank you！