seungjunnah/deepdeblur_release

Deep Multi-scale CNN for Dynamic Scene Deblurring

deepdeblur_release's Issues

When str will be 2 in shortcut(str)?

Hi,
what will be the case when input and output will not be with same third dimension? there is always conv at the beginning of every scale-level so when this case will happen?
Also, i don't understand what the bold part should do

i'm talking on this function :

function ResBlock(filtsize, nStates, inStates)
local function shortcut(str)
local str = str or 1
if str == 1 then
return nn.Identity()
else
local str = 2
return nn.Sequential()
:add(nn.SpatialAveragePooling(1,1, str,str))
:add(nn.Concat(2))
:add(nn.Identity())
:add(nn.MulConstant(0))
end
end

Thanks!!
Sivan

[Question] Can this method work with unpaired training set?

Hi,
I am trying to apply this method on a training set that does not have a sharp image ground truth pair for every blurred image. The training set only has a bunch of different blur and sharp images. Would the method still work? Any insights and suggestions will be helpful.
Thank you.

experiment algorithms

Thanks for your work. I ’m comparing different deblurring algorithms. and I didn't find the Sun' implementation in github about his paper “Learning a convolutional neural network for non-uniform motion blur removal”，which is used in your experiment. did you realize
it yourself ?

nil value

Hey,Thank you for your shared code.I have modified the opts.lua:
cmd:option('-threads', 0, 'number of main threads')
cmd:option('-nDonkeys', 0, 'number of donkeys to initialize (data loading threads)')
And I got an error when make this command: th main.lua -nEpochs 451 -save scale3

the error is :
epoch begins : 1
==> training!
==> doing epoch on training data:
==> online epoch # 1 [mini-batchSize = 1]
/home/username/torch/install/bin/lua: train.lua:259: attempt to call field 'log10' (a nil value)
stack traceback:
train.lua:259: in function 'cumulate_error'
train.lua:283: in function 'f2'
data.lua:117: in function 'addjob'
train.lua:275: in function 'train'
main.lua:89: in main chunk
[C]: in function 'dofile'
...name/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: in ?

the error mentioned above is located in train.lua:259: cPSNR = cPSNR - 10math.log10(mse),
I rewrote it as cPSNR = cPSNR - 10math.log10(math.abs(mse)+0.01),but makes no difference.
Could you help me for this problem?Thank you very much.

can provide a pre-trained CPU model？

As I do not have a GPU machine, so can provide a pre-trained CPU model？

thanks.

Can this run on windows?

Ubuntu troubles me a lot ...

Is there any other source of datasets?

Failed to download the dataset for some reason.
Can you provide another link to download the datasets

Large values in output

Hello again,

during my implementation i encountered a problem
the input value is between [0,1] but the output value of the net (predictions, images after moving through the net) is between [-10^7,10^7] .
i tried to do sanity check and set Scales = 1. the training was on 2 images. the purpose was to make sure the net can learn 2 images and over fit those images.
even after 40,000 iterations i got values of the output between [-7,7] and not smaller than that.
maybe there is some relu or BN that is not in the paper Or did you had the same problem?

Thanks
Sivan

Hi！Can this method be used for defocused images?

Thank you for your code!
I have some problems:
1.I notice that this method is kernel-free method.In theory, if there is a dataset of defocused images, can this method be used for defocused images?
2.A lot of work is tested on the GOPRO dataset. Does the GOPRO dataset have defocused images and corresponding sharp images?

몇가지 궁금한게 있어 글을 씁니다.

안녕하세요, 좋은 논문과 좋은 코드 공개해주셔서 감사합니다.
몇가지 궁금한 점이 있어 글을 습니다.

1. Table2,3 를 보면 K=1일때 가장 느리고, K=3일때 가장 빠른데요, K=3가 의미하는게 3-level이지 않나요? 하지만, K=1일때 가장 느리기 때문에, K=1일때가 3-level인 것 같은데 뭐가 맞는지 궁금합니다.

2. K=2일때 가장 성능이 좋았고, K=1일 때 가장 성능이 좋지 않았는데(느리고, PSNR이 낮음), 왜 기본 코드를 공개하실때나, 논문의 그림의 경우 3-level로 하신건가요?

3. Adversarial Loss의 경우 PSNR을 떨어트리는 효과가 있는데, 결과 비교가 PSNR로 비교가 되어있고, 다른 논문보다 PSNR이 높은데 이러한 것을 어떻게 해석해야 하나요? Adversarial Loss가 PSNR을 낮췄음에도 다른 논문보다 결과가 좋다고 해석을 해야될지, 아니면 Adversarial Loss에 곱해진 lambda값이 너무 작아서 효과가 미미한건지 궁금합니다.

4. 보통 3x3 convolutional layer가 성능이 더 좋고, 빨라서 많이 사용하는데,(3x3 conv layer 2개 = 5x5 conv layer), 5x5 convolutional layer를 쓰신 이유가 궁금합니다.

3주째 이 논문에 대해 분석을 하고 있는데, training하는데 시간이 오래걸리고, 막히는 부분이 많아 이렇게 글을 씁니다. 감사합니다.

Help in getting (building) torchx package

Dear Seungjun,

You have listed the Dependencies:
torch7
torchx
cudnn
Every thing is done, except torchx, I have no idea how to build it. It is a package which is implemented using lua, there is a cmakelists file, but I am not able to build it and generate the package torchx. Could you please guide me in this regards?
Thanks in advance

Implementation details of upconv

Hi, Thanks to your great work.
I am not familiar with lua, I want to know the implementation details of upconv.
What is the input of upconv? Restored low resolution image or its high-level features from the resblock?
Is the process of upconv as described below:
(batch ,3, h, w) ——>conv——>(batch, 322, h, w)——>PixelShuffle(2)——>(batch, 3, 2h, 2w)?

Provide the blurring generation code？

Hi, @SeungjunNah . Thanks for the work.
I'm working on experiment which need to generate blurring effect.
Do you think it is OK for you to provide the code of blurring generation ?
Thank you~

Duty cycle of REDS

Hi, when creating REDS, you said

we average the 1920 fps video frames to produce virtual 24 fps blurry video with duty cycle τ= 0.8

Does it mean for every 80(1920/24=80) frames in the 1920 fps video, 64(80*0.8) frames are used to create one blurry frame, and 16 frames are skiped? If so, what is the corresponding clean frame?

Thanks!

Provide Docker image?

Thanks in advance.

Could you provide the raw videos using in generating GoPro datasets?

Hi, @SeungjunNah.
Do you think it is OK for you to provide the raw videos using in generating GoPro datasets?

Got errors

stack traceback:
...h/install/share/lua/5.1/cudnn/SpatialFullConvolution.lua:31: in function 'resetWeightDescriptors'
...h/install/share/lua/5.1/cudnn/SpatialFullConvolution.lua:105: in function <...h/install/share/lua/5.1/cudnn/SpatialFullConvolution.lua:103>
[C]: in function 'xpcall'
/home/harry/torch/install/share/lua/5.1/nn/Container.lua:63: in function 'rethrowErrors'
/home/harry/torch/install/share/lua/5.1/nn/ConcatTable.lua:11: in function </home/harry/torch/install/share/lua/5.1/nn/ConcatTable.lua:9>
[C]: in function 'xpcall'
/home/harry/torch/install/share/lua/5.1/nn/Container.lua:63: in function 'rethrowErrors'
/home/harry/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function </home/harry/torch/install/share/lua/5.1/nn/Sequential.lua:41>
[C]: in function 'xpcall'
/home/harry/torch/install/share/lua/5.1/nn/Container.lua:63: in function 'rethrowErrors'
/home/harry/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function </home/harry/torch/install/share/lua/5.1/nn/Sequential.lua:41>
[C]: in function 'xpcall'
/home/harry/torch/install/share/lua/5.1/nn/Container.lua:63: in function 'rethrowErrors'
...e/harry/torch/install/share/lua/5.1/nn/ParallelTable.lua:12: in function <...e/harry/torch/install/share/lua/5.1/nn/ParallelTable.lua:10>
[C]: in function 'xpcall'
/home/harry/torch/install/share/lua/5.1/nn/Container.lua:63: in function 'rethrowErrors'
/home/harry/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'forward'
test.lua:49: in function 'get_output'
test.lua:128: in function 'get_output_img'
test.lua:268: in function 'demo'
demo.lua:38: in main chunk

WARNING: If you see a stack trace below, it doesn't point to the place where this error occurred. Please use only the one above.
stack traceback:
[C]: at 0x7f337fb50f50
[C]: in function 'error'
/home/harry/torch/install/share/lua/5.1/nn/Container.lua:67: in function 'rethrowErrors'
/home/harry/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'forward'
test.lua:49: in function 'get_output'
test.lua:128: in function 'get_output_img'
test.lua:268: in function 'demo'
demo.lua:38: in main chunk
Lua 5.1 Copyright (C) 1994-2008 Lua.org, PUC-Rio

Please helo me with this!

threads.lua:183: [thread 4 callback] bad argument #2 to '?'

Hi, I am training my model by your shared code, but I have a problem. When I use "th main.lua -nEpochs 450 -save scale3", the result is :
==> online epoch #1 [mini-batchsize = 4]
/root/torch/install/bin/luajit : threads.lua:183: [thread 4 callback] bad argument #2 to '?' (cannot sample n_sample > prob_dist:size(1) samples without replacement at /root/torch/pkg/torch/lib/TH/generic/THTensorRandom.c:204)
How to solver the problem?

SSIM result issue

Hi, I would like to confirm what metric is used in paper, cause I use SSIM get about 0.87, but use CW_SSIM get 0.93 (by Pyssim package) in GoPro gamma subset (1111 images for testing, 720 x 1280 resolutions), and the value in paper is about 0.9135
Thanks for replying!

Code or pre-trained model for "RNN with Intra-Frame Iterations for Video Deblurring"

Hi Seungjun,

I was wondering when you will be able to release the code or pre-trained model for the CVPR19 paper titled "Recurrent Neural Networks with Intra-Frame Iterations for Video Deblurring". I was interested to see how it performs for video consistency and temporal artifact if any.

I noticed in the paper, it is claimed that code will be released publicly, wanted to check if it is coming soon-ish.

Thanks,
Touqeer

python implementation

Hi , do you know if there is an implementation in python for your code?
Thanks!

qlua：cannot find the command

Dear author：When I run the demo, error: cannot find 'qlua' command，the command 'lua' is from package 'lua5.1' (universe). I set up the lua on 5.1 version, but it can not identify the 'qlua' command, do you know which causes the problem? Waiting for your reply. Thank you！

Is there other edition of code?

who can provide a tensorflow edition?

Can this algorithm be implemented on a Windows system?

Can this algorithm be implemented on Windows system?

number of epoches

Hi!

i wanted to ask what was the number of epochs you used for train?
in the code it is 1 but it doesn't look right

thanks!
Sivan

What's the difference of linear and gamma2.2?

hello, @SeungjunNah

thanks for great work! i just see we need set -blur_type during training, so what does it(camera response function) actually mean? does it mean we need add some noise on the input data which is blurred? and different blur_type means different noise?

and what's the difference of linear and gamma2.2? Does it mean we need train the corresponding model for each? but in blind deblur area we don't know the camera response function in advance, so how to use it?

i just read the paper about the process of gamma2.2 blur image generate, it first use do inverse gamma2.2 of the sharp image, and then do average using equation 2, then get the blurred image. what I am confused is that: do we need do gamma2.2 after do the average? （it seems that we should not do that from paper, but in the real world, i think we need fitting the CRF.）

another, which model(gamma2.2 or linear) is better evaluated on Kohler Dataset dataset(both are trained on GOPRO dataset)？

thanks.

How to test

I want to test the model on my picture,and how to test? Thanks

How to test the GoPro dataset using your pre-trained model?

Hello ，good work, but I have a question.
I would like to test the GoPro dataset using your pre-trained model, and what is the file structure of the test data?
Looking forward to your reply！

error

Non-uniform blind deblurring with CNN
==> processing options
Starting main thread with seed: 5449690409643447296
==> switching to CUDA
==> define parameters
==> construct model
==> loading model from /mywork/DeepDeblur_release-master/experiment/release_scale3_adv_lin/models/model-450.t7
==> defining some tools
==> configuring optimizer
optimization algorithm : ADAM
==> defining training procedure
==> defining test procedure
nil
qlua: /home/cui/torch/install/share/lua/5.1/paths/init.lua:26: bad argument #1 to 'dir' (string expected, got nil)
stack traceback:
[C]: at 0x7f13024a8940
[C]: in function 'dir'
/home/cui/torch/install/share/lua/5.1/paths/init.lua:26: in function 'iterfiles'
test.lua:263: in function 'demo'
demo.lua:39: in main chunk
Lua 5.1 Copyright (C) 1994-2008 Lua.org, PUC-Rio
Please help me ,thank you!

What is the name of your dataset?

Hi, seungjun. Thank you for your works.
I want to use your blur dataset for my research purpose. But is 'GOPRO' really the name I should use when I cite this? Please let me know if you prefer another name.

Permission to reuse figures from the paper

Hi! I'm studying at Chalmers University of Technology and I'm writing a report about motion deblurring in images. I've been using your report as a template for how to carry out my work. So firstly I would like to thank you for the work you have done. Since I'm using a very similar architecture as yours, I would need to show a diagram of the architecture in my report. Is it okay if I reproduce the images of the architecture that you represent in your paper?

Thanks, Daniel

processing speed and image resolution

I tried to run the demo with GPU with my own image. However, it gave me some observations:

• image resolution cannot be as large as 3000x3000
• running speed is about 5sec per picture.

I run the program with Ubuntu 16.04 + TITAN-X. I wonder are the above two observations is as expected?

excuse me ,Do you have python code?

Training with a different dataset

Hi Seungjun,
I am currently trying to train the network with a different dataset than the GoPRO to help deblur the images I am using. I'm having a problem in getting the code to recognise my dataset though. I currently have my images saved in DeepDeblur_release/dataset/TrainingPairs/train/1/ then I have the images split into blur and sharp like the GOPRO dataset. When training the model I change the dataset option to be "TrainingPairs" and expected everything else to work fine due to the same layers of directories etc. being used as the GoPRO dataset.

However, in running this, I receive the error "/home/armstrong/DeepDeblur_release/code/donkey.lua:138: attempt to get length of global 'train_list' (a nil value)". This seems to imply that it cannot find the data I am trying to train with. I was wondering if there is something off about how I am structuring the path to my training data? I hope you can help.

Kind regards,
John Armstrong

Problem with the dataset

The sharp frames provided are not continuous?
Because after I averaged the 7 frames and got i terrible blur images and them are much different with the blur images provided~

Where can I get the REDS dataset?

Thanks for you work, your dataset is very useful for image deblurring. I am very interested in your work and want to follow your work. So, I want to know where can i get your new REDS dataset.

Measure the number of float-point-operations of the generator?

I am finding the right way to measure/estimate the number of FLOPs of your model generator.
As you know, there is some unofficial method for measuring the number of FLOPs of a model such as this one: https://github.com/apaszke/torch-opCounter.
However, your model using muti inputs (3-time scales) so it is quite hard to measure the number of FLOPs.
Could you please provide me a solution to measure FLOPs?
I would appreciate it if you provide me some instructions, directions, code, or something like that.
thank

function for evaluating the discriminator loss

Line 198 of train.lua that computes the entropy.fake is using the output_label which was computed using the generator output (line 179) before the backward pass (line 189).

I was wondering if that is correct? Shouldn't entropy.fake for discriminator (line 198) be estimated using the generator output which is obtained with the updated weights of the generator after the backward pass of line 189?

Unable to download pretrained models

http://cv.snu.ac.kr/~snah/Deblur/DeepDeblur_models/experiment.zip mentioned in Readme file. Is there any mirror link to download files?

How can I get the same results in your paper ?

Hi! I compiled your code and got deblurred results.
Next, I calculated PSNR between each deblurred result and sharp image, and averaged them.

However, I got different results.

GOPROSET
PSNR (paper, K=1) 28.93 dB
PSNR (test, K=1, blurtype & model = blur) 28.49 dB
PSNR (test, K=1, blurtype & model = blur_gamma) 29.12 dB

OS : ubuntu 16.04
VGA : titan X

How can I get the same results in your paper ?

Thanks!
HH.

Camera calibration?

Hi,
Do you have intrinsic parameters of cameras used to record REDS datasets?
If you have, can you post it to the web for downloads ?

Demo code doesn't work for images of resolution 1920 × 1080

The demo code doesn't run if I use images of resolution, 1920 × 1080. However, it works for the images provided which have a dimension of 1280 × 720.
Torch version used: 7.0
cudnn 5.1
GPU: NVIDIA GeForce GTX 1080 Ti GPU

test

Why not use the tanh activation at the last layer?

Normally we add tanh after the last convolution to get an output within range -1~1, I just wonder if there is any particular reason not using it anywhere in this architecture.

opts.lua in function parse

Hi！Thank you for your awesome code, when I run the demo, I meet the problem, can you help me?
in mode r at /tmp/buildd/torch7-0.9.99/pkg/torch/lib/TH/THDiskFile.c:649
stack traceback:
[C]: at 0x7f85d59e74f0
[C]: at 0x7f85cc86c220
[C]: in function 'DiskFile'
/usr/share/lua/5.1/torch/File.lua:405: in function 'load'
.../code/opts.lua:63: in function 'parse'
demo.lua:14: in main chunk

can not download pretrained model

hello, i try to run your pretrained model. but seems can not download. is there other link for pretrained model.

data processing

Hi!
in the paper you write that you subtract 0.5 from input and output but in the code you don't do it. which one is correct?

also you relate to the images in the data augmentation part as they have values between [0,1]
in the test set i noticed that there are images with values higher than 1

thanks!

Why the discriminator's architecture in the code is not identical to the paper's version?

Hey @SeungjunNah, thanks again for releasing this code. I noticed that in the code, first two layers of the discriminator are conv 3-32 5*5 stride1, 32-32 5*5 stride2,which is not identical to the paper's version that places only conv 3-32 5*5 stride2.
Is this a miss or there is a reason for it? And which version has a better performance in terms of PSNR or SSIM in practice?

Training time

I am curious about the training time，but I cannot find the answer in the paper. Could you please give me the approximate time?

segmentation Fault with Cudnn

While i am trying to run the demo I am getting this error.

Found Environment variable CUDNN_PATH = /usr/local/cuda/lib64/libcudnn.so.5Segmentation fault (core dumped)