Giter Site home page Giter Site logo

vsr-duf's Introduction

Deep Video Super-Resolution Network Using Dynamic Upsampling Filters Without Explicit Motion Compensation

This is a tensorflow implementation of the paper. PDF

directory

./inputs/G/ Ground-truth video frames
./inputs/L/ Low-resolution video frames

./results/<L>L/G/ Outputs from given ground-truth video frames using depth network
./results/<L>L/L/ Outputs from given low-resolution video frames using depth network

test

Put your video frames to the input directory and run test.py with arguments <R>, <L> and <T>.

python test.py <R> <L> <T>

<R> is the upscaling factor of 2, 3, 4. <L> is the depth of network of 16, 28, 52. <T> is the type of input frames, G denotes GT inputs and L denotes LR inputs.

For example, python test.py 4 16 G super-resolve input frames in ./inputs/G/* using 16 depth network with upscaling factor 4. (Possible combinations for <R> <L> is 2 16, 3 16, 4 16, 4 28, and 4 52.)

This code was tested under Python 2.7 and TensorFlow 1.3.0.

video

supplementary video

bibtex

@InProceedings{Jo_2018_CVPR,
	author = {Jo, Younghyun and Oh, Seoung Wug and Kang, Jaeyeon and Kim, Seon Joo},
	title = {Deep Video Super-Resolution Network Using Dynamic Upsampling Filters Without Explicit Motion Compensation},
	booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
	year = {2018}
}

vsr-duf's People

Contributors

yhjo09 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

vsr-duf's Issues

Training code release

Hi,
Just gave your method a try, the core idea is quite elegant and the results look really nice!

I was wondering, are you planning on releasing the training code to allow retraining other networks?

This algorithm creates artifacts

Hi @yhjo09 I have been testing your algorithm a lot and I notice artifacts in the result images that are not present in the input images.

Your own repo has a good example of what I am talking about. In the result image here you can notice green and black artifacts at the top of the image. This is not present in the original input here

I notice the same thing on my images while trying every single supported combination of <R><L> values.

Captureaaa

On some (i.e. 4 52) the artifacts are on the left of the image.

Your paper does show full images - did they have the same issue?

If not how did you solve this issue?

Why 512*288 image frames can't work?

This is the log:
Caused by op 'G/MatMul', defined at:
File "test.py", line 81, in
GH = G(L, is_train)
File "test.py", line 49, in G
t = DynFilter3D(x[:,T_in//2:T_in//2+1,:,:,c], Fx[:,0,:,:,:,:], [1,5,5]) # [B,H,W,RR]
File "/home/disk1/hym/pycharm_remote_files/VSR-DUF/utils.py", line 214, in DynFilter3D
x = tf.matmul(x_localexpand, F) # b, h, w, 1, R
R
File "/home/disk1/hym/anaconda/envs/python37/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 2015, in matmu l
a, b, adj_x=adjoint_a, adj_y=adjoint_b, name=name)
File "/home/disk1/hym/anaconda/envs/python37/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 1245, in b atch_mat_mul
"BatchMatMul", x=x, y=y, adj_x=adj_x, adj_y=adj_y, name=name)
File "/home/disk1/hym/anaconda/envs/python37/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 78 7, in _apply_op_helper
op_def=op_def)
File "/home/disk1/hym/anaconda/envs/python37/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in ne w_func
return func(*args, **kwargs)
File "/home/disk1/hym/anaconda/envs/python37/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3272, in crea te_op
op_def=op_def)
File "/home/disk1/hym/anaconda/envs/python37/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1768, in in it
self._traceback = tf_stack.extract_stack()

InternalError (see above for traceback): Blas xGEMMBatched launch failed : a.shape=[147456,1,25], b.shape=[147456,25,16], m=1, n=1 6, k=25, batch_size=147456
[[{{node G/MatMul}} = BatchMatMul[T=DT_FLOAT, adj_x=false, adj_y=false, _device="/job:localhost/replica:0/task:0/device:G PU:0"](G/ExpandDims, G/strided_slice_8)]]
[[{{node G/add_18/_317}} = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", sen d_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_2365_G/add_18", tensor_type= DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Release: practical, user and system friendly version of test.py

  • no excessive system memory hogging
  • adjustable video memory fraction to maximize video memory utilization (read the file)
  • can load all images from the inputdir "at once", no splitting to separate directories to avoid memory hogging
  • eyes-convenient representation powered by tqdm
  • can take other image formats (such as bmp) directly from the console
  • frame-count can be specified directly on console
    tested on python 3.6/TF1.15.4
    requirements: place the new test_opt.py into the directory of VSR-DUF
pip3 install tqdm

start inferencing:

python test_opt.py 2 16 dataset 0 33492 png

where:

2 = upscale factor
16 = depth
dataset = "./inputs/dataset"
0 = start frame
33492 = end frame
png = image format (any that will work with LoadImage()

Download

limitation: designed for 6-digit frame numbers for file names, to adjust, modify str("%06d"%counter) and (outdir + '/{:06d}.png'.format(counter)) from default 6 to what your movie requires

TODO and near future plans:
improve tqdm representation to work in terms of fps not it/s - done
allow to specify digit count on the console

updated:

  • added functionality to operate on a specific range of frames
  • added error protection, now testing for existence of first/last frames
  • re-designed the progress bar
  • variable names friendly to programmer

Results in MSU Video Super Resolution Benchmark

Hello,
MSU Video Group has recently launched Video Super Resolution Benchmark and evaluated this algorithm.

DUF-28L takes 5th place by subjective score, 11th place by PSNR, and 9th by our metric ERQAv1.0. DUF-16L takes 8th place by subjective score, 12th place by PSNR, and 11th by our metric ERQAv1.0. You can see the results here.

If you have any other VSR method you want to see in our benchmark, we kindly invite you to participate.
You can submit it for the benchmark, following the submission steps.

Some question about making training datasets?

I have read your inspiring paper. How do you select areas of motion in paper's part 4.Implementation? Because I try your method training with randomly cropping the blocks in videos. And the performance is not really good like yours.

Testing with 16L x2 and GT inputs didn't work properly

Hi, just read your great work and thanks for sharing the code.
However, when I tried to run the test code on your data with the arguments python test.py 2 16 G, the results didn't look well as I show below.
Frame001
I've tried all the other parameters you offer, and they seem to be good. So I wonder if there's something wrong with 'params_16L_x2.h5'.

Code implementation

Hi, I have read the source code and paper.

I have questions about Filter generation network and Residual generation network. Figure 3 in the paper shows that 2d convolution blocks (1x1 and 3x3) follow the denseLayer, however, there are 3d convolution blocks in the code.
Which one should I follow?

Thanks!

What are the gaussian filter parameters for x2, x3 ?

Hi~Thanks for your greate work and sharing the codes.
I'm using your project and my own data to train video super-resolution X2、X3 and X4 model,
the X4 model works well, but the X2、X3 models don't .
when running the X2、X3 models, I find that the output residual looks abnormally.
I notice that you comment "h = gkern(13, 1.6) # 13 and 1.6 for x4" for the gaussian filter, but I don't change this parameter in the X2、X3 models.
I am wondering if the improper parameter causes the wrong residual.
Would you please tell me what the gaussian filter parameters that you used for training the X2、X3 models?
Thank you very much!

image
image

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.