jihyongoh/fisr

[AAAI 2020] Official repository of FISR.

Hi there 👋

• 👨🏻‍💻 I am currently an assistant professor at CMLab (Creative Vision and Multimedia Lab.) in Chung-Ang Univ. (CAU).
• 👨🏻‍⚕ Please visit my personal homepage (here).
• 🔬 I primarily focus on a variety of deep-learning-based Computer Vision research areas, such as:

   Neural Radiance Fields (NeRF)
   Video Frame Interpolation / Super Resolution / Deblurring / Colorization  
   Optical Flow Estimation
   Computational Photography
   SDR-to-HDR Inverse Tone Mapping
   Generative AI; Diffusion Models, GANs
   GAN/CNN-based Synthetic Aperture Radar (SAR) Target Recognition/Generation

• 💻 If you are interested in collaborating with me, please don't hesitate to send the email below.
• 📧 Contact: [email protected]

fisr's Issues

Some questions about preprocessing operations

First of all, thank you very much for your public and maintenance of the project.
I see that your input data is PWC-Net processed for continuous frames, then warp, and then send these data to the network together. Whether these two steps of preprocessing can be regarded as a frame interpolation operation. The subsequent network just played a super-resolution effect?
So I still have to complete the tasks of super resolution and frame interpolation in two steps. I do n’t know how you think about this problem

pretty bad PSNR and Result Image - Test Only

Hi.
Thanks for uploading your codes.
However, i follow guide regarding only test, but it seems result images really bad and PSNR on logs are also bad.

Attach sample one result image and some logs below.

Total size of variables: 48316251
Total bytes of variables: 193265004
Model: FISRnet
[] Testing starts
[] Reading checkpoints...
WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
[*] Success to read FISRnet-122000
Start to read flow data (test).
Reading 10 x 8 x 1080 x 1920 x 2 flow file in .flo format
Successfully load.
Start to read warped data (test).
Successfully load.
[ 0/ 30]-th image, scene: 0-0, time: 1.0577(minutes), test_PSNR: fr1 (FI-SR) 13.82336405[dB], fr2 (SR) 14.73211411[dB], fr3 (FI-SR) 16.87715632[dB]
--------------------------------------------------------------- test_SSIM: fr1 (FI-SR) 0.87482150, fr2 (SR) 0.78183737, fr3 (FI-SR) 0.86297231
[ 1/ 30]-th image, scene: 0-1, time: 2.3114(minutes), test_PSNR: fr1 (FI-SR) 15.92068400[dB], fr2 (SR) 15.90569003[dB], fr3 (FI-SR) 16.80912121[dB]
--------------------------------------------------------------- test_SSIM: fr1 (FI-SR) 0.85897002, fr2 (SR) 0.78195489, fr3 (FI-SR) 0.83428669
[ 2/ 30]-th image, scene: 0-2, time: 3.5859(minutes), test_PSNR: fr1 (FI-SR) 17.18495496[dB], fr2 (SR) 15.63897788[dB], fr3 (FI-SR) 17.93267987[dB]

what is ./data/LR_sample_5seq.mat

In the main.py file I found the following：

""" to load a lighter train data to reduce time consumption """

   args.train_data_path = './data/LR_sample_5seq.mat'

   args.train_label_path = './data/HR_sample_5seq.mat'

what is ./data/LR_sample_5seq.mat，Where can i get them？
Without them, the program will report an error

The code for creating dataset

Hi, thanks for your nice work! I want to make my own dataset. Could you please share your code for creating the dataset?

Some Links for Test Code are Failure

As title,
Some Links for Test Code are Failure.

1. Step 3 in Test code, for downloading pre-train weights:
   Download the pre-trained weights from this link and then unzip it to place in <source_path>/checkpoint_dir/FISRnet_exp1.

2. Description in Test code:
   How to make flow and warped files by using PWC-Net: you can download all the related tensorflow files from this link including weights, and then modify the folder name ‘tfoptflow’ as ‘FISR_tfoptflow’.

Could you please share these two links for running test code.

Result problem

Hi author,
I run below.
python main.py --phase 'FISR_for_video' --exp_num 1 --frame_num 3 --frame_folder_path './FISR_test_folder/scene1'
The result image size was (2048,3840) ,but only quarter of original image.
My purpose is to run SR on single image .
Did you have any idea about the problem

main.py: error:

hello,why i have
main.py: error: argument --phase: invalid choice: "'FISR_for_video'" (choose from 'train', 'test', 'FISR_for_video')
?

Training set in PNG format

Thanks for your excellent work and your 4K dataset.

1. I would like to train other VFI and SR models on your 4K dataset to evaluate their performance on UHD videos. Could you upload the source .png files of the training set?
2. I ran the test code but got bad results similar to #4. Following your suggestion, I regulated "--test_patch (2,2)" into smaller patches (4,4) but got no improvement. The experiments were conducted in Centos. Is there anything wrong when running in Linux?
   Thank you.

No module named 'FISR_tfoptflow.ckpt_mgr'

in model_base.py line 18

from .ckpt_mgr import BestCheckpointSaver

where is the module ckpt_mgr?

Is there any plan to open source the code? When is it about?

Is there any plan to open source the code? When is it about?