jbhuang0604/selfexsr

Single Image Super-Resolution from Transformed Self-Exemplars (CVPR 2015)

Single Image Super-Resolution from Transformed Self-Exemplars (CVPR 2015)

Introduction

This is the research code for the paper:

Jia-Bin Huang, Abhishek Singh, and [Narendra Ahuja] (http://vision.ai.illinois.edu/ahuja.html), "Single Image Super-Resolution from Transformed Self-Exemplars", CVPR 2015 PDF

The proposed algorithm achieves the state-of-the-art performance on image super-resolution without the need of any external training dataset, feature extraction and complicated learning algorithms. For more details, please visit our Project page.

All the datasets (Set5, Set14, Urban 100, BSD 100, Sun-Hays 80), precomputed results and visual comparisons can be found in the following sections.

Citation

If you find the code and dataset useful in your research, please consider citing:

@inproceedings{Huang-CVPR-2015,
    title={Single Image Super-Resolution From Transformed Self-Exemplars},
    Author = {Huang, Jia-Bin and Singh, Abhishek and Ahuja, Narendra},
    booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
    pages={5197--5206},
    Year = {2015}
}

Contents

Folder	description
cache	cached data for vanishing point detection
data	Testing images of five datasets (Set5, Set14, Urban 100, BSD 100, Sun-Hays 80). All the images have been cropped according to the desired super-resolution factor. This avoids misalignment of the groundtruth high-resolution images and the super-resolved images
external	We use the vgg_interp2 from imrender to perform bilinear interpolation
quant_eval	Quantitative evaluation code
reference	A copy of the CVPR paper and the bibtex
source	MATLAB source code

To run the algorithm on all datasets, simply run the sr_demo_bacth.m. Note that it is an educational code that is not optimized for speed. If timing is a concern, you can achieve visually similar results with small numbers of iterations, e.g., set the number of iterations opt.numIter = 5; in the file sr_init_opt.m. An example of the speed and quality trade-off can be found in Fig. 10 in the paper.

Feedbacks and comments are welcome! Feel free to contact me via [email protected].

Enjoy!

Note: For vanishing point detection only windows executable is provided (from Image Completion using Planar Structure Guidance), a cross-platform version will be included later.

Comparison with the state-of-the-art

Datasets

The full super-resolution results on Set 5, Set 14, Urban 100, BSD 100 and Sun-Hays 80 are available.

| Dataset | Image source | Download full results | |---- | ---|----| ----| | Set 5 | Bevilacqua et al. BMVC 2012 | link (16.1 MB)| | Set 14 | Zeyde et al. LNCS 2010 | link (86.0 MB)| | Urban 100 | Huang et al. CVPR 2015 | link (1.14 GB)| | BSD 100 | Martin et al. ICCV 2001 | link (568 MB)| | Sun-Hays 80 | Sun and Hays ICCP 2012 | link (311 MB)|

Set 5 dataset - link

Set 14 dataset - link

Urban 100 dataset - link

BSD 100 dataset - link

Sun-Hays 80 dataset - link

State-of-the-art image super-resolution algorithms

In each dataset, we include results of the state-of-the-art single image super-resolution algorithms:

Image	Description
HR	High-resolution images. All images were cropped so that each dimension is a multiplication of the super-resolution factor. This avoids the misalignment problem in the quantitative comparison.
LR	Low-resolution test images generated with bicubic kernel downsampling.
bicubic	Bicubic interpolation
nearest	Nearest-neighbor interpolation
SelfExSR	Our result
A+	R. Timofte, V. De Smet, and L. Van Gool, A+: Adjusted Anchored Neighborhood Regression for Fast Super-Resolution. In Asian Conference on Computer Vision (ACCV 2014). Code available here
Abhishek	A. Singh and N. Ahuja, Super-Resolution Using Sub-Band Self-Similarity. In Asian Conference on Computer Vision (ACCV 2014). No publicly implementation available. Results were provided by the authors.
Kim	K. I. Kim and Y. Kwon, “Single-image super-resolution using sparse regression and natural image prior”, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 32, no. 6, pp. 1127-1133, 2010. Code available here
Glasner	Daniel Glasner, Shai Bagon, Michal Irani. Super-Resolution From a Single Image, In International Conference on Computer Vision (ICCV 2009). No public implementation available. Results were generated by our own implementation.
ScSR	Jianchao Yang, John Wright, Thomas Huang, and Yi Ma. Image super-resolution via sparse representation. IEEE Transactions on Image Processing, Vol 19, Issue 11, pp2861-2873, 2010. Code available here
SRCNN	Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang. Learning a Deep Convolutional Network for Image Super-Resolution, in European Conference on Computer Vision (ECCV 2014). Code available here

Qualitative comparison

In our supplementary material, we includde 120 sample comparisons with the state-of-the-art algorithms. Download the document here.

You can browse and compare our results with other methods via the following links.

• Urbana 100 - Super-resolution factor 4x
• BSD 100 - Super-resolution factor 3x
• Sun-Hays 80 - Super-resolution factor 8x

Quantitative comparisons

We report three types of metrics

• PSNR: Peak signal-to-noise ratio
• SSIM: Structural similarity index
• IFC: Information fidelity criterion

Results on Set 5

Scale	Bicubic	ScSR	Kim	Sub-band	Glasner	SRCNN	A+	Ours
2x - PSNR	33.64	35.78	36.24	Sub-band	35.43	36.28	A+	36.50
3x - PSNR	30.39	31.34	32.30	Sub-band	31.10	32.37	A+	32.62
4x - PSNR	28.42	29.07	30.07	Sub-band	28.84	30.08	A+	30.33
2x - SSIM	0.9292	0.9485	0.9518	Sub-band	0.9452	0.9509	A+	0.9537
3x - SSIM	0.8678	0.8869	0.9041	Sub-band	0.8811	0.9025	A+	0.9094
4x - SSIM	0.8101	0.8263	0.8553	Sub-band	0.8210	0.8525	A+	0.8623
2x - IFC	5.72	6.94	7.05	Sub-band	6.70	6.85	A+	7.83
3x - IFC	3.45	3.98	4.25	Sub-band	3.68	4.11	A+	4.76
4x - IFC	2.28	2.57	2.82	Sub-band	2.42	2.76	A+	3.19

Results on Set 14

Scale	Bicubic	ScSR	Kim	Sub-band	Glasner	SRCNN	A+	Ours
2x - PSNR	30.22	31.64	32.14	Sub-band	31.41	32.00	A+	32.23
3x - PSNR	27.53	28.19	28.96	Sub-band	28.21	28.90	A+	29.16
4x - PSNR	25.99	26.40	27.18	Sub-band	26.43	27.13	A+	27.40
2x - SSIM	0.8683	0.8940	0.9031	Sub-band	0.8881	0.9012	A+	0.9036
3x - SSIM	0.7737	0.7977	0.8140	Sub-band	0.7926	0.8124	A+	0.8197
4x - SSIM	0.7023	0.7218	0.7434	Sub-band	0.7163	0.7395	A+	0.7518
2x - IFC	5.74	6.83	6.92	Sub-band	6.47	6.68	A+	7.60
3x - IFC	3.33	3.75	3.92	Sub-band	3.59	3.81	A+	4.38
4x - IFC	2.18	2.46	2.57	Sub-band	2.30	2.50	A+	2.90

Results on Urban 100

Scale	Bicubic	ScSR	Kim	Sub-band	Glasner	SRCNN	A+	Ours
2x - PSNR	26.66	28.26	28.74	28.34	27.85	28.65	28.87	29.38
4x - PSNR	23.14	24.02	24.20	24.19	23.58	24.14	24.34	24.82
2x - SSIM	0.8408	0.8828	0.8940	0.8820	0.8709	0.8909	0.8957	0.9032
4x - SSIM	0.6573	0.7024	0.7104	0.7115	0.6736	0.7047	0.7195	0.7386
2x - IFC	5.72	6.98	6.86	7.08	6.17	6.66	8.02	7.96
4x - IFC	2.27	2.75	2.71	2.72	2.35	2.63	3.16	3.33

Results on BSD 100

Scale	Bicubic	ScSR	Kim	Sub-band	Glasner	SRCNN	A+	Ours
2x - PSNR	29.55	30.77	31.11	30.73	30.28	31.11	31.22	31.18
3x - PSNR	27.20	27.72	28.17	27.88	27.06	28.20	28.30	28.30
4x - PSNR	25.96	26.61	26.71	26.60	26.17	26.70	26.82	26.85
2x - SSIM	0.8425	0.8744	0.8840	0.8774	0.8621	0.8835	0.8862	0.8855
3x - SSIM	0.7382	0.7647	0.7788	0.7714	0.7368	0.7794	0.7836	0.7843
4x - SSIM	0.6672	0.6983	0.7027	0.7021	0.6747	0.7018	0.7089	0.7108
2x - IFC	5.26	6.20	6.30	6.36	5.56	6.09	7.15	6.84
3x - IFC	3.00	3.37	3.49	3.17	2.72	3.39	3.92	3.81
4x - IFC	1.91	2.22	2.20	2.18	1.86	2.18	2.51	2.46

selfexsr's Issues

about Urban100

Hi, thanks for ur sharing!!!
I was wondering whether u had the x3 downsapled urban100 dataset?

Error in compute_difference.m: unknown variable/function 'warpedImage'

Hi,

First of all, thank you for the excellent work and code! I was trying to evaluate my own method with you quantification script. But there seems to be an error in the function compute_difference.m

At some point in the code the function tries to use variable warpedImage:

% Compute Structural similarity index (SSIM index)
45	% =========================================================================
46	[ssim, ~] = ssim_index(imgt, warpedImage);
…	
49	% Compute information fidelity criterion (IFC)
50	% =========================================================================
51	ifc = ifcvec(imgt, warpedImage);

But warpedImage was not defined before. I am guessing it should just be replaced with 'im' such that the code becomes

% Compute Structural similarity index (SSIM index)
45	% =========================================================================
46	[ssim, ~] = ssim_index(imgt, im);
…	
49	% Compute information fidelity criterion (IFC)
50	% =========================================================================
51	ifc = ifcvec(imgt, im);

Is this correct, or did you intend to have another preprocessing step which produces warpedImage?

Thanks again for the great work!

Cheers

code doesn't work on Linux systems (EdgeDetectTest.exe)

hi all!
the code needs to execute EdgeDetectTest.exe , so it needs Windows SO. Is there a workaround or can anyone give the exe source code so that I can compile it on linux? If not, anyone could give me another solution?
tnx

Urban100 HR images seem to be bicubic upscalings of the LR images?

Hello, from the zip file it seems that HR images of urban100 seem to be bicubic upscaled versions of the LR images.

Simply running bicubic upscaling on the images gives a 97 SSIM.
What's going on?

the links of datasets were not opened

the links of datasets were not opened

problem with running the code

Hello,
The cvpr paper is an excellent work and I tried to run the code. However, it appears errors as follows.
??? Undefined variable "vision" or class "vision.ShapeInserter".

Error in ==> sr_extract_plane>sr_detect_plane_from_vp at 179
shapeInserter = vision.ShapeInserter('Shape', 'Lines','BorderColor', 'White');

Error in ==> sr_extract_plane at 48
modelPlane = sr_detect_plane_from_vp(vpData, img, opt);

Error in ==> sr_demo at 38
modelPlane = sr_extract_plane(filePath.dataPath, filePath.imgFileName, opt);

Error in ==> sr_demo_batch at 63
imgHiRes = sr_demo(filePath, opt);

It seems some code were missing or I made some mistakes to miss something. Would you help me solve it .Thank you very much! @jbhuang0604

source/EdgeDetectTest.exe doesnot generate complete *_vanishingpoints.txt

The matlab code in sr_extract_plane.m invoking source/EdgeDetectTest.exe generates only partial info
vx vy vz score numLines
-0.99157554 0.12952947 0.00024355 650.54 47
0.76044877 -0.64939690 -0.00115497 488.17 39
-0.43647291 0.89971575 0.00172559 331.19 31
But expected is 4KB file as present in "cache\vpdetection\text\img_100_SRF_2_LR-vanishingpoints.txt"
The start , end and length of lines in 3 sets not present.
Tried in x64 machine.
Behavior in matlab code and CMD for the source/EdgeDetectTest.exe is same.
Any suggestion?
Thanks !

What version of matlab is running?

I got an error using the 2015 version

Readme: Dataset links do not work

The links corresponding to datasets 'Set 5' and 'Set 14' give a 404 error. It would be great if you could share the dataset for open testing.