atiyo/deep_image_prior

Image reconstruction done with untrained neural networks.

PyTorch Deep Image Prior

An implementation of image reconstruction methods from Deep Image Prior (Ulyanov et al., 2017) in PyTorch.

The point of the paper is to execute some common image manipulation tasks using neural networks untrained on data prior to use.

Architectures differ from those used in the actual paper. The authors use some specific networks for specific tasks. This repo uses a couple of alternative architectures to produce similar results. One where upsampling is done on the basis of pixel shuffling, and the other using transposed convolutions. Pixel shuffling results in some hotspots that do not disappear with further training.

Requirements

Python3 with PyTorch, torchvision and NumPy. CUDA and cuDNN are optional (settable within the script in a self-explanatory way) but strongly recommended.

To use

It's relatively easy to play around with the settings from within the scripts. To reproduce the results in the repo, do the following.

Make a directory to hold the network output:

mkdir output

Generate output images with:

python3 deep_image_prior.py

Consolidate output images into a training gif and sample some actual data with:

python3 parse_ec2_results.py

Results

Note that the images here have been downsampled for formatting sensibly in the README. Full sized samples are in the repo if you would like to have a closer look.

Training was done over 25k iterations on an Amazon GPU instance. Takes roughly an hour on 512x512 images.

Upsampling with transposed convolutions:

Note the grid like speckles during training. These are caused by convolutional kernels overlapping with one another during upsampling.

Ground truth	Input	Output	Training

Upsampling with pixel shuffling:

No speckles, however there is a hotspot (in the out of focus region towards the bunny's behind) that becomes a black spot. The appearance of these hotspots seems commonplace through both architectures, but the extra smoothness given by the convolution transpose layers repairs these more effectively.

Ground truth	Input	Output	Training

deep_image_prior's Issues

Why 124 instead of 128 in U-Net deconv

In https://github.com/atiyo/deep_image_prior/blob/master/deep_image_prior.py#L188 you have 124 instead of 128.

Can you explain why?

about input data size

hi,
I am a student
I want to input a 600x600 size picture as input
but it requires 512x512 size input
can you help me that where can I change about input size from 512x512 to 600x600?

definition of u_deconv_2 is repeated

deep_image_prior/deep_image_prior.py

Line 197 in c50e5e4

self.u_deconv_2 = nn.ConvTranspose2d(32, 16, 4, stride=2, padding=1)