rlct1/gin-sg

AIM2020 ECCV Extreme Image Inpainting Track 2 Semantic Guidance

Deep Generative Inpainting Network with Semantic Guidance (GIN-SG) for Extreme Image Inpainting

For AIM2020 ECCV Extreme Image Inpainting Track 2 Semantic Guidance
This is the Pytorch implementation of our Deep Generative Inpainting Network with Semantic Guidance (GIN-SG) for Extreme Image Inpainting. We have participated in AIM 2020 ECCV Extreme Image Inpainting Challenge. Our GIN is used for reconstructing a completed image with satisfactory visual quality from a randomly masked image.

Overview

Example of Image Inpainting using our GIN-SG

• An example from the validation set of the AIM20 ECCV Extreme Image Inpainting Track 2 Semantic Guidance
• (left: masked image, middle: segmentation map, right: our completed image)

Preparation

• Our solution is developed using Pytorch 1.5.0 platform
• We train our model on two NVIDIA GeForce RTX 2080 Ti (with 11GB memory)
• Apart from Pytorch and related dependencies,
• Install natsort

pip install natsort

• Install dominate

pip install dominate

• Install scipy 1.1.0

pip install scipy==1.1.0

• If you would like to use tensorboard for logging, please also install tensorboard and tensorflow
• Please clone this project:

git clone https://github.com/rlct1/gin-sg.git
cd gin-sg

Testing

• An example of the validation data of this challenge is provided in the datasets/ade20k/test folder
• Please download our trained model for this challenge here (google drive link), and put it under checkpoints/gin_sg/
• For reproducing the test results for this challenge, please put all the testing images under datasets/ade20k/test/
• You can test our model by typing:

python test_ensemble.py --name gin_sg 

• The test results will be stored in results/test folder
• If you would like to test on other datasets, please refer to the file structure in the datasets/ade20k/test folder
• Note that the file structure is for AIM20 IC Track 2
• You can download our test results for this challenge here (google drive link)

Training

• By default, our model is trained using two GPUs
• Examples of the training images from this challenge is provided in the datasets/ade20k/train folder
• If you would like to train a model using our warm up for initialization, please download our warm up for this challenge here (google drive link), and put it under checkpoints/warmup/

python train.py --name yourmodel --continue_train --load_pretrain './checkpoints/warmup' 

• If you would like to train a model from scratch,

python train.py --name yourmodel 

• If you would like to train a model based on your own selection and resources, please refer to the options/base_options.py and options/train_options.py for details

Acknowledgment

Our code is developed based on the skeleton of the Pytorch implementation of pix2pixHD

gin-sg's Issues

Loss explodes after several hundred iterations

Hi. First of all, thanks for sharing your code.

I'm training a model from your warmup checkpoint, with my own dataset. I adapted the ade20k dataset code, including the mask generation (I'm loading masks from files instead of randomly generating them during training). One modification worth noticing is that I'm not using the concatenation of the three images/segmentation images (the 9 channels input tensors), but just the 3 channel image and segmentation mask.

Also, I'm training in one GPU, with --batchSize 1.

The model seems to train well, including the output images shown after 100 iterations, giving very reasonable output images. But at some point, the loss explodes, and the output images degrade (my input images are in gray tones, and after the loss explodes the model starts to inpaint colored regions). This same behavior happened in 3 different training procedures.

Here, the console output for my last training:

(epoch: 1, iters: 200, time: 1.3572) G_GAN: 0.0000 G_COARSE_L1: 0.1753 G_OUT_L1: 0.4198 G_STYLE: 0.0117 G_VGG: 0.1646 G_TV: 0.0001 D_real: 2.0362 D_fake: 1.9544 
(epoch: 1, iters: 300, time: 1.3532) G_GAN: -0.0000 G_COARSE_L1: 0.0972 G_OUT_L1: 0.5641 G_STYLE: 0.0125 G_VGG: 0.1339 G_TV: 0.0012 D_real: 1.9728 D_fake: 2.0429 
(epoch: 1, iters: 400, time: 1.3499) G_GAN: -0.0001 G_COARSE_L1: 0.1616 G_OUT_L1: 0.4204 G_STYLE: 0.0049 G_VGG: 0.1384 G_TV: 0.0002 D_real: 1.8953 D_fake: 2.1278 
(epoch: 1, iters: 500, time: 1.3494) G_GAN: -0.0001 G_COARSE_L1: 0.1268 G_OUT_L1: 0.2278 G_STYLE: 0.0048 G_VGG: 0.1313 G_TV: 0.0004 D_real: 1.9033 D_fake: 2.0789 
(epoch: 1, iters: 600, time: 1.3490) G_GAN: -0.0002 G_COARSE_L1: 0.2029 G_OUT_L1: 0.3248 G_STYLE: 0.0084 G_VGG: 0.1644 G_TV: 0.0006 D_real: 1.7782 D_fake: 2.2434 
(epoch: 1, iters: 700, time: 1.3456) G_GAN: 0.0001 G_COARSE_L1: 0.2666 G_OUT_L1: 0.7073 G_STYLE: 0.0081 G_VGG: 0.1698 G_TV: 0.0003 D_real: 2.0981 D_fake: 1.8844 
(epoch: 1, iters: 800, time: 1.3353) G_GAN: 0.0001 G_COARSE_L1: 0.4171 G_OUT_L1: 0.4439 G_STYLE: 0.0153 G_VGG: 0.1921 G_TV: 0.0008 D_real: 2.0663 D_fake: 1.8784 
(epoch: 1, iters: 900, time: 1.3418) G_GAN: 0.0001 G_COARSE_L1: 0.4716 G_OUT_L1: 0.4312 G_STYLE: 0.0022 G_VGG: 0.0901 G_TV: 0.0016 D_real: 2.1602 D_fake: 1.8772 
(epoch: 1, iters: 1000, time: 1.3460) G_GAN: -0.0004 G_COARSE_L1: 0.2798 G_OUT_L1: 0.6536 G_STYLE: 0.0056 G_VGG: 0.1369 G_TV: 0.0002 D_real: 1.4287 D_fake: 2.4171 
(epoch: 1, iters: 1100, time: 1.3425) G_GAN: 0.0009 G_COARSE_L1: 0.2045 G_OUT_L1: 0.3318 G_STYLE: 0.0043 G_VGG: 0.1242 G_TV: 0.0002 D_real: 2.8307 D_fake: 1.1338 
(epoch: 1, iters: 1200, time: 1.3384) G_GAN: 0.0005 G_COARSE_L1: 0.1394 G_OUT_L1: 0.2315 G_STYLE: 0.0016 G_VGG: 0.0948 G_TV: 0.0002 D_real: 2.4711 D_fake: 1.5035 
(epoch: 1, iters: 1300, time: 1.3350) G_GAN: 0.0009 G_COARSE_L1: 0.2395 G_OUT_L1: 0.7470 G_STYLE: 0.0415 G_VGG: 0.1887 G_TV: 0.0008 D_real: 2.6512 D_fake: 1.1111 
(epoch: 1, iters: 1400, time: 1.3352) G_GAN: 0.0006 G_COARSE_L1: 0.2906 G_OUT_L1: 0.4269 G_STYLE: 0.0054 G_VGG: 0.1482 G_TV: 0.0002 D_real: 2.3596 D_fake: 1.4024 
(epoch: 1, iters: 1500, time: 1.3351) G_GAN: 0.0011 G_COARSE_L1: 0.2057 G_OUT_L1: 0.2507 G_STYLE: 0.0088 G_VGG: 0.1236 G_TV: 0.0018 D_real: 3.1009 D_fake: 0.9248 
(epoch: 1, iters: 1600, time: 1.3462) G_GAN: 0.0003 G_COARSE_L1: 0.3154 G_OUT_L1: 0.2537 G_STYLE: 0.0031 G_VGG: 0.1319 G_TV: 0.0001 D_real: 2.3537 D_fake: 1.6636 
(epoch: 1, iters: 1700, time: 1.3684) G_GAN: 0.0010 G_COARSE_L1: 0.2320 G_OUT_L1: 0.2390 G_STYLE: 0.0047 G_VGG: 0.1333 G_TV: 0.0005 D_real: 3.0459 D_fake: 0.9691 
(epoch: 1, iters: 1800, time: 1.3773) G_GAN: 0.0013 G_COARSE_L1: 0.2808 G_OUT_L1: 0.8809 G_STYLE: 0.0336 G_VGG: 0.2725 G_TV: 0.0003 D_real: 3.3564 D_fake: 0.7259 
(epoch: 1, iters: 1900, time: 1.3969) G_GAN: 0.0012 G_COARSE_L1: 0.1810 G_OUT_L1: 0.4566 G_STYLE: 0.0059 G_VGG: 0.1399 G_TV: 0.0001 D_real: 3.2172 D_fake: 0.7872 
(epoch: 1, iters: 2000, time: 1.3465) G_GAN: 495.6933 G_COARSE_L1: 7.3341 G_OUT_L1: 56887712.0000 G_STYLE: 146619840869323970644541440.0000 G_VGG: 1852339.6250 G_TV: 122902388736.0000 D_real: 1.8178 
(epoch: 1, iters: 2100, time: 1.3384) G_GAN: 726.3157 G_COARSE_L1: 5.7460 G_OUT_L1: 83326952.0000 G_STYLE: 2926275191578642603348852736.0000 G_VGG: 3801816.0000 G_TV: 761112363008.0000 D_real: 1.3165

Thanks in advance for any comments you could provide.