baldassarrefe/deep-koalarization

Keras/Tensorflow implementation of our paper Grayscale Image Colorization using deep CNN and Inception-ResNet-v2 (https://arxiv.org/abs/1712.03400)

🐨 deep koalarization

Impementation of our paper Deep Koalarization: Image Colorization using CNNs and Inception-ResNet-v2 (2017)

Federico Baldassarre^*, Diego Gonzalez Morín^* and Lucas Rodés-Guirao^*

^{* Authors contributed equally}

deep-koalarization was developed as part of the DD2424 Deep Learning in Data Science course at KTH Royal Institute of Technology, spring 2017.

The code is built using Keras and Tensorflow.

Consider starring this project if you found it useful ⭐!

Table of contents

• Citation
• Abstract
• Project Overview
• Results
• Use the code
• Community

Citation

If you find Deep Koalarization useful in your research, please consider citing our paper as

@article{deepkoal2017,
  author          = {Federico Baldassarre, Diego Gonzalez-Morin, Lucas Rodes-Guirao},
  title           = {Deep-Koalarization: Image Colorization using CNNs and Inception-ResNet-v2},
  journal         = {ArXiv:1712.03400},
  url             = {https://arxiv.org/abs/1712.03400},
  year            = 2017,
  month           = dec
}

arXiv e-print

Abstract

We review some of the most recent approaches to colorize gray-scale images using deep learning methods. Inspired by these, we propose a model which combines a deep Convolutional Neural Network trained from scratch with high-level features extracted from the Inception-ResNet-v2 pre-trained model. Thanks to its fully convolutional architecture, our encoder-decoder model can process images of any size and aspect ratio. Other than presenting the training results, we assess the "public acceptance" of the generated images by means of a user study. Finally, we present a carousel of applications on different types of images, such as historical photographs.

Project overview

Inspired by Iizuka and Simo-Serra et al. (2016), we combine a deep CNN architecture with Inception-ResNet-v2 pre-trained on ImageNet dataset, which assists the overall colorization process by extracting high-level features. In particular, Inception-ResNet-v2

The fusion between the fixed-size embedding and the intermediary result of the convolutions is performed by means of replication and stacking as described in Iizuka and Simo-Serra et al. (2016).

We have used the MSE loss as the objective function.

The Training data for this experiment could come from any source. We decuded to use ImageNet, which nowadays is considered the de-facto reference for image tasks. This way, it makes easier for others to replicate our experiments.

Results

ImageNet

Historical pictures

---

Use the code

Refer to INSTRUCTIONS to install and use the code in this repo.

Community

Thanks to the people who noticed our work!

We are proud if our work gets noticed and helps/inspires other people on their path to knowledge. Here's a list of references we are aware of, some of the authors contacted us, some others we just happened to find online:

• François Chollet tweeted about this project (thank you for Keras)
• Emil Wallnér on FloydHub Blog and freecodecamp
• Amir Kalron on Logz.io Blog
• sparkexpert on CSDN
• Eryk Lewinson on Medium

Projects originated from here

• Coloring Black and White Images with Neural Networks, by emilwallner.
• Ensemble Image Colorization using Convolutional Neural Networks with Refinement Network, by Kriztoper.

deep-koalarization's Issues

[REQ] Colab version

Hi there, congratulations for your great study/work !

It would be great to let users test its powerfulness/precision by a colab notebook (as, for example, DeOldify already allows):
https://github.com/jantic/DeOldify

I've also found this git that by @animalecs may help.

Hope that inspires.

koalarization.dataset.lab_batch issue

Hi! I want to run

python -m koalarization.dataset.lab_batch -c '../data/inception_resnet_v2_2016_08_30.ckpt' '../data/resized' '../data/tfrecords' --verbose

I have the following Traceback:

Using TensorFlow backend.
2021-12-01 00:26:43.453183: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2021-12-01 00:27:03.174050: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2021-12-01 00:27:03.174278: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2021-12-01 00:27:03.174446: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2021-12-01 00:27:03.174590: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
Finished writing 0 images in 0.54s
Traceback (most recent call last):
  File "/home/lucas/anaconda3/envs/koalarization/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/lucas/anaconda3/envs/koalarization/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/mnt/c/Users/Lucas Ariel Saavedra/Documents/GitHub/deep-koalarization/src/koalarization/dataset/lab_batch.py", line 255, in <module>
    ).batch_all(examples_per_record=args.batch_size)
  File "/mnt/c/Users/Lucas Ariel Saavedra/Documents/GitHub/deep-koalarization/src/koalarization/dataset/lab_batch.py", line 89, in batch_all
    self._run_session(sess, operations, examples_per_record)
  File "/mnt/c/Users/Lucas Ariel Saavedra/Documents/GitHub/deep-koalarization/src/koalarization/dataset/lab_batch.py", line 172, in _run_session
    coord.join(threads)
  File "/home/lucas/anaconda3/envs/koalarization/lib/python3.6/site-packages/tensorflow/python/training/coordinator.py", line 389, in join
    six.reraise(*self._exc_info_to_raise)
  File "/home/lucas/anaconda3/envs/koalarization/lib/python3.6/site-packages/six.py", line 719, in reraise
    raise value
  File "/home/lucas/anaconda3/envs/koalarization/lib/python3.6/site-packages/tensorflow/python/training/queue_runner_impl.py", line 238, in _run
    enqueue_callable()
  File "/home/lucas/anaconda3/envs/koalarization/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1235, in _single_operation_run
    target_list_as_strings, status, None)
  File "/home/lucas/anaconda3/envs/koalarization/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/home/lucas/anaconda3/envs/koalarization/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [string_input_producer requires a non-null input tensor]
         [[Node: input_producer/Assert/Assert = Assert[T=[DT_STRING], summarize=3, _device="/job:localhost/replica:0/task:0/cpu:0"](input_producer/Greater, input_producer/Assert/Assert/data_0)]]

I don't know how to fix it. Thank You!

PD: I'm working without GPU

Pre-trained model

Is there a pre-trained model for this? I want to do transfer learning on my dataset w/o training from scratch.

Inception output size

Hi! I'm confused as to the fusion layer. I thought Inception Resnet v2 was supposed to have 1000 classes, so the output is supposed to be of the shape (1000, ), not 1001. Can you explain this?

How do you train your neural network?

I've managed to prepare the images and run the batching tests.

When I run the main > python3 -m colorization.main. I get a couple of errors. I've used the folder structure you suggested, but I believe I have the wrong configuration. I'd appreciate if you could outline the steps to train the network. Thanks 🚀

training on my own data

for my own dataset, I don't have any label , just color images, any instructions for this ? thank you

Labels from Inception at the beginning of training seem incorrect

When training the network and looking at the outputted images from the validation set, i see labels from inception at the top of the images, are they supposed to be objects recognized in the images from the pretrained inception features? If so they seem very incorrect, i see objects listed which are not present in the images and vice versa. Is this because the network is untrained and doesn't utilize the features from inception correctly? Also how long does it usually take for the network (on say 10.000 images, i have a larger dataset but I've started of with a small portion to test it out) to converge to some colors being generated because after about 200 iterations all i see is a redish-brownish color on all the images. Is it possible I'm doing something wrong?

train.py?

Hi!what does the "from colorization import Colorization" means?I did not find the Colorization....

train on multiple gpu

Have you tried to train it on multiple gpu?How?

queue_single_images_from_folder

it seems the functions queue_images_from_folder in images_queue.py is out of date with tensorflow v2 and I'm not sure how to recreate its functionality. Any tips?

Lab_Batch file is not creating Tf-records

Training

Hi,
I am a student in deep learning and neural network.
Could you help me with some questions I wonder myself ?

how many lab_images_.tfrecord do we need to do before training ?
For val_lab_images_.tfrecord, do we just need to rename one lab_images_.tfrecord ?
Where are the grey images ?

Cheers

Simon

from resizeimage import resizeimage

I con't find the model resizeimage .

TypeError: Can not convert a LabImageRecordReader into a Tensor or Operation.

TypeError: Fetch argument <dataset.tfrecords.images.lab_image_record.LabImageRecordReader object at 0x0000000074CF3940> has invalid type <class 'dataset.tfrecords.images.lab_image_record.LabImageRecordReader'>, must be a string or Tensor. (Can not convert a LabImageRecordReader into a Tensor or Operation.)

i got this error when i ran train.py.

the solution is below@charleshamesse .check it out